Hun, your little boy loves his dad, sees him as his male role model, looks up to him, is proud of him--these are all healthy, normal, and good for him!

His dad's cool, loves him, spends time with him--and their time together is FUN time--this is also normal!
But there is one thing his father will NEVER be & that is the mother he loves/adores--who nurtures him, is there for him, kissed his cuts and scrapes, loves him in a way that no-one else ever will! You will NEVER be hated by this little boy & every woman he ever meets will be compared to you--you have to hide your anger with his dad & not tell him what a jerk his father is--regardless of how much you'd love too!.....
I also left my sons father when he was very little--only 4 months old--and for many of the same reasons you mentioned--add to that he was VERY controlling & though he never laid a hand on me he frightened the heck out of me enough to not wait around for what I knew was inevitable if I stayed!...I too was the bad guy as far as his father was concerned--but some things are better left to be explained when you're talking to your son in 15 or 20 yrs from now! until then--let him have his perfect dad--he needs that in his life & by not telling him now that his dad is the bad guy--you are making this sacrifice for your son's mental well being
As far as his father is concerned--if you have actual proof of him saying these things--you can always go back to court and demand parenting classes

you can amend your agreement at any time!
I hope this helps?