Your husband's behavior would be a deal-breaker for me.
He did not get "lured" into a one night stand. He did not develop an "emotional which turned physical" relationship with a co-worker.
He actively sought out multiple sexual partners, did not use protection, passed himself off as single...it's too disgusting to even think about.
If you need him to finalize the adoption, then hang in there till that's done. If you don't need him to finalize the adoption, then kick his sorry, slimy, disgusting, pathetic, lying, cheating butt to the curb, get your baby girl and move on to a wonderful life with someone who will appreciate you and all the good you have to offer.
Let him stay with his pathetic "swinger" friends and wait for karma to kick him in the ****...and believe me, it will. And that wil be a good day. And he'll deserve every bit of what he gets, and then some.
This isn't a man you want as a partner. This is a sicko. Do you want this sicko near your child????