Just remember, when you feel like everyone else is mad at you for not taking him back --
THEY AREN'T MARRIED TO HIM -- YOU ARE!
And, maybe, just maybe, when he's living with his parents, they may get a closer view of how he really is. Of course, they will probably stick up for him because he's their family!
Bottom line is you do what's best for YOU! Don't let any guilt trip get to you. Stick to your guns -- you're the only one that can please YOU!