I also think the wings are underrated! They are usually much quieter...if you're in the Tower and on one of the floors just above chef mickeys be prepared to musically "celebrate with Mickey Mouse" every 20 minutes or so from about 7 am LOL. or at least hear the happy tots below doing so. I happen to love that noise when I'm at WDW but some people like to sleep in on vacation.
Also some families don't like the 'grand central station' feel of the tower. I love the energy in both the tower and the calm of the wing rooms.
If you're on a higher floor than the noise isn't so much an issue. It is more convienient to get to the monorails if you're in the Tower. But it can be more convenient to get to the pool/quick service food/arcade from the wings, so it all depends on what is important to your family.
I was shocked when I asked my child if she wanted me to try to get a tower room for our next trip and she said she didn't want to be "up high"! she wanted to stay in the wing! And this is a kid who has previously stayed in the main building at GF and the CR tower before: the wings are her very first choice and my DH's too.
Now...if they offered us an upgrade to the tower for free...I'd try to talk her into it. LOL
As far as whether the OP should leave their group in the wings, that's a decision only they can make. If you're going to travel with in-laws or someone else who will be greatly offended by it, it might not be worth it to make the trade off. But if they are easy going and not likely to care (or pout the rest of your trip!) then go for it! It's your vacation too after all!
IMO you just can't go wrong staying at the Contemporary, wing or tower.