I can certainly relate to your situation/dilema. All I can say is you never know unless you try it out.
Back in '98, our best friends wanted to take DH & I to WDW for the weekend for us letting them have their wedding in our home. I thought we'd have a blast...I mean, we're all best friends, same interests, no kids, small group (4). Boy, was I wrong. Talk about the trip from hell. I said never again, we go alone or not at all.
Well, my hubby's massive family has been trying to plan a get-together for years at the World with the whole family. We're all Floridians all across the state, and we all go all the time, just not together. Well, after nine years, we finally had to give in. My excuse book had been exhausted. There were 11 of us, and I'll tell ya, we had the best time. I can't wait until we get to do it again. Felt wonderful to go with seasoned disney vets who knew what they were doing!
With the large group, we all did have some other things planned that didn't include everyone. We stayed at our time share; everyone else stayed on site. DH & I had a couple must-do restaurants over at CityWalk, so we did that with a couple from his family while the grandparents hung close to the disney hotel; a few of us hit parks while others hit DTD. But generally we were all together at least in the same parks communicating by cell phones. Honestly, I wish I'd done it years earlier, and I would never hesitate to do it again.
So you never know until you give it a shot!