I think it depends on the time of year. If you are going to be there during a school vacation, you might want to stay at the Swan/Dolphin. They have the most adult atmosphere of any of the on property hotels, plus you have the bonus of being able to walk to Epcot and MGM.
We stayed at the Yacht Club right after Labor Day, and it was not crowded at all. However, we've been there in the summer and during the holidays and many, many kids were running around the lobby, pool, etc. It just was a different (not as relaxing), noisier atmosphere. We'll be at WL in a few weeks and are hoping for low crowds.
The monorail hotels have a great appeal to families with small children.
As for moderates, we liked the Coronado Springs and POFQ. But the food courts at both were mobbed with families and very loud.
The Swan and Dolphin both host many conventions and have great restaurants that are less likely to attract families. We like that because sometimes after a day at the parks, we are glad to be surrounded by grown ups again! We prefer the S/D and can sometimes get great rates there. When they are fully booked or aren't offering discount rates, we go for the hotels listed above.