Unknowingly, we booked our first family trip to WDW in 2003 during Pop Warner week and actually stayed right in the middle of all the kids at ASMo. Other than it being louder than normal (from what I've seen on our last three trips) at the pool area, I can't complain about staying at the same resort. The food court was obviously prepared for the breakfast rushes each morning, and food was hotter and fresher, and took less time to get than I've had at POP or ASMu my last two stays.
Tell DH to stop worrying...you'll have a great time! That first family trip was the start of our Disney obsession...we'll be on trip #5 in a few weeks...and that's just in two years time! We've even started planning our trip to DL next summer.