I have to say though that we soooo many times looked at couples struggling with younger ones.
I think a lot of that is self-inflicted. We took DD-soon-to-be-six on her first trip when she was about 3 weeks short of two. It was our first trip with a child and we pushed her too hard...for which we were richly rewarded with Magic Kingdom Meltdown. We also had Epcot Meltdown (duh

), and I had to call Lost and Found to ask if anyone had turned in Cinderella's lost slipper (seriously). They hadn't.
However, we learned a lesson on that trip, and so did she. Mommy and Daddy haven't forgotten that we were mostly to blame.
Our Princess has made four
very successful trips since then, with her sixth trip coming up in May -- two months after her sixth birthday. She's also been to California twice, Boston twice, and some other, shorter trips. She's a seasoned trooper!
I think you just have to adjust your pace and recognize that little people get tired. They're built very close to the ground, so the stimulation of walking down Main Street, for example, is much greater for them than it is for us. Plus, they have to take a LOT more steps! You just have to do Disney (or any other travel)
at the kid's pace, and they'll have a great time.
And, of course, every kid and every set of parents are different, so there's no equation you can plug numbers into and get a good answer.