We have never left any of our kids behind either. Our youngest took her first trip at 3 1/2 mos and has been 5 times in her 5 years. I think the younger the easier
Once you have a second or third, fourth, etc. they have to be taken to the siblings games, practices, dance classes, etc. and so they are already used to not having a great schedule. (or at least that is how it is for us...Im not hiring a sitter so that the youngest dosent have to get off her "schedule"). We are a busy family and all the kids go with the flow at home or on vacation.
Our kids as babies or toddlers have always been great to travel with to WDW or anywhere. They ride in the stroller (nap there if they need to) and move from place to place with us as we go.
I have precious pictures and memories of all the kids at WDW as babies, toddlers, and preschoolers and wouldnt trade that for anything.
Even at 3 mos, dd loved the shows and characters and the business of her surroundings while at WDW.
I know that none of my kids will look back at old pictures and say "why didnt you take me to WDW when you took my brother/sister?" I would hate to have to say...you were too much trouble to take
The first trip we took without all our kids was this past October. My dd23 had gotten married in July and they couldnt get off from work after just being off for their Honeymoon. It was very hard and sad for me to be at our favorite place without one of my kids...and she was 23!! I cant imagine being there without one of my littles.
You have to do what you think is best for your family, OP. We can only give our opinions...but you have to do what you find best. This is a topic that will be very divided.