Everybody is different and what works for one won't work for another. I loved taking our children when little, under 4. It is all so real to them. They see it so differently. For example, when DS#1 was 3 and he got of Star Tours he couldn't believe the robot took them the wrong way. At 4, he was floored because "he took the wrong turn again!". Now at 5, "it's all pretend mom."
When DS#2 was 15 months, he didn't really care about WDW, even though he had gone before. But after that trip, it's excitement over all the characters. He actually recognizes everything Disney, including the word & all the theme music. It's "go Disney?"
Babies are so portable. I don't find diapers a hindrence. It's actually easier before they are potty trained, imo. With a little planning there is next to no extra work. And hey, before 3 they are FREE!
Our next trip is in Dec; we're sneaking it in a few weeks before DS#2 turns 3 so we don't have to pay. And we don't want DS#3 to be too old before his first trip. I'd take them all NOW if I could for a bonus trip.