My three yo DD has been four times, fifth coming up in September. At that time, our baby will be 3.5 months, same age as DD was on her first visit. To us, the experience has nothing to do with what the child will remember later in life, rather the experience they have while they are there. Even at three months, their face lights up at the colors, sounds, characters, etc.
When my family first went to Disney World I was 6, my brother 4. I was at an age where I questioned all the "magic". I could see the strings in 20,000 Leagues Under the Sea, knew the characters were in costume etc. My brother on the other hand was at an age that it was all real to him. That was truly magical for my parents to see his amazement. (and the fact that I thought Mission to Mars, "could be real"

) Now I get to see that amazement on my own daughter's face! She loves everything about Disney, and remembers everything from her last trips. We talk about them all the time, because there is always another trip being planned. (Just ask her, she can tell you every detail, down to what Cindy said to her in the castle before lunch!)
To my family it is no bother to have the baby (babies) with us, it is truly a pleasure, and we treasure every moment. PLUS, I do not know how on earth we could stay away ourselves until our kids were older! My husbands cousin has us beat though, her babies have gone at only weeks old!