We first went as kids when I was 8 1/2 and my brother was a month shy of 6. I was in LOVE with theme park rides and Disney, so I remember everything from waiting in line for 2 hours for Splash (back in 1991), meeting characters,seeing the Beauty and the Beast show,etc. My brother has less memories of the actual rides and parks. This is partly due to his age at the time, but also because he had a couple of significant moments that were not related to rides: he learned to swim while in Florida, and on our first day at the parks he touched a bug lamp at Sea World and burned his hand (had to keep his hand in a cup of ice all day).
I feel that at about age 5, kids will remember the stand out moments, and photos/videos of the trip will help them "remember" more than they actually do. As for people who say "Maybe the kids won't remember being there as a baby/toddler, but we will". Fair enough. However, I think it depends on your situation. If you can make Disney a day trip, or drive there in a few hours, then yeah go for it, take the 2 year old while they are free-why not? But if you're like me, and a trip to Disney for a 2 year old includes 8-10 hours of travel in airports/planes each way, and full plane fare on top of that, it wouldn't be worth it. I would rather wait until they were old enough to DEFINITELY remember some big things, and the equally important memories of a family time. But if I lived in Florida, my children would be seasoned Disney-goers by the time they required an actual ticket.