We took my younger child at 7 months as my older one was 3 and the rest of the family was going. I chalked it up thinking that we didn't really have to "pay" for him and this trip would be for the kids, so I was prepared to do very little of what I wanted to do, and made my peace with that ahead of time.
I was shocked to find that BOTH children absolutely LOVED it! We had the best trip ever! It was hotter than ever, in the middle of July, but we took our time and alternated indoor/outdoor waiting lines and attractions to stay cool, hung out in the Baby Care stations and grabbed a snack and watched cartoons when it got too hot, and everyone did great the whole week! The baby LOVED the colors, lights, and motion on the rides (He kicked, cooed , and babbled while taking it all in all through "It's A Small World"- precious!) and my daughter truly thought EVERYTHINg was "magic"- HER own world at that age!
Each trip at each age has been different, but they've always loved it. I just go knowing the kids WILL get tired, cranky, and hot, but it's up to me to manage that. I let THEM lead the trip for the most part and it has always created magical memories. I get VEY upset when I see parents yelling at their kids for being tired and crabby at Disney World! If you can't fall in love with ALL children there, you are in big trouble!

PLUS, I did get to ride everything I wanted thanks to stroller naps for the baby with DH while we child swapped the older one or she took a break. It truly is our favorite FAMILY vacation place!
That being said, if this is your only child, you are only going with your immediate family, and/or your budget is not such that you might be able to go again in the next 2-3 years, waiting until closer to 4 might be better as your child may remember more. Either way, I will almost guarantee you will all have a blast if you just know to be flexible and laid-back ahead of time.
(BTW- My 7 month old is now 3.5 yrs. old! He was back at Disney at 2 and will be there again this fall at almost 4 yrs. old. He continues to ask at least once a week, "Can we go to Mickey's House today?"
