I wouldn't take a "baby-baby" if you know what I mean, but we took our kids at some point with an 18-19 month old (different kids at different times). That was actually a glorious time, because as a previous poster said - it was so much fun for us (the parents). I just remember one staring at the big "Pooh" character in awe, another just overwhelmed (in a positive way) by the big Coke Cups at the All Star resort in FL. So it's definitely the little things, but if you don't go then you'll never get to see them at that stage enjoying it in the same way.
A warning about 3 year olds - twice we've gone with a 3 year old and both times they were terrified about every ride with "scaries". This could be a little darkness, lightning, loud noises, etc. I remember standing in lines assuring a kid there would be no scaries while he screamed and tried to get out of the line. Rides I swore wouldn't be scary (The Land at EPCOT) would have a 30 second scene that would leave him terrified for the next hour. Meanwhile my 18 month old at the time was perfectly calm, as long as Mom or Dad was there to hold on to him. So this age, at least with my kids, was really iffy. Old enough to understand scary, but not to deal with it.
On the sad side, by the time we went back again, my oldest was 6. He tolerated Dumbo, Character Breakfasts, etc. with his younger brothers, but when I suggested he put them in his journal for 1st grade he was making, he was mortified - "I can't write that - everyone will make fun of me!" So they grow up too fast (maybe it's different with girls!) Had we not gone when he was little, I wouldn't have all the fun memories before he got too "cool" for this stuff. Now he's almost 11 and we're going out to DL for the first time & I don't know how he'll do. At least we're doing MM so he can do some cool rollercoasters too!
But overall I'd go when she can walk competently and notice/be fascinated by her surroundings and just realize you'll have fun but may not accomplish great things!