I think as long as you realize that your time at WDW will be different than when you've gone pre-kids, any age will be OK once your Pediatrician says it's OK for travel like that. Some things to consider:
1) More frequent "breaks" for feeding, diaper changes, naps, etc.
2) Riding on the "big" rides will either be out or you and DH will have to take turns.
3) More "stuff" to carry, although with midday hotel breaks you don't have to bring as much as you might think. In fact, since you'll need to bring a stroller, you'll have something to serve as a "pack mule" for your child's and even your own things.
My DD will be 1 month short of 3 when we make our first trip to WDW in Sept. I have to admit I'm glad she'll still qualify for free park admission as well as free meals at many restaurants (family service and buffet). We've also gone on an Alaskan cruise when she was 11 mo and gone to DLR when she was 23 mo. We had so much fun on each trip that I see no reason why we should put off WDW (Europe will wait until she can appreciate the cultural attractions, though).
I tried to wait to tell her about this trip until closer to our travel date for my sake because I knew she'd want to go now. I didn't do very well keeping the secret due to my own excitement

. She's already very excited about this trip and looking forward to breakfast with the Princesses, one with Mickey and another with Pooh.
As for not remembering things at such a young age, I thought that too until my DD started mentioning things about our DLR trip with details I had forgotten. I know they were her memories and not just repeating what I'd said. Admittedly, she doesn't remember everything but enough to look forward to the trip ahead and talk about riding the teacups again.