My first dd's trip was at 2 and she had a wonderful time. Meeting Cinderella was like a dream come true for her. She was 2 1/2 at the next trip and also had a great time, she was willing to go on all the dark rides and loved them (Pirated, HM, etc). Her next trip, she was 5 and still had a great time but her imagination had kicked in and she really didn't like HM or Pirates and the theming on Thunder Mountain freaked her out even though she loves roller coasters. Her sister was 21 months on that trip and had a wonderful time. She enjoyed everything, even the characters. Our last trip, dd's were 5 1/2 and 28 months and again both had a great time. Our 2 year old loved the characters and all the rides, our 5 year old loved the characters even though she now knows that "there are really people in there". I think she still believes the face characters (ie: princesses) are real.
I do think it depends on your child. Our kids were good stroller nappers, which is a big help. I wouldn't trade the memories of the first trip for anything, even if my kids don't really remember. Although, my 2 1/2 year old still pretends we are going to Dizzy Worl every day and our trip was almost 2 months ago. Everytime we hear "Small World" in the car she says "we ride on that!", so obviously she remembers something! I don't think you can go wrong either way.
~Ellen