I didn't get to go to WDW until I was in my early 20s. Oldest DD was 4 on that trip- she doesn't remember it, but loves looking at the pictures and knows some of the details from videos and from what have told her about it. She remembers some of the things we did on trips we took after that (ages 6 and 7), and pretty much everything from the trips ages 10 and up.
ETA: my other kids also have "vague" memories of our last trip (when they were ages 5 and 8). They remember some of the attractions and a few details, but not everything. I'm hoping the memories we make on our trip this month will stay with them forever!
Oh, and to keep the memory thing in perspective - we moved to TX from NJ when DS(9) was 5. When he was 8 we went back to NJ to visit, and he remembered NOTHING of our old house (which we lived in his entire life until our big move to TX), nothing of our old neighborhood and favorite restaurants, his old preschool, etc. He really didn't remember living in NJ at all. So, one trip to WDW when they are 3 or 4 will likely only be remembered if you show photos and videos and talk about it a lot!