My daughter, who is now 12, took her first trip at 4. My son, who is now 7, took his first trip at 3. My daughter remembers a bit about her first trip, my son does not! However, when my son was 3 and we decided to go, we decided that although he might not remember it, he would still enjoy himself a lot while there. . .which he definitely did! After that trip, though, my intention was to go every year or every other year. It turned out that although my parents took my 2 children next year for a day or 2 long trip, so many personal things had happened, that I wasn't able to take him again.
Here's something funny. . .there were a couple of rides we knew our son would be too short for, just by a hair. So, we got the highest tennis shoes we could find for him (still boys, though) and inserted 1/4 -1/3 inch lifts inside them at the shoe repair guy's store! I know that some of you might disapprove of that, but we knew he'd enjoy some of these rides, and I didn't think 1/4 inch would compromise any safety issues. As it turned out, the measuring sticks the cms use are actually incorrect, and he towered over the height requirement. . .didn't even need those lifts!
However, we're going in October! YAY!
It depends what your motivation is. If it's just to have a great time, go when he's three. You can probably go again later. If it's so he'll remember the trip for sure, then wait at least another year.