This year we went the week after (checking in the day before Memorial Day), in 2004 we're going the week before (checking out the Friday before Memorial Day). Crowds weren't too bad. On the good side, I remember that DS went on TT using the singles line, once it was 15 minutes, once practically a walk-on. No need for fast pass all week. The only bad part came on Enites-SM had a 1/2 hour wait, which I thought was a lot for an Enite!
The reason we're going before this year is, for one thing, Memorial Day is much later this year. Also, it'll be a few days less hot (hard to use "cooler" and May in the same sentence). I hate heat and wish we could go a different time, but work and school schedules dictate otherwise.