My favorite time to go is the second week of May. We have gone at this time every year for the past four years. The crowds aren't at their absolute lowest at this time, but they are manageable and we have never waited in line for more than 20 minutes (using smart timing, Magic Hours and fast passes). It is after spring break and before summer vacation. The weather is also just about perfect (mid to upper 80s and little to no rain). I think my secret has been getting out, though. Last year the crowds were a little heavier than in the past. I'm hoping it was just a fluke. DH and I will find out in just 62 more days though!