Both of my more recent trips have been in May! We went the first week, and the third week of May. Both times, the crowds were moderate or low during the weekdays. On weekends, it picked up a little. We were ok with our fastpasses and a little planning, though. Longest line we had was 40 minutes for Peter Pan. Just make sure you plan!
It was never cold, or anything. Hot weather, but not as bad as other summer months. There was rain in the afternoon a lot, but it always went away after an hour or so. I think that's pretty typical for Florida, though.
I would suggest the first two weeks of May... most kids are still in school then, so smaller crowds in general.