We went Memorial Day weekend, which was crowded, but do-able. But the minute we hit Tuesday the place was a ghost town. We had the place to ourselves.
I would absolutely go back again after the long weekend !!!!
Florida weather is still great then, not too hot !
Kids in Florida were still in school, as is many parts of the country.
What you must check is the dates for the "Gay Day at Disney World". It is around the 5th of June and probably begins several days before through the weekend. Besides that it may not be for you and your family, I think is draws something like 100,000 people to the Disney World area.