We used to go in October and then moved to the two weeks after Thanksgiving, but last year we did a last minute trip in May (15-21) and although it was busier than we're used to, it was very, very tolerable - Saturday at MK the day we left was awfully busy though.
I was also told by a cast member that it gets insanely crazy Memorial Day week, but prior to that it isn't bad at all.
I think of it this way: would I want to pull DS out of school for a week a few weeks before it ended for the year if he were older? Nah. And the nunber of schools out in May is negligible compared to June.
I think you're okay. We're heading there the second week of May and I've had little problem with PS, etc. I think June, etc. will be crazy though with #50!