We went for that timeframe (May 17 - 20) last year and it rained a couple of mornings on us. We wore LS/sweatshirts during the cooler parts of the day and only ended up in plain shortsleeves mid-day the last day (forgot to sunscreen my arms, and they burned). We're from Arizona, though, so we have a different interpretation of 'shorts weather'.
Crowds weren't bad. I have seen them much smaller (a Thursday in early November and the day after MLK day (which was dead, I mean walk-on to Soarin mid-afternoon) in January), but they were still quite manageable. We were only there on weekdays, though. We're going again (slightly earlier May 13 - 17) this year. One last trip before the APs expire!