We just got back this weekend from WDW and were able to swim. It was hot almost every day. On two mornings it was cool enough for a light sweatshirt for the kids, but it was warm by lunch time.
We live in New Orleans and are used to hot weather, and Orlando STILL seemed hot to us! I wouldn't expect the weather to change much in the next couple of weeks, so you should be fine. I think there were a couple of evenings where it was a bit too cool and windy for us to want to swim, but if you plan to do it early enough in the afternoon, you should be fine.