We'll be there at same time...this is my 5th trip during the week before and/or including part of Memorial weekend-we've always gone then because our kids' school system was out for summer vacation between the 18th-20th every year, so we'd leave the afternoon they get out or the next day. Most east coast schools are still in session, so crowds are just not that bad.
Kids are grown, but we're still going! We'll be there May 19-25 this time. The weather has always been super--plenty warm enough to swim (mid 80's) but not broiling hot yet (although we usually leave parks in hottest part of the day anyway to head for pool hotel.) Some years there has been some rain, but the typical Florida pour-for-10 minutes, then done & drying up fast. Most trips were : I think the real rainy season starts more in June. The crowds are manageable, not near as bad as the descriptions of summer and holiday crowds - we never waited more than 30 min for a ride. (we do get to park as early as possible, using EMH, and use FASTPASS, get done by early afternoon, leave park to swim etc, and come back in evening. Has worked well every time.) We think it's a wonderful time to go.