We have gone the week leading to Memorial Day twice. The crowds are very manageable til friday then it gets pretty busy though still not horrible. Definitely use fastpass wisely but our longest wait was 45 minutes and that only happened once or twice.
The last trip 2 years ago was that same week but central Florida got 11.5 inches of rain during our trip which definitely kept crowds low. The weather people kept saying we were witnessing a historical event-i could have done without being a part of that history!!!!
Aside from that I think it is a great week to go!