I agree with rie'smom. Of all of the U.S. holidays, Memorial Day and Labor Day are the two least crowded in WDW. Although crowds will be elevated, they will not be unmanagable.
Count on rain in the afternoon. Make sure to check out the Flower and Garden Festival in Epcot. Have fun, and enjoy the trip!
See you there! I'll be at the Boardwalk from May 26-30.