We've been heading down to WDW every May since 2004 and have been during the first part of the month, the middle of the month, and the end of the month. Unfortunately we have seen a very steady increase in crowds each year no matter when we go. I really think that with all the discounts available, there really isn't a slower time at Disney any longer.
You need to remember the Flower and Garden Festival is on until May 15th, then Star Wars weekends start the weekend of May 20th for 4 weeks - both of which bring crowds to the parks.