I like the Flower and Garden Festival - so - I'd pick dates once that has started.
AVOID the week before and after Easter.
The weather - normally beautiful. Yes - it's warmer as you progress farther into May. You can swim both months (you can swim in March on most days). Again - there's a chance of cool day/morning/evening - more in early April - but really - that would be an "odd" day - not the norm for a week.
Star Wars weekends - if you LOVE Star Wars - going during a SWW is fun. Yes - there are some lines for meet n greets - but hey - that's what you are going for. Yes, the park is a tad busier on the weekend. But if SW is not your thing - just hit HS on a M-Th and really, there's no concern.
I went second week of May last year - and it was great. Couple days were hot, but not "summer miserable," a couple days were perfect all day, and it was rainy/cooler one day. Besides the one morning of solid rain - we really ranked the weather as gorgeous the whole trip. And frankly, the rainy morning gave us a much needed break as we opted to not leave the resort until it was more at the drizzle stage. SWW hadn't started yet. We had wanted to plan to attend when we could do both - but work schedules didn't allow for it. We didn't think crowd levels were bad most of the trip. There was one MK day that WAS quite busy - this was being driven by an ideal evening schedule of MSEP, fireworks, and EMH for those planning to stay late -- but we were prepared to hit MK on a busy day - had our FP's, and meal and touring plans pre-set up - and that made dealing with the crowds no big deal.
I'd pick a week I could hit both the F&G Festival and a SWW if I could (we are Star Wars fans). Work doesn't normally let that happen for me. BUT - when traveling with one other set of occasional traveling companions - then I would go in April primarily because it IS cooler then and they would think even April is too hot.
If you normally travel in Sept - I think WEATHER wise - you'll love either April or May. Sept is hot and humid in comparison.
PS - by gorgeous weather - I mean shorts and a t-shirt all day comfortable.