If you are staying off site I would suggest making sure you go before may 15. The crowds will be less as the area schools have not gotten out yet.
If you are staying on site. I would recommend going no later than the first week of june. From my experience I find the hotel rates tend to go up to a premium after the first week in june and so do the crowds.
We always go the week prior to memorial day . This year it is may 22-29. Sunday to Sunday. The weather isn't to bad. The crowds aren't that bad either.
Have a great trip. If you have any questions feel free to email me. I'll try to help if I can.
Maybe someone else out there has some more suggestions out there as well.