I think any time you can plan a trip which doesn't coincide with a holiday, school vacations, marathon or other big event (parties) can be pretty decent crowd level. Next year Easter is actually early on 4/1. Next year we are planning a trip to go during our kids spring vacation, which is usually the 3rd week of April. This year is like the perfect storm -- Easter, many on school vacation and Marathon weekend. I'm going only for a conference, but i am fearing the worst for crowds. Next year has to be better.
I would say early May is an ideal time to go -- no holidays or school vacations, weather is good and flower and garden is going on. Aside from early May, September might be the next best time to go -- kids are back in school and there are no holidays, but it is in hurricane season.