Since we are from New Jersey, we usually go during Jersey week in November and love it - great weather, no issues with crowds, food and wine festival, Super Soap weekend - just great!
We have also been in late June (too hot and rainy and crowded), late September - just hot - but no crowds made it great; and just recently this October over Columbus day weekend - perfect trip.
Our next trip hopefully will be in the first week of December, so we can enjoy the Christmas decorations and events.