If I could go whenever I wanted crowds wouldn't be my only consideration. Weather and special events would have to come into play too, and of course cost.
Of our trips, the lowest crowds were in January by far but the weather was really iffy. We got lucky and most of the trip was warmish, with just a couple of days out of 12 that we needed jackets, but that same week the next two years had a long stretch of sub-freezing lows. I'd rather not go back at that time of year because of the uncertainty of the weather (not saying never, though, because it is the easiest month for DH to be away from work).
Next would be the beginning of our December trip. We don't have a lot of patience with other people's kids, so we wedged our trip into the gap between Pop Warner and Christmas and the crowds did get a little nuts for the last two days (start of Christmas break in many places). The holiday decorations were great and the weather was perfect! I wouldn't hesitate to go back again at that time of year, but next time we'll either brave overlapping with PW or make the trip a little shorter rather than run into the holiday crowds.
After that comes late Sept/early Oct. The crowds were fairly light but the MK closes early several days a week for the Halloween party which is a bummer on the nights you aren't attending. And for us northerners, it was H-O-T! Food & Wine is a nice perk and we'll likely plan another Oct trip once the kids are grown (can't do it sooner because Oct is testing month in MI public schools
).
Late August was barely more crowded than late September and I didn't think the weather was noticably different. Once the southern schools go back, the summer crowds disappear, which makes the last week or two before Labor Day a great time for those of us with kids in late-starting northern/midwestern schools. If DH could reliably get away this would be my choice every time; the heat is managable, the crowds (and prices) are low, and the days (and park hours) are longer than in the off-peak winter months.
Our next trip will be for Flower & Garden. We were planning on early April, but now we're thinking opening weekend because of the overlap with the free dining offer. The weather won't be as warm, but the price difference is worth it.
Once we checked in Fort Wilderness the Sunday after Thanksgiving and stayed until Thursday. It was a great trip, mild weather and no crowds. The other time we checked into Hard Rock at Universal on the Wed of the first week after New Year's. That trip they didn't even make you get off the rides and go back into the queues. My DD rode Dueling Dragons something like 17 times in a row. 
We have also been in mid-May like some other posters are reporting. I don't like going at that time of the year because the parks are packed with local school kids on school trips with discounted tickets. We have found them to be poorly supervised, rude and inconsiderate. Not my idea of "Magic". 
