We went Sept 17-21 last year, and I went Jan 25-26 this year.
Sept was definitely warmer; Jan was chilly and it rained.
On the dates we went, the crowds both times were on the light side, in my opinion. In Sept I could definitely see a difference between weekdays and getting to the weekend. Then again, the 21st was the day Haunted Mansion opened up again with its Nightmare Before Christmas overlay, so it was probably crowded with HM fans that day!
Swimming depends on where you're from and what you're used to.
Rain can be a bummer, but it clears out the parks, so I find it very helpful. We bought ponchos and carry them with us, so when it rains we throw those on and keep going. OK, well, to be exact, when it rained on our trip in December (we've been a couple times this last year, LOL) my hubby and son were miserable, so they went back to the hotel while I stayed and played in DCA. Then it stopped raining, I went back to the hotel, and hubby went off for a couple hours on his own as the park filled back up with people who didn't like the rain.
I think that after labor day you're more likely to run into attractions being down, but then if you get into Halloweentime they know it will be busy so they try to have things up and running. Depending on early or late January, you'll either hit the end of the holiday season (which will still have some crowds) or you'll be in a quieter, more-attractions-down time.
You can see there's no one, solid, answer.

Hope my thoughts helped somewhat anyway!