We've been in Nov several times. (Though this year, we're doing US/IOA in November instead). We do the week right after T-giving or the week right before T-giving. Weather is perfect! Layers can be your friend in the morning but we normally ended up in shorts and t-shirts partway through. A jacket was needed some mornings though. We've had the occasional too hot day - like mid 80's and high humidity. We live in the southeast and the humidity that day was horrible even for us. We've had the occasional day where we wear pants all day instead of shorts. We've never had a lot of rain on our November trips. It's possible, but just a little rain here and there and mostly dry. Swimming has always been fine for us - and we're in South Carolina so we're cold weather wimps. We've also been the first-second week of December before and the weather has been fine then too, just pants a little more often.
The crowds have always been low when we've gone, not as low as end of January, but still on the low side. We're always gone well before T-giving though, leaving the Sunday before. Or don't arrive until the Sat after T-giving. We did the parks once the Sat after T-giving and it really was quite manageable - but we picked which park carefully. We do that even with the lower crowds on other days as well, so that likely makes a difference in me saying that it's not too crowded. We still tour smart - but that just means doing certain rides at certain times and figuring the rest as we go. Since we have kids, we like flexibility!
(Just teasing, if it's inappropriate, I will remove)
We went to 
