We were there Nov. 1-8 this year and the weather was nice - a bit cool (65-70's during the day and 50's at night, breezy). The crowds were pretty light, with the exception of the F&W Festival on Sunday. We avoided Epcot on Friday night & Saturday for just this reason. The super-headliners like Toy Story Mania and Soarin' still built up pretty big wait times by noon or so, but most everything else was easily accessible with FP's or a short wait.
Check out easywdw.com and take a look at their monthly calendars. There are some great park recommendations, if you prefer to avoid EMH (or are staying off-site). If you do want to take advantage of EMH, then I suggest park hoppers so you can still tour the ideal park for half the day.
I thought it was a great time to go overall. If you want to swim every day, maybe not so much. But it still helps tremendously to plan which parks you'll be hitting on each day to maximize your crowd minimizing.
November is a great time to visit IMHO. We especially like to visit after the F&W event in that narrow week or so before the Thanksgiving holiday crowds. It was pretty warm this year - about 85, but pretty comfortable. Nights were chilly, but not cold. No crowds, except for MVMCP.