We were there the same week last year that you're going. Most days we were able to wear shorts and T-shirts with a sweatshirt added for early in the morning and late at night. We only had rain one day, and that day it was cooler so we were in pants and sweatshirts all day. We were able to swim. From what I've read, November weather can vary a lot. The forecast won't be accurate this far out, but hopefully it will be in the day or two before your trip. Plan on bringing both pants and shorts and a sweatshirt/fleece jacket, but adjust the quantities of wam vs cold weather items once we get a better forecast.
ETA: I'll add that we're from near Washington, DC, so we were leaving weather in the '50's to go down to '70's and '80's in Orlando. YMMV if your home state's weather is different than ours.