A couple things to remember relative to the cold, even if from the north.
There is a lot of water as WDW, making the cold even more chilling feeling as the wind whips off the various bodies of water. Also, when you are home and it is cold, you typically ar in and out of cars, home, work, shopping, whatever. You are not walking long distances in the cold (typically) and standing around in it. You are doing both at WDW, thereby making the cold feel really cold, moreso than at home.
Just my observations after about 15 December trips over the years.