If you get wet (whether from sweat or a water ride or rain) being inside a frigid restaurant or store is miserable! I always bring a sweatshirt or jacket. The few times I haven't, I've had to buy one.
This trip I'm bringing a sweatshirt as well as a lightweight hooded knit jacket that I can easily throw in my backpack.
Even some of the shows (PhilharMagic, It's Tough to Be A Bug, etc.) get cold sometimes.