I always pack a couple nicer tops for the days we have PSs at fancier digs, but that's just a personal preference.
The issue for us is that it is very cool in some of the spots, and can be quite a contrast to how we've had to dress for the day to survive the heat. My DD just bought a lite sweatshirt jacket to wear for the extreme.
WDW ON JUNE 24
My husband prefers to dress in long khaki's and a polo shirt for the nicer places, and in cooler months will usually wear a sports jacket as well. We dress like we would if we were at home going to a restaurant of the same calibre.