We've stayed at SOG twice and are going back next Nov. We walked to Poly and the TTC every day. It's an easy walk and the first time we had our 12 and 14 year olds with us (1998).
We even had our car last time since we came on the auto train but we never moved it the whole week.
BTW, we wear pedometers every day and we did around 25,000 steps each day (around 10 miles) and only a small portion of that was walking back and forth to the Poly. So, yeah, in the grand scheme of things, the walk is not a significant factor.