We're also debating this issue. We're leaning towards staying around the hotel and swimming, napping, etc. But, we'll be taking friends with us-- their first time to WDW-- and I'm not sure if they'll be offended if we sort of dump them for a day in the middle of vacation. Of course, after two days with a Disney commando (me) they might be glad to get rid of us for a while.
I'd say if you have little kids (ours will be 10 and 6) definitely skip the parks the day of the party. It's probably what we'll end up doing because our 6 year old will be a wreck if we run around all day.
Julie