We have 3 kids -- DS - 15, DS - 12. and DD - 9. We drive down and normally leave very early on Sat and make it all the way down to HH, and check in early on Sunday. Unfortunately we won't be able to leave until later Saturday (still not sure what time) because my DS is in a regatta (on HS crew team). Are you driving down?
We are usually down there the week of the golf tournament. We have never gone, but we avoid Harbortown towards the end of the week. It never really has had an impact on our trip.