We are going to be in Mount Vernon in May. Is 4 hours enough time to enjoy Mount Vernon or is that rushing it. I want to be able to get back on the road for 1:00pm to head back home to CT. I do not want to hit rush hour traffic in NJ or NY so I want to take route 81 home and that is a 9 hour drive.