We just did both resorts in mid-May. We didn't drive direct because we did 5-day stays at each. When we checked out of HH we stayed around and enjoyed the resort, and finally left mid afternoon and drove to Savannah where we stayed downtown in a hotel for the friday night.
Saturday we drove to St. Augustine, saw the sites, and spent the night there. On Sunday we finished the drive and arrived at VB.
Splitting it up this way made it very relaxing, plus we saw other attractions on the way.
Biggest tip I can give is to try and time things so you don't wind up driving through Jacksonville during rush hour. Outside of that, it's an easy drive between the two resorts.