You're right, flying into Charleston or Savanah is very expensive, and often connects through Atlanta. Flying into Jacksonville, FL is reasonable, but still a 3-hour drive. When you figure in the time and effort to check into the airport, flying, getting luggage, renting a car, and then driving 3 hours (repeat on the way back), I'd just as soon drive.
We have driven to HHI from the Hartford, CT area on two occasions. The first time was with our 1 1/2 month old daughter, and the second when she was 1 yr. She slept most of the way, so it was great! I'm not sure if I would drive to WDW very often (we've done it once) since it a long 2-day drive, and you can pick up incrediblly cheap airfare if your diligent about watching fares. HHI can be reasonably be done in one long day, or two fairly short ones.
Its a long drive, but not difficult if you avoid rush hour times in the NYC and Baltimore/DC metro areas. If you drive on a weekend down, its not bad as long as you leave fairly early and miss the NJ shore traffic. Driving on a weekend back up is OK too, especially on Sat. On Sunday, make sure to be through NYC by mid-afternoon.
Without major traffic problems, it would take 12-14 hrs depending on how many stops you make and for how long. A long day drive, but doable. My best luck has been to bugger out of work early on a Friday, get through NYC by 3pm, stop for dinner as traffic starts getting heavy in Maryland, get back on the road by 7pm, and stop in Fredericksburg or Richmond, VA around 9-10pm for the night. From there it's an easy 7 hour drive to HHI.
Coming back we usually bite the bullet and drive straight through (because we end up wanting to stay one extra day and use one of our driving days), but doing the drive above backwards in 2 days works too.