I would imagine the drive down would be pretty easy. We have done it numerous times with 3 kiddos. BUT, we drove home (SE CT) from Myrtle Beach last April break, and the traffic around DC was HORRIBLE!!! (like 3 miles in 1 1/2 hours horrible!!!) Fortunately, we were staying with relatives in Northern VA, so we didn't have to continue in the mess - can't even imagine how long it would have taken to get home. If you can somehow plan to drive at "off" times (not so sure that exists from DC through East Haven, CT), it might not be too bad. We will most likely be joining the masses again this April break as we head to the Fort Myers area.
Also depends how the kids do in the car. My kids have always been very good in the car, so we have had some really enjoyable experiences. It also depends how the parents do in the car - trips with my father behind the wheel were not always so pleasant!!!
It is a very do-able trip, but you may have to psych yourself up! Guess you just have to prepare yourself for the worst, but hope for the best!