I used to live in DC and my wife grew up ~1/2 mile from Mt Vernon.
I think you need to pick one or the other. If you've never been to DC, then I would suggest you spend the day on the mall in DC. The Smithsonian is huge, comprised of many different buildings. Each one would take a few hours to tour in a meaningful way. Pick the topic most appealing..Air & Space, American History, Natural History, Art, etc... and immerse yourself.
Depending on whether or not you have an accident on the beltway, traffic is a crap shoot. Where are you staying? It may be worthwhile to park outside the city at a metro stop, and take public transportation in. The metro has a station that is dead center on the mall, between the capital & Wash. Momument. (I worked three blocks from there). IF you are heading to connecticut, I'd suggest either New Carrolton or Greenbelt. That will allow you to avoid navigating out of town. Plus the Metro is pretty darn nice. Worth the ride.
Rush hour begins ~3 in earnest. You're biggest concern though will be getting through Baltimore during rush hour. Listen to the am news station there. The Ft. McHenry Tunnel is the main route, but the Harbor Tunnel (I-895) is a good alternate. Both end you up in the same place eventually, but if there is an acident on one route, the other will be much faster. I live in Philly now, but get baqck to DC often. The run is ~3-3 1/2 hours. Add the time necessary to get from Philly to CT. and you have your travel time. )BTW - 95 from DC to Philly is a NASCAR race. Buckle up and watch out for the other guy !!!)
Here is a map of the Metro Stations....
http://www.wmata.com/metrorail/systemmap.cfm
Here is a link to the Smithsonians.....
http://www.si.edu/
Have fun !!!