It's possible, but I wouldn't really recommend it, unless you want to take a commuter train in.
Baltimore is only about 35 miles from DC, but you wouldn't want to start making the drive until after rush hour is over - lots of commuters from north of DC clogging the roads into town in the morning. That means you wouldn't get into DC until late morning, and then unless you want to sit in outbound rush hour, you're looking at leaving by 3:00 PM or waiting until late evening to get back to Baltimore.
A