I guess weekends would be a little better. If you're thinking Friday night as weekend, don't. Travel to DE and MD beaches can get intense. That can include Saturday morning...and the reverse coming out Sunday night.
I don't usually let weekdays scare me driving to the South from NJ/PA. The main thing is you don't want to get caught in the DC (and sometimes Baltimore) rush hour. Actually, I've found that the worst stretch isn't the DC beltway itself (I-495/95), but I-95 south of the beltway to about Dumfries. If you're going south in the PM rush, that's where the HOV lanes merge back in. If you want to avoid the DC area altogether, you could consider taking US-301 and merging back in to I-95 a little north of Richmond. There's various ways to do that: pick up DE-1/DE-99/US-301 or I-97 in Baltimore, then US-50 or MD-3 to US-301, among others.