Looks like we are going to make our 1st road trip to WDW from exit 8A of NJ Turnpike. Airfares are out of sight!!!

It will be myself (44), my husband (53), my son (9) and daughter (9). Neither of us are drive all night kind of drivers. I am ok driving early hours and my husband is ok driving the later hours, but neither of us wants to drive the graveyard shift from 1 am-4am.
We are going for a week beginning Memorial Day weekend. How bad is 95 traffic?
I can take Friday off from work. Are we better leaving Thursday night around 6 pm and driving 5-6 hours to get past Wash DC/Richmond (staying in a hotel overnight) OR better leaving Friday around 4am and trying to drive straight thru?
Concerned Memorial Day weekend traffic might bring a lot of traffic.
Leaving WDW on Sat (the following weekend), what is the best time to leave?
Thanks!