Going by the last two years, reports were that the crowds built up through the week before Mem Day weekend. So Wed will probably be heavier than it was the week before, but it's just going to get more crowded the later into the week you go. And then the holiday monday has been much lighter, as it seems people start to go home on that day, having taken a couple days before the weekend, rather than taking the Tuesday after for travel.
However, this year they have more things going on around that time, and of course it's closer to summer breaks, etc etc, so what happened the last two years might not be as good to go by!