Hmm..I think the latter week would be much more likely to be more crowded. Lots of schools are getting out just prior to Memorial Day. Attendance charts in the Unofficial Guide show a major spike at Memorial Day and no real dip again until August. I know everything is an educated GUESS based on past patterns, but I would go early if you can.
I've been once in May and attendance was definitely building towards the end of our visit, which was towards the end of the month.