We go every year in April or May. The closer you get to Memorial Day, the busier it gets as more schools on traditional schedules are out. On the front end, you want to pay attention to when Easter is. If you get past about 10 days after Easter, the crowds from that are pretty well gone. I haven't noticed Mother's Day making any real impact on crowds.
Easter this year is late - April 23rd. That's still a full week before the first week in May, but if you arrive that weekend before the 1st you may still encounter some crowds from those who arrived on Easter weekend and stayed over. The second week might be slightly less crowded, but either one should be just fine.