We were there in May 2001 for two weeks. Mother's Day was our second day there. It was great! The crowds were thin, the weather was perfect.
I believe most people would go the week after Memorial Day, because of school schedules. In the south, schools let out around the end of May. Although recently there has been a push to start early August and get out early May.
There are still enough schools that let out late May or early June, so you're picking a good time.
Have fun!