I would 100% go. DS went in a decent rain storm with a friend when he came home on spring break. They saw everything they wanted to in DL including ROtR, then went back and upgraded to park-hoppers and saw everything they wanted to in DCA too. That's pretty typical.
Crowds will be nil - BUT - be prepared! The rain is cold here and a cold air mass always follows the rain - you don't want to be walking around in soaked clothes. Umbrellas and ponchos won't cut it. You want actual rain gear - head to toe. Wet feet are the worst.
It's already cold. It will only be getting colder as the week goes on.