Hi Simon
Last week in April first week in May are the 2 weeks we usually go, going to Egypt this April/May though, the weather is really nice, warm with very little rain in our experience.
As you said the crowds are less as easter is out of the way and most of the spring breakers have gone but it's still by no means quiet, you'll still need fast pass, express pass though.
We would whole heartedly recommend this time of year to anyone hope you have a really great time.
Calvin & Anne