We went the week after Easter last year, and it was a great week. Crowds steadily declined throughout the week.
The Rival Run is at the end of the week, but it is also the least popular of the Disney Race weekends and the crowds aren't affected as much as they are during say Marathon weekend.