We've gone during Spring Break for the past 3 years, once Easter week, the others the week before. It is crowded, yes, but it is not that bad and we were able to everything we wanted. The key is to have a plan and take full advantage of fast pass. The biggest problem at this time of year is that the fast passes for the more popular rides are usually gone by early afternoon (or the return time is 6:00 or so). We would enter the park and immediately grab fastpasses and ensure that we got our next set as soon as we were able. Ride the big attractions early in the day. This especially works well at EPCOT since so many of the attractions there have a very short wait, even during peak season.
Once in a while, we did the single rider lines with my older children. That can help cut down on the waits although sometimes we found the single rider line to be no shorter than the regular one.
Everyone will tell you to avoid EMH at this time of year. In fact, I've not heard one good thing about the evening EMH's during peak season; however, we've always had a good experience, maybe we've just been very lucky. My DD and I went to MGM's EMH last year and TOT was a walk-on - we rode it at least 5 times in a row. RNRC had a line, but it was maybe 20 minutes. Don't rule out EMH - give it a try and if it's crowded leave. Also, sometimes the crowds are heavy at the beginning but thin out considerably later on.
The biggest drawback to me is the weather. Yes, it is beautiful and rarely rains but it is unpredictable and often gets chilly in the mornings and evenings. Therefore, it is necessary to pack clothes for warm weather as well as jackets, jeans and sweatshirts. This can be very inconvenient if you stay in the parks all day like we often do.
I was very nervous the first time we went during Easter, too but we had a wonderful trip and it wasn't as bad as I thought. I guess I was prepared for the worst. Also, I do not handle crowds well at all so if I can do it, anyone can.