Heck no, never for me. I have only ever gone to Disney during SB but it's defintely not nearly as bad as people make it out to be. Logic says that it will be more crowded than usual but it's really not bad.
As far as wait lines go, it really depends on the ride. When I last went (spring break of this year) EE was new, we went on twice, and the total wait time for both times was 30 minutes TOPS. We had fastpasses but never used them. Soarin was about 40 minutes but there are trivia screens in the queue so we made a contest out of it and it went by pretty quickly. We walked on POTC, literally, no wait at all. IASW we waited maybe 10 minutes max. Splash and B Thunder were about 20 minutes, BTMR was maybe a few minutes longer. Spaceship Earth, walked on; Mission: SPACE I walked onto [mind you, I was in the singles line (my entire whimpy group left me)], ToT was 20-30 minutes. Kilimanjaro Safari was probably our longest wait at 45 minutes-an hour. TTBAB 3D show was about 30 but that's a really great time to admire the Tree of Life. We obviously went on more but the other wait times are gone in my mind.
Food lines were the worst for us. We at twice the 2 days we were there [both counter service (Electric Umbrella in EPCOT, and some place in Dinoland in AK) and the lines were REDICULOUS. Luckily, the place in AK had the burger condiments bar so I snacked on the pickels until our food came
So I would suggest going during SB-the poor thing gets a bad rep. Just start getting in line for food half an hour before you're hungry!