It hasn't been crowded IME.
Well maybe the specific events are, but the events are OUTSIDE of the parks or during times that aren't open to other people. We've been there as non-participants, and you really do not notice that anything is going on. (the events keep the crowds away from the parks during open hours)
(except for maybe possibly running into Evanna Lynch randomly behind the 3 Broomsticks and getting a picture with her because she is the loveliest* sweetest person ever...and then later realizing she's staying at your resort)
(*seriously, she's lovely.)