We were there right before last Thanksgiving, Sat-Mon (Tues we went to Kennedy Space Center). The crowds did get progressively heavier with each day. We stayed on-site and had express passes, but the crowds weren't unbearable. Single-rider lines for the HP rides worked great (especially Gringotts). The longest lines we waited in were Forbidden Journey (about 45 min, but it was really bad timing on our part) and the Shrek 3-D movie (also about 45 min, and we used express passes!). If Shrek and Minions are important to you, I would recommend doing those first thing in the morning, since the lines really seemed to back up, and they move slowly since those are shows and not continuously-loading rides.
My only other comment would be that the HP areas do get elbow-to-elbow crowded with people. Part of this is because of the design/theming. It takes away from the experience, so try to spend some time in the HP areas in the early morning hours so that you can really take in the design/attention to detail.