We just got back from our Harry Potter Pilgrimage. I grew up with WDW and have taken my son on three week-long trips there. He had been asking to do HP for a few years, but I didn't want to try to do both WDW and HP in one trip, so we made a special trip and spent three days there from 11/22-11/24. We stayed at the Royal Pacific, so we got early entry and express pass. I thought the hotel was great and a better value than Disney hotels.
We had three days but pretty much did everything we wanted in two. We spent most of our time in the HP areas, since that's the reason we went. My son and I are also not big coaster fans, so we didn't ride those. We only did early entry once and slept in and did regular park entry the other two days. With early entry, single rider lines, and express pass, we didn't wait more than 20 min for anything (including Gringotts) except for Forbidden Journey. That wait was an hour because we had bad timing. We could have timed it better and had a minimal wait. The second time we rode, it was a 20 min wait.
It was really nice to not feel rushed in the morning and to not feel "guilty" about leaving the park at 3 or 4 pm because we were tired and needed to rest. By day 3 we were just leisurely revisiting our favorites.
We also took a day to drive over to Kennedy and see the space shuttle Atlantis exhibit. I highly recommend that! We also spent the evening of our arrival day at Magic Kingdom, so the trip wasn't completely Disney-free. I really wanted to see the castle all decorated for Christmas, so we did the Very Merry Christmas Party on 11/21. I don't think I could have justified paying for full one-day admission tickets, but the party tickets are a little cheaper.
We had a great time and I would highly recommend Universal for the Potter fans. I've never read any of the books, but I was still amazed by the theming and level of detail in Diagon Alley and Hogsmeade. I think those two areas are better than anything Disney has right now (I was a little underwhelmed by the non-HP parts of the parks, though).