The start of the Dec. is not a very busy time to visit. Harry Potter should be just fine. Heck, when I went this summer, Harry Potter was fine except for Friday and Saturday.
If you are going to be staying onsite, I suggest you take advantage of the early entry into the park so you can see and do almost everything before other guests are allowed into the park.
If you are not staying onsite, then I suggest you do if you can afford it. You will get FREE express access to nearly all of the rides. Just show your room key at each ride (except for 3 that don't have express) and you will skip the line and be on the ride in a matter of minutes. You also get the early entry as mentioned above.
If you can not stay onsite for any reason, then I suggest seeing Harry Potter in the evening. It is the first place everybody goes to, and in the evening the area can empty out (not to mention the area looks so much better at night!).
Keep in mind Islands of Adventure (which has HP) will close at 7pm that sunday and Universal will close at 6pm. If you are only visiting for one day (and not staying onsite) then I would visit Universal right when they open at 9am and start leaving the park around 1pm. You should be able to do everything in those 4 hours. If you start leaving around 1pm, you should make it over to Islands by 1:30 which is more than enough time to see all the major rides at the at park. I would get to the Harry Potter area around 4-5pm.
Enjoy your trip!
