I'm in a similar boat. I'm a die hard HP fan going to WWoHP with kids who aren't crazy about rollercoasters, but are willing to try Forbidden Journey of Harry Potter. We're going for 3+ days and here's the plan:
Day one, enter IOA as early as possible, ride FJ as many times as we can before lines get crazy then ride Hippogryff (which is a kiddie friendly ride) a few times. Then spend the rest of the morning shopping at Hogsmeade before having lunch at Three Broomsticks. Afternoon doing other IOA rides before heading back to Potterland for end of day.
Day two, early IOA for breakfast at Three Broomsticks, then off to Olivander's for wands (something that will be just as good as a ride, and probably a line as long as a ride - but you're kids will love it so don't miss it!). Go through line for Dragon Challenge just to see the sets, then chicken out before ride. Then spend afternoon at Universal Studios.
Day three, early IOA for more HP rides and shopping, then do anything else we may have missed. hehe
I think you'll be fine with two mornings at HP if you get in at early entry time for on site guests, but if your kids are anything like the uber fans I'm going with, you'll be happy for three, then at least you can have some time to try a few of the other rides and attractions at Universal.