Well, it depends on how into Harry Potter you are. I'm a big fan and I could spend an entire day in just the HP areas of the parks (and have done so in the past). There's so much to see and do besides the rides.
First you have all the stores in both sections, which are worth going into even if you don't want to buy anything. It took about an hour for my friends and I just to visit all the stores in Diagon Alley, and we didn't even buy anything.
You should definitely eat at both Leaky Cauldron in Diagon Alley and the Three Broomsticks in Hogsmeade. If you don't want a full meal at both, at least get a snack. The atmosphere is great. You can also get ice cream at Florean Fortesque's in Diagon Alley and some neat beverages at the Hopping Pot (which is right next to the Diagon Alley stage area).
There are two performances in Hogsmeade, the Frog Choir and the Triwizard Champions. The Diagon Alley performances are The Tale of the Three Brothers (a dramatized puppet show) and Celestina Warbeck and the Banshees (a jazz style singing group). All four of them are definitely worth a viewing. Performances alternate every half hour and start around 11 or so, I think.
Then of course there are the rides, which could have long waits, and Ollivander's wand shop.
If you want to do other attractions as well, I'd want to spend the entire day there. However, if you're not a huge HP fan and just want to walk through the area and do the rides, you could probably see it in less time.