I also spent a long time at the London set, 6 hrs. Going there was what convinced me that I had to take my younger sister who is a huge HP fan to Universal. I had already been to Universal, and just kept thinking as we toured, "If she is this into this place, she is going to die when she sees the HP sections at Universal." The warehouse and the parks are very different experiences, and I didn't feel like I was repeating anything or that one place was better or could substitute for another.
I've gone twice with my sister to Universal now because once just wasn't enough for her. The first time HP was the main purpose of the trip. The second, we spent a good amount of time in the HP sections which we both love, but we also did other things as well.
My mother also wanted to go only to see the HP sections, so I went once during a Disney trip with her, and we just went over for the day to see Harry.
So basically, I've gone twice now with people who at first were only interested in HP and both had a great time.
If you are a big HP fan, it is definitely worth it. I spent 6 hrs in Hogsmeade the first time I visited it. We get really into exploring everything, casting spells, shopping, eating in the restaurants, sending owl mail, watching the performers, and experiencing the rides.