I completely disagree with you. The first time I saw hogsmead, I had seen one Harry Potter movie, the first one, when it first came out so several years before. It did not matter one whit that I thought the area was well done, and exciting.
This past nov. our group was 16 people. Including some kids down to 10, up to my mother of 88.
Most of those people had never watched ANYTHING HP, maybe never even heard of it. Everyone of their eyes lit up when we went through those brink walls at Diagon Alley.
I will say, through the years I have now watched all of the HP films. Has it increased my enjoyment of the area. No. Not one bit. (frankly, I thought the movies got stupider and stupider as I moved thru them.) I still like the rides, I still like the food, I still like the theming, but even though I now get a few more references, it is no way increases my enjoyment of the area from the first time I saw it and didn't know squat.
BTW, f and f is a horrible ride no matter how you look at it. I won't spend any time there again.
Just because you eat and breath everything HP, you need to accept that you don't even have to like the franchise to enjoy HP land.