If you are a great fan of fantasy, magic, wizards and mystical/magical creatures you will like Harry Potter.
I love that type of thing so HP was a natural draw for me and the grands.
I went with someone who had only seen part of one movie and it just didn't have the same impact with them that it did with me. I understood the nuances of Moaning Myrtle in the loo, the owlery and Buckbeak. It meant nothing to the other person and could have been just generic theming.
I read the books as they came out, with the grands, then got the movies as they came out and we've watched them many, many times.
Really it is great theming but not knowing anything about the story behind it seems to me that you could go to any well themed attraction and enjoy it, maybe not as much as this exceptionally themed portion of the park but why not maximize it?
My advice is to at least watch one movie, you might like it. If you don't like the movies/books chances are you'll be wondering what the hullabaloo is all about. FJ is a great ride, well themed and will be fun, but the different facets won't have much meaning.
As for the OP, we went on Nov 7 a Sunday and it was low crowds, despite it being NJ week. However, can't say the same about the crowds at WDW at that time.