We went exactly that time last year an it was busier than I expected. I am sure that might have had to do with Diagon Alley being open for only 3 months. If you want to experience Harry Potter with a small amount of people and no wait for the major attractions, then staying on property is worth it.
We stayed at Cabana Bay, so we did not have the express pass and found that the lines were reasonable for all other rides, except Forbidden Journey and Rockit and Gringots. All others were 15-30 minutes tops, including Hogwarts Express, I think because the trains are so high capacity that the line never gets too long.
Also Universal closes early on Sunday and Thursday, Friday and Saturday for Halloween Horror Nights, which is amazing by the way, this year I believe they are adding Wednesday to Halloween Horror nights as well. So Universal will only be open from 8 AM to 5 PM most of the days of your visit. Islands of Adventure will be open later but make sure you factor that in.