Do the things that will make or break your day first thing. I've never been there when WWOHP hasn't died down at night.. but it could happen. If missing Ollivanders will ruin your day then wait in the line.
Here is how I did it when my family was in town- they only had one day. I'm not saying it is the best way.. just the way I did it.
-Got there at rope drop. Stopped in the store at the front of the park. I told them they had just about everything HP related there and this was the ONLY time all day they would get to shop without people everywhere around them. They looked at everything and sort of knew what they would want later in the day when it was a zoo and they'd miss things.
-After shopping they were allowing non-resort guests to walk through Suess. Walked that way. While everyone was dashing to FJ we walked all the stores, got pictures in Hogshead, picture with the train, ect.
-Walked back to FJ and it had an hour wait. They are big enough fans that they wanted to wait through the full line to see the things that are missed on the Castle Tour.
-After FJ the crowds were NUTS.

We exited through Jurassic and explored IOA and US until 6:30pm. (park closed at 9pm)
- Headed back to WWOHP for the rest of the night. Took pics in the Dragon Challenge line, pics of castle in light and dark, watched the school chiors, drank butterbeer, ect. The later it got the less busy it was. By 7:30pm we were able to get into Ollivanders with a 10 minute wait. We stayed until closing then shopped and picked up everything they had picked out earlier in the day.