Yes, do Olivanders first. That line only gets worse as the day goes on.
As far as FJ goes, walking through the main line of the castle is fun, but one thing I enjoyed as well is walking through in the single riders line. If you are not in a race to get back on the ride you can linger more casually since you don't have a crowd urging you on. Anyone coming down that line you can just have go around you.You don't get to go through 100% of the castle this way, but I did find it easier to stay in the parts of the castle containing characters dialog where in the main line the crowds got antsy if you paused to listen. So go in the main line one time to see everything, but be sure to use the single rider line for a second time to catch everything.