My wife and children are not, either, but my DD8 is such a Potterphile that she happily got up with me to be there at 7AM each day and we then met my wife and son later in the morning once they were up and going.
I will say that the single rider line for FJ moves very quickly, and since there is no way to interact with others during the ride even if you're riding with family, it's a great way to do the ride with a shorter line (although you miss seeing a lot of the castle that way). My daughter did single rider 15 times (no exaggeration), in addition to the four times I rode it with her, while I waited in the Child Swap room, and each time it took her less than 15 minutes in line, so that may be a strategy for you to ride FJ without getting there at 7AM. As for Ollivander's, I think if you don't get there early, you'll just have to deal with a longer line, but it never looked awful, in my opinion.