BUT for almost every ride we had to remove and store the bag which is a huge pain.
Wow, that's not our experience. We don't find it to be a pain, and we find that there are far more rides that you can bring things in than not. You must have ONLY been hitting the bigger or spinniest-of-them-all rides.
My girls don't ride roller coasters and we are really only going for the Wizarding World.
Gringott's, Dragon Challenge, and Forbidden Journey don't allow you to bring stuff. Hippogriff, the train, of course Ollivanders are fine.
If after two days of all things Harry they are board then we might venture into other places on the third day.
I beg of you to perhaps open your mind to the rest of the parks earlier than your last day. We are huge HP fans (do you belong to the facebook Hogwarts Running Club? if not, you should!) but couldn't do 2 solid days in Wizarding World. Gotta break it up, hit Men in Black, see the sights, etc etc.
Do you know if the non-riding lines for the rides allow backpacks?
If you are not riding, you can take a bag with you. That's what DH does, since he's of a size that doesn't work with any of the thrill rides (and so far not Gringott's or Forbidden Journey either). He does have to tell each and every Team Member that he's not riding (though in general they can tell already). And you don't have to be in a "non riding line" (you mean the castle tour line? I read that that's been closed, FYI); ANYONE can go all the way through ANY of the lines at Universal. You can bring your infant into the line for The Hulk. You'll exit with the infant before the seating area, but the infant can go through. And a non-rider just goes on through the line until the exit point for that line. At Forbidden Journey it's at the sorting hat. Dragon Challenge you actually pass close to the seating area to leave.
Just tell the TMs that you're not riding and it should be fine.
though I will state that we have NOT yet done the above since metal detectors went in. I don't think DS and I rode RRR on our last trip, and if we did DH didn't join us (that line is boring and he would rather wander). And Hulk was down.