I completely understand. One of the big selling points to me was the zippered pocket on the back of the bag. When wearing it like a backpack, I can reach back one handed, unzip the pocket, and pull out my phone. It is very nice.
To get into the main compartment, you'd have to pull the bag around to the front of you if it is behind you because the zippers (at least on ours) are pretty secure and don't slide down as easy as a traditional backpack, so you'd have to hold the bag with one hand and unzip with the other.