I'll tell you what the difference is in your scenario -
If we are there to watch the fireworks/parade, we are not going to deliberately stand behind tall people because, being petite, we know I can't see.
We'll get behind shorter people. But, if people, at the last moment, hoist a child on their shoulders, you have NO way to plan for that. Does that make the situation clearer? Of course, taller people have the same rights as anyone, but they don't suddenly 'grow' when the fireworks start!
"...they don't suddenly 'grow'..'" made me laugh. I do find shoulder kids rude as well as constant cell phones and iPads.