It is probably the loudness that is getting to you most, not necessarily that it’s a cell phone, on speaker or not. If the cell phone user is speaking at a normal conversation level, you probably wouldn’t even notice it. And the same applies to face to face conversations that are really loud. Maybe it is part of the trend where we don’t always seem to remember there are other people in the world.
Granted there are occasions where something urgent is going on, and you have to take that call right then and there, But we have definitely gotten used to immediacy: instant answers, instant communication, etc. A phone call, e-mail, or text beckons us to respond/answer it right now.