Legally? Maybe not. Morally? That's debatable. Since there were plenty of open seats they could have moved to toward the front of the bus (I know, because that's where I was forced to sit, with the rollator crushed on top of me), it wouldn't have impinged upon their "safety" whatsoever. No, they just didn't want to be inconvenienced by getting up. You can play devil's advocate all you want, but the fact of the matter is they were inconsiderate and caused myself physical discomfort because of it, and delayed the rest of the people boarding the bus behind me because it took me some time to try to turn around when they refused to move.
I'm sorry but I don't feel it is unreasonable for me to expect that I should be able to sit in the handicapped accessible seats on the buses, particularly when there are signs posted above the seats which state verbatim that "priority seating" for those is given to passengers with disabilities. There is no reason why I should have to wait for the next bus when there are plenty of other available seats, just because I can't strap my mobility device to the wall. In almost every case, literally all the people would have had to have done was go across the aisle, step up to the seats in the back of the bus, or have one person move down one seat. I refuse to take the blame for their rudeness.