For reasons that I will not get into here, I need a GAC and I have to say that I have never heard these comments. I also have friends that need wheel chairs, walkers, ECVs, etc. and again, I do not hear them getting these comments. That being said, those in wheelchairs, ECVs, etc. can be stopped off to the side of the walkway, looking at a map, etc. and people will run into them and cuss them out, when the person walking was the one not paying attention.
Now if you do have someone complaining about it to you, ask them to rent a wheelchair for a day, push themselves the entire day, do not get out of the wheelchair, except to transfer to ride vehicles, transfer to the toilet (they must take the wheelchair in with them, pushing themselves until they get into the stall) and see how many attractions they can get on in a day, they will quickly find out that it is not as easy as they think. Explain that those with GAC cards generally have just a little bit lower difficulty moving out of people's way, but since their disability is not visible, no one will move out of their way. This will give them a general idea of what you go through. In addition those with GACs will wait longer on some attractions than those in the regular lines. Overall, it balances out in the long run.
Another thing to remember is that it ALWAYS easier for someone that is walking to move out of the way (even with a stroller) than it is for a person in an ECV, wheelchair or that is suing a walker or cane to move out of the way.
As for those in wheelchairs switching places and such, please note that things may not always be what they seem. ECV and wheelchair rentals are expensive and sometimes there are multiple people in a party that can walk for a while, so they will rent one ECV/wheelchair and switch off as needed throughout the day. That being said there are indeed those that abuse the system, we all see it, but in general, those of us that do need it are pretty good at catching those that are abusing the system and reporting it. Why? Because the abuse makes it harder on all of us that have real needs.
One thing that I do wish Disney would do is either print people's pictures on their GAC cards or require ID when they go to use them. I often have seen people hand their friends their GAC card and said here use my card, but I don't want to (or can't) ride.