The chances of are far greater that the bottom of a shoe is far more unclean than some else's hands. As others have pointed out, most of us can wipe ourselves without transferring bodily fluids or solids onto the handle.
Your shoe however may well be standing in urine or solid waste while you are relieving yourself. Then that gets transferred onto the handle for the next person, who in turn can transfer it to the door while exiting the stall and before washing their hands.
Your shoe however may well be standing in urine or solid waste while you are relieving yourself. Then that gets transferred onto the handle for the next person, who in turn can transfer it to the door while exiting the stall and before washing their hands.