Depends on the person pointing the finger at me, of course. Someone I know will get a swat and a discussion, a stranger will get a step back, or more, then a call to the police. The call would be about being verbally assaulted, with the possibility of physical assault. I wouldn't just walk away - who knows what the whackadoodle would do to the next person. If there are no consequences for the whackadoodle behavior, then that behavior is reinforced and allowed to continue and possibly escalate.