I've had that happen before. I didn't swat the persons finger, but I did verbally question them as to why they were pointing their finger in my face.
Their response indicated that they really didn't even realize they were doing it, until I said something.
Regardless, it's rude to point your finger in anyone's face.