Well, apparently I am in the VAST minority, but I was born and raised in a very northern state, and my brother and I were taught "Yes/No, Ma'am" and "Yes/No, Sir". It was what we were expected to respond with when given a directive or asked for our attention by our parents or any other adult. It never caused me to fear any adults, and I do not recall any time when we used it sarcastically except with each other, when one of us would get a bit bossy (

)
I ceased using it to address adults when I became an adult. Except for Mom and Dad. They still get sir and ma'am when the occasion is appropriate.