Eh, I don't find that rude either. Sometimes you have something you want to say privately to someone and doing it quickly and quietly via a whisper instead of both leaving, saying what you need to say, and returning is just easier. The later is just as obvious anyway and sometimes you either can't or don't want to wait until later to say whatever it is. I really don't know why people care so much about that kind of thing. I couldn't possibly care less if someone whispered something to someone in front of me or broke off into some other language.