In general it drives me crazy, but overall I think "your house, your rules" is the way it should be.
Two things that I think are REALLY rude are tracking mud into someone's home, and/or automatically dumping your shoes in someone's front entryway.
We definitely wear our shoes in our house. I've never had anyone track mud into my home. However, it drives me nuts when people leave a pile of shoes in my front entry. Other guests can't get in the door because of a huge pile of shoes. Not exactly the first impression I want to make.
I've seen signs to remind people to take off their shoes, where are the signs that say "please don't pile your shoes in my doorway!"