Actually, someone I know told me that they get asked this all of the time too but they go to Vegas a lot. So I guess that it can happen with other places too.
People who make comments like these are just plain rude IMO. People should be free to enjoy anything they wish.