In our regular daily interactions with people, I completely agree.
On a city street late at night, SHOULD I react the same way to someone who LOOKS LIKE a stereotypical "gangbanger" (isn't saying that we know he is, obviously), as to someone dressed/groomed like a typical businessman?? In that one, quick instance, does it MAKE SENSE for me to react EXACTLY THE SAME WAY? Statistically speaking? Doesn't mean I'd be correct in every instance about each of those individuals!
We're fighting humans' self-preservation and self-protection instincts here. Animals (including humans!) size up potential danger very quickly, based on MYRIAD cues, and react accordingly. In some instances, this makes sense to maintain. In others, they may be prejudices to work against.