I could even see taking dress into account, when it comes to areas with gang activity. I have no problem with the fact that my father is stopped more often because he chooses to advertise his affiliation with a criminal organization on his back (biker vest - colors and patches). It is when it is rooted entirely in factors over which people have no immediate control and assumptions about where certain types of people "belong" that it crosses a line, IMO.
But it isn't just inconvenience. Even if you set aside the psychological aspect, there are real consequences of "profiling" stops. Everything from receiving more tickets (something that is often done to justify the stop - ticket for a minor infraction that is routinely overlooked in most circumstances) to missing work time or even losing jobs because of delays on the commute. When I was 18-20, I lived and worked in Detroit. I missed one job interview and almost lost another job because of traffic stops that turned into searches because "the only reason a white girl would be in this neighborhood - one that was on my route from home to work - is to get drugs" (and yes, one officer said that in so many words). For someone who needs every dime of their hourly wage to make ends meet, that's more than an inconvenience. And what's worse, it not-so-subtly discourages profiled populations from going into those places where they don't belong - not a huge deal for a white girl in a low-income Detroit neighborhood, but a serious obstacle to a better future for minorities who are harassed for being in the suburbs where most of the jobs have gone.
Ferguson was about much, much more than just that one case, though. Look at the policing numbers there, the number of tickets written for crimes of poveryy (broken taillights, loud mufflers, etc), the number of residents with suspended licenses over unpaid fines, the rate of warrants for those same unpaid fines.... The rioting in Ferguson was about the overall relationship between the community and the police, a relationship that is common in poor communities all across the country thanks to political leadership that aims to "solve" poverty by fining it out of existence with so-called "broken windows" policing. And in that sense, law enforcement is victimized by our messed up national conversations about race and poverty as the black communities beaten down by these policies - they're forced into an adversarial relationship where they are viewed as the enemy, in existence to harass and write tickets rather than protect and serve.