Really? My city is currently the murder capital of the country, but I still agree with what Brian said:
I'm not going out of my way to just hang out in the sketchier areas, but you're right, some of the very best restaurants (or bars) are in those neighborhoods. My current neighborhood is "officially" sketchy, but we have the best neighbors in the world (and a fantastic corner store with an excellent deli counter across the street). And my previous neighborhood was supposedly one of the "better" areas, but someone got shot outside the bar next door--the same bar I frequented regularly. Crime happens in every city in the world, and I'm not going to live in fear.