You just have to be careful. Since you're a photographer, you're already observant, so keep an eye on who and what is around you.
I used to make money committing photography, and I've been to many places where safety was questionable (favorite quote from local, as I was setting up a shot in Durban, South Africa: "Well it's pretty safe-we're down to one murder here a week") and came out okay. It helps a lot to know the area and to have a "native guide". And it's not only the "bad element" you have to watch for - one time in Mexico we were stopped by Federales (right out of an old movie - dark, lined, moustachioed faces, complete with submachine and jeep-mounted guns) and questioned extensively. Our Mexcian team member (thank God he was with us) chatted with them for some time, money changed hands and we were on our way. Generally a large city will have good and not-so-good parts, just don't go into the bad parts alone, in the middle of the night, with a really large white lens <g>. I like using a backpack that doesn't scream "camera", like the Tamrac Adventure series.
IIRC, Toronto usually ranks near the bottom of the Top North American Crime Cities sort of lists, while NY and Boston rank in the middle. So you're most likely safe with your camera and should have no problem.
Another point - the two times I've been mugged have both been within a hour of home (I live near Boston, MA) and both times they weren't interested in the camera or equipment; once they took all the cash and the other time took all the cash and my friend's travel toothbrush kit. Still haven't figured that one out. Both times were in populous shopping areas. I've walked many times down streets through "bad areas" in Boston (where there are still some very cool old architectural details) without any incident, so you never know.
HTH.
---Ritch