My understanding is that the one on Victoria Street in Downtown Toronto is generally the least busy.
We got our passports there last spring. I got there shortly after they opened. There was a line-up already but nothing like I've heard people describe at some of the other locations. I waited in line for about 20 minutes in the line-up for them to do a preliminary check of your documents. I was then given a number and sat down in the waiting area and did some work. My number was called within about 45 minutes.
By the way, if you are bringing any photocopies (rather than originals) of supporting documents for an application for another family member and the document is a two-sider (for example, drivers licences and photo health cards), make sure you copy both sides of the document and have the guarantor sign both.