PP is pretty much right. The way I learned it, with most service animals that come from actual service animal schools it goes like this (Just fyi I'm basing this off how Guiding Eyes for the Blind works/worked)
A puppy is born

and goes at a few months old to a puppy raiser. The puppy then pretty much becomes the puppy raisers world. They're house broken, taught basic commands (sit, down, stay) and socialized. They'll start wearing "blankets", which is that little jacket looking thing that will typically say something like "Guide Dog in Training" on it. The purpose of that jacket is not just to prove it's in training, or tell you what its in training for, but also to get them used to eventually wearing a harness. It's almost like being in uniform for them. They start to associate wearing the blanket or the harness with a work mentality. After their stay with the puppy raiser, they go back to school where they go into intense training, learning how to actually guide someone. This is where they learn stuff like stopping at curbs, when to ignore a command if it may put their future partner in danger, etc etc. At some point in this a visually impaired person will come to the school and start training how to use a dog (usually using a dog that's trained to listen to anyone). Once they get the basics down, they're matched with their own dog. GEB takes into consideration how fast you walk, how tall you are, even if you have mobility issues when pairing you with a dog. For instance if someone has a bad hip, they're not going to pair that person with a dog that likes to walk quickly. Once paired, the rest of the time is spent working with your dog, bonding, and learning how to really respond to each other.
I know that's going a lot further than your question asked, but I figured I'd tell you to explain there's a TON of stuff that goes into training those dogs. When they're in that harness, they are totally switched into work mode. If a person with a guide dog were to be walking, and someone tried to pet it, that's an obvious distraction and could put the person at risk of getting hurt. A lot of people understand that, but when they see someone with a dog say at a restaurant where the dog isn't actively doing work, they assume it's ok to approach it. The way you have to think of it is the dog is still at work but on a break, he's not off the clock. I'll tell you with both our dogs, they knew the difference between being in that harness and being out of it. Our second one was very mellow in his harness, did his work and all, but man when that harness came off he was bouncing around like a moron, butt wiggling and tail knocking stuff off the table.
Now as a disclaimer of sorts, I'll say this. I haven't been around a puppy in training in a long time, and GEB may do stuff differently now. My experience is really around fully trained dogs. Other schools may do things completely different than the one I'm familiar with. I also can't speak to schools that train any other kind of service animal (dogs for the deaf, mobility assistance etc). I've known people that won't let anyone touch their dog, some that will let someone like a handicap child pet their dog, or will let anyone pet their dog. It can really just be a matter of preference.
I think the best thing to do is if you see a dog and it looks like its doing anything that could be considered work, ignore it. If it's at rest, you could politely ask the owner about them or if you could say hi to the dog. Most of the time, you'll probably be turned down, so please don't be angry if you are. While it may not seem bad if one person pets the dog in harness, over time the training will be broken and the dog will stop making that distinction between work and play. Also I'd like to stress that if you should ever approach someone with any type of service animal, address the person, not the animal.