Hmm. I understand where you're coming from, but I respectfully disagree in some ways.
I completely understand why some choose to be very strict, and I also understand that for those people contamination of their foods (for example, in non-designated friers) can make them sick, and absolutely if someone cooking for you buys food that will physically make you sick, I completely understand why that makes it very complicated, and why it's important that non-strict vegans don't make it harder.
However, I make my choices based on what I think causes the least suffering to animals in the aggregate, and I think that this happens when a larger group of people make more vegan choices. This will increase the availability of vegan products overall, and will make it easier for strict vegans to be strict. It also will have the greatest impact on the industry.
What I will say is that I'm very careful about who I eat what around. When I'm around people who I don't know very well and have only had short conversations about veganism with, I absolutely don't do things like put milk in my coffee around them. If it's a situation of hospitality and I haven't self-identified as vegan in advance (which is rare, but happens every once in a while because life is complicated), I won't always have the conversation with people if there's nothing else that I can eat around (because I don't want to make someone feel badly and they don't necessarily know that I'm vegan in the first place). I would never self-identify as vegan in a restaurant and then have a bakery product or dairy. However, I have very in-depth conversations with a lot of people about veganism, and they understand that the choices I make, although few and far between, are done for specific reasons and they also know that many vegans do choose to be very strict.
I don't like labels, but considering how passionate I am about vegan food, considering how much I cook it, share it with others, talk about it, and spread awareness, I DO call myself a vegan--both to myself, and often in public situations if I think it's appropriate (certainly not while I've got milk in my coffee if I'm talking to someone for the first time). I think it's really important to have conversations with people about abstaining from meat, eggs, and dairy, but please understand that this does not mean that I'm not VERY careful to make sure that the people I interact with do not have a faulty understanding of what veganism is, for the exact reason you mentioned.
Ultimately, I feel more comfortable and the people around me are more comfortable when I am occasionally flexible, and I think that does a LOT to promote veganism in general. I've found that when I'm not flexible, occasions often end up revolving around me and my eating habits, and people often find that kind of distasteful and self-absorbed of me and associate being vegan with being self-absorbed, and so I try to temper that by keeping things in perspective. (The birthday girl does not need to choose a restaurant around me. The server at that restaurant should be focusing on her, not me. The conversation should be about the birthday girl, not my veganism. I will order the best approximation I can off the menu, but the bread *might* have egg.)
This has been my personal experience, and although I really respect why you feel as you do, I really would hope that you'll reconsider being so harsh about labeling "vegan" or "not a vegan". I think it kind of undermines what we're all hoping for, which is that this lifestyle becomes more popular, that people are healthier, the environment is healthier, and fewer and fewer animals suffer with each year that passes.