Yes, it does matter, not necessarily in what brand of clothing you wear, but if you took care in your appearance. No matter what people may say in polite society, we really do judge based on appearance, even if it is for the first second you see someone. Some people may not let the appearance influence how they treat a person, but there are others that do let appearance influence their interactions.
For example, I am fat. When you look at me, you see a fat woman. What you will not see is a fat woman with greasy, stringy hair, sloppy dirty clothing, and an oily face devoid of any makeup to accentuate anything positive upon that face. This would be a fat woman that does not care about her appearance, and say what you will, but I know this woman will be judged in a negative light before she even opens her mouth. She may be the nicest person you will ever meet, but most people won't wish to know that because all they will see is an ugly fat woman and assume she doesn't care about herself or anything else.
Knowing this, I do not leave the house unless my clothing is clean, my hair is washed, some attempt of styling said hair is made, and I put on a bit of makeup. Since I happen to believe my eyes are my best feature, I always wear at least eyeliner and mascara, in the hopes of drawing one's gaze away from my fat rolls!
I also walk with my shoulders back, head held up, and try to give out an air of confidence. People will treat you differently if you act like you are confident rather than being hunched over and timid. I may not feel confident when surrounded by beautiful slim people, but I won't let them know I don't feel that way!
All bets at beauty are off though when I am gardening. Then the dirty, sweaty fat woman is present, but I won't be going off my property that way!