I'm a SAHM and I can't stand SAHM groups (for myself). Whenever I tried to join one it felt like I was going straight back to high school. There would be the popular ones that led the group. I live in a fairly wealthy area, and you'd get the moms in designer clothes, perfect makeup and hair who would look at me like I was something they found stuck to their shoe. Oh my goodness! How could have I have gone out of the house with infant twins wearing a sweatshirt, no makeup, with my roots showing?! Hey, if I got to take a shower back then, it was a good day.
Many of the women I met also took this time to bash their husbands and get into a contest as to who works the hardest, sleeps the least, and has the smartest baby. That's not to say that there weren't some nice people in the groups, just that the other people seemed to take over, at least in the groups I tried to join. I wonder if the women in your group don't want the men around because they felt inhibited when it comes to complaining about their husbands? I also wonder if any of the women have husbands who don't like them spending time with other men, no matter how innocent the situation. My brother was like that. DSIL worked as a secretary at West Point and he "made" her quit because he didn't want all those men around her. I know, he's

, but men like that do exist, just like there are a few women who freak out every time their husband even says Hi to another woman.
Whatever the reason, I would either demand that the men be kept in the group, and the women who don't like it can leave (if it's an organized group with written rules), or I would do as others have said and send a letter to all the members of the group telling them why you're leaving and invite the men and any others who feel the same to start a new group.