For what it's worth, I do hear what you're saying.
I think what's making this debate so very heated is because of all the social and cultural baggage that comes packaged with what should - in a perfect world - be just a personal, private, consensual choice between a couple of people. You're absolutely right - it's not a hardship or a hindrance, when it's just you and your partner and no one else is impacted.
But there's an important cultural context in which this debate is taking place. We're less than a generation removed from the "Boys Club", when women could not advance in their careers, because they were denied the social networking opportunities and mentoring that men had available to them. Even now, women who try to make inroads into traditionally male environments (ie, truckers) are often harassed and isolated.
We live in a world where certain countries have even codified this morality into law. Where if an employer wishes to hire a woman, he is legally obliged to provide a separate work environment and break room. Supporters of this kind of segregation are found in both genders. The women aren't comfortable socializing with men, and appreciate the accommodations made for them. Men feel they're making reasonable efforts to accommodate women in the workplace. Both feel it is immoral for the genders to mix. It is a matter of faith, culture, ethics, modesty and propriety.
I had one woman tell me that she felt protected and valued and cherished. That she never had to worry about sexually harassed, or even left destitute (since her husband was legally obligated to provide for her). If she broke the law, her husband would be held accountable, as a parent is held accountable for a wayward child. She described woman as a "protected class of people". I had a man explain to me that the women of his culture are different from Western women. They're not as strong or as independent. They're more emotional and delicate and precious, and therefore need protection in a way that Western women don't.
If you truly value something and want it to last, you must protect it and cherish it and keep it safe from the elements, right?
On both sides of this argument, people are hearing both subtext, and text-text:
"My husband respects me (more than yours respects you)." "I value my marriage (you don't)." "I'm dedicated to protecting and preserving my marriage (you aren't)." "You may think you'd never cheat, but anyone can slip given the right opportunity." "Why even take that risk?" "I'm more moral than thou art."
"Your marriage is weak." "You don't trust your partner." "You don't trust yourself." "You must be totally obsessed with sex."
It's no wonder people are getting offended with each other!
If the two of you have decided to structure your lives so that you don't spend any unnecessary amount of time in the company of the opposite gender, and if you choose not to have intimate friendships with any others (I'm assuming this includes your friendships with other women?), then that should absolutely be your right. And if it makes you both happy, wonderful!
But, where it becomes problematic for me is when too many other people decide that your way is the right way to do things. Or even worse, the only way to do things. Because that's what leads to a gender-segregated society. It's the reason why women are
still harassed and treated like sex objects when they step outside of their expected roles. Or else politely isolated, avoided, and regarded as "unicorns", with all the associated consequences for their careers. "Women just don't make good executives. We hire plenty of them, but they never advance. Now, this young man, he reminds me of me when I was his age! Plus, he's a member of my golf club." And yes, many people today
are content living in that sort of world, but I'm not, and I'd hate to see us slipping back into it.
Sometimes our decisions are symbolic of more than we intend. And I'm saying this as a woman who both worked in the military and later quit her job and stayed home to keep house and raise our children. I've had a lot of time to think about the significance of my choices in a broader cultural context.