So in your opinion, every affair (and btw, affairs can be much more than "keeping your pants zipped") starts with the full intention of going out and finding someone to cheat with? All cheaters are habitual cheaters? Never at any time the history of affairs have any started innocently with work, or friendship (which no one said anyone couldn't have a friend of the opposite gender)?
I can't answer for others that say they have an "agreement" but it sounds like for most that have said it, its more of a decision than an agreement. As in: I make the decision for me, my spouse makes the decision for himself. And as everyone has said, its not about trust.
Affairs do not start "innocently". There is always a point at which you realize this has become more than a friendship, and that's the point at which you have the option of saying, "Whoa, baby, I'm putting the brakes on this!" or of saying, "Imma just let my libido be my guide! Woot!"
If you choose the second, that's on you. It doesn't matter if it's your first time. And it doesn't matter if you didn't shake hands with this person on your first day of work and think, "I'm totally doing this person!" It doesn't even matter if you didn't realize how badly on the rocks your relationship was before you embarked on this affair. You still made that choice. This relationship, in some way, is filling a need that's not being met in your marriage.
Wouldn't it have been nice if you could have sorted that out before you cheated?
Anyway, as I said before, it can make sense to avoid being alone with certain individuals you find personally tempting, especially if they have indicated particular interest in you as well. Ultimately, this is always about the individual.
I don't know about you, but I am most certainly NOT sexually attracted to every human being with a set of XY chromosomes. And of the small subset of XY human beings that I do find delightfully attractive, as a happily married woman I am quite capable of maintaining an appropriate emotional and romantic distance between us. Even over lunch. Even if they're serving oysters!
I expect you are much the same. If we cheat on our spouses, it's
not because we've spent too much time talking to members of the opposite sex. It's because we made the choice to let a particular friendship before more. So, it makes no sense at all to avoid being friendly with half the human race, just because you're afraid you might unexpectedly find yourself experiencing an uncontrollable desire to have sex with them. Your choices and your actions are always within your own control.
For that matter, I have friends who smoke. I have friends who drink. I do neither, and I have no desire to start. Does this mean I should avoid hanging out with my friends when they're smoking or drinking? Of course not! For me, there's nothing tempting about what they're doing. If I was a recovering alcoholic or nicotine addict, that might be a different issue, but I'm not. While I fully respect the right of others to say, "I can't be around smokers/drinkers!" I'm not going to live by their rules. I don't need to. And I'd be most irritable if that person then told me that it was in my nature to drink or smoke, and the attraction of cigarettes and booze was "dangerous", and that I'm willfully ignoring how easy it is to "slip" and that if I don't want to drink or smoke I should avoid any situation where I could easily get a drink or a cigarette. (Aw... no more backyard BBQs!

)