It all comes down to boundaries and character.
If you are married, and MEANT those vows, when you find yourself attracted to a person of the opposite sex you take yourself out of the situation. End of story. If it is a work situation and you can't literally remove yourself, you keep it business. No joking, teasing, flirting, drinks after work.
Abuse is not an excuse. An affair is just dropping a match on a powder keg in that example. Get out if that is the case.
Spouse not meeting your needs? Talk to him/her. Go to counseling. Search your heart and soul. Still nothing changes? Leave. Get out there and find the person or people who can make you happy.
If you decide to stay in the marriage for kids or whatever, YOUR VOWS STILL APPLY. Do you think your kids discovering an affair is going to hurt them less than an above-board divorce?
Need to feel good about yourself? Feeling insecure? Crave the attention to reaffirm your womenhood/manhood? An affair might be a temporary fix but when your spouse finds out, or your mom, or your church...how will you feel about yourself then?

Affairs destroy more than the betrayed.
As for the other person who knowingly participates...affairs don't just happen. You know if you are initially attracted, and you know when you are developing an attraction. You know if your heart races when you're getting ready in the morning, looking forward to seeing this person. You know this person is taken and you do not care. You are looking out for you. Textbook selfishness.
If you choose to go ahead and chase someone and not care if their spouse or children get hurt as long as you get whatever it is that you want?
That seems pretty black and white to me as well--it's wrong.
At least it is by MY moral code.
Just because it is not as big a crime/sin/breach of integrity as the spouse does not mean what she is doing is not WRONG WRONG WRONG.
The world would be a better place if people did not justify wrong actions with entitlement reasoning and 'but this case is different.'