We have talked in many threads over the years, so hopefully you know I respect you. Please bear that in mind as you read this post because I am going to us some strong words.
You are not just lending an ear. You are having and affair with a married man, plain and simple. An affair does not require sex. In fact, the most damage done by an affair is done by shared emotional intimacy. There was once a thread on the DIS which asked women which would hurt them more, if their husband cheated emotionally or physically. The overwhelming answer was that emotional cheating would be more painful. That is what you are taking part in.
You are having deep, emotion conversations with a married man behing his wife's back. If she knew and saw you as close firends, it would be different, but that is not the case. It doesn't matter who is at fault. It doesn't matter if she is the worst wife in the world. He is married to her and until he gets a divorce, you have no place having the relationship you have with him.
You are having an affair. I know you don't intend to, but that is what you are doing. Stop it. Stop it now. If you continue, you will cause pain to his wife, him and yourself, more pain than you know. There is no other possible outcome, even if sex is never involved.
Their marriage may be spinning apart, but don't allow yourself to play a part in its destruction. You don't want to be a part of so many tears, so much pain. Walk away and don't look back.