The feelings that come up during affairs are complicated, strong and difficult. The pain and hurt from a break up can be so difficulty and consume your thoughts which makes it harder. I think you are doing a really good thing right now by avoiding him. He doesn't seem to be able to give you the things you want and need in a relationship. I am not talking about the erectile dysfunction, but more he is not emotionally dedicated to you because he has been seeking other women while you were together and since then. At this point I think you need the time away from him to figure out your emotions and figure out what you need. As far as your husband, I think you need to look at what made you step out of the marriage and is that something you think you both could fix. Is he willing or are you willing to go into couples counseling to look at how to mend the relationship if that is the route you want to take. I encourage you to work through the feelings and thoughts you are having and mainly take care of yourself first, which I know is easier said than done. But regardless you need to make sure that either man is able to meet your wants and needs so that whatever relationship you end up in will be one that is satisfying to you. Does that make sense?