I would like to help you with your question.
I'm sorry for the confusion and doubt that this affair has brought into your life. I don't think this older man is a stable choice for a relationship. His history suggests that he is more interested in chasing women and attempting sexual relationships then in anything else. His issues with impotence are likely very real and that is why he is chasing women...trying to desperately convince himself that he does not have a problem.
Because he acknowledges multiple sex partners, you should seriously consider being tested for sexually transmitted diseases. While this may sound embarrasing/shameful...it is the right thing to do to protect your own health. Please consider this as soon as possible.
As to your marriage...you acknowledge a rough patch with your husband. And...you also acknowledge that you feel you love the older gentleman. Unfortunately, this older man does not seem at all interested in a permanent relationship. While he questioned if you were using him by just trying to make your husband jealous, his own motives for having so many sexual partners are suspect and I gather that he is using women to bolster his own ego and sexual prowess.
Is marital therapy something you would consider? You don't write as to how serious the marital difficulties are, yet I can conclude that they were serious enough to lead you to seek satisfaction with another man. I encourage you to think through your real feelings about your marriage and to seek help if you want to heal those wounds.
I can understand how you could get swept off your feet. Those feelings of falling in love are wonderful ... and a welcome break when a marriage is not going good. However, the decision to leave a marriage is a serious undertaking that will have long lasting impacts.
Please let me know your thoughts.