Hello, I'd like to help you with your question. The guy you are with sounds like he likes you but he is not willing to put you and the relationship first in his life. If each time you express your needs and want the relationship to progress he responds with slowing things down and basically put the relationship on the back burner, then he probably is only interested in his own needs and not what you want out of the relationship.
Add to that the fact that he has already strayed outside of your relationship and basically left you to deal with the situation on your own says he is putting his needs first.
Although it is understandable that you have strong feelings for him, he is not returning those feelings. From your description it sounds like he is in the relationship for his own reasons. If you feel that you do not mind his behavior, staying in the relationship is an option. However that means never knowing where you stand with him because he is not invested in the relationship. On the other hand, you might want to consider moving on if you want a relationship that is more balanced and healthy. You are supposed to be able to get your needs met in your relationship and this relationship is not giving you that option.
I hope this has helped you,