Hello, I'd like to help you with your problem.
It sounds like he is approaching this relationship on his own terms and does not seem concerned with how you feel. It sounds like you reacted to him as you did because of his confusing behavior. And since he did not give you answers as to why he acted that way and said those things, you had no choice but to contact him if you wanted answers.
This guy is showing you some red flags with his behavior. For one, your impression is that the two dates went well. But he seemed to not feel the same when he didn't contact you for a third date and sent you that strange and somewhat hurtful message when he was leaving for his weekend away. It is not normal for someone to pull back from a good beginning of a relationship and not respond to messages or be concerned with what the other person is feeling. Because he was not responding to your messages it left you with no way to know what had happened. So it's natural for you to question him and see if you could find out his side of things.
His reactions when you contact him are also of concern. The response after you contacted him online was odd, saying he thought you'd never contact him but in reality, he never made the effort to contact you. He also never gives you a clear reason why he doesn't make an effort in the relationship and instead avoids talking about your feelings by claiming he is busy and that you have never given him a chance to speak or gather his thoughts. These responses sound like avoidance and that he is not taking responsibility for his own behavior towards you. Instead he is blaming you for wanting answers to his behavior. That is a sign that he is either avoiding owning up to why he left the relationship or that he feels his behavior should not be questioned.
At this point, you only acted based on his dysfunctional behavior around the relationship. He is the one refusing to give you answers, won't own up to not contacting you and continues to blame you for wanting answers. You got upset with him, which is a natural response to his frustrating behavior. There is nothing wrong with what you did and you are not an idiot for asking him for answers. At this point though, you may want to move on. He is showing all the signs of possibly having issues and if he is acting like this now, it's not going to improve if you try to make a go of the relationship. You deserve someone who respects your feelings and who will give you answers when asked.
I hope this has helped you,