You may benefit from seeing a solicitor on this matter. Trust law cases are quite complex, and this falls under trust law if you need to go to court for a remedy.
In terms of what your reasonable expectations should be, as you have had the benefit of occupation, and he has contributed half to repairs and upkeep along the way, it would be unusual for you to be able to claim back your mortgage payments (given that he would have had to make rental payments elsewhere having not been able to live in the house he actually owned).
The deposit is less clear cut, as there was no formal deed of trust, but under the scrutiny of the court are you going to be able to say that this was not your understanding? You've told me that you said he could have the deposit back. That's probably enough for him to ask for this back as equitable principles apply - i.e. what is fair and what was agreed verbally for example, even if not written down.
My initial view is that he is correct in his assumptions, I'm afraid.
But you should go to a solicitor locally with all o the facts and figures and they will be able to give a more definite view for you.
You can search for one here: https://solicitors.lawsociety.org.uk/
Thank you for your enquiry today. I am happy to answer follow-up questions - please do get in touch with requests for extra information or further queries and I will do my best to help you.