If you have proof that you have repaid the deposit amount, then he has no hold over you.
If you are paying the mortgage then you can simply ask him to leave.
You would have to give reasonable notice, but nothing more.
You can ask the police to remove him if he refuses to go.
You can change the locks if you wish, this is your asset, your house, not his.
He has no right to remain there and ask you to leave.
I hope this now clarifies things for you.
If you have further questions, you can always come back to me.
In the meantime, if you could take a second to provide a 5 star rating (at the top right of your screen), I'll be credited for my time spent responding to your question.