Could we have the background detail please?
He appears to allege that you asked for extra work which he says is outside the contract and you are saying that it was a fixed-price contract.
As much background detail is possible would be helpful.
Whether you have gone ahead with this or not is not relevant. He agreed to do the design for a fixed cost and that not unreasonably included any changes. Further, he agreed that the changes would be at no extra cost.
I don’t know when you’ve paid his £4500 or not but I get the impression you have. You are entitled to your plans.
I would tell him that you have no intention of paying him any more money and if he thinks he is entitled to it, then you must sue you in the Small Claims Court and who will defend those proceedings. Further, you will, your discretion, ask for a court order for him to either 1: return the full amount of money to you and he can do what he likes with the plans, 2: an order for him to give you the plans back, 3: if you have to get the plans done somewhere else at a greater cost, as the court to return the £4500 to you plus any additional costs charged by another architect.
Can I clarify anything for you?
Please don’t forget to rate the service positive. It’s an important part of the process by which experts gets paid.