A holding deposit can br refundable, yes.
If the landlord pulls out or any reference checks return negatively etc. If the tenant pulls out you loose the holding deposit. As you say, one weeks rent only.
But you wording does not state it was a holding deposit. It says a deposit for first months rent. That is different.
The problem you have is lack of any clear concise written rental agreement.
Ultimately, if it was a holding deposit then you should receive 3 weeks back, if it was a deposit to cover first months rent as you also state, you could forfeit that money.
If you feel head done by, and disagree with the landlord, the you can commence proceeding against him.
The first thing to do, is to send an official "letter before claim". This sets out your claim against the landlord, what has happened, why you are entitled to a refund, how much and a date to return payment by, failure to do so would result in legal action in court.
If he ignores or refuses to return payment, then you can start your online money claim against him.
The link below would take you to the money claim website. There is a £35 fee to pay to start your claim.