Its a hard one.
For you is the fact that you didnt sign a contract or indemnity. Neither did you ask them to do any work
However against you is that you agreed to pay and that may indicate that there was a contract which was implied by law.
Ordinarily you would defend the claim and counter sue the builder, but clearly there is an issue as you agreed that wouldnt claim against him any more.
There is not an answer one way or another here, it really depends on the Judge you get.
As I have indicated there are points good and bad. Because the builder is bad does not mean that the company ought not to be paid - it was not their fault. They delivered what they promised to, even if it was wrong because of the builder.
It can go either way. A Court may say because there is no written contract you are ok. On the other hand a Judge may say because you agreed to pay that means there is a contract.
On balance, I think you have a defence, but I must warn you I think it is weak.
Your case is that there is no signed contract or indemnity
Can I clarify anything for you about this today please?