If you had insurance at the time you must inform your insurance company. Your policy will say you must let them know about any accidents and also any impending property damage claims. So if you have insurance, you should let them know otherwise if they find out about this they could void your cover.
You say the owner of the fence wants you to pay £1650. You could pass this to you insurer to deal with and that's my best advice. If not then you can ask them how they arrived at that figure and ask them to confirm whether they obtained 3 quotes. For all you know they have used the most expensive one - as a claimant they have to minimise their losses so they need to claim the average sum from 3 quotes for repairs.
If they do sue, you will receive court documents (a response pack) which you must complete and return to the court. It is a tick box exercise for the most part and there is a short section to write a defence which is easy enough though please feel free to come back to this site if you need any more help. A claim will also take 9-12 months to be decided at court. If you lost then you would get 14 days to pay the judgment before the claimant can enforce the order, and 30 days to pay in full before it is registered with credit agencies. The claimant cannot recover legal costs if they win, in a claim under £10K (a small claim), all they can claim are the court fees and interest.
If the claim has no merit then you have an option (after your defence is filed) to apply to strike the claim out. The court will consider an application if the claim has no merit, or is misconceived. The application costs £255 but this is recoverable if your application succeeds. If you are on a low income, have low savings or in receipt of benefits then you can ask the court to waive the court fee. If you won the application, the claim is struck out.
The hearing (if the case gets that far) is likely to be held remotely, it's you, a district judge (who is a practising solicitor or barrister) and someone from the claimant company. The Judge decides and if you lost, you get 14 days to pay the sum. If paid in full within 30 days then nothing goes on your credit record.
If you lost and if you were unable to pay the full sum owed at that point you can also ask the court to pay by instalments (a simple form is sent in, form N245, and a fee of £50 unless you qualify for a fee exemption so if you are on a low income, have low savings or in receipt of benefits then you would qualify). If the CCJ was not paid in full though bear in mind the credit record would contain the CCJ details for up to 6 years. After that it comes off the credit registers.
I can assist you going forwards if it gets to the point they issue a claim. If you need a law firm to assist you to defend the claim, here is an example for you and they offer a fixed fee : https://stormcatcher.co.uk/practice-areas/dispute-resolution/small-claims-lawyer-service/