Are you asking if you are liable?
No. You are not.
Apart from anything else he can't prove your car is responsible.
In any event, cats are semi wild animals in law and you are not liable for their conduct whilst roaming.
I wouldn't worry about this. Some people just have nothing better to do than worry about their gravel. If they had hobbies and a job they would not care about that type of nonsense.
Can I clarify anything for you?
all the best.