You can refuse for the reasons you have given but I would not.
Whilst I appreciate you will not have a company car and driving is not part of your job, I think that they want to put you on the company policy in case you should you ever have to drive a company vehicle in , say, an emergency. This way you are insured to do so.
If there are any further points please respond I will be happy to reply further.
Yes, you can refuse as it is not something that reasonably forms part of your duties