A verbal agreement is binding in exactly the same way as a written agreement.
The difficulty with his claim is that you didn’t tell him to go ahead and he says you did.
There is no proof that you told him to go ahead so if this went to court, it would depend who the judge believed on the day.
These matters are decided on the balance of probabilities and hence, the judge only needs to prefer one person’s version of events slightly more than the other and that persons claim or defence will succeed.
You ask whether you’re legally bound to pay for his time, not based upon what you have told me where you quite simply didn’t tell him to proceed.
If he is minded to issue Small Claims Court proceedings, you will have to defend them.
Can I clarify anything else for you?
Please rate the service positive. It is an important part of the process by which experts get paid. It does not cost you anything but helps us greatly.