You don’t have a duty to maintain hedgerows. There may be a problem if they become a nuisance and overgrown the highway but other than that, you don’t have to maintain them.
You don’t have to lock the gate to object to this, you can simply put a notice on the board and nail it to a stick and bang it into the ground in the middle of the plot or better still, put several notices saying that this is trespass a new object and they must leave immediately.
Take photographs and include a current newspaper in one of the photographs so that the can be no dispute over the date it’s better than having a photograph dated because that can be changed by altering the date on the camera..
That would be sufficient, for court purposes, to be an objection. If you know the person’s name and address, so much the better .
Ironically enough, if you go and dig all the spuds up and leave them on the ground, that is probably criminal damage and you could face prosecution if the person makes a complaint.
You dig the spuds up and take them that is theft.
There is an argument, a technical legal argument that by growing these things and taking them out of the ground and taking the spuds away (or whatever) they are stealing your nutrients from the ground! However, unless you want a technical legal argument, probably better not go there.