The fence is trespassing.
Provided he can simply remove the fence without damaging it, then he can do that. However if he cannot, and he cuts it, it is criminal damage which is a police matter.
However if you will not allow him to cut it, he can apply to court to get you to remove the trespass and he can ask the court to award costs against him.
If the route of your plants or trees goes over onto his land, he is entitled to remove that trespass. However, if he cuts the roots and the plant dies, whilst the police are probably not going to be interested from a criminal damage point of view, you are entitled to be compensated for the dying plants.
Can I clarify anything else for you?
Please rate the service positive. It is an important part of the process by which experts get paid. It does not cost you anything but helps us greatly.