Hello, I am a solicitor with 20 years experience I will try to answer this for you.
If you are certain that the fence is yours and not your neighbours then it is unlikely that he can prevent you replacing or altering it. He may have had an agreement with the person you bought from but unless there is a formal restriction at the land registry (your conveyancing solicitor would have told you about this) then you are not bound.
Even if there is a restriction it would not be a crime to work on the fence but arguably a civil wrong.
You can not cause criminal damage to your own property.