thank you. The length of time the fence has been in place I will not pretend does not present a problem. The potential problems are twofold-if the fence has been in place for more than 20 years, then they may be able to claim a prescriptive right over the land in question to continue to use the land. If they can demonstrate that they have used a particular area of your garden as of right without your permission which includes permission of previous owners of your property, then they can seek to establish a prescriptive right to continue to do so and registered this as easement as against your title.
In addition, if they can demonstrate that they have used the land in question for 10 years or more and have enclosed it and treated it as their own in the mistaken but reasonable belief that the lannd belonged to them, then they could seek an application for adverse possession in respect of the land.
I wonder whether this issue was picked up when you bought the property as regards ***** ***** in the correct location? it may also be worth contacting your seller if you still have the means of contact for them. If you can find some form of permission granted by a previous owner, that could serve to mitigate one or both of the above risks significantly or absolutely.
On the other hand, if they are looking to sell the property, this does give you a potential advantage in that people are often very reluctant to have disputes raised particularly when they are trying to sell a property as this can put off potential buyers. This may therefore be a good time to strike and if you are not able to agree the position of the boundary, consider raising a boundary dispute using the land registry is boundary dispute resolution service which costs £90 initially though in the event the neighbours disputes your claims, there is also the cost of a surveyor to prepare a detailed plan and report to consider:
of course, even if this application are settled in your favour, it does not guarantee or preclude the neighbours seeking an application for adverse possession potentially.