Thanks for your enquiry.
Under law, if a party possesses land which does not belong to them for 10 years or more, unopposed, that party has the right to apply to the Land Registry for "possessory title" of the land.
So, even if the boundary fence has been in the wrong position for all these years, you would have the right to take this action to claim the land in question, if you wished to.
I would hope that your neighbour does seek legal advice, as he will be told of the above fact and any Solicitor will tell him that he is wasting his time 9and money) if he did pursue the matter against you.
I hope this assists and sets out the legal position.
If I have helped, please don't forget to rate my answer.