You will need to establish that the fence is yours. Does your title deeds confirm this? If so, the neighbour cannot remove it without your consent.
You do not have to allow your neighbour to out scaffolding on your land but do bear in mind that there is a statute, access to neighbouring land, which allows a court to grant access, including erection of scaffolding, in certain circumstances. This is mainly to carry out works to existing buildings.
Any further questions or queries?
Fences are tricky as they were not always adequately dealt with. You can buy a copy of your title from the Land Registry, £6, maybe they hold an old conveyance or transfer with a plan showing a T mark.
Other than than it is difficult to establish who owns the boundary fence.
Please can I kindly request that you accept /positive rate my reply otherwise the website will not pay me for assisting you. Happy to assist further if you have any subsequent questions or queries.