Legally, the neighbour does not have to have anything on his land which he doesn’t want. If there is a fence on the boundary he cannot remove it without removing your half so it remains.
The situation now is that if he doesn’t want the fence back, if you put it back and it’s half on his land, which is how it would be, that is trespass. He does not have to have a fence there and if you want a fence there then it will have to be wholly on your land.
If the neighbour thinks he is being smart grabbing a few millimetres, you can tell him that you are not bothered about the fence either but you propose to leave it down. Call his bluff. Then, if he wants the fence back he is going to have to put it on his land. He cannot compel you to put the fence back wholly on your own land.
If you put it back without his consent, it is trespass but if he wants to take action, it’s unlikely that the court are going to make you remove the fence and are unlikely to offer him substantial compensation if anything quite simply because he was happy with it for years and has only now changed his mind. Indeed, the court could well give him nominal compensation of say, £1 but still ordered the court costs against him on the basis that he is being unreasonable. It depends on the view the judge takes in court.
If you put the fence back where it was, and he simply removed it, then if he damages it, that is criminal damage which is a criminal offence and becomes a police matter.
Please don’t forget to rate the service positive. It’s an important part of the process by which experts get paid. We can still exchange emails.