If this is your trellis and provided it hasn’t caused any damage, then you are entitled to let it fall to rack and ruin (unless the property is leasehold and there is a covenant to repair the trellis which seems unlikely).
It may be an eyesore but that’s unfortunate for him. He shouldn’t look at it! There is plenty of case law which says that there is no right to a view.
If the trellis has damaged his plants or whatever, he may be entitled to compensation for that and the cost of removing the trellis although the cost of taking it to court and the risk is going to be totally disproportionate to the cost of him simply picking it up, throwing it over the fence (he is not entitled to dispose of it or keep it) to you and replacing the few broken plants.
My suggestion would be to email him back to tell him that you have taken legal advice and that in the circumstances, you no longer want the trellis in place and you have decided not to replace it. You can tell him that if he doesn’t like it, you don’t mind if he removes it at his own expense.
You can tell him to take legal advice as you have done.
Can I clarify anything else for you?
Please take a moment to look at the top right hand corner of the page and rate my service by clicking one of the stars at the top of the screen. Then press Submit. Thank you. If you still need any point clarifying, I will still reply because the thread does not close.