Hello, i am a solicitor with over 15 years experience. I will try to help you with this.
Is the fence actually on the boundary, or is it on your or your neighbour's land abutting the boundary.
The position is that if the fence is on your land then he can not touch it. If it is on his he can do pretty much what he wants to with it. If it is actually straddling the boundary then the usual position is that it is a joint fence but that one of you (you from the sound of it) will have the responsibility of maintaining/replacing it periodically.
If it is truly on the boundary then it will be a party fence but the protections under the Party Wall Act will not apply. I think that in practical terms there is not much you can do about this. The duty to maintain the wall may be yours but this does not mean that he is not allowed to do work on it. You need to look at the sales contract to see what the rights etc regarding the fence are. I would not suggest taking it down unilaterally as this could give rise to an allegation of criminal damage. I think a court would be unlikely to order that it be taken down and the costs would be very high. The neighbour has a clear right to build anything on his land so if you were to insist on him moving the trellis he could simply build an even more intrusive structure just 1 mm from the fence.
There is no right to light/views etc in your garden.