It is very difficult to say who has responsibility for any of the boundary fences.
The only way to establish this is if it states it specifically in the deeds. Either in words or by an inward 'T'. However, the vast majority of deeds I read do not have T marks.
Other than that it is convention, I.e who has been maintaining the boundary fence. This varies, sometimes it's the left and others it's the right. No hard and fast rule I'm afraid. One indication could be the side which has the best side/flush side facing inward is yours. This is because if you were to put a fence up you'd put the best side facing in.
I hope this helps. Do come back to me if you have any further questions / queries.
Very helpful. It establishes your boundary. Technically you could ask him to remove the fence he has erected on your wall. He cannot drill into your wall.
With regard to the neighbour's fence, technically you cannot attach anything to it as it is your neighbour's fence. However it should not be on your boundary wall in the first place.