Whether you have the “good side” or the “bad side” of the fence does not indicate ownership. If there is no proof of who put the fence up and if it’s on the boundary, then it is likely to be a joint liability. The neighbour is entitled to let it fall down if he doesn’t want to spend money on it and you are entitled to repair it but you can’t actually do anything without his consent.
Hence, provided the neighbour agrees, you would be able to replace the fence on the boundary but if he does not agree, you would have to offset the fence over to your side. It depends whether he is just being awkward or not.
If the fence is attached to your wall and it has been there for over 20 years without consent or objection, then there will be an easement for the fence attachment.
If it hasn’t been there for 20 years and there is no specific even mentioning the deeds, then you can require it to be removed if the neighbour owns the fence which, for what you have described, doesn’t appear to be the case.
Can I clarify anything for you?
Please don’t forget to rate the service positive. It’s an important part of the process by which experts get paid.