I will try to assist you with this.
If you are not prepared to grant access to your side for this wall to be rebuilt, how would they be able to facilitate the build and for example carry out the pointing of the brickwork to make it look acceptable? Or are you now not bothered about it being done?
The situation under the Access to Neighbouring Land Act is that you must grant access for any work required to preserve a property. That means to maintain it. It does not allow for new building.
Under the Party Wall Act anyone who was building up to the line of junction is allowed access to the neighbouring land to facilitate the build which is the situation that you were originally in.
At present this is neither preserving the property nor a new build and it would be for the court to decide which, if any, Act applied here.
There is actually no reason why the whole build could not be carried out from one side but it is highly likely that what you would then be faced with looking at could look a mess as the builders may not be able to finish it properly and you may be shooting yourself in the foot by not allowing access to facilitate this build. If you are happy to continue to look at the wooden fence the local authority are unlikely to try to force you to grant access unless they were adamant that this must be replaced with redbrick and they decided to have the court determine the matter under one of the statutory provisions. It is unlikely but possible.
What you have to decide of course is whether you are happy with the fence and want things to stay as they are or not.
Can I clarify anything for you?
Please do not forget to rate the service positive. It’s an important part of the process by which experts get paid.