If it is your wall, I cannot see that any action they bring to stop you taking the wall down will succeed.
If you start to take the wall down, and the neighbour seeks an injunction, you will have to defend it on the basis that it is your wall and you are entitled to do what you like with it.
You may need to prove to the satisfaction of a judge, if they do apply to court for an injunction, that the wall is indeed owned by you and it is not joint or party.
You also need to bring to the attention of the court that this is unsafe and hence, that is why it is being removed. The court cannot compel you to keep boundary features which belong to you, if you don’t want them.
The granting of planning permission or not, as the case may be, is not relevant.
Can I clarify anything else for you?
Please rate the service positive. It is an important part of the process by which experts get paid. It does not cost you anything but helps us greatly.