Hello my name is ***** ***** I will help you.
Is it your fence or their fence please?
Ok. If it's shared boundary fence then they need your permission. If it's their fence then they do not need permission and yes they can't take it down.
So you need to check the deeds. Unless it says otherwise or marked with a T then it's a shared boundary.
Can I clairfy anything for you about this today please?
If I could ask you to rate my answer before you go today, by clicking on the 5 stars or happy face, otherwise the site does not pay me for the time spent with you today. Thank you in advance and good luck!