That's a very good question. Are you referring to the Trustee or the Power of Attorney ?
If both the answer is probably similar.
Trust - ideally compelling him to act, though I can't see that working because of the past experiences. So, removing him would not be ideal but may be the only way for it to work for my mothers benefit.
Power of Attorney - as above really, only problem my mother has said that she won't sign another document because of the stress that she was put under to sign the form for my brother.