Firstly, you don't need to separate or move out to register your home rights, the notice simply records your interest in the property on the property charges register.
You fill in the HR1 form to do this: https://www.gov.uk/government/publications/notice-of-home-rights-registration-hr1
There's no fee for the application.
If you don't do this before you divorce, your home rights expire on the pronouncement of the decree absolute. If you divorce without resolving the house/ property issues, the judge can extend home rights until you have done so past the date of the decree absolute, but they need to be registered in the first place.
Secondly, as to whether you have a legal right to ask his mother to leave, I'm afraid this is more tricky. He (and you for that matter) can have in the house any guests or family that you wish.
The only way to legally exclude them would be by way of a non-molestation order for example if they were causing harassment, alarm and distress. But it seems that this is not as serious as that - in that your mother in law couldn't be said to be harassing you to a level where she could face prosecution. So a non-molestation order is not in my view appropriate.
You are entitled to privacy in your home, and you are entitled to ask that she move out. If he refuses, then it's a choice you have to make as to whether you are able to continue living with him like this.
I hope that this answers your questions?