He is anxious because he is aware that you can apply to the court for an order allowing you to remain in the property and forcing him to leave.
This is because the housing needs of the chidlren are the priority so far as the court is concerned.
Alternatively he could be forced to purchase another property for your and the children to live in until the youngest child is 18.
From your point of view the biggest drawback to this is that once the youngest child was 18 the property would revert back to you ex and you would be left with nothing.
On that basis if he wants you to leave you should ask for a lump sum that allows you to se up in rented accommodation elsewhere - think £5,000 to £8,000
If you have a low income you will still be entitled to Housing Benefit - so most if not all of your rent will be covered.
So far as child maintenance is concerned he is in fact liable to pay 19% of his gross income from all sources as Child Maintenance - divided as to one third to the other mother and two thirds to you
If you have been the main care of the chidlren until now then that is what will continue - but you will have to consult him on all major issues and if he is not happy with a decision you have made then he can apply to the court for the court to decide (on such things as education etc)
You can read more here
I hope that this is of assistance - please ask if you need further details