The court could be persuaded by CAFCASS but it takes alot to change residence without a full Section 7 report during which your children will be spoken to.
I have dealt with quite a number of cases similar to yours with varied outcomes.
My only concern would be acceptance of your partners alleged behaviour and the truth of any alleged disclosure. If you entirely ignore even the remotest possibilities that something could have happened the court would be concerned. You need to accept that something might have happened even if just accidental or something that has been misinterpreted.
The truth is with your partner in prison for violence he clearly has the propensity to act in the alleged manner.
You also need to be very careful with what you say. For instance you say he barely remembers the offence he committed as he was under the influence of drugs. If he is under the influence of drugs how can you safeguard your children.
I know you appreciate all of this and I do not want to be coming over as negative.
As for now he is not on the scene therefore there are NO safeguarding issues. So on the face of it no need to remove the children.
Moving forward and more long term you need to fully accept your partners offending and not belittle or excuse his behaviour. Your partner has to prove himself that he wants to change. You need to be proactive. If I was acting for you I would want him to go on a parenting course and anger management course upon his return. I would want him getting all the help available whilst in prison. In short he has to prove he is committed and a good person. He needs to walk the walk.
It will be a long process and sometime before he would be allowed near the children but he will get there if he determined and genuine.
The courts love nothing more than people who have turned their life around.