If there is a court order stating that the child lives with you, then legally you can ask for him to be returned, but the response from the father will be that there are safeguarding concerns, so to get him returned you will have to go to court.
If there is no current court order in place, then you're both seen as equal in the eyes of the law as you both have parental responsibility (if he's named on the birth certificate), so he can withhold the child if he believes he has valid reason for doing so. And in light of the incident with your boyfriend, the father clearly has valid reasons for withholding the child given that he saw you with the person you are not supposed to be with until an assessment has been carried out. It's June, the incident occurred in January, the assessment should have been completed by now, but it would appear that it has not been started even.
Let me put it this way, if it was the father asking for advice, I'd tell him to withhold the child and go to court for the court to assess the situation with your partner to ensure that you have complied with the SS requirement for an assessment. As it is you that is asking for advice, I will tell you that the only way to resolve this, is for you and your boyfriend to have the assessment and pass it with flying colours, and then the father won't have a safeguarding issue. At present, he has a very valid safeguarding issue.
If the assessment is positive, then the father should be satisfied - if he's not, the option to go to court is still open to him.