Is the supervised access continuing? Where does it take place? What is the social workers' intention? Is there a child protection order in place or are they just monitoring the position with your ex?
Why would they think you're lying?
Unless there is any order for supervision or similar, you can legally do what you consider to be in the best interests of your family. Given the situation they will probably stay in contact with you and think that they are helping even when they are in fact not being helpful, such as them accusing you of having your ex in the house when in fact it was your son who answered the phone. I would suggest that that may be worthy of a formal complaint which would demonstrate that you aren't going to be intimidated by them and that may cause them to back away from their present interference. I hope that helps. Please leave a positive rating so that I am credited for my time.