They are harassing you. Harassment is a course of conduct (more than twice) which either harasses intentionally or unintentionally that the average man in the street would consider to be harassment. It’s a criminal offence under the Protection from Harassment Act and is a police matter. You need dates and times of all the incidents and also what has been said.
The police may not want to pursue this if they don’t think there’s enough evidence to bring successful prosecution but they should follow up every complaint and hopefully give the culprits a warning.
You can also make a civil application for a court order to stop them speaking to you or mentioning you to any third party. Unfortunately, there is no legal aid for this.
They are also slandering you and in conjunction with the court order to stop them speaking to you or mentioning you to any third party under the Protection from Harassment Act you can also apply for a court order (injunction) to prevent any common-law defamation.
The threat of those court applications and a report to the police, coming from a solicitor may do the trick without the need to go to court.
If you do get the court order and they carry on doing what they are doing, they are liable to be arrested.
Can I clarify anything for you?
Please don’t forget to rate the service positive. It’s an important part of the process by which experts gets paid.