Do you know of any reason?
No, but has there been an incident?
OK. I presume this is a public pool?
In that case, they can withdraw your invitation to attend for any reason they like unless it is an unlawful reason like sex or race discrimination.
They do not have to provide a reason.
However, you can always turn these allegations round upon people by effectively threatening a defamation action.
But it depends how much you are prepared to spend on this.
A solicitor could write to them for about £100 plus VAT. That letter would effectively need to say that you are considering a defamation action and need this information in order to consider your position.