My wife and I have a house that we rent out - in a town

Hi
My wife and I have a house that we rent out - in a town 60 miles from where we live - using a letting agent.
We have been with this agent for about four months.
After the first letting fell through, they let the property to a company; for the directors (a man and his wife) to live in.
We discovered, through a trusted contractor working on the outside of the house, that the house was being used by the company as a sort of dormitory - presumably for incoming migrant employees of the company, contrary to the terms of the tentants agreement
There have been various breaches of the agreement, and the agents haven't really performed to the standards expected – keys have been lost, rent has been late, and our concerns about use of the property has not being taken seriously it seems.
We requested an urgent property visit to confirm the situation, and the agent seemed reluctant to comply, stating that the information was 'hearsay'.
We eventually received notification that a property visit had been undertaken, and that all was well with the house.
The photographs that the agency submitted showed that clearly this was not the case, with six beds in three rooms, (which should have rung alarm bells, and a check of the tenancy agreement).
We have given notice to the tenants (through the agent), and they are due to vacate 26th April.
Along with other incidents of incompetance, we don't feel that the agency has been working in our best interests, and really don't want to continue using them.
We signed a contract with them when we first started dealing with them,
We have drafted out a letter (email) to the area manager of the agent wishing to terminate our agreement when this tenancy ends, but have not yet sent it, as we are unclear on the legal implications.
Given that the terms of the contract have been breached, we would be grateful for some advice upon how we should proceed.
Best regards