There are racially aggravated offences in criminal law but they are variations on offences that exist in a racially aggravated form.
S4A of the Public Order Act 1986 deals with intentionally causing harassment alarm and distress
“(1)A person is guilty of an offence if, with intent to cause a person harassment, alarm or distress, he—
(a)uses threatening, abusive or insulting words or behaviour, or disorderly behaviour, or….”
So your use of language could be fit within this but the Crown would need to prove you were intending to cause harassment alarm or distress to a specific person. That will be difficult. The fact that the person at the other end of the phone was black could be a problem but your assertion that you did not know this would amount to a defence.
Another possible section is s5 of the same act. A person commits an offence if he
“(a)uses threatening, abusive or insulting words or behaviour, or disorderly behaviour, …..,
within the hearing or sight of a person likely to be caused harassment, alarm or distress thereby…”
It is a defence for the person accused to prove that he had no reason to believe that there was any person within hearing or sight who was likely to be caused harassment, alarm or distress.
So you have a defence to this offence as well in that you did not know there was any person likely to be caused harassment alarm or distress. It is possible to argue that using that language is capable of causing distress to anyone regardless of their ethnicity and if the court took that view then you could be convicted.
It may seem ridiculous to you but the law can intervene in the use of this sort of language. In the way you describe it being used. You have a clear defence to any allegation under s4A of the Act and a less sure one under S5. This applies to the racially aggravated version of the offences as well.
However this is such a minor incident that even if reported the police may be reluctant to devote resources to it.
In other words technically you could be in some trouble but in reality you would need to be unlucky to be prosecuted in these circumstances.
If you are questioned by the police you have a right to free legal advice from a solicitor and it would be sensible to use that right.
I hope this answer has been useful but please do not hesitate to ask follow up questions.