Have you agreed to him taking them? Has he taken them before? Are there any court orders in place?
Do you think he will take them and not return?
What basis do you have for that fear. If he has threatened not to bring your son back then you could apply to the court to prevent him going. The court could then make an order that he calls every few days and if he ignores that he wouldn't be able to take them away in the future. You are right to be concerned but you have to have a basis in evidence for your concern.
In that case all you can do is refuse to hand him over and let him take you to court for a contact order. You can speak in court of your concerns.