My name is ***** ***** I have been a solicitor for more than 35 years.
Regardless of the age of the children the position is that you do indeed need his permission or an order from the court to take the children out of the Uk even on holiday (just as he needs yours).
Equally you are entitled to have the passports to allow you to take them abroad - and since they spend more time with you than their father it is reasonable that in general you should hold them.
You cannot report them as stolen since you know where they are!.
Your starting point is to write a polite letter informing him that you wish to take the chidlren on holiday - giving the basic details and asking him to confirm his permission and provide the passports within 14 days. Say that i view of his past behaviour a promise to provide the passports just before the holiday will not be good enough
If he refuses or does not reply the you need to attend a Mediation Information and Assessment Meeting with a local Family Mediator.
Of mediation is not successful then you can apply to the court for a Specific issue Order confirming the holiday - and forcing your ex to hand over the passports
This is done using a Form C100 and paying the fee of £215
You need to be applying in early June is you wish the matter to be resolved in time!
Please ask if you need further details