You certainly do not "owe" him for days that you have had them because he was away.
You are also entitled to know where the children are at night - it is a simple courtesy to ensure that if there is a disaster somewhere you know whether or not you should be worried.
Equally children are not packages.
I am sure that they enjoy spending the occasional night with their Grandmother but if your ex is routinely not able to have them then you are correct - they should be at their other home.
Your starting point is to discuss this with your ex to agree a parenting plan which deals with all of this using Family Mediation
If that fails then you may need sto apply for a Child Arrangement Order dealing with this point - although it is to be hoped that this will not be necessary
You can find more information on the website
I hope that this is of assistance - please ask if you need further details