If you have not been sent a P800 calculation for each of the tax years concerned, call the tax office on the number here and ask for them. That will give you a starting point. Tell the tax office again that you no longer work the second job and to review your tax code for the current tax year, 2014/15.
I suspect that you may have been given too much in the way of personal allowances between the two jobs or your income was at a level which took you into the 40% tax band and you were only paying 20%. Without the figures and the P800s its impossible for me to say definitively what the problem is and what has caused it.
When you started the second job, you should have been asked to complete a P46 to confirm whether you had more than source of income or not. Unless the agency employer informed the tax office electronically of your status as their employee, they should have sent the P46 to the tax office for tax coding instructions. Unfortunately, many agency employers cut corners with their PAYE responsibilities and leave it to employees to pick up the pieces.
When you have received the P800 tax calculations check that they are correct. Then, write to the tax office and ask for an explanation as to why you are underpaid. Ask the tax office to consider your eligibility for relief from paying the tax under Extra Statutory Concession A19 which you can read about here. I doubt you will qualify to be frank. Secondly, ask them to consider whether the employer is at fault for PAYE errors and which HMRC think they should pay the tax underpayment caused by their errors. You can read about employer PAYE error here.
If the tax office won't write off the tax or make your former employer pay it, then you are left with writing to The Adjudicator's Office and ultimately to The Parliamentary Ombudsman but while your case is considered the tax office will still want you to pay the tax owing.
I hope this helps but let me know if you have any further questions.