In the lack of the clarification requested, I will try to provide a general answer.
If your passport renewal application was rejected because of a "poor" photo, or because the photo lacked the requirements as passport photo, you should have received a communication informing you of the rejection.
You might have been given a deadline to re-submit the photos with the requested specifications, and any indication about the fee should normally be in the communication.
If you are uncertain about what to do, you should contact HMPO, which you can do either via an online form (https://eforms.homeoffice.gov.uk/outreach/Passport_Enquiries.ofml) or calling Passport advice on 0300(###) ###-####
This is of course applicable if your application was refused only because of a poor photo.
In case it was rejected (also) for any other reason, you should seek professional advice from a solicitor.
I hope this information is helpful and I wish you the best.
Please do not hesitate to contact JA again if you need help and you may ask to speak with me directly if you wish.