There isn't really a yes or no answer to this question. It really depends on both the precise circumstances of the individual (you) and whether or not HMRC ever take a look at it. Luck pays a big part in these things.
The finance director is clearly a conservative type. He's concerned about the potential for trouble should HMRC carry out a PAYE audit at which they may see documentation that has you on its books as an employee, albeit part-time, and as a contractor operating through your own limited company. HMRC may argue that all your pay should be put through the payroll and subjected to income tax and employee and employer national insurance contributions.
If you have other clients that you work for and you actually turn down work offered by the institution, your own position is strengthened somewhat. It would be quite another thing if you were a full time employee and did a substantial amount of work through the company. HMRC would argue that you were simply an employee and that all your pay should either go through the payroll or be subjected to PAYE through IR35
which you can read about here. The director is concerned that a PAYE audit may end with a bill for PAYE and NIC, backdated for several years, for his institution.
If I were you, I'd sit tight and try to develop a bigger client base. Ultimately, you may have no choice in the matter and the institution may ask you to work solely as a private contractor or as an employee. I've had several clients in that situation and they have had to go with the flow.
I hope this helps but let me know if you have any further questions.