Seeking expert testimony is a sign of strength. A personal relationship with a caring professional is proven clinically effective.
I believe that I can help you.
It is always frightening when someone intrudes into your life and threatens you in the worst possible way - to have your child taken from you.
If your girlfriend is an adult, then he has no say over her. You have never mentioned her age or yours, but unless she is a child or his ward because she has been declared incompetent (and I doubt that is the case), then he has no more control over her life than he has over mine.
It your son is not abused, if your wife wants to be with you and your son, then there is nothing he can do about, unless he starts trouble by making up untrue stories, for which he could get into trouble if it shown that he as lied to authorities.
You talked about unresolved anger towards your own father. If this manifests towards your wife or towards your girlfriend's adoptive father, then this may be the issue that is frightening this man or turning him against you. Perhaps he fears you and fears for his daughter.
This should not be grounds to take your child, but should be reason to see a counsellor, if you have an issue.
Your girlfriend has probably spoken to her father in some negative way and you are having problems with her, which you did not mention. If there were problems then you would not be separated.
Go to the counsellor or therapist, work on your issues, and let the situation calm down. Learn to change your behavior and temper your anger and you will succeed.
Do this slowly and at the right time so as to minimize friction and anger. If you have anger issues you must learn to control them.
Finally, let me recommend an excellent book for you available at www.amazon.co.uk and elsewhere.
I wish you good fortune and will keep you and your family in my prayers.
Elliott, MAE, LPCC, NCC, CCMHC