Ask an Immigration Solicitor. Get an Answer ASAP.
You can obviously marry who you choose but very unlikely he would be married in the UK because of his criminality. If you are able to get married in the U or elsewhere that would help towards an application to join you or remain in the UK under EU law but it is likely the Home Office will refuse on the basis of a marriage of convenience. It will be for you to prove that your relationship is genuine and subsisting and not of convenience. Why have you waited until he got caught to decide to get married? Were you aware of his immigration status? How long have you been in a relationship with him? Are you in employment?
OK that's useful to know that your relationship has existing for 2 years. Clearly when you do get married you need to provide evidence of your relationship. Also as your relationship has existed for 2 years he is able to apply under EU law as an unmarried partner but this option is not as flexible as the married option.