Hello I am a solicitor with 20 years experience. I wlil try to answer this for you.
I am a little unclear here. Is he intending to build a driveway that is entirely on his own land or to improve in some way the right of way over your land?
Ok they are restricted to only using the area defined as the right of way. They can not use, go onto or overhang any of your land off the right of way. The right of way passes over your land it is not his land in any sense, except that he can pass over it. He cannot park on it either under the terms of the easement.
They can not legally use any part of your land except for the defined right of way.