As this access has always been used with permission and for only a relatively short period of time, then unless it is mentioned in the deeds, or the neighbour agrees, you can not claim any right to use it under the Prescription act (needs 20 years without consent or objection) or any other provision.
With regard to the planning permission only being granted if there was a right-of-way, that’s something for the local authority to deal with.
If ever you come to sell the house, you are going to need to get this right-of-way sorted and hence, it really depends on what the neighbour who owns the land over which you require the access is defining as special circumstances. It may be that taking rubbish out and gardening rubbish he would class as special circumstances and that what he simply doesn’t want is you using it as your daily access rather than using the front of the property or whatever access you currently use.
The neighbour can make what stipulations he likes. Normally, a right-of-way would be stipulated with words such as “a right of way on foot for all reasonable purposes associated with the land”.
If he’s going to specify that it’s only to be used for taking rubbish out or when moving house, that is very difficult to enforce without blocking it altogether.
Can I clarify anything else for you?
Please rate the service positive. It is an important part of the process by which experts get paid. It does not cost you anything but helps us greatly.