As I understand it you land stops at your back door, so the neighbours are correct.
I think what you are confusing is that today from a planning permission perspective you’d not be allowed to build to within 1m (ie 3ft) of the boundary. So today you couldn’t build your property right up to the boundary, this stops this sort of issue.
I’ve never hear of a permissable boundary, waht you are suggesting is that you can use some of your neighbours land.
I see you pay a fee for access, this makes sense to the other landowner as it affirms the fact that the land is theirs and you are being granted temporary access. I assume this was all apparent at the time you bought the property. To be honest I don’ see why you are describing the landowner as a git. I imagine your property was cheaper than it would have been if you owned the access.