Bear in mind, they’re probably totally unaware of the arrangement / deeds.
Why not invite them round for a barbecue, get a charcoalburger and a glass of wine inside them and then drop it into conversation?
“Meant to say, watch out for those steps, they’re lethal. We should probably put a fence panel in place of that gate in fact, it’s only there to provide us with right of access and we never use it. What do you think?” They’ll either agree or ask about their right of way, in which case you can explain.
Just, y’know, be subtle. If you think the entire reason for inviting them over is just an excuse to tell them to GTFO, it might backfire.