OP seems to have developed a sense of ownership over this local trails. (Been riding here 15 years, dug with my own hands etc).
Hell, as they say, is other people, but they’ve got the same right to use them as you. I also get annoyed by people who block trails, happens all the time in summer near me, groups of lads (usually) used to having their local wood to themselfs arrive at Cwmcarn for their road trip weekender and don’t really adapt stop en mass on hairpins or to look at a rock or something, it’s annoyingly but they’ll usually move when they see you coming.
Have to admit when I’m riding with my kids I’ll block the trail, but most people smile and say hi when they see I’m introducing a child to our sport.
I suspect your “missed by inches” line was hyperbolic, I’ve never been to Swinley, but I suspect it’s not a WC level DH venue, if you can’t see another rider on the trial ahead you’re either riding with your eyes glued to the front wheel, or you should check your brake pads.