Would it be a sensitive trail if you rode it in the day, conflict and argument guaranteed? If so, I can see how that might influence your feelings riding it in the evening, even if it’s just on a subconcious level.
I noticed a thread up today about Brown Knoll in the Peak – I rode that once at a weekend and ran into a dickhead convention walking across it – must have had 4 or 5 confrontations in abar 2 miles, including one trekking pole in the chest from an old bloke, and a 5 min frank exchange of views with a pair of Teds who didn’t understand that we were on the slopes of Kinder Scout. So on the odd occasion I’ve ridden it since it doesn’t seem that enjoyable a trail [it’s just a link-up in any case, not a notable MTB trail].
If it isn’t one of those trails, though, then you need to rethink your approach. Anxiety over where you’re riding just on general principles, which are bogus anyway, isn’t right.