I do not feel like a pariah, ever. I have access to an extensive network of rights of way; whole areas of country which are maintained for access and where riding is encouraged as long as it doesn't get silly; spots specifically provided in which it can get as silly as it wants; entire settlements taken over for enormous occasional mass-participation rides and dozens of races held on private land. In my regular riding spots there are pubs, cafes and shops that are happy to serve me food and youth hostels, inns and B&Bs that are happy to deal with my dripping gear and muddy tyre-prints.
In about 12 years of riding I've had just the slightest hint of hassle on just a couple of occasions. Mostly other people, out and about enjoying the same countryside in different ways have been no worse than indifferent and often kind and friendly. Perhaps it's just that I'm a particularly lovely human being, but that doesn't seem to explain someone building the trail centres… 🙂