I once got a bollocking of a guy because i was riding with a few mates near ambleside last year on one of the evans "ride it" days without paying. I lived near by at the time and didn't feel i needed to pay for my local trails, (i didn't ride the whole route and had forgotten it was on)
I do feel its good manners to put a couple of quid in a bucket somewhere though if its for charity. Especially if you are tagging along with people who have paid.
But if you are only crossing paths and are getting on with your on agenda then don't pay.