Been riding where I shouldn’t for a very long time (when I say a long time I mean pre MTB days, where a pushbike fitted with cowhorn bars and the thickest tyres available up the top of kinder drew amazed onlookers rather than unwelcome gestures) .. Unless aggression is shown to me I always plead ignorance in a polite way.
If they’re very rude.. I’m rude in return. If they’re aggressive (that’s only happened the once mind) I answer in kind.
Rule of thumb for me is that I ALWAYS show courtesy, slow down, give a smile and a thanks for people who I inconvenience along the way.