TBH prett much all of my road miles are with just the one mate or solo, we've both tried bigger groups and clubs but not gotten on with it, in large part due to the way drivers deal with it when faced with more than two cyclists.
We tend to try and plan interesting routes on quieter roads, but it is sort of sad that avoiding dickish driving has to be such a consideration when choosing a route. And you are right I think roadies sort of do accept inconsiderate and dounright dangerous driving as "The Norm" and hence don't challenge it.
We did a Charity ride last Sunday which was pretty well planned, executed and routed IMO, but where it did have to take in busier road sections there were still issues, mainly due to impatient and stupid drivers, perhaps also partly due to the mix of cyclists you get on a Charity ride, Big groups that vary in speed (and control), with lots of gaps inbetween do sort of serve to tempt bad drivers into sketchy overtakes, only to then sit in the gap revving or just hang it out and try to take the lot on a bend, I did see some interesting displays...
Ultimately it is down to driver education and temprament, as a cyclist you can "choose to make yourself less of a target" by riding solo or in small groups, taking the quietest routes and picking less busy times, but then you really shouldn't have to do that.
Cycling is a legitimate use of the roads, if a driver can't read the road ahead, spot that an overtake isn't safe, and in that split second accept that they are going to have to wait a little bit, rather than dip their right toe and hope for the best, then then they really shouldn't have a licence.