I'd say it's as much who you ride with and what the dynamic of the group is. I've lived and ridden in the North East, the Peak, The North Downs and now (for the second time) Brighton and the South Downs. I'm really lucky in that the group I ride with now is full of both technically good and just plain old fashioned fit & strong riders. Riding with these guys and trying to follow their back wheels has made me a whole world faster.
It doesn't really matter how hard/easy the terrain is. If the terrain is "easier" a strong and good rider will just go faster until they reach the same limit of their skill. This is all within reason of course, a fire road drag won't tax anyone.
Within the group it seems to be self perpetuating as we all try and stay with, compete with and beat each other but always making sure the back of the group never spits people out and regular re-groups at gates and lots of chatting are also the norm.
Just my tow penneth, and something no-one has mentioned