There’s a “London Calling” thread on Bike Radar with a bunch of folk just informally arranging rides around the area and includes Swinley, and there’s a Facebook group to go with it.
Not specifically DH but there are downhillers & jump enthusiasts there. Generally just a mix of all kinds, abilities and styles.
Also on Facebook is ‘DH Group’. It’s not specifically based on an area, but there are a lot of people on there from around London. Some do Swinley on and off.
If DH is your thing, there are spots in the Surrey Hills that are quite DH, along with gaps, jumps and drops. Further away is Aston Hill or Rogate. Again the guys in the above groups do these places also.