At my regular DH spot, probably more than 50% the riders on any given day will partake in a race/timed event or two at some point during the year, regardless of their ability.
Compare this with the massively popular trail centre down the road and it’s more like 15-20% of them who race.
I’d guess the reason for this would be the amount of fitness that it’s perceived you need to enter a XC race and not come last by an embarrassing margin. Compare that to DH where there’s always a chance you may do alright if you can keep it upright as there always plenty of people falling off.