Just thought I would share my similar experience.
I’ve just started riding again after a big off left me with concussion, a broken arm and collarbone.
I went through exactly the same as you and decided to ride more xc stuff and in a much more conservative way.
I’ve just started trail centre and natural dh riding again but mentally I put it down to the same as any other sport- knowing when to attack (familiar rides, with other riders) and when to back off ( blind or new trails or solo riding).
Confidence coming back slowly but still have the odd squeaky bum moment where the fear gets the better of the bravery- just walk away and wait for the right time to go for it. It does come back!