OP there’s no-one to impress by walloping down the side of said hill and ending up in a ditch. However, maybe there are things you can do to increase your sense of control on the bike (1 finger braking, footwork, looking further down the trail, using the front brake as the dominant/rear brake only when appropriate, not staring at things to avoid, not hanging off the back like it’s the 90s etc)?
There’s always someone faster who will think you’re slow, and someone slower who will think you’re fast. There’s a speed at which you feel you can respond to things and control the bike and beyond that you will be beyond your perceived range of control.
I now aim to ride smoother than faster, which happens to mean I ride faster than I used to be but ride more calmly. It’s amazing what a difference staying loose on the bike can do and it can only be done if you’re relaxed.