It might be worth reflecting a bit on why you started riding off road in the first place. What was it about the experience that you enjoyed and that made you want to do it again? Do you still have that feeling and if not, why not?
Also, what is it that is pushing you to keep riding stuff that is beyond your comfort zone? Do you need to be better/faster than other riders to feel good about yourself? Do you believe that the only way to improve is to keep pushing your boundaries?
Somebody mentioned guitar playing on another thread, which is something else that I dabble in. Although guitar playing and mountain biking seems poles apart there are some interesting parallels. Lots of people try and take up guitar (or any instrument) in their middle age, but get frustrated because they don't sound like a rock god after a few months and give up. The problem is often that they are always trying to play stuff that's way beyond their ability in a mistaken belief that if they keep playing hard pieces they'll get better quicker. In fact, as any teacher will tell you, playing hard pieces badly just ingrains bad habits. It's much better to start with something simple and really nail that before moving on.
I don't really know how well that translates to mountain biking, but it does seem as though an awful lot of people take it up in later years and then beat themselves up because they are not a riding god after a few months. If you can hold onto the simple enjoyment of riding a bike and just keep doing it, then after 20 years you'll be pretty good without even trying. What's the hurry?
Oh and if pads make you feel more comfortable then wear them. I've taken to wearing G-form knee pads on pretty much every off-road ride now as I found that I hardly ever crash on the bits that I think are dangerous. It's the simple bits that get me every time.