My most recent off (broken arm) has given me a bit of perspective on this issue.
I used to worry about crashing because I didn't fancy hurting myself. However, having done it I realise the pain isn't really the issue. Yes it hurts, but it's hardly unbearable. In fact, if you have the sort of job/life where you can drop out for a month or so then it's really no big deal. A&E will patch you up, you get to act all heroic for a bit with your scars/bruises and your local GP will happily sign you off work for weeks on end at the drop of a hat. So, you can sit around taking it easy until you are better.
The problem comes when you have people who rely on you. I've not taken any time off work with the latest fracture, simply because I don't have the sort of job where I can do that without causing major headaches for other people. So I've muddled through as best I can. But it's pretty miserable trying to carry on normal life when you are injured and more importantly, it still caused major disruption for the rest of the family. Mainly as I couldn't drive.
So, ironically, having had a decent crash I'm now less scared of crashing, but much more aware of the impact injuring myself has on those around me. Basically, for me, doing something that I know has a good chance of causing serious injury just because it's fun is pretty selfish.
I'm not ready to retire to the golf course just yet, but I think Garry got it right.
Broken bones are a young man's game.