I've worn one for the 24 years I've been riding a mountain bike. I have yet to hit my head while riding. There are plenty of times when riding when I do not need it, at which point I look for a convenient place to put it. So far the best place I've found is on my head.
I've started to notice a lot more people riding on the South Downs without helmets. They are just people out for a stroll on a bike, they are not in any more need of a helmet than someone out for a walk or a run.
If you feel you need one then wear one and if you don't then don't.