Driving at 100mph on the M40 at 2am in the morning with no other cards around is not unsafe.
So you can see that there are no other cars. What about someone crossing the road? Illegal, perhaps. But that doesn't make it safe for you to flatten them and carry on with your journey.
Would you travel at 100mph on the A1 at 2am (which is barely any different to the M1 in places, or I dare say the M40), where someone may be legally cycling, lit up by a set of lights £4.99 from ASDA?
I got caught speeding entering a 30 limit by a school on a Sunday afternoon. Cursed myself at the time, because I'd normally be doing 20-25mph on a weekday. Then cursed them for putting a mobile van there on a Sunday. But I was surprised to learn at the subsequent Speed Awareness course that as many kids are killed outside schools on weekends as there are on weekdays.
That 'fact' might not even be true. I dunno. It's what they told us. But the problem is, and what this thread highlights as well, is that everyone has different definitions of what is safe. We all see things in different ways. And almost 20 years after passing my driving test I'm still learning things about driving all the time, and I cringe at some of the stuff I've done in the past. The upper limit is there to protect us from our own stupidity, and even simple lack of knowledge, which applies to us all. I think part of being a good driver, is accepting that there are probably variables that you haven't even thought about. So I think it pays to stick to the limits. Unless you're travelling serious distances, it makes barely any difference to the time getting there anyway.