I agree such that to say that there are many factors that influence safety such as skill, conditions, judgement, awareness, speed etc.
Varying these factors will have an effect on road safety eg increase skill, or reduce speed or change attitude.
So if we accept that these changes will bring about safety we have to legislate for each one as we are able to control it.
Speed is one of those, it is easy to control and has a significant effect on safety no matter what your skill. EG a very skilled person at 70 MPH on a motorway is less likely to cause harm in the event of an accident than the same person at 90 MPH. That is an indisputable fact. Therefore pragmatics and logic indicate that reducing speed is safer. The rights of the public to be safe outweigh the rights of the individual to get there a few minutes sooner.