working links to bmj discussion
http://www.bmj.com/search/compulsory%2520cycle%2520helmets
http://www.bmj.com/search/cycle%2520helmets
http://www.bmj.com/search/bicycle%2520helmets%2520cochrane%2520review%2520
Discussion fro the cernow paper published in 2005 critiquing the cochranre review
I have put in bold a little of the discussion on the point you didn’t understand about why the use of the sample they did introduces possible bias
The review’s conclusion that its case–control studies establish scientific evidence that helmets protect against brain injury and its recommendation that cyclists should be encouraged to wear them are not supported. First, the studies’ primary choice of outcome of interest is inappropriate and disregards the science; second, bias in their control groups is likely; third, their findings are too limited and outdated to be useful now.
Referring to the first point, the review dismisses both randomised controlled trials and cohort studies because head injury is a relatively rare outcome, but this is a reason based on cost advantages of the case–control design when cyclists with head injury are cases, rather than on need for thorough scientific assessment of preventive health care. Indeed, such advantages enter into the choices of head injury (of all kinds) as cases and other injury as controls. This conforms with popular belief that equates head injury and the risk of the even rarer death and chronic disability, but it jumbles together problems for scientific research that involve disparate injuries and risk factors and it distracts attention from the important outcome of interest, intracranial trauma of severity AIS 4–6. Such trauma would be the appropriate choice for cases, but none of the studies collects, treats and interprets data in accordance with scientific knowledge of it and its causes, and findings of empirical association between reduced head and brain injury and the wearing of helmets say nothing about their efficacy against it.
Second, sub-analysis of data of Thompson 1996, like that made by Thompson 1989, suggests that the cases and controls of those studies differ in the prevalence of risk factors for brain injury; unmeasured factors may exist and add significant bias. This implies that the review’s flat conclusion that helmets reduce brain injury by 88% (the estimate of the small 1989 study) may be grossly in error. Further, that conclusion disregards the 75% derived from the larger sampling of 1996 and the possibility that the difference is just due to chance. Yet Cummings says that I do not consider the role of chance in the decline from 1989 to 1996 in the stated protective effect and that I suggest it is evidence. I do not; I acknowledge that the confidence intervals overlap—but the p-value of 0.08 is very suggestive of a real decline.
Use of case–control studies to obtain data derogates from the normal standard for Cochrane systematic reviews, randomised controlled trials—which Cummings suggests would be unethical. I comment on the ethics elsewhere (Curnow, 2006). Here, I point to the contrast with compulsory wearing of helmets, despite experimental evidence that helmets can increase axonal shear injuries (Corner et al., 1987). Further, the review finds only an empirical association between the wearing of helmets and brain injury. To establish scientific evidence, data would need to be obtained, treated and interpreted to test a hypothesis in accord with scientific laws and knowledge of brain injury. Neither the review nor its studies do this. Also, some in the Collaboration seem to have lost sight of this point, on the specious argument that Cochrane reviews need to focus on “real-world effectiveness” without concern about scientific theory. Though no coherent theory may be available for evaluating some kinds of health care, disregard of available scientific knowledge cannot be justified.
Third, the review’s conclusions are too limited and outdated to be useful. They are limited because they are explicitly restricted to cyclists who crash. The conclusions would hold for the whole population of cyclists only if those who crash are representative of it, which would not be so if wearing a helmet affected the risk of having an accident. The risk would be less if the more cautious cyclists chose to wear helmets, or greater if a helmet gave its wearer undue confidence, resulting in a little less care being taken. Such risk compensation has been suggested as an explanation for some safety devices in motor vehicles not performing as expected. It would appear to be likely for cyclists too; a study conducted in Victoria following publicity about helmets found that teenagers believed helmets would save them in a serious accident with a bus or a truck! (Elliott and Shanahan Research, 1986). I therefore suggest that an assumption that wearing a helmet does not affect the risk of having an accident is of dubious validity. Hence, there can be no certainty that the conclusions of the review can usefully be applied to the whole population of cyclists, contrary to its aim of “establishing the real-world effectiveness of helmets … for the promotion of helmet programs”. See also the comments in my Section 2.3.
The review’s estimates that helmets reduce the risk of head and brain injury by 85% and 88% date from the 1989 study, when almost all helmets had hard shells capable of protecting the skull. The indications are that present-day soft helmets cannot do this (Curnow, 2003), but the 1989 estimates continue to influence policy makers. For example, a leaflet issued by the US Department of Transportation in 1998 to promote the wearing of helmets states these estimates as fact and the U.S. Consumer Product Safety Commission cites them in a press release of 29 March 2006. But the reviewers continue to uphold these estimates, despite knowing their influence on policy and that the adequacy of the control groups used to derive them has not been confirmed by sub-analysis like that used in Thompson 1989.
5. Conclusions
In view of the influence of a Cochrane review of bicycle helmets on policies for wearing, it should have the utmost reliability. All evidence should be obtained from experiment or randomised controlled trial based on relevant scientific knowledge. The current review should be removed from the Cochrane library.
Acknowledgements
