Evaluating Evidence in Medicine

Responsive Mode

Project Team:

Principal Investigator:

Professor Jon Williamson, University of Kent

Award Information:

This project seeks to improve the way that evidence is evaluated in medical research and health policy by developing a recent line of work in philosophy.

Evidence-based medicine has transformed the way in which statistical evidence is used in medicine. Hierarchies of evidence are now routinely used by medical researchers and health policy makers to assess evidence for the effectiveness of treatments and health policies: studies that simply observe patients after treatment are ranked lower than studies that randomly decide who to treat, and these in turn rank lower than studies that review the evidence obtained by a series of trials. Evidence hierarchies have become so widely endorsed that they are now being used across the social sciences and in public policy, as well as in medicine.

While there has been some debate about which sorts of trials should be placed at the top of the hierarchy, this project focuses on the bottom level, which is normally occupied by evidence that is not obtained from a statistical trial. In our view, while it is appropriate to relegate anecdotal evidence and hearsay to this lowest level, other, better quality evidence is also being ignored, simply because it is often not obtained from statistical trials.

In particular, evidence of the underlying physiological and biochemical mechanisms is often classified as inferior to statistical evidence. This is because evidence of mechanisms is normally obtained, not simply via statistical trials, but in a complex way, by integrating a mixture of laboratory experiments, basic scientific knowledge and case studies as well as past trials. Recent work suggests that it is wrong to view evidence of mechanisms as inferior. Philosophers of causality and historians of medicine have argued that evidence of mechanisms is required alongside statistical evidence in order to evaluate whether treatments or health policies are effective. This is because such evidence helps to determine whether positive results of a trial are due to genuine effectiveness or are simply a statistical blip; such evidence is also crucial when designing and interpreting a statistical trial, and when determining effectiveness in a new population or a particular patient.

In an exploratory project the project team put forward these arguments and consulted with medical researchers and health policy advisers, who verified the importance of mechanistic evidence and confirmed that it was often being tacitly used, against the explicit recommendations of the hierarchies of evidence.

But how can one formulate explicit guidelines for considering mechanistic evidence alongside statistical evidence? One reason why non-statistical evidence is relegated to the bottom of the hierarchies is that it is very hard to weigh against evidence obtained from statistical trials. In this project we seek to understand how to evaluate mechanistic evidence alongside statistical evidence in medical research and health policy.

This task needs a variety of methods. It requires work on philosophical theories of causality and causal discovery, to provide the theoretical underpinning. It requires historical case studies of causal discovery in medicine, to understand the full variety of ways in which evidence needs to be evaluated. It requires close cooperation with medical researchers and health policy advisers, to assess the practical needs and concerns of those who use the guidelines. It requires formal methods to address technical concerns.

The makeup of the project team is crucial if this task is to succeed. Our team involves the National Institute of Health and Clinical Excellence (NICE) and the WHO International Agency for Research on Cancer (IARC) as well as philosophers of causality at the Centre for Reasoning at the University of Kent, researchers in history and philosophy of science and medicine at UCL and at the University of Ferrara, and an expert in evidence-based medicine at Leiden Medical Centre.

Project dates: June 2015- May 2018