Researchers have created a star-based metric that assesses the quality of evidence for an association between a given behavior — like eating red meat — and a given health outcome.Credit: Education Images/Universal Images Group/Getty
Does Eating Red Meat Shorten Lifespan? Some researchers certainly think so. Works such as the Global Burden of Diseases, Injuries, and Risk Factors Study1 has prompted the World Health Organization and the US Department of Agriculture to advise people to limit their consumption of unprocessed red meat to protect against diseases such as type 2 diabetes and various types of cancer.
Other researchers are less sure. Targets for red meat consumption set by health authorities and expert panels vary widely, with some advising eating no more than 14 grams per day and others not giving a recommended limit. This sends out a confusing message that in itself is not good for public health.
It’s not just red meat: the evidence base surrounding much dietary and broader health advice is similarly contentious. Now, a new approach could help health policymakers better assess the quality of studies evaluating potential health risks. A team from the Institute for Health Metrics and Evaluation (IHME) at the University of Washington in Seattle has created a star-based metric that assesses the quality of evidence linking a specific behavior — like eating red meat or smoking . and a specific health outcome2. A five-star rating means the connection is clearly made; One star means there is either no association between the two factors, or the evidence is too weak to draw a firm conclusion.
The burden of proof studies
What the researchers call “weight of evidence” analysis doesn’t resolve inherently irritating questions like the risks of red meat or the benefits of vegetables. But as an assessment of the quality of available research, it can be helpful to point out areas to research funders where better evidence is needed to make more informed conclusions.
How is the star rating structured? What are its parameters – and can the methodology itself be considered rigorous research? The IHME team took several steps to try to quantify the impact of various biases in the studies being evaluated. For example, an epidemiological study may be biased in different ways than a study testing the outcomes of health interventions. The researchers also eliminated a common source of research bias, the assumption that health risks increase exponentially with the parameter being studied, such as blood pressure or unprocessed red meat consumption. And they tried to account for the bias that can occur when sample sizes are small.
Applying this framework to studies evaluating a total of 180 questions produced results that are largely unsurprising. For example, studies examining a link between smoking and a variety of cancers receive a five-star rating3. Similarly, high systolic blood pressure — the force exerted by the heart to pump blood — has a five-star association with the narrowing of blood vessels, known as ischemic heart disease4.
Studies evaluating the diet and its health effects receive significantly lower star ratings. For example, the IHME analysis finds only weak evidence of an association between the consumption of unprocessed red meat and outcomes such as colon cancer, type 2 diabetes and ischemic heart disease5. It finds no association in studies examining whether eating unprocessed red meat leads to either type of stroke. There is stronger, but not overwhelming, evidence that eating vegetables reduces the risk of stroke and ischemic heart disease6.
Let’s move beyond rhetoric: it’s time to change the way we judge research
In some cases, the lower star ratings could be due to effect size: for example, any health risks from eating red meat are likely small compared to the huge toll smoking takes on the body. Above all, the poorly rated results show that studies in these areas have to improve if they are to deliver convincing results.
It is difficult to separate the effects of a single dietary component from those of the complex array of exposures over a person’s lifetime. Larger studies with a diverse pool of participants and tight control of their daily diet would be needed. Such studies will involve collaboration between research groups with different expertise and access to participants in different environmental settings – a move that funders need to encourage. This is a company worth prioritizing. A low risk for an individual does not mean a low public health impact: a low-risk behavior can have a large impact on the population if it is very common.
The literature in the field of responsible research and innovation shows how metrics in science must always be tested for robustness and rigor. Wide consultation must take place, and the unintended consequences of using metrics must be anticipated as much as possible, as demonstrated by initiatives such as the San Francisco Declaration on Research Assessment and the Leiden Manifesto. That work needs to come sooner rather than later.
We have evidence that clinical trials that are underpowered and lack the necessary controls to understand the data are not helping. When funders don’t focus their efforts on producing quality data, the public remains confused, tired, distrustful and deprived of the information they need to make informed health and lifestyle decisions.
