Case Study: Evaluation of a meta-analysis of the association between soy protein and cardiovascular disease

by   S. Stanley Young, et al.

It is well-known that claims coming from observational studies most often fail to replicate. Experimental (randomized) trials, where conditions are under researcher control, have a high reputation and meta-analysis of experimental trials are considered the best possible evidence. Given the irreproducibility crisis, experiments lately are starting to be questioned. There is a need to know the reliability of claims coming from randomized trials. A case study is presented here independently examining a published meta-analysis of randomized trials claiming that soy protein intake improves cardiovascular health. Counting and p-value plotting techniques (standard p-value plot, p-value expectation plot, and volcano plot) are used. Counting (search space) analysis indicates that reported p-values from the meta-analysis could be biased low due to multiple testing and multiple modeling. Plotting techniques used to visualize the behavior of the data set used for meta-analysis suggest that statistics drawn from the base papers do not satisfy key assumptions of a random-effects meta-analysis. These assumptions include using unbiased statistics all drawn from the same population. Also, publication bias is unaddressed in the meta-analysis. The claim that soy protein intake should improve cardiovascular health is not supported by our analysis.


The reliability of an environmental epidemiology meta-analysis, a case study

Summary Background Claims made in science papers are coming under incr...

Standard meta-analysis methods are not robust

P values or risk ratios from multiple, independent studies, observationa...

Evaluation of a meta-analysis of air quality and heart attacks, a case study

It is generally acknowledged that claims from observational studies ofte...

Statistical Methods for the meta-analysis paper by Itzhaky et al

This document describes the statistical methods used in Itzhaky et al ("...

Reliability of meta-analysis of an association between ambient air quality and development of asthma later in life

Claims from observational studies often fail to replicate. A study was u...

The reliability of a nutritional meta-analysis study

Background: Many researchers have studied the relationship between diet ...

Please sign up or login with your details

Forgot password? Click here to reset