J Clin Epidemiol 2019: (Journal)
Ewald H., Ioannidis J. P., Ladanie A., Cord K. M., Bucher H. C., Hemkens L. G.
OBJECTIVES: To evaluate how estimated treatment effects agree between non-randomized studies using causal modelling with marginal structural models (MSM-studies) and randomized trials (RCTs). STUDY DESIGN: Meta-epidemiological study. SETTING: MSM-studies providing effect estimates on any healthcare outcome of any treatment were eligible. We systematically sought RCTs on the same clinical question and compared the direction of treatment effects, effect sizes, and confidence intervals. RESULTS: The main analysis included 19 MSM-studies (1039570 patients) and 141 RCTs (120669 patients). MSM-studies indicated effect estimates in the opposite direction from RCTs for 8 clinical questions (42%), and their 95% CI did not include the RCT estimate in 9 clinical questions (47%). The effect estimates deviated 1.58-fold between the study designs (median absolute deviation OR 1.58; IQR 1.37 to 2.16). Overall, we found no systematic disagreement regarding benefit or harm but confidence intervals were wide (summary ratio of odds ratios (sROR) 1.04; 95% CI 0.88 to 1.23). The subset of MSM-studies focusing on healthcare decision-making tended to overestimate experimental treatment benefits (sROR 1.44; 95% CI 0.99 to 2.09). CONCLUSION: Non-randomized studies using causal modelling with MSM may give different answers than RCTs. Caution is still required when non-randomized "real world" evidence is used for healthcare decisions.