Prevalence of Avoidable and Bias‐Inflicting Methodological Pitfalls in Real‐World Studies of Medication Safety and Effectiveness

Many real‐word evidence (RWE) studies that utilize existing healthcare data to evaluate treatment effects incur substantial but avoidable bias from methodologically flawed study design; however, the extent of preventable methodological pitfalls in current RWE is unknown. To characterize the prevalence of avoidable methodological pitfalls with potential for bias in published claims‐based studies of medication safety or effectiveness, we conducted an English‐language search of PubMed for articles published from January 1, 2010 to May 20, 2019 and randomly selected 75 studies (10 case‐control and 65 cohort studies) that evaluated safety or effectiveness of cardiovascular, diabetes, or osteoporosis medications using US health insurance claims. General and methodological study characteristics were extracted independently by two reviewers, and potential for bias was assessed across nine bias domains. Nearly all studies (95%) had at least one avoidable methodological issue known to incur bias, and 81% had potentially at least one of the four issues considered major due to their potential to undermine study validity: time‐related bias (57%), potential for depletion of outcome‐susceptible individuals (44%), inappropriate adjustment for postbaseline variables (41%), or potential for reverse causation (39%). The median number of major issues per study was 2 (interquartile range (IQR), 1–3) and was lower in cohort studies with a new‐user, active‐comparator design (median 1, IQR 0–1) than in cohort studies of prevalent users with a nonuser comparator (median 3, IQR 3–4). Recognizing and avoiding known methodological study design pitfalls could substantially improve the utility of RWE and confidence in its validity.