menu
Female doctor at computer

How the peer reviewed GRACE checklist is making it easier for researchers to decide which observational comparative effectiveness research studies are worth reviewing.

When it comes to non-interventional research, it can be difficult to know which studies are worth reviewing. In interventional clinical studies there are obvious factors to look for, like randomized controlled trials that indicate a study has followed accepted protocols and standards of practice. But nothing like this exist in non-interventional research.

This is becoming a growing challenge for researchers, particularly as demand increases for information about comparative effectiveness of medical treatments for a various subgroups of interest. In response, my colleague Priscilla Velentgas and I launched the GRACE Initiative (Good ReseArch for Comparative Effectiveness), to enhance the quality of observational comparative effectiveness research (CER), and to facilitate its use for decision-making about therapeutic alternatives. Our hope was to provide a well-defined set of principles that would both guide good practice for the researchers conducting observational studies of comparative effectiveness, and provide direction to those who need to understand how such studies can be used appropriately when considering therapeutic options.

We began by developing the GRACE Principles, which is high-level guidance document intended to help decision-makers evaluate the quality of observational studies comparing the effectiveness of various medical products and services. This document is broken into three groups of questions that can be used to evaluate the quality of non-interventional comparative effectiveness studies, in terms of the accuracy of information generated and how to consider areas of uncertainty. The three primary questions include:

  1. Were the study plans specified in advance of conducting the study?
  2. Was the study conducted and analyzed in a manner consistent with good practices, and reported in enough details for evaluation and replication?
  3. How valid is the interpretation of comparative effectiveness for the population of interest, assuming sound methodology and appropriate follow-up?

From there we developed the 11-point GRACE checklist, which investigators can use to determine whether an observational study of comparative effectiveness merits consideration for its informational value. At the heart of every question on this list is whether the data provided is adequate for the study purpose. The checklist challenges the reader to think critically about key issues, such as whether the primary outcomes are adequately recorded for the study purpose. Whether the primary outcomes are validated, adjudicated, or otherwise known to be valid in a similar population. And is the classification of exposed and unexposed person-time free of “immortal time bias.”  We don’t provide a pass-fail type of scoring system, since interpretation of these studies requires weighing available evidence and needs to be tempered by judgment of their applicability to routine care.

We complied this checklist items through an extensive literature review and consultation with experts from academia and other health professionals to ensure its validity. Since no gold standard exists for this process, we used three validation tests to evaluate each checklist item:

  1. Quality assessments from published systematic reviews
  2. Individual recognized experts
  3. Concordant expert assessments

After being tested by hundreds of volunteers from five continents, we can confidently say that the GRACE Checklist provides a validated tool for screening the quality of observational comparative effectiveness studies. And we aren’t the only ones who think so. The GRACE Principles have been widely cited as a valuable research assessment tool by NICE and EMA, and the checklist was recently named the preferred reporting standard for observational comparative effectiveness studies in The Journal of Managed Care and Specialty Pharmacy.

Perhaps one of the reasons that this work stands out from its competitors is that we actually tested the elements of the checklist to understand their predictive value, individually and collectively.  Unlike consensus-driven guidance documents, of which they are plenty, the GRACE work stands out because of these validation efforts.

We believe that this checklist will give researchers a more productive way to judge observational data, while providing the biopharmaceutical community with a framework for conducting observational research that will withstand scrutiny by experts when evaluating evidence on new or competing products. Our hope is that eventually all observational research studies will come with a GRACE checklist stamp of approval.