JABSOM Library: Systematic Review Toolbox: Quality Assessment

Critical Appraisal Questions

Is the study question relevant?
Does the study add anything new?
What type of research question is being asked?
Was the study design appropriate for the research question?
Did the study methods address the most important potential sources of bias?
Was the study performed according to the original protocol?
Does the study test a stated hypothesis?
Were the statistical analyses performed correctly?
Do the data justify the conclusions?
Are there any conflicts of interest?

The University of Sydney Library, Systematic Reviews: Assessment Tools and Critical Appraisal

Taylor, P., Hussain, J. A., & Gadoud, A. (2013). How to appraise a systematic review. British Journal of Hospital Medicine, 74(6), 331-334. doi:10.12968/hmed.2013.74.6.331

Young, J. M., & Solomon, M. J. (2009). How to critically appraise an article. Nature Clinical Practice Gastroenterology and Hepatology, 6(2), 82-91. doi:10.1038/ncpgasthep1331

Quality Assessment

Assessing the quality of evidence contained within a systematic review is as important as analyzing the data within. Results from a poorly conducted study can be skewed by biases from the research methodology and should be interpreted with caution. Such studies should be acknowledged as such in the systematic review or outright excluded. Selecting an appropriate tool to help analyze strength of evidence and imbedded biases within each paper is also essential. If using a systematic review manuscript development tool (e.g., RevMan), a checklist may be built into the software. Other software (e.g., Rayyan) may help with screening search results and discarding irrelevant studies. The following tools/checklists may help with study assessment and critical appraisal.

Assessing the Methodological Quality of Systematic Reviews (AMSTAR 2) is widely used to critically appraise systematic reviews.
Centre for Evidence-Based Medicine (CEBM) contains a collection of critical appraisal tools for studies of all types and examples of usage.
Cochrane risk-of-bias (RoB 2) tool is the recommended tool for assessing quality and risk of bias in randomized clinical trials in Cochrane-submitted systematic reviews.
Critical Appraisal Skills Programme (CASP) has 25 years of experience and expertise in critical appraisal and offers appraisal checklists for a wide range of study types.
Joanna Briggs Institute (JBI) provides robust checklists for the appraisal and assessment of most types of studies.
National Academies of Sciences, Health and Medicine Division provides standards for assessing bias in primary studies comprising systematic reviews of therapeutic or medical interventions.
Newcastle-Ottawa Scale (NOS) is also used in non-observational studies of cohort and case-control varieties.
Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2) tool surveys diagnostic accuracy studies on four domains: index test, reference standard, patient selection, and flow and timing.
Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) framework is often used to measure the quality of cohort, case-control and cross-sectional studies.

Cochrane RoB 2 (RCT)

RoB 2.0: A revised tool to assess risk of bias in randomized trials pt1 from Cochrane Training on YouTube

Rayyan for Study Selection

[Webcast 67] Management/Analysis Tools for Reviews: Rayyan with Martin Morris from KTDRR and KTER on YouTube

JBI SUMARI (Multiple)

13 JBI SUMARI Tutorial: Critical appraisal from JBI on YouTube

QUADAS-2 (Diagnostic Test Accuracy)

6.3 Assessment of risk of bias and applicability using the QUADAS-2 tool from Cochrane Training on YouTube

Requesting Research Consultation

The Health Sciences Library provides consultation services for University of Hawaiʻi-affiliated students, staff, and faculty. The John A. Burns School of Medicine Health Sciences Library does not have staffing to conduct or assist researchers unaffiliated with the University of Hawaiʻi. Please utilize the publicly available guides and support pages that address research databases and tools.