Evidence Synthesis in the Social Sciences
Critical Appraisal / Risk of Bias Assessment
This page provides an overview of screening studies for eligibility, highlighting key steps and considerations to help researchers navigate this complex methodology. While not comprehensive, it is designed to offer a foundational understanding of the process. A thorough literature search often produces many potentially relevant studies, sometimes numbering in the hundreds, thousands, or even tens of thousands. These guidelines help manage documents, focusing on methods to identify and exclude studies that are irrelevant to your research.
The risk of bias assessment (sometimes referred to as "quality assessment" or "critical appraisal") helps establish the transparency of evidence synthesis results and findings. It is a defining element of systematic reviews and is often performed for each study included in the review. Simply put, to critically appraise is to measure the quality of the study.
Evidence Syntheses strive to eliminate bias in their findings. Individual studies included in a synthesis may contain biases in their results or conclusions, for example, design flaws that raise questions about the validity of the findings or an overestimation of the intervention effect. Risk of bias assessment is generally not required with evidence synthesis outside of systematic reviews. However, this may depend on the evidence synthesis method that you are utilizing.
Tools
This Cochrane Training presentation helps navigate the steps for assessing the risk of bias. It also discusses health studies-based sources of bias that may not apply to other disciplines. For disciplines outside of human medicine, the Critical Appraisal Skills Programme offers checklists that can be applied to a diverse range of studies. The Mixed Methods Appraisal Tool can be a helpful resource if the included studies employ mixed methods.
Screening
Screening Steps to Consider:
Reconcile disagreements by having the screeners conference together and/or having a third person reconcile disagreements. *Asking the screener who disagreed with the group to articulate a rationale for the decision will sometimes reveal problems in item wording or terms that need to be more clearly defined.
- Obtain the full text of all documents that were not ruled out based on information in their titles and abstracts.
- Develop a full-text screening guide. This will involve adding any questions related to the PICOS criteria that were not part of the title and abstract screening.
- Evaluate the eligibility of studies using the full-text screening guide and the full-text of the studies. You can use the PRISMA Flow diagram to visually summarize the screening process.
- Identify software to support screening.
- Develop a screening guide: The screening questions will reduce the cognitive burden on screeners, allowing them to process documents accurately, reliably, and efficiently (4-6 questions in total provide a good balance). There are four general principles in survey design that are highly applicable to the creation of a screening guide:
- Word questions clearly
- Ensure that screeners understand questions in a similar way
- Ask one question at a time (no double-barrel questions)
- Use low inference questions (questions that do not require screeners to make complex judgments)
- Train screeners
- Pilot-test the screening process, gather feedback, and revise as needed.
- Screen the titles and abstracts of each document using the screening guide; documents are typically screened by two trained screeners who work independently of one another.
Examples:
The image above depicts a table with three columns labeled "Document title," "Keep or Drop?" and "Rationale." It contains three rows, each detailing separate documents with corresponding decisions and explanations. The table has a white background with blue borders and black text.
The image above is a table with ten columns and six rows. The first row contains headers for each column. The first column header is "Screener", followed by "Doc 1" through "Doc 10". The first column lists names: Dola, Alex, Mike, Bob, and Ted. The table displays decisions labeled as "Keep" or "Drop" for each document column and screener.
Representing Risk of Bias Assessment
Risk of bias assessments can be represented in a table format, showing each included study and its strength across several quality criteria for that particular study type.
If a high proportion of studies are assessed as having a high risk of bias, caution should be exercised when interpreting the results for your evidence synthesis. More information can be found in Chapter 8 of the Cochrane Handbook.
Resources
You can use the PRISMA Flow diagram to visually summarize the screening process.
View the CASP (Critical Appraisal Skills Programme) to illustrate what people will look for when they appraise your review.