Evidence Synthesis in the Social Sciences

What is a protocol? 

A protocol is a detailed plan outlining the steps for a research project. Developing a protocol for a systematic review can take several weeks or months, depending on the project's complexity and the availability of time for team members. A detailed protocol should be developed and made public before the team begins conducting the systematic review. This helps to ensure that the search was designed a priori and not in response to studies identified for inclusion. 

Why use protocols? 

Systematic reviews are complex research projects that require advanced planning, teamwork, and a series of critical judgments and decisions. When these plans are developed in advance, they provide essential guidance for the review team, limit ad hoc decisions, and increase accountability to readers. 

Protocols can: 

• Facilitate planning and communication by allowing members of the review team (and perhaps advisory members) to contribute to plans for a systematic review. 
• Enhance the conceptualization and methodological rigor of systematic review plans through collaborative teamwork (and possibly peer review) at the protocol stage.
• Document initial plans
• Minimize bias in the review process by limiting ad hoc and post-ad hoc decisions.
• Provide transparency and accountability of review authors to readers.
• Avoid research waste by publicly declaring plans for a review. 

Public protocols can later be compared to the completed review to assess reviewers' decisions and methods, helping to detect any instances of selective reporting. Access to an a priori protocol enables readers to evaluate the rationale and fairness behind reviewers' decisions on study inclusion and exclusion, their assessment of bias risks, adherence to the planned review methodology, and whether all results are fully reported or selectively presented. 

When plans change: 

When reviewers decide to deviate from the plans described in the protocol, their final report should document the changes they made, the timing of these changes (whether changes were made before or after data collection and before or after data analysis), and a rationale for each change. 

References: 

Littell, J. H., & Valentine, J. C. Unit 2: Problem formulation. In J. C. Valentine, J. H. Littell, & S. Young (Eds.), Systematic reviews and meta-analysis: A Campbell Collaboration online course. Open Learning Initiative, 2023.

Stewart, L., Moher, D., & Shekelle, P. (2012). Why prospective registration of systematic reviews make sense. Systematic Reviews. Vol. 1.

Evidence Synthesis Protocol Template

This document can be used as a template to prepare a protocol for a range of evidence synthesis methodologies (such as systematic reviews, scoping reviews, or systematic maps).

PRISMA Checklist

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) is an evidence-based minimum set of items for reporting in systematic reviews and meta-analyses. The 27 checklist items pertain to the content of a systematic review and meta-analysis, including the title, abstract, methods, results, discussion, and funding.

• PRISMA-P - This is an extension of the PRISMA checklist, which provides a general overview of recommended items to include in the protocol. 
• PRISMA Extension for Network Meta-Analysis

  The PRISMA extension for network meta-analysis, or PRISMA-NMA, provides guidance for reporting systematic reviews comparing multiple treatments using direct and indirect evidence in network meta-analyses. In addition to providing guidance, it also highlights educational information related to key considerations in the practice of network meta-analysis.

Additional Guidance:

• Guidance from the Campbell Collaboration 
• Guidance from Collaboration for Environmental Science 
• Guidance from Cochrane
• Guidance from JBI

References:

• University of Minnesota Guide to Evidence Synthesis
• Shamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., & Stewart, L.A. (2015) Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015: Elaboration and explanation. British Medical Journal. V 349 https://doi.org/10.1136/bmj.g7647. 
• Peters, M.D., Godfrey, C., McInerney, P., Khalil, H., Larsen, P., Marnie, C., ... & Munn, Z. (2022). Best practice guidance and reporting items for the development of scoping review protocols. JBI Evidence Synthesis, 20(4), 953-968. doi.org/10.11124/JBIES-21-00242

There are various platforms where you can share or publish a protocol. Ultimately, your choice will depend on the specific journal requirements or the norms within your discipline. Some journals require you to publish the protocol in a specific registry, while others mandate using registered report journals hosted by evidence synthesis organizations such as the Campbell Collaboration, Cochrane, JBI, etc.

Places to Register: 

• Campbell Collaboration 
• PROSPERO - a title and protocol registry for systematic reviews with at least one health outcome. 
• Open Science Framework Registries - One of the more common places for protocols. To learn more, review the template and the OSF Registries help page. Additionally, consider this template for synthesizing non-intervention studies. 
• Protocols.io

• View the Participating Journals tab on this website for a list of journals offering a registered reports submission model. 

Protocols published in various registries have different expectations. Those published in journals may require a more extensive rationale section and more detailed content in each section. 

Protocols in some registries are not "verified" for content or completion, resulting in many examples of missing details in certain sections, often failing to meet PRIMA-P reporting standards. 

The existence of a protocol does not guarantee completeness or detailed methodology. Incomplete protocols are frequently shared, similar to reviews.