Evidence Synthesis in the Social Sciences

Systematic review searches are designed to comprehensively locate all studies addressing the review question, with a focus on reproducibility and transparency.  These searches prioritize sensitivity—maximizing the retrieval of all relevant studies—to achieve a thorough overview of the evidence, unlike traditional searches, which focus on precision (reducing irrelevant results). This approach ensures no relevant studies are overlooked, even if it means retrieving some irrelevant records.

Your initial search should be designed for a primary database (the one most likely to contain the most studies on your topic). It should be a single, multi-line search with all relevant terms, rather than running multiple separate searches. Then, adapted or 'translated' for use in other databases. 

Steps to developing a search strategy: 

1. Identify a known set of studies:

   Locate between 3 and 5 relevant studies that meet many or all of the stated inclusion criteria and use these examples to identify search terms (keywords and their synonyms) and database-specific subject headings. You can also use them to determine whether the concepts of interest are mentioned in the title or abstract of the articles (as opposed to the full text) and what study designs are used in this field. These "known articles" can also eventually be used to test the search strategy that will be developed.

2. Harvest terms: Locate terms relevant to each of your search concepts.

   • Brainstorming by the research team: Collaborate with your team to generate ideas.
   • Disciplinary thesaurus or encyclopedia: Consult specialized reference materials for terms in your field.
   • Harvesting terms from known articles: Retrieve terms by scanning article records in databases, focusing on titles, abstracts, author keywords, and subject terms. Collect these terms into a working file (e.g., notepad, word processor, or spreadsheet) and organize them by concept, possibly using a table format for clarity.
   • Database entries: Use database searches to identify additional terms related to your concepts. 

3. Verify search concepts: 

   • Verify that your intended search concepts appear in each of your known articles' titles, abstracts, and author-supplied keywords. If several articles lack relevant terms in these fields, you should reconsider including that concept in your search strategy and reduce the number of search concepts to streamline your approach. 

4. Locate and apply subject headings: 

   • Databases often use controlled vocabulary (subject headings) to organize (index) articles on the same topic under one preferred term, regardless of what term the author provided. This helps group related records and index entire articles, not just titles or abstracts. Subject headings can be found by reviewing known articles or browsing a database thesaurus. They should be selected based on their understanding of the topic and the database's definitions. Some databases offer descriptions and examples of terms. However, some concepts may not have a suitable subject heading, especially if they are new or lack the right level of specificity. **Indexing is a time-consuming process; indexers may not be topic experts, and indexing errors can occur. ** Therefore, systematic searches should include both controlled vocabulary and associated keywords. This approach can help mitigate the potential bias that results from using only one of these approaches alone.**

The image presents a table divided into four rows and three columns. The first row contains headers: “Keywords” and “Subject headings.” The “Keywords” header is described as found in the title, abstract, and author-supplied keywords fields. The “Subject headings” header indicates information from ERIC via ProQuest. The subsequent rows list search concepts:

• Search Concept 1 is associated with "(Primary or Secondary school students/settings)".
• Search Concept 2 refers to "(games or gamification)".
• Search Concept 3 is about "(Student engagement)".

The table is outlined with thin blue lines.

Example: 

The example below combines all the keywords and concepts into a single search. Lines 1, 2, and 3 combine the keywords for each concept, and Line 4 combines all concepts using the 'AND' operator. Documenting your search in a line-by-line fashion, using concept blocking, can help avoid misplaced parentheses or AND/OR operators, which can lead to significant errors in search results.

The image depicts a table with four rows and two columns. The left column contains numbers from 1 to 4, and the right column contains text strings comprised of keywords and phrases. The first row lists terms related to digital entertainment, including "Video games," "digital games," "gaming," "virtual reality," and "augmented reality," all separated by "OR." The second row lists educational levels, including "Primary school," "first grade," "second grade," "third grade," and "fourth grade." The third row focuses on engagement with "student engagement," "curiosity," "interest," and "excitement." The fourth row connects the previous categories with "1 AND 2 AND 3."

One thing to note is that database search operators are specific to a database AND its platform. Therefore, it is essential to look up the available operators in a database before you begin. All databases provide information about database-specific operators and other syntax, but the location of this information will vary. In PsycINFO (via the ProQuest interface), the database operators can be found by clicking the "Search tips" link. 

The image is a screenshot of the APA PsycInfo Advanced Search interface provided by ProQuest. Below the header, "APA PsycInfo" is prominently displayed, followed by "Advanced Search" in large text, as well as "Search Tips," which is circled in red. The interface features two search fields with drop-down menus for search parameters, labeled with "AND" for conjunction. Options to limit the search by "Peer reviewed" and "Impact statements" are available. A "Publication date" filter offers a default option for "All dates." Additional text fields include "Location," "Classification," and "Test & measure," each with a "Look up" link. At the bottom right, there are "Search" and "Clear form" buttons. 

General Guidance

The general guidance is to search the title field, the abstract field, and, if available, the author-supplied keywords field (which may also be referred to as identifiers or keywords).

In some cases, other fields may be searched, but this is evaluated on a case-by-case basis. For example, suppose your population of interest is children with a specific disease. In that case, you might consider searching for terms such as 'child,' 'youth,' or 'pediatric' in the publication name/source title/journal name field, as well as the above fields.

Again, the execution of this command varies across databases and platforms, as does the availability of specific individuals or combinations of search fields.

Why Title, Abstract, and Keywords?

Full-text vs. bibliographic databases: Databases do not always contain the full text of every article and often only offer bibliographic details rather than the complete content of the articles. The article record will include common metadata elements, such as document title, author, publication name, volume, issue, page numbers, publication date, DOI, author-supplied keywords, and other relevant fields.

On the other hand, some databases offer full-text access to specific journals or certain date ranges within those journals. The extent of this full-text coverage varies by institution, depending on the licenses negotiated, and also differs among databases and platforms. While searching within the full text can yield more precise results, it may also compromise the transparency and reproducibility of your search process. Therefore, focusing on bibliographic metadata fields—such as title, abstract, and keywords—is recommended rather than including the full text in your search to maintain clarity and reproducibility.

Bibliographic databases provide limits and filters that can be applied to narrow results by date, publication type, language, and other attributes. Limits or filters can be used before or after running the search, depending on the database.

The quality and accuracy of these limits and filters depend on the quality of the indexing done by the database provider, which can vary widely. It is not recommended to rely heavily on limits and filters, as these are not consistently applied in databases, and relevant studies may be lost from the search results. While limits and filters can be applied in systematic reviews, clear justification should be given for their use.

Date Limits

You may choose to limit your searches to a specific time frame for justifiable reasons, such as the introduction of new technology, significant socio-political changes, or shifts in paradigms related to your topic. However, avoid setting date limits solely to reduce the number of search results to screen. 

Publication Type Limits

Bibliographic databases often allow searches to be limited by publication type; however, in systematic reviews, these limits are typically avoided to ensure that grey literature is also captured. The exception is when non-relevant sources, such as news or wire feeds, are excluded. The team should decide if applying a publication type limit is appropriate for each database. Ask yourself, "Will the publication type limit remove a substantial amount of clearly non-relevant literature from the search, saving the team significant time in screening without risk of losing relevant research?"

Language Limits

Campbell advises against using language limits in bibliographic database searches, as these can introduce bias by excluding non-English studies, thus impacting review findings. Instead, reviewers are encouraged to include non-English research when possible, leveraging translation tools or team resources to assist with abstract and full-text translations if needed. However, in many cases, there are no funds for translation, nor is there time to do this, so it is relatively common for a filter to be applied to limit results to just the English language. This limitation is acknowledged in the review's methodology.

Geography

Your review may focus on research in a particular geographic region or set of countries that meet specific criteria, such as Latin America and the Caribbean or Low- and Middle-Income Countries (LMICs). To limit your research to these regions, you can use keywords, country names, and other relevant geographic terms to apply a geographic search hedge.

Reasoning

The search should be documented for the following reasons:

• To report it correctly in the review
• To enable the search to be reproduced
• To enable others to assess the thoroughness of the search
• To enable the search to be updated
• To demonstrate compliance with standards

Databases

• List all the databases searched, including the platform and vendor, e.g., PsycINFO, ProQuest
• Note the period searched and the date of the last search for each resource
• Note any restrictions/limits applied, e.g., language
• The entire strategy for each database with total hits and set numbers
• Copy and paste exactly as run
• Report the date of the search – the last date the search was conducted
• If text mining software is used, document the software used, including its version

Other sources

• All searches were conducted by hand-searching
• Experts contacted

Reporting

There are several places where searches can be reported. These include:

• The appendix
• The review abstract 
• The methods section 
• The results section, or supplementary material

• Searching for studies: A guide to information retrieval for Campbell systematic reviews
• Search Methodology for JBI Evidence Synthesis 2.4 

Page content adapted from: Young, S., Premji, Z. & Engelbert, M. Unit 3: Searching the Literature. In J. C. Valentine, J. H. Littell, & S. Young (Eds.), Systematic reviews and meta-analysis: A Campbell Collaboration online course. Open Learning Initiative, 2023.