Evidence Synthesis in the Social Sciences
Planning a Search Strategy
Systematic review searches are designed to comprehensively locate all studies addressing the review question, with a focus on reproducibility and transparency. These searches prioritize sensitivity—maximizing the retrieval of all relevant studies—to achieve a thorough overview of the evidence, unlike traditional searches, which focus on precision (reducing irrelevant results). This approach ensures no relevant studies are overlooked, even if it means retrieving some irrelevant records.
Your initial search should be designed for a primary database (the one most likely to contain the most studies on your topic). It should be a single, multi-line search with all relevant terms, rather than running multiple separate searches. Then, adapted or 'translated' for use in other databases.
Steps to developing a search strategy:
-
Identify a known set of studies:
Locate between 3 and 5 relevant studies that meet many or all of the stated inclusion criteria and use these examples to identify search terms (keywords and their synonyms) and database-specific subject headings. You can also use them to determine whether the concepts of interest are mentioned in the title or abstract of the articles (as opposed to the full text) and what study designs are used in this field. These "known articles" can also eventually be used to test the search strategy that will be developed. -
Harvest terms: Locate terms relevant to each of your search concepts.
- Brainstorming by the research team: Collaborate with your team to generate ideas.
- Disciplinary thesaurus or encyclopedia: Consult specialized reference materials for terms in your field.
- Harvesting terms from known articles: Retrieve terms by scanning article records in databases, focusing on titles, abstracts, author keywords, and subject terms. Collect these terms into a working file (e.g., notepad, word processor, or spreadsheet) and organize them by concept, possibly using a table format for clarity.
- Database entries: Use database searches to identify additional terms related to your concepts.
-
Verify search concepts:
- Verify that your intended search concepts appear in each of your known articles' titles, abstracts, and author-supplied keywords. If several articles lack relevant terms in these fields, you should reconsider including that concept in your search strategy and reduce the number of search concepts to streamline your approach.
-
Locate and apply subject headings:
- Databases often use controlled vocabulary (subject headings) to organize (index) articles on the same topic under one preferred term, regardless of what term the author provided. This helps group related records and index entire articles, not just titles or abstracts. Subject headings can be found by reviewing known articles or browsing a database thesaurus. They should be selected based on their understanding of the topic and the database's definitions. Some databases offer descriptions and examples of terms. However, some concepts may not have a suitable subject heading, especially if they are new or lack the right level of specificity. **Indexing is a time-consuming process; indexers may not be topic experts, and indexing errors can occur. ** Therefore, systematic searches should include both controlled vocabulary and associated keywords. This approach can help mitigate the potential bias that results from using only one of these approaches alone.**
The image above presents a table divided into four rows and three columns demonstrating one way to organize items within your search strategy. The first row contains headers: “Keywords” and “Subject headings.” The “Keywords” header is described as found in the title, abstract, and author-supplied keywords fields. The “Subject headings” header indicates information from a particular resource: ERIC via ProQuest. The subsequent rows list different search concepts:
- Search Concept 1 is associated with "(Primary or Secondary school students/settings)".
- Search Concept 2 refers to "(games or gamification)".
- Search Concept 3 is about "(Student engagement)".
Example:
The example below combines all the keywords and concepts into a single search. Lines 1, 2, and 3 combine the keywords for each concept, and Line 4 combines all concepts using the 'AND' operator. Documenting your search in a line-by-line fashion, using concept blocking, can help avoid misplaced parentheses or AND/OR operators, which can lead to significant errors in search results.
Text version of the image above representing a combined keywords and concepts single search:
- "Video games" OR "digital games" OR gaming OR "virtual reality" OR " augmented reality".
- "Primary school" OR "first grade" OR "second grade" OR "third grade" OR "fourth grade".
- "student engagement" OR curiosity OR interest OR excitement.
- 1 AND 2 AND 3.
Database Operators
One thing to note is that database search operators are specific to a database AND its platform. Therefore, it is essential to look up the available operators in a database before you begin. All databases provide information about database-specific operators and other syntax, but the location of this information will vary. In PsycINFO (via the ProQuest interface), the database operators can be found by clicking the "Search tips" link, which will open a popup window.
The image is a screenshot of the APA PsycInfo Advanced Search interface provided by ProQuest. Below the header, "APA PsycInfo" is the "Advanced Search" link list. "Search Tips" is circled in red at the end. The interface features two search fields with drop-down menus for search parameters, labeled with "AND" for conjunction. Options to limit the search by "Peer reviewed" and "Impact statements" are available. A "Publication date" filter offers a default option for "All dates." Additional text fields include "Location," "Classification," and "Test & measure," each with a "Look up" link. At the bottom right, there are "Search" and "Clear form" buttons.
Searching Specific Fields
The general guidance is to search the title field, the abstract field, and, if available, the author-supplied keywords field (which may also be referred to as identifiers or keywords).
In some cases, other fields may be searched, but this is evaluated on a case-by-case basis. For example, suppose your population of interest is children with a specific disease. In that case, you might consider searching for terms such as 'child,' 'youth,' or 'pediatric' in the publication name/source title/journal name field, as well as the above fields.
Again, the execution of this command varies across databases and platforms, as does the availability of specific individuals or combinations of search fields.
Why Title, Abstract, and Keywords?
Full-text vs. bibliographic databases: Databases do not always contain the full text of every article and often only offer bibliographic details rather than the complete content of the articles. The article record will include common metadata elements, such as document title, author, publication name, volume, issue, page numbers, publication date, DOI, author-supplied keywords, and other relevant fields.
On the other hand, some databases offer full-text access to specific journals or certain date ranges within those journals. The extent of this full-text coverage varies by institution, depending on the licenses negotiated, and also differs among databases and platforms. While searching within the full text can yield more precise results, it may also compromise the transparency and reproducibility of your search process. Therefore, focusing on bibliographic metadata fields—such as title, abstract, and keywords—is recommended rather than including the full text in your search to maintain clarity and reproducibility.
Improve Search Results
AND
If you are getting too many choices and want to narrow your results, use the "AND" operator. In doing this, you ensure that all terms are present in the results list. For example: hormones AND female AND age AND memory
The image shows an APA PsycInfo database search results page with a teal header. The search bar near the top contains the query "hormones AND female AND age AND memory." Below this, the results section displays 10 results sorted by "Relevance." Options for filtering the results, such as "Peer reviewed" under "Limit to" and "Scholarly Journal" under "Source type," are visible on the left.
OR
To broaden your search, use the "OR" operator. For example, "attachment behavior" OR "attachment disorder" will provide a list of results with either term.
The image shows an APA PsycInfo database search results page with a teal header. Below the header, the "Advanced Search" feature includes search boxes where users can input search terms. Two search fields are visible, containing the phrases "attachment behavior" and "attachment disorder," connected by an "OR" operator. Each search box has a dropdown to select where to search for the term. Below the search fields, there are options to add additional rows and limit search results by peer-reviewed articles or impact statements.
NOT
To narrow your results, use "NOT," but be cautious. NOT eliminates all results containing a specific word, so be very careful using the NOT operator because you might unintentionally exclude records.
Example: (testimony OR witness+) NOT expert.
The image shows an APA PsycInfo database search results page with a teal header. A search bar with the query "(testimony OR witness*) NOT expert" in the search field. Below the header, it indicates that 16,303 results are available. On the left is a sidebar, which contains filter options such as sorting methods, source type, and publication date. The main section to the right shows the list of search results, with brief information about each item, such as the title, authors, publication source, and options for accessing full text or more details. Each entry is numbered, and options to "Get it @ ASU" are displayed for the listed works.
Nesting | Parentheses ( )
You can enclose search terms in parentheses to specify the order in which they are interpreted. Information within parentheses is read first, then information outside parentheses is read next. For example:
memory AND (optimis* OR pessimis* AND personality)
The computer first searches for everything that has either optimis* OR pessimis* AND personality. Then, from that pool of results, it will limit it to only those results that also include memory.
The image shows an APA PsycInfo database search results page with a teal header. A search bar with the query “memory AND (optimis* OR pessimis* AND personality)”, resulting in 631 entries. On the left, sorting options include "Relevance," "Peer reviewed," "Source type," and a date range slider from 1890 to 2023. The right side lists search results, each entry featuring a title, author, and publication details. Icons for citation, email options, and scholarly journal indications are present alongside each entry. Three results are visible, discussing topics that relate to memory, personality, optimism, pessimism, and neuropsychological performance.
Truncation*
Truncation means "to shorten" and is a function that allows you to substitute any character (or number of characters) from the point where the truncation symbol is placed. In the ProQuest interface, the truncation symbol is an asterisk (*), which represents up to five characters within or at the end of a word. For example, therap* will locate records with therapy, therapies, therapists, therapeutic, etc.
The image shows an APA PsycInfo database search results page with a teal header. A search bar with the query "(artificial intelligence OR AI) AND therap*". To the right, options for modifying the search, viewing recent searches, or saving the search are available. The main panel displays a total of 1,537 results, sorted by relevance. Each search result is listed with a title, authors, source information, and publication details. Options to view abstracts, full text, and citations are underneath each entry. The left side features filters like peer review options, source type, and publication date with a bar graph for visual guidance.
Plural +
The "+" symbol instructs the database to search for both the plural and singular forms of the words. For example, witness and witnesses:
The image shows an APA PsycInfo database search results page with a teal header. A search bar with the query "(credibility OR reliability) AND (testimony OR witness*) NOT expert." Below the search bar, it shows "755 results." The results are sorted by relevance, with an option to limit to peer-reviewed sources. There are various icons for sorting, sharing, and saving searches.
Quotations
Use quotations " " to search for the exact phrase. For example, "stress scale" will NOT retrieve "scale of perceived stress," or any other variation. It will retrieve only "stress scale."
Tip* When using the ProQuest database platform that hosts both PsycINFO and PsycARTICLES, you can also use curly brackets { } and the search will include spelling (humor v humour) and grammatical variants (tall will retrieve tallest) of the exact term. Example: compare "stress scale" with "{stress scale}"
The image is a screenshot of the APA PsycNet search interface, affiliated with the ASU Library. Below, there is a navigation menu with options like "Search," "Browse," and "APA Thesaurus." The main section shows an "Advanced Search" form and a search bar with the query "stress scale." Two search fields are visible, with drop-down menus allowing the user to specify the search parameters. Options for filtering results are present, including checkboxes for "Peer-Reviewed," "Impact Statement," and "Open Access." A checked box labeled "Test Available" is visible. On the right, there are sections labeled "Search Tools," "Recent Searches," and "Trending," with sub-options like "Author Lookup" and "Journal Lookup."
PRE/X: finds terms with X number of words between them, in the order they are entered in the search.
NEAR/X: finds terms with X number of words between them, regardless of the order in which they occur.
Specifying NEAR/1 can be helpful for finding results that include specific phrases because the two terms are next to – or “one word away” from — each other.
Increasing the proximity search limit to NEAR/2 will return records that have the search terms appearing within two words of each other (i.e., next to each other or separated by a single word between them). For example, searching for child* NEAR/2 trauma* {in Abstract field} AND experience {in Any Field} will retrieve records with useful variations due to both the word order and the truncation operator.
Matching search results for this proximity search would include (but aren’t limited to)
- Experience of childhood trauma
- Traumatic childhood experiences
- Experience of children with traumatic [brain injuries]
- Trauma on child [emotional health]
- Trauma-exposed children
- [Psychological] trauma for children
- Children experience traumatic [stress]
- Children after trauma
- Children’s traumatic experiences
The image shows an advanced search page from the PsycINFO database. Below, there are search fields with drop-down menus for selecting search criteria. The first search field contains the text "child* NEAR/2 trauma*" with the option "in Abstract - ABSTRACT". A second query line displays "experience" with "in Anywhere". There is a button labeled "Add a row" to include additional search criteria. Below the fields, options to limit search results are displayed, with checkboxes for "Peer reviewed" and "Impact statements".
Similar to truncation, a wildcard character replaces any single character inside or at the end of the word. One single "?" will retrieve only one more character. It can be used within or at the end of a word. In PsycINFO (on the ProQuest interface), the wildcard character is a question mark (?) and substitutes zero or one character.
- For example, searching gr?y would retrieve “grey” and “gray.”
**Note** Some databases use the # symbol for Wildcard searching.
Limiters and Filters
Bibliographic databases provide limits and filters that can be applied to narrow results by date, publication type, language, and other attributes. Limits or filters can be used before or after running the search, depending on the database.
The quality and accuracy of these limits and filters depend on the quality of the indexing done by the database provider, which can vary widely. It is not recommended to rely heavily on limits and filters, as these are not consistently applied in databases, and relevant studies may be lost from the search results. While limits and filters can be applied in systematic reviews, clear justification should be given for their use.
Date Limits
You may choose to limit your searches to a specific time frame for justifiable reasons, such as the introduction of new technology, significant socio-political changes, or shifts in paradigms related to your topic. However, avoid setting date limits solely to reduce the number of search results to screen.
Publication Type Limits
Bibliographic databases often allow searches to be limited by publication type; however, in systematic reviews, these limits are typically avoided to ensure that grey literature is also captured. The exception is when non-relevant sources, such as news or wire feeds, are excluded. The team should decide if applying a publication type limit is appropriate for each database. Ask yourself, "Will the publication type limit remove a substantial amount of clearly non-relevant literature from the search, saving the team significant time in screening without risk of losing relevant research?"
Language Limits
Campbell advises against using language limits in bibliographic database searches, as these can introduce bias by excluding non-English studies, thus impacting review findings. Instead, reviewers are encouraged to include non-English research when possible, leveraging translation tools or team resources to assist with abstract and full-text translations if needed. However, in many cases, there are no funds for translation, nor is there time to do this, so it is relatively common for a filter to be applied to limit results to just the English language. This limitation is acknowledged in the review's methodology.
Geography
Your review may focus on research in a particular geographic region or set of countries that meet specific criteria, such as Latin America and the Caribbean or Low- and Middle-Income Countries (LMICs). To limit your research to these regions, you can use keywords, country names, and other relevant geographic terms to apply a geographic search hedge.
Documenting Your Searches
Reasoning
The search should be documented for the following reasons:
- To report it correctly in the review
- To enable the search to be reproduced
- To enable others to assess the thoroughness of the search
- To enable the search to be updated
- To demonstrate compliance with standards
Databases
- List all the databases searched, including the platform and vendor, e.g., PsycINFO, ProQuest
- Note the period searched and the date of the last search for each resource
- Note any restrictions/limits applied, e.g., language
- The entire strategy for each database with total hits and set numbers
- Copy and paste exactly as run
- Report the date of the search – the last date the search was conducted
- If text mining software is used, document the software used, including its version
Other sources
- All searches were conducted by hand-searching
- Experts contacted
Reporting
There are several places where searches can be reported. These include:
- The appendix
- The review abstract
- The methods section
- The results section, or supplementary material