Skip to Main Content

Systematic Reviews: The steps of a systematic review

An introduction to systematic reviews and the steps involved in undertaking a systematic review

Steps

The eight steps of a systematic review

  1. Establish a team
  2. Formulate the research question
  3. Develop a protocol and register it
  4. Formulate the search strategy
  5. Screen the results
  6. Extract the data
  7. Assess the risk of bias
  8. Analyse and Report

This guide will cover some steps in detail and signpost resources that will help you with others.

Step 1. Establish a team

At least two people and preferably three+

  • Researcher/s
  • Screener/s
  • Statistician/s
  • (Librarian/s)

Step 2. Formulate the research question

Developing a protocol (plan) for a systematic review underpins many of the individual processes that need to be undertaken within the review process.

There are several available frameworks for developing questions, including:

  • PICO  = Population, Intervention, Comparison, Outcome (Intervention questions)
  • PECO = Population, Exposure, Comparison, Outcome (Risk questions)
  • PIRT   = Population, Index Test, Reference Test, Target Condition (Diagnostic questions)
  • SPIDER = Sample, Phenomenon of Interest, Design, Evaluation, Research Type (Experiential/qualitative questions)

i) Define inclusion and exclusion criteria  - used to further focus the research question and select studies and helps to develop the search strategy

ii) Has the topic already been reviewed?

iii) Is your topic (the FINER criteria):

  • Feasible - i.e. is the question one that the team is capable of addressing using the available evidence

  • Interesting - authors need enough commitment to see the work through

  • Novel - is there a genuine gap in knowledge

  • Ethical - implication of results

  • Relevant - translation of finds to inform decisions

Further reading:

Richardson, W. S., Wilson, M. C., Nishikawa, J., & Hayward, R. S. (1995). The well-built clinical question: a key to evidence-based decisions [Editorial]. ACP Journal Club, 123(3), A12-13. https://www.acpjournals.org/doi/10.7326/ACPJC-1995-123-3-A12

Counsell, C. (1997, Sep). Formulating questions and locating primary studies for inclusion in systematic reviews. Annals of Internal Medicine, 127(5), 380-387. https://doi.org/10.7326/0003-4819-127-5-199709010-00008

Samson, D., & Schoelles, K. M. (2012, Jun). Chapter 2: Medical Tests Guidance (2) Developing the Topic and Structuring Systematic Reviews of Medical Tests: Utility of PICOTS, Analytic Frameworks, Decision Trees, and Other Frameworks. Journal of General Internal Medicine, 27, S11-S19. https://doi.org/10.1007/s11606-012-2007-7

Booth, A., Noyes, J., Flemming, K., Moore, G., Tuncalp, O., & Shakibazadeh, E. (2019, Jan). Formulating questions to explore complex interventions within qualitative evidence synthesis. BMJ Glob Health, 4, Article e001107. https://doi.org/10.1136/bmjgh-2018-001107

Step 3. Develop and register your protocol

Once you have formulated your question, you will need to start developing a protocol to guide the conduct of your review. This will cover inclusion/exclusion criteria, screening methods, risk of bias and data analysis.

It is good practice to prospectively register your protocol and, in many cases, is a requirement for future publication of the review.

There are a number of places where you can register your protocol, for example:

Further reading:

Stewart, L., Moher, D., & Shekelle, P. (2012). Why prospective registration of systematic reviews makes sense [Editorial]. Systematic Reviews, 1(1), Article 7. https://doi.org/10.1186/2046-4053-1-7

Booth, A., Clarke, M., Dooley, G., Ghersi, D., Moher, D., Petticrew, M., & Stewart, L. (2012). The nuts and bolts of PROSPERO: An international prospective register of systematic reviews [Article]. Systematic Reviews, 1(1), Article 2. https://doi.org/10.1186/2046-4053-1-2

Step 4. Formulate a search strategy

The search should be thorough, objective, reproducible, include a range of sources. The search should balance thoroughness of the search with efficiency in use of time and funds, and it should minimise publication and language bias

Peer review the search strategy to ensure that the research question is translated properly, that Boolean and Proximity operators are appropriate, that Subject Headings and Textword search terms are appropriate, that spelling is correct, that limits and filters have been used appropriately

PRESS (Peer Review of Electronic Search Strategies 2015 statement) McGowan, J., Sampson, M., Salzwedel, D. M., Cogo, E., Foerster, V., & Lefebvre, C. (2016). PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement [Article]. Journal of Clinical Epidemiology, 75, 40-46. https://doi.org/10.1016/j.jclinepi.2016.01.021 evidence/press

Many systematic reviews need a search for grey literature

Seek unpublished literature by hand-searching, contacting experts, looking for conference proceedings and checking clinicaltrials.gov

Check for publication bias i.e. there is a tendency to submit or accept studies for publication based on the direction of strength of the study findings.

Once you have defined your question you can start the searching process. To see appropriate databases that you can use to start searching, consult your Subject Guide.

A systematic search strategy

All quotes below are from the following article: Bramer, W., de Jonge, G. B., Rethlefsen, M. L., Mast, F., & Kleijnen, J. (2018). A systematic approach to searching: an efficient and complete method to develop literature searches. Journal of the Medical Library Association, 106(4), 531-541. https://doi.org/10.5195/jmla.2018.283

i. Determine a clear and focused question [see Step 2]

ii. Describe the articles that can answer the question

  • "A good starting point for a search is hypothesizing what the research that can answer the question would look like."

iii. Decide which key concepts address the different elements of the question

  • "Key concepts are the topics or components that the desired articles should address, such as diseases or conditions, actions, substances, settings, domains (e.g., therapy, diagnosis, etiology), or study types. Key concepts from the research question can be grouped to create elements in the search strategy."

iv. Decide which elements should be used for the best results

  • "Not all elements of a research question should necessarily be used in the search strategy. Some elements are less important than others or may unnecessarily complicate or restrict a search strategy."

v. Choose an appropriate database and interface to start with

  • "Important factors for choosing databases to use are the coverage and the presence of a thesaurus" [controlled vocabulary].

vi. Document the search process in a text document

  • Design and create "the complete search strategies in a [text] document, instead of directly in the database itself, to register the steps taken and to make searches accountable and reproducible."

vii. Identify appropriate index terms in the thesaurus of the first database

  • "Searches should start by identifying appropriate thesaurus [subject heading] terms for the desired elements"

viii. Identify synonyms in the thesaurus

  • "Most thesauri offer a list of synonyms on their term details page"

ix. Add variations in search terms

  • "e.g., truncation, spelling differences, abbreviations, opposites [antonyms]"

x. Use database-appropriate syntax, with parentheses, Boolean operators, and field codes

  • "Different interfaces require different syntaxes, the special set of rules and symbols unique to each database that define how a correctly constructed search operates."

xi. Optimize the search

  • "The most important question when performing a systematic search is whether all (or most) potentially relevant articles have been retrieved by the search strategy. This is also the most difficult question to answer, since it is unknown which and how many articles are relevant. It is, therefore, wise first to broaden the initial search strategy, making the search more sensitive,"

xii. Evaluate the initial results

  • "The results should now contain relevant references. If the interface allows relevance ranking, use that in the evaluation. If you know some relevant references that should be included in the research, search for those references specifically' i.e. create a 'must-be-present, or 'gold-standard' list of references that your search must find

xiii. Check for errors

  • "Errors might not be easily detected. Sometimes clues can be found in the number of results, either when the number of results is much higher or lower than expected or when many retrieved references are not relevant. However, the number expected is often unknown, and very sensitive search strategies will always retrieve many irrelevant articles. Each query should, therefore, be checked for errors."

xiv. Translate to other databases

  • "To retrieve as many relevant references as possible, one has to search multiple databases. Translation of complex and exhaustive queries between different databases can be very time consuming and cumbersome."

xv. Test and reiterate

  • "Ideally, exhaustive search strategies should retrieve all references that are covered in a specific database. If additional papers have been identified through other non-database methods...results that were not identified by the database searches should be examined. If these results were available in the databases but not located by the search strategy, the search strategy should be adapted to try to retrieve these results"
Searching for Systematic Reviews

The first step in searching for studies is to locate previously conducted systematic reviews in your area of interest. This has three main purposes:

  1. To verify that your question hasn't already been answered
  2. To verify that there are no other review protocols registered with researchers already looking at the same question
  3. To identify related systematic reviews that will need to be accessed so that you can review the reference lists for relevant primary studies.

Useful databases for identifying systematic reviews:

Health Sciences

Biological and Environmental Studies
 
 
 
 

Tips for Database Searching for Systematic Reviews

Some databases have limits that allow you to filter your search for Systematic Reviews, for example:

  •  PsycINFO and PsycArticles (under Methodology)
  •  CINAHL. (under Publication Type) 
  • Scopus and Web of Science (filter by article type (Review Article) and then search within the results for "systematic review")
 
Searching for pre-prints
There are a number of pre-print servers that can be searched. A fairly comprehensive list is available at https://asapbio.org/preprint-servers.
In Health, MedRxiv was launched in 2019 and has rapidly established itself as the most prominent in the field.

If you are looking for studies in institutional repositories, OpenDOAR from the UK's JISC is perhaps the most well known search engine.

 

Searching for primary studies

The next part of searching for studies is to find other 'primary studies' (pieces of original research not included in those systematic reviews e.g. trials, diagnostic accuracy studies). The identification of all studies relevant to your question is made up of two broad stages:

  • Development of a structured search strategy that is run across multiple databases
  • An iterative process for finding further studies through hand search of relevant publications, reviewing reference lists, citation searching, contact with authors and location of on-going studies.

The guides below provide an introduction to locating studies for different types of systematic reviews (therapeutic, diagnostic, qualitative...). You can also contact your subject librarian for help and advice.

Cochrane Handbook. Chapter 4 Searching for and selecting studies

Relevo, R. (2012, Jun). Chapter 4: Effective Search Strategies for Systematic Reviews of Medical Tests. Journal of General Internal Medicine, 27, S28-S32. https://doi.org/10.1007/s11606-011-1873-8

 
Searching for qualitative studies

Qualitative ystematic reviews have become much more common . Many databases allow you to filter or limit your search to qualitative studies.

Finding qualitative articles (U Washington)

 
Bulk export from library databases

Systematic review searches may involve you exporting from one to many thousands of results from databases to a reference management tool like EndNote, Mendeley or Zotero. This document shows how the big database companies permit this.

 
Searching for grey literature

There is a separate guide on grey literature available. This separate guide includes information on places to search for grey literature.

Briefly, grey literature is generally considered to be material published or made available by organisations or individuals, not through commercial publishers. This can include clinical study reports, conference abstracts, government documents, organisational reports, personal communications, pre-prints, regulatory documents, study registrations, theses, blogs and social media posts.

Searching for grey literature can be problematic as it isn't collected, organised or stored in a consistent way. You will need to be flexible in your approach, depending on the type of grey literature you need – this will differ from review to review.

Reporting your grey literature search is not as straightforward as reporting a search on a bibliographic database. You should aim to record the following information:

  • Name of web-site or resource used
  • URL (if appropriate)
  • Date of search
  • Keywords  used for searching or details of how you browsed
 
Further reading:

Aromataris, E., & Riitano, D. (2014). Constructing a search strategy and searching for evidence. A guide to the literature search for a systematic review. The American journal of nursing, 114(5), 49-56. https://doi.org/10.1097/01.NAJ.0000446779.99522.f6

Lefebvre, C., Glanville, J., Wieland, L. S., Coles, B., & Weightman, A. L. (2013). Methodological developments in searching for studies for systematic reviews: past, present and future? Systematic reviews, 2, 78, Article 78. https://doi.org/10.1186/2046-4053-2-78

Booth, A. (2010). How much searching is enough? Comprehensive versus optimal retrieval for technology assessments [Review]. International Journal of Technology Assessment in Health Care, 26(4), 431-435. https://doi.org/10.1017/S0266462310000966

Toth, B., Gray, J. A., & Brice, A. (2005). The number needed to read-a new measure of journal value [Editorial]. Health information and libraries journal, 22(2), 81-82. https://doi.org/10.1111/j.1471-1842.2005.00568.x

 

A search strategy checklist:

1. Translation of the research question
  • Does the search strategy match the research question?
  • Are the search concepts clear?
  • Are there too many or too few PICO elements included?
  • Are the search concepts too narrow or too broad?
  • Does the search retrieve too many or too few records? (Please show number of hits per line.)
  • Are unconventional or complex strategies explained?
2. Boolean and Proximity Operators (these vary based on search service)
  • Are Boolean or Proximity operators used correctly?
  • Is the use of nesting with brackets appropriate and effective for the search?
  • If NOT is used, is this likely to result in any unintended exclusions?
  • Could precision be improved by using proximity operators (e.g., adjacent, near, within) or phrase-searching instead of AND?
  • Is the width of proximity operators suitable (e.g. might adj5 pick up more variants than adj2)?
3. Subject Headings (database-specific)
  • Are the subject headings relevant?
  • Are any relevant subject headings missing e.g. previous index terms?
  • Are any subject headings too broad or too narrow?
  • Are subject headings exploded where necessary and vice versa?
  • Are major headings (“starring” or restrict to focus) used? If so, is there adequate justification?
  • Are subheadings missing?
  • Are subheadings attached to subject headings? (Floating subheadings may be preferred.)
  • Are floating subheadings relevant and used appropriately?
  • Are both subject headings and terms in free text (see below) used for each concept
4. Text word searching (free text, natural language)
  • Does the search include all spelling variants in free text (e.g. UK versus US spelling)?
  • Does the search include all synonyms or antonyms (i.e. opposites)?
  • Does the search capture relevant truncation (i.e. is truncated at the correct place)?
  • Is the truncation too broad or too narrow?
  • Are acronyms or abbreviations used appropriately? Do they capture irrelevant material? Are the full terms also included?
  • Are the keywords specific enough or too broad? Are too many or too few keywords used? Are stop words used?
  • Have the appropriate fields been searched e.g. is the choice of the text word fields (.tw.) or all fields (.af.) appropriate? Are there any other fields to be included or excluded (database-specific)?
  • Should any long strings be broken into several shorter search statements?
5. Spelling, syntax and line numbers
  • Are there any spelling errors?
  • Are there any errors in system syntax e.g. the use of a truncation symbol from a different search interface?
  • Are there incorrect line combinations or orphan lines (i.e. lines that are not referred to in the final summation that could indicate an error in an AND or OR statement)? 
6. Limits and filters
  • Are all limits and filters used appropriately and are they relevant given the research question?
  • Are all limits and filters used appropriately and are they relevant for the database?
  • Are any potentially helpful limits or filters missing? Are the limits or filters too broad or too narrow? Can any limits or filters be added or taken away?
  • Are sources cited for the filters used?
Methodological search filters

Methodological search filters (aka Hedges or optimised search strategies) are designed to selectively retrieve different types of evidence from specific bibliographic databases, and usually consist of a tested combination of subject headings and natural language terms (keywords). They are designed to be used with a more specific topic of interest, and can be used to retrieve studies with a specific study design (e.g. RCTs), studies capable of answering a specific type of question (e.g. prognosis) or focused on a narrow subject area e.g. for systematic reviews of therapeutic interventions it is customary to restrict the search to randomised controlled trials.) 

They are experimentally derived, based on the use of a 'gold standard' set of citations with known relevance to the focus of the filter; they provide evidence of their reliability with performance measures, calculated on the filter's ability to retrieve (or not retrieve) both relevant and irrelevant citations from the gold standard set.

The guides below have information on methodological search filters. Please contact your subject librarian to verify that you are using the most appropriate filters for your topic.

 
Performance measures for filters
 
 
 
 
 
 

Sensitivity

  • How effective a search strategy is at finding relevant citations - the proportion of the the relevant citations correctly retrieved

Specificity

  • How effective a search strategy is at not retrieving irrelevant citations - the proportion of irrelevant records correctly excluded

Precision

  • The number of relevant citations retreived as a proportion of all the citations retrieved.
  • This is not the same as sensitivity or specificity e.g. retrieving only 5 citations from a database of millions may be 100% precise, but have very low sensitivity

Searching is always a trade-off between sensitivity and specificity i.e. highly sensitive searches usually result in low specificity and vice-versa. In everyday searching, we can only approsimate high sensitivity or high specificity.

 

Critical appraisal tools

Retrieving material is easy. Reading material critically is not as easy. The following resources may help.

 
 
 
General tips on literature searching
 
Text mining tools

Step 5. Screen the results

Include or exclude studies based on the protocol's pre-specifice criteria

Use two or more members of the review team

There are a number of strategies to select studies e.g. Titles alone, or Titles and Abstracts

Document the decisions made about each study

 
Screening tools
  • Rayyan
  • Covidence
  • DistillerSR

Bond University developed this free tool to speed up steps in conducting a systematic review

 

Step 6. Extract the data

The assessment of included studies and extraction of data should be done by at least two people.

Data collection forms should be designed carefully e.g.

example of a data extraction form

Step 7. Assess the risk of bias

It is important to examine and report on the risks of bias in the review.

Useful tools

There are several sections in the Cochrane Handbook for Systematic Reviews of Interventions that look at assessment of bias

There are some other resources that may help with assessing bias:

Step 8. Analyse and Report

The review should be reported as per the PRISMA statement  and PRISMA checklist e.g. "Our systematic review was reported in accordance with the 2020 PRISMA statement. Our Review protocol was registered with PROSPERO in February 2021 (registration number CRD42017057687)"

There are guidelines on how to report your research methods and findings.

 

See below for the EQUATOR Network Introduction to Reporting Guidelines

 

Other guidance:

License