systematic literature review step by step

Systematic Review

• Library Help
• What is a Systematic Review (SR)?

Steps of a Systematic Review

• Framing a Research Question
• Developing a Search Strategy
• Searching the Literature
• Managing the Process
• Meta-analysis
• Publishing your Systematic Review

Forms and templates

Image: David Parmenter's Shop

• PICO Template
• Inclusion/Exclusion Criteria
• Database Search Log
• Review Matrix
• Cochrane Tool for Assessing Risk of Bias in Included Studies

   • PRISMA Flow Diagram  - Record the numbers of retrieved references and included/excluded studies. You can use the Create Flow Diagram tool to automate the process.

   •  PRISMA Checklist - Checklist of items to include when reporting a systematic review or meta-analysis

PRISMA 2020 and PRISMA-S: Common Questions on Tracking Records and the Flow Diagram

• PROSPERO Template
• Manuscript Template
• Steps of SR (text)
• Steps of SR (visual)
• Steps of SR (PIECES)

Adapted from  A Guide to Conducting Systematic Reviews: Steps in a Systematic Review by Cornell University Library

Source: Cochrane Consumers and Communications  (infographics are free to use and licensed under Creative Commons )

Check the following visual resources titled " What Are Systematic Reviews?"

• Video  with closed captions available
• Animated Storyboard
• << Previous: What is a Systematic Review (SR)?
• Next: Framing a Research Question >>
• Last Updated: May 8, 2024 1:44 PM
• URL: https://lib.guides.umd.edu/SR

Reference management. Clean and simple.

How to write a systematic literature review [9 steps]

What is a systematic literature review?

Where are systematic literature reviews used, what types of systematic literature reviews are there, how to write a systematic literature review, 1. decide on your team, 2. formulate your question, 3. plan your research protocol, 4. search for the literature, 5. screen the literature, 6. assess the quality of the studies, 7. extract the data, 8. analyze the results, 9. interpret and present the results, registering your systematic literature review, frequently asked questions about writing a systematic literature review, related articles.

A systematic literature review is a summary, analysis, and evaluation of all the existing research on a well-formulated and specific question.

Put simply, a systematic review is a study of studies that is popular in medical and healthcare research. In this guide, we will cover:

• the definition of a systematic literature review
• the purpose of a systematic literature review
• the different types of systematic reviews
• how to write a systematic literature review

➡️ Visit our guide to the best research databases for medicine and health to find resources for your systematic review.

Systematic literature reviews can be utilized in various contexts, but they’re often relied on in clinical or healthcare settings.

Medical professionals read systematic literature reviews to stay up-to-date in their field, and granting agencies sometimes need them to make sure there’s justification for further research in an area. They can even be used as the starting point for developing clinical practice guidelines.

A classic systematic literature review can take different approaches:

• Effectiveness reviews assess the extent to which a medical intervention or therapy achieves its intended effect. They’re the most common type of systematic literature review.
• Diagnostic test accuracy reviews produce a summary of diagnostic test performance so that their accuracy can be determined before use by healthcare professionals.
• Experiential (qualitative) reviews analyze human experiences in a cultural or social context. They can be used to assess the effectiveness of an intervention from a person-centric perspective.
• Costs/economics evaluation reviews look at the cost implications of an intervention or procedure, to assess the resources needed to implement it.
• Etiology/risk reviews usually try to determine to what degree a relationship exists between an exposure and a health outcome. This can be used to better inform healthcare planning and resource allocation.
• Psychometric reviews assess the quality of health measurement tools so that the best instrument can be selected for use.
• Prevalence/incidence reviews measure both the proportion of a population who have a disease, and how often the disease occurs.
• Prognostic reviews examine the course of a disease and its potential outcomes.
• Expert opinion/policy reviews are based around expert narrative or policy. They’re often used to complement, or in the absence of, quantitative data.
• Methodology systematic reviews can be carried out to analyze any methodological issues in the design, conduct, or review of research studies.

Writing a systematic literature review can feel like an overwhelming undertaking. After all, they can often take 6 to 18 months to complete. Below we’ve prepared a step-by-step guide on how to write a systematic literature review.

• Decide on your team.
• Formulate your question.
• Plan your research protocol.
• Search for the literature.
• Screen the literature.
• Assess the quality of the studies.
• Extract the data.
• Analyze the results.
• Interpret and present the results.

When carrying out a systematic literature review, you should employ multiple reviewers in order to minimize bias and strengthen analysis. A minimum of two is a good rule of thumb, with a third to serve as a tiebreaker if needed.

You may also need to team up with a librarian to help with the search, literature screeners, a statistician to analyze the data, and the relevant subject experts.

Define your answerable question. Then ask yourself, “has someone written a systematic literature review on my question already?” If so, yours may not be needed. A librarian can help you answer this.

You should formulate a “well-built clinical question.” This is the process of generating a good search question. To do this, run through PICO:

• Patient or Population or Problem/Disease : who or what is the question about? Are there factors about them (e.g. age, race) that could be relevant to the question you’re trying to answer?
• Intervention : which main intervention or treatment are you considering for assessment?
• Comparison(s) or Control : is there an alternative intervention or treatment you’re considering? Your systematic literature review doesn’t have to contain a comparison, but you’ll want to stipulate at this stage, either way.
• Outcome(s) : what are you trying to measure or achieve? What’s the wider goal for the work you’ll be doing?

Now you need a detailed strategy for how you’re going to search for and evaluate the studies relating to your question.

The protocol for your systematic literature review should include:

• the objectives of your project
• the specific methods and processes that you’ll use
• the eligibility criteria of the individual studies
• how you plan to extract data from individual studies
• which analyses you’re going to carry out

For a full guide on how to systematically develop your protocol, take a look at the PRISMA checklist . PRISMA has been designed primarily to improve the reporting of systematic literature reviews and meta-analyses.

When writing a systematic literature review, your goal is to find all of the relevant studies relating to your question, so you need to search thoroughly .

This is where your librarian will come in handy again. They should be able to help you formulate a detailed search strategy, and point you to all of the best databases for your topic.

➡️ Read more on on how to efficiently search research databases .

The places to consider in your search are electronic scientific databases (the most popular are PubMed , MEDLINE , and Embase ), controlled clinical trial registers, non-English literature, raw data from published trials, references listed in primary sources, and unpublished sources known to experts in the field.

➡️ Take a look at our list of the top academic research databases .

Tip: Don’t miss out on “gray literature.” You’ll improve the reliability of your findings by including it.

Don’t miss out on “gray literature” sources: those sources outside of the usual academic publishing environment. They include:

• non-peer-reviewed journals
• pharmaceutical industry files
• conference proceedings
• pharmaceutical company websites
• internal reports

Gray literature sources are more likely to contain negative conclusions, so you’ll improve the reliability of your findings by including it. You should document details such as:

• The databases you search and which years they cover
• The dates you first run the searches, and when they’re updated
• Which strategies you use, including search terms
• The numbers of results obtained

➡️ Read more about gray literature .

This should be performed by your two reviewers, using the criteria documented in your research protocol. The screening is done in two phases:

• Pre-screening of all titles and abstracts, and selecting those appropriate
• Screening of the full-text articles of the selected studies

Make sure reviewers keep a log of which studies they exclude, with reasons why.

➡️ Visit our guide on what is an abstract?

Your reviewers should evaluate the methodological quality of your chosen full-text articles. Make an assessment checklist that closely aligns with your research protocol, including a consistent scoring system, calculations of the quality of each study, and sensitivity analysis.

The kinds of questions you'll come up with are:

• Were the participants really randomly allocated to their groups?
• Were the groups similar in terms of prognostic factors?
• Could the conclusions of the study have been influenced by bias?

Every step of the data extraction must be documented for transparency and replicability. Create a data extraction form and set your reviewers to work extracting data from the qualified studies.

Here’s a free detailed template for recording data extraction, from Dalhousie University. It should be adapted to your specific question.

Establish a standard measure of outcome which can be applied to each study on the basis of its effect size.

Measures of outcome for studies with:

• Binary outcomes (e.g. cured/not cured) are odds ratio and risk ratio
• Continuous outcomes (e.g. blood pressure) are means, difference in means, and standardized difference in means
• Survival or time-to-event data are hazard ratios

Design a table and populate it with your data results. Draw this out into a forest plot , which provides a simple visual representation of variation between the studies.

Then analyze the data for issues. These can include heterogeneity, which is when studies’ lines within the forest plot don’t overlap with any other studies. Again, record any excluded studies here for reference.

Consider different factors when interpreting your results. These include limitations, strength of evidence, biases, applicability, economic effects, and implications for future practice or research.

Apply appropriate grading of your evidence and consider the strength of your recommendations.

It’s best to formulate a detailed plan for how you’ll present your systematic review results. Take a look at these guidelines for interpreting results from the Cochrane Institute.

Before writing your systematic literature review, you can register it with OSF for additional guidance along the way. You could also register your completed work with PROSPERO .

Systematic literature reviews are often found in clinical or healthcare settings. Medical professionals read systematic literature reviews to stay up-to-date in their field and granting agencies sometimes need them to make sure there’s justification for further research in an area.

The first stage in carrying out a systematic literature review is to put together your team. You should employ multiple reviewers in order to minimize bias and strengthen analysis. A minimum of two is a good rule of thumb, with a third to serve as a tiebreaker if needed.

Your systematic review should include the following details:

A literature review simply provides a summary of the literature available on a topic. A systematic review, on the other hand, is more than just a summary. It also includes an analysis and evaluation of existing research. Put simply, it's a study of studies.

The final stage of conducting a systematic literature review is interpreting and presenting the results. It’s best to formulate a detailed plan for how you’ll present your systematic review results, guidelines can be found for example from the Cochrane institute .

Jump to navigation

Cochrane Cochrane Interactive Learning

Cochrane interactive learning, module 1: introduction to conducting systematic reviews, about this module.

Part of the Cochrane Interactive Learning course on Conducting an Intervention Review, this module introduces you to what systematic reviews are and why they are useful. This module describes the various types and preferred format of review questions, and outlines the process of conducting systematic reviews.

 45-60 minutes

What you can expect to learn (learning outcomes).

This module will teach you to:

• Recognize features of systematic reviews as a research design
• Recognize the importance of using rigorous methods to conduct a systematic review
• Identify the types of review questions
• Identify the elements of a well-defined review question
• Understand the steps in a systematic review

Authors, contributors, and how to cite this module

Module 1 has been written and compiled by Dario Sambunjak, Miranda Cumpston and Chris Watts,  Cochrane Central Executive Team .

A full list of acknowledgements, including our expert advisors from across Cochrane, is available at the end of each module page. 

This module should be cited as: Sambunjak D, Cumpston M, Watts C. Module 1: Introduction to conducting systematic reviews. In: Cochrane Interactive Learning: Conducting an intervention review. Cochrane, 2017. Available from https://training.cochrane.org/interactivelearning/module-1-introduction-conducting-systematic-reviews .

Update and feedback

The module was last updated on September 2022.

We're pleased to hear your thoughts. If you have any questions, comments or feedback about the content of this module, please contact us .

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, automatically generate references for free.

• Knowledge Base
• Methodology
• Systematic Review | Definition, Examples & Guide

Systematic Review | Definition, Examples & Guide

Published on 15 June 2022 by Shaun Turney . Revised on 17 October 2022.

A systematic review is a type of review that uses repeatable methods to find, select, and synthesise all available evidence. It answers a clearly formulated research question and explicitly states the methods used to arrive at the answer.

They answered the question ‘What is the effectiveness of probiotics in reducing eczema symptoms and improving quality of life in patients with eczema?’

In this context, a probiotic is a health product that contains live microorganisms and is taken by mouth. Eczema is a common skin condition that causes red, itchy skin.

Table of contents

What is a systematic review, systematic review vs meta-analysis, systematic review vs literature review, systematic review vs scoping review, when to conduct a systematic review, pros and cons of systematic reviews, step-by-step example of a systematic review, frequently asked questions about systematic reviews.

A review is an overview of the research that’s already been completed on a topic.

What makes a systematic review different from other types of reviews is that the research methods are designed to reduce research bias . The methods are repeatable , and the approach is formal and systematic:

• Formulate a research question
• Develop a protocol
• Search for all relevant studies
• Apply the selection criteria
• Extract the data
• Synthesise the data
• Write and publish a report

Although multiple sets of guidelines exist, the Cochrane Handbook for Systematic Reviews is among the most widely used. It provides detailed guidelines on how to complete each step of the systematic review process.

Systematic reviews are most commonly used in medical and public health research, but they can also be found in other disciplines.

Systematic reviews typically answer their research question by synthesising all available evidence and evaluating the quality of the evidence. Synthesising means bringing together different information to tell a single, cohesive story. The synthesis can be narrative ( qualitative ), quantitative , or both.

Prevent plagiarism, run a free check.

Systematic reviews often quantitatively synthesise the evidence using a meta-analysis . A meta-analysis is a statistical analysis, not a type of review.

A meta-analysis is a technique to synthesise results from multiple studies. It’s a statistical analysis that combines the results of two or more studies, usually to estimate an effect size .

A literature review is a type of review that uses a less systematic and formal approach than a systematic review. Typically, an expert in a topic will qualitatively summarise and evaluate previous work, without using a formal, explicit method.

Although literature reviews are often less time-consuming and can be insightful or helpful, they have a higher risk of bias and are less transparent than systematic reviews.

Similar to a systematic review, a scoping review is a type of review that tries to minimise bias by using transparent and repeatable methods.

However, a scoping review isn’t a type of systematic review. The most important difference is the goal: rather than answering a specific question, a scoping review explores a topic. The researcher tries to identify the main concepts, theories, and evidence, as well as gaps in the current research.

Sometimes scoping reviews are an exploratory preparation step for a systematic review, and sometimes they are a standalone project.

A systematic review is a good choice of review if you want to answer a question about the effectiveness of an intervention , such as a medical treatment.

To conduct a systematic review, you’ll need the following:

• A precise question , usually about the effectiveness of an intervention. The question needs to be about a topic that’s previously been studied by multiple researchers. If there’s no previous research, there’s nothing to review.
• If you’re doing a systematic review on your own (e.g., for a research paper or thesis), you should take appropriate measures to ensure the validity and reliability of your research.
• Access to databases and journal archives. Often, your educational institution provides you with access.
• Time. A professional systematic review is a time-consuming process: it will take the lead author about six months of full-time work. If you’re a student, you should narrow the scope of your systematic review and stick to a tight schedule.
• Bibliographic, word-processing, spreadsheet, and statistical software . For example, you could use EndNote, Microsoft Word, Excel, and SPSS.

A systematic review has many pros .

• They minimise research b ias by considering all available evidence and evaluating each study for bias.
• Their methods are transparent , so they can be scrutinised by others.
• They’re thorough : they summarise all available evidence.
• They can be replicated and updated by others.

Systematic reviews also have a few cons .

• They’re time-consuming .
• They’re narrow in scope : they only answer the precise research question.

The 7 steps for conducting a systematic review are explained with an example.

Step 1: Formulate a research question

Formulating the research question is probably the most important step of a systematic review. A clear research question will:

• Allow you to more effectively communicate your research to other researchers and practitioners
• Guide your decisions as you plan and conduct your systematic review

A good research question for a systematic review has four components, which you can remember with the acronym PICO :

• Population(s) or problem(s)
• Intervention(s)
• Comparison(s)

You can rearrange these four components to write your research question:

• What is the effectiveness of I versus C for O in P ?

Sometimes, you may want to include a fourth component, the type of study design . In this case, the acronym is PICOT .

• Type of study design(s)
• The population of patients with eczema
• The intervention of probiotics
• In comparison to no treatment, placebo , or non-probiotic treatment
• The outcome of changes in participant-, parent-, and doctor-rated symptoms of eczema and quality of life
• Randomised control trials, a type of study design

Their research question was:

• What is the effectiveness of probiotics versus no treatment, a placebo, or a non-probiotic treatment for reducing eczema symptoms and improving quality of life in patients with eczema?

Step 2: Develop a protocol

A protocol is a document that contains your research plan for the systematic review. This is an important step because having a plan allows you to work more efficiently and reduces bias.

Your protocol should include the following components:

• Background information : Provide the context of the research question, including why it’s important.
• Research objective(s) : Rephrase your research question as an objective.
• Selection criteria: State how you’ll decide which studies to include or exclude from your review.
• Search strategy: Discuss your plan for finding studies.
• Analysis: Explain what information you’ll collect from the studies and how you’ll synthesise the data.

If you’re a professional seeking to publish your review, it’s a good idea to bring together an advisory committee . This is a group of about six people who have experience in the topic you’re researching. They can help you make decisions about your protocol.

It’s highly recommended to register your protocol. Registering your protocol means submitting it to a database such as PROSPERO or ClinicalTrials.gov .

Step 3: Search for all relevant studies

Searching for relevant studies is the most time-consuming step of a systematic review.

To reduce bias, it’s important to search for relevant studies very thoroughly. Your strategy will depend on your field and your research question, but sources generally fall into these four categories:

• Databases: Search multiple databases of peer-reviewed literature, such as PubMed or Scopus . Think carefully about how to phrase your search terms and include multiple synonyms of each word. Use Boolean operators if relevant.
• Handsearching: In addition to searching the primary sources using databases, you’ll also need to search manually. One strategy is to scan relevant journals or conference proceedings. Another strategy is to scan the reference lists of relevant studies.
• Grey literature: Grey literature includes documents produced by governments, universities, and other institutions that aren’t published by traditional publishers. Graduate student theses are an important type of grey literature, which you can search using the Networked Digital Library of Theses and Dissertations (NDLTD) . In medicine, clinical trial registries are another important type of grey literature.
• Experts: Contact experts in the field to ask if they have unpublished studies that should be included in your review.

At this stage of your review, you won’t read the articles yet. Simply save any potentially relevant citations using bibliographic software, such as Scribbr’s APA or MLA Generator .

• Databases: EMBASE, PsycINFO, AMED, LILACS, and ISI Web of Science
• Handsearch: Conference proceedings and reference lists of articles
• Grey literature: The Cochrane Library, the metaRegister of Controlled Trials, and the Ongoing Skin Trials Register
• Experts: Authors of unpublished registered trials, pharmaceutical companies, and manufacturers of probiotics

Step 4: Apply the selection criteria

Applying the selection criteria is a three-person job. Two of you will independently read the studies and decide which to include in your review based on the selection criteria you established in your protocol . The third person’s job is to break any ties.

To increase inter-rater reliability , ensure that everyone thoroughly understands the selection criteria before you begin.

If you’re writing a systematic review as a student for an assignment, you might not have a team. In this case, you’ll have to apply the selection criteria on your own; you can mention this as a limitation in your paper’s discussion.

You should apply the selection criteria in two phases:

• Based on the titles and abstracts : Decide whether each article potentially meets the selection criteria based on the information provided in the abstracts.
• Based on the full texts: Download the articles that weren’t excluded during the first phase. If an article isn’t available online or through your library, you may need to contact the authors to ask for a copy. Read the articles and decide which articles meet the selection criteria.

It’s very important to keep a meticulous record of why you included or excluded each article. When the selection process is complete, you can summarise what you did using a PRISMA flow diagram .

Next, Boyle and colleagues found the full texts for each of the remaining studies. Boyle and Tang read through the articles to decide if any more studies needed to be excluded based on the selection criteria.

When Boyle and Tang disagreed about whether a study should be excluded, they discussed it with Varigos until the three researchers came to an agreement.

Step 5: Extract the data

Extracting the data means collecting information from the selected studies in a systematic way. There are two types of information you need to collect from each study:

• Information about the study’s methods and results . The exact information will depend on your research question, but it might include the year, study design , sample size, context, research findings , and conclusions. If any data are missing, you’ll need to contact the study’s authors.
• Your judgement of the quality of the evidence, including risk of bias .

You should collect this information using forms. You can find sample forms in The Registry of Methods and Tools for Evidence-Informed Decision Making and the Grading of Recommendations, Assessment, Development and Evaluations Working Group .

Extracting the data is also a three-person job. Two people should do this step independently, and the third person will resolve any disagreements.

They also collected data about possible sources of bias, such as how the study participants were randomised into the control and treatment groups.

Step 6: Synthesise the data

Synthesising the data means bringing together the information you collected into a single, cohesive story. There are two main approaches to synthesising the data:

• Narrative ( qualitative ): Summarise the information in words. You’ll need to discuss the studies and assess their overall quality.
• Quantitative : Use statistical methods to summarise and compare data from different studies. The most common quantitative approach is a meta-analysis , which allows you to combine results from multiple studies into a summary result.

Generally, you should use both approaches together whenever possible. If you don’t have enough data, or the data from different studies aren’t comparable, then you can take just a narrative approach. However, you should justify why a quantitative approach wasn’t possible.

Boyle and colleagues also divided the studies into subgroups, such as studies about babies, children, and adults, and analysed the effect sizes within each group.

Step 7: Write and publish a report

The purpose of writing a systematic review article is to share the answer to your research question and explain how you arrived at this answer.

Your article should include the following sections:

• Abstract : A summary of the review
• Introduction : Including the rationale and objectives
• Methods : Including the selection criteria, search method, data extraction method, and synthesis method
• Results : Including results of the search and selection process, study characteristics, risk of bias in the studies, and synthesis results
• Discussion : Including interpretation of the results and limitations of the review
• Conclusion : The answer to your research question and implications for practice, policy, or research

To verify that your report includes everything it needs, you can use the PRISMA checklist .

Once your report is written, you can publish it in a systematic review database, such as the Cochrane Database of Systematic Reviews , and/or in a peer-reviewed journal.

A systematic review is secondary research because it uses existing research. You don’t collect new data yourself.

A literature review is a survey of scholarly sources (such as books, journal articles, and theses) related to a specific topic or research question .

It is often written as part of a dissertation , thesis, research paper , or proposal .

There are several reasons to conduct a literature review at the beginning of a research project:

• To familiarise yourself with the current state of knowledge on your topic
• To ensure that you’re not just repeating what others have already done
• To identify gaps in knowledge and unresolved problems that your research can address
• To develop your theoretical framework and methodology
• To provide an overview of the key findings and debates on the topic

Writing the literature review shows your reader how your work relates to existing research and what new insights it will contribute.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the ‘Cite this Scribbr article’ button to automatically add the citation to our free Reference Generator.

Turney, S. (2022, October 17). Systematic Review | Definition, Examples & Guide. Scribbr. Retrieved 6 May 2024, from https://www.scribbr.co.uk/research-methods/systematic-reviews/

Is this article helpful?

Shaun Turney

Other students also liked, what is a literature review | guide, template, & examples, exploratory research | definition, guide, & examples, what is peer review | types & examples.

How to Write a Systematic Review of the Literature

Affiliations.

• 1 1 Texas Tech University, Lubbock, TX, USA.
• 2 2 University of Florida, Gainesville, FL, USA.
• PMID: 29283007
• DOI: 10.1177/1937586717747384

This article provides a step-by-step approach to conducting and reporting systematic literature reviews (SLRs) in the domain of healthcare design and discusses some of the key quality issues associated with SLRs. SLR, as the name implies, is a systematic way of collecting, critically evaluating, integrating, and presenting findings from across multiple research studies on a research question or topic of interest. SLR provides a way to assess the quality level and magnitude of existing evidence on a question or topic of interest. It offers a broader and more accurate level of understanding than a traditional literature review. A systematic review adheres to standardized methodologies/guidelines in systematic searching, filtering, reviewing, critiquing, interpreting, synthesizing, and reporting of findings from multiple publications on a topic/domain of interest. The Cochrane Collaboration is the most well-known and widely respected global organization producing SLRs within the healthcare field and a standard to follow for any researcher seeking to write a transparent and methodologically sound SLR. Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA), like the Cochrane Collaboration, was created by an international network of health-based collaborators and provides the framework for SLR to ensure methodological rigor and quality. The PRISMA statement is an evidence-based guide consisting of a checklist and flowchart intended to be used as tools for authors seeking to write SLR and meta-analyses.

Keywords: evidence based design; healthcare design; systematic literature review.

• Evidence-Based Medicine* / organization & administration
• Research Design*
• Systematic Reviews as Topic*

Systematic Reviews

• Introduction

Step by Step: Systematic Review Handbooks

Basic steps.

• 1. Planning a Review
• 2. Defining Your Question & Criteria
• 3. Standards & Protocols
• 4. Search Terms & Strategies
• 5. Locating Published Research
• 6. Locating Grey Literature
• 7. Managing & Documenting Results
• 8. Selecting & Appraising Studies
• 9. Extracting Data
• 10. Writing a Systematic Review
• Tools & Software
• Guides & Tutorials
• Accessing Resources
• Research Assistance

Templates and checklists are provided throughout this guide to support the many tasks required to complete a review.

• Systematic Review Workbook: Medical Sciences Library, Texas A&M University A detailed guide and template for developing, conducting, & reporting reviews.

A systematic review uses specific procedures to locate, evaluate and synthesize the results of relevant research to address your research question. Procedures are explicitly defined in advance, in order to ensure transparency, reproducibility, and minimize bias. The below handbooks provide step by step guidance for conducting a systematic review. 

You may be required to follow a specific review standard or protocol, or you may use handbooks for more general guidance. This guide frequently references the Cochrane Handbook .

• Cochrane Handbook for Systematic Reviews of Interventions This handbook provides detailed guidance on conducting a Cochrane systematic review, however is a valuable resource for all authors,
• Systematic Reviews of Health Promotion and Public Health Interventions (Cochrane Collaboration) Guidelines specific to advising and supporting systematic reviews of health promotion and public health interventions within the Cochrane Collaboration.
• Systematic Review Study Guide Concise 16 page step-by-step guide from Monash University Library.

Planning a Review & Research Question Development 

(Guide tabs 1-3)

• Assemble a team 
• Develop a specific question: Frameworks like  PICO and FINER can help guide this process
• Specify inclusion/exclusion criteria
• Choose a standard (for exp.,  IOM , MECIR )
• Develop a protocol
• Consider registering your protocol ( PROSPERO ) 

Searching for Literature

(Guide tabs 4-6)

• Construct individual search strategies for  Cochrane Summaries , PubMed and any other relevant databases ( PsycINFO , ClinicalTrials.gov, ICTRP , etc.)
• Run your searches in all databases
• Search grey literature (association websites, conference proceedings, theses and dissertations, contact researchers in the field) 

Managing & Documenting Search Results 

(Guide tab 7 )

• Export all citations to a citation manager (see Tools & Software )
• De-duplicate records, keeping track of original numbers found, duplicates, and numbers after de-duplicating

Screening & Selecting Studies​

(Guide tab 8)

• Screen citations by title/abstract using inclusion/exclusion criteria 
• Obtain full-text of possibly relevant citations 
• Screen citations a second time by full-text. Record reasons for each excluded study

Extracting Data

(Guide tab 9)

• Extract findings and data from studies 
• Evaluate the quality and risk of bias of studies
• Tabulate characteristics of included studies

Writing your Review

(Guide tab 10)

• Synthesize findings and data; summarize findings 
• Write article (refer to reporting protocols such as the PRISMA checklist)
• Update searches if it's been more than 6 months and add relevant studies 
• Submit to journal

Adapted from MD Anderson Cancer Center

• << Previous: Introduction
• Next: 1. Planning a Review >>
• Last Updated: Feb 26, 2024 2:04 PM
• URL: https://libguides.ucmerced.edu/systematic-reviews

• Open access
• Published: 01 August 2019

A step by step guide for conducting a systematic review and meta-analysis with simulation data

• Gehad Mohamed Tawfik 1 , 2 ,
• Kadek Agus Surya Dila 2 , 3 ,
• Muawia Yousif Fadlelmola Mohamed 2 , 4 ,
• Dao Ngoc Hien Tam 2 , 5 ,
• Nguyen Dang Kien 2 , 6 ,
• Ali Mahmoud Ahmed 2 , 7 &
• Nguyen Tien Huy 8 , 9 , 10  

Tropical Medicine and Health volume  47 , Article number:  46 ( 2019 ) Cite this article

800k Accesses

293 Citations

92 Altmetric

Metrics details

The massive abundance of studies relating to tropical medicine and health has increased strikingly over the last few decades. In the field of tropical medicine and health, a well-conducted systematic review and meta-analysis (SR/MA) is considered a feasible solution for keeping clinicians abreast of current evidence-based medicine. Understanding of SR/MA steps is of paramount importance for its conduction. It is not easy to be done as there are obstacles that could face the researcher. To solve those hindrances, this methodology study aimed to provide a step-by-step approach mainly for beginners and junior researchers, in the field of tropical medicine and other health care fields, on how to properly conduct a SR/MA, in which all the steps here depicts our experience and expertise combined with the already well-known and accepted international guidance.

We suggest that all steps of SR/MA should be done independently by 2–3 reviewers’ discussion, to ensure data quality and accuracy.

SR/MA steps include the development of research question, forming criteria, search strategy, searching databases, protocol registration, title, abstract, full-text screening, manual searching, extracting data, quality assessment, data checking, statistical analysis, double data checking, and manuscript writing.

Introduction

The amount of studies published in the biomedical literature, especially tropical medicine and health, has increased strikingly over the last few decades. This massive abundance of literature makes clinical medicine increasingly complex, and knowledge from various researches is often needed to inform a particular clinical decision. However, available studies are often heterogeneous with regard to their design, operational quality, and subjects under study and may handle the research question in a different way, which adds to the complexity of evidence and conclusion synthesis [ 1 ].

Systematic review and meta-analyses (SR/MAs) have a high level of evidence as represented by the evidence-based pyramid. Therefore, a well-conducted SR/MA is considered a feasible solution in keeping health clinicians ahead regarding contemporary evidence-based medicine.

Differing from a systematic review, unsystematic narrative review tends to be descriptive, in which the authors select frequently articles based on their point of view which leads to its poor quality. A systematic review, on the other hand, is defined as a review using a systematic method to summarize evidence on questions with a detailed and comprehensive plan of study. Furthermore, despite the increasing guidelines for effectively conducting a systematic review, we found that basic steps often start from framing question, then identifying relevant work which consists of criteria development and search for articles, appraise the quality of included studies, summarize the evidence, and interpret the results [ 2 , 3 ]. However, those simple steps are not easy to be reached in reality. There are many troubles that a researcher could be struggled with which has no detailed indication.

Conducting a SR/MA in tropical medicine and health may be difficult especially for young researchers; therefore, understanding of its essential steps is crucial. It is not easy to be done as there are obstacles that could face the researcher. To solve those hindrances, we recommend a flow diagram (Fig. 1 ) which illustrates a detailed and step-by-step the stages for SR/MA studies. This methodology study aimed to provide a step-by-step approach mainly for beginners and junior researchers, in the field of tropical medicine and other health care fields, on how to properly and succinctly conduct a SR/MA; all the steps here depicts our experience and expertise combined with the already well known and accepted international guidance.

Detailed flow diagram guideline for systematic review and meta-analysis steps. Note : Star icon refers to “2–3 reviewers screen independently”

Methods and results

Detailed steps for conducting any systematic review and meta-analysis.

We searched the methods reported in published SR/MA in tropical medicine and other healthcare fields besides the published guidelines like Cochrane guidelines {Higgins, 2011 #7} [ 4 ] to collect the best low-bias method for each step of SR/MA conduction steps. Furthermore, we used guidelines that we apply in studies for all SR/MA steps. We combined these methods in order to conclude and conduct a detailed flow diagram that shows the SR/MA steps how being conducted.

Any SR/MA must follow the widely accepted Preferred Reporting Items for Systematic Review and Meta-analysis statement (PRISMA checklist 2009) (Additional file 5 : Table S1) [ 5 ].

We proposed our methods according to a valid explanatory simulation example choosing the topic of “evaluating safety of Ebola vaccine,” as it is known that Ebola is a very rare tropical disease but fatal. All the explained methods feature the standards followed internationally, with our compiled experience in the conduct of SR beside it, which we think proved some validity. This is a SR under conduct by a couple of researchers teaming in a research group, moreover, as the outbreak of Ebola which took place (2013–2016) in Africa resulted in a significant mortality and morbidity. Furthermore, since there are many published and ongoing trials assessing the safety of Ebola vaccines, we thought this would provide a great opportunity to tackle this hotly debated issue. Moreover, Ebola started to fire again and new fatal outbreak appeared in the Democratic Republic of Congo since August 2018, which caused infection to more than 1000 people according to the World Health Organization, and 629 people have been killed till now. Hence, it is considered the second worst Ebola outbreak, after the first one in West Africa in 2014 , which infected more than 26,000 and killed about 11,300 people along outbreak course.

Research question and objectives

Like other study designs, the research question of SR/MA should be feasible, interesting, novel, ethical, and relevant. Therefore, a clear, logical, and well-defined research question should be formulated. Usually, two common tools are used: PICO or SPIDER. PICO (Population, Intervention, Comparison, Outcome) is used mostly in quantitative evidence synthesis. Authors demonstrated that PICO holds more sensitivity than the more specific SPIDER approach [ 6 ]. SPIDER (Sample, Phenomenon of Interest, Design, Evaluation, Research type) was proposed as a method for qualitative and mixed methods search.

We here recommend a combined approach of using either one or both the SPIDER and PICO tools to retrieve a comprehensive search depending on time and resources limitations. When we apply this to our assumed research topic, being of qualitative nature, the use of SPIDER approach is more valid.

PICO is usually used for systematic review and meta-analysis of clinical trial study. For the observational study (without intervention or comparator), in many tropical and epidemiological questions, it is usually enough to use P (Patient) and O (outcome) only to formulate a research question. We must indicate clearly the population (P), then intervention (I) or exposure. Next, it is necessary to compare (C) the indicated intervention with other interventions, i.e., placebo. Finally, we need to clarify which are our relevant outcomes.

To facilitate comprehension, we choose the Ebola virus disease (EVD) as an example. Currently, the vaccine for EVD is being developed and under phase I, II, and III clinical trials; we want to know whether this vaccine is safe and can induce sufficient immunogenicity to the subjects.

An example of a research question for SR/MA based on PICO for this issue is as follows: How is the safety and immunogenicity of Ebola vaccine in human? (P: healthy subjects (human), I: vaccination, C: placebo, O: safety or adverse effects)

Preliminary research and idea validation

We recommend a preliminary search to identify relevant articles, ensure the validity of the proposed idea, avoid duplication of previously addressed questions, and assure that we have enough articles for conducting its analysis. Moreover, themes should focus on relevant and important health-care issues, consider global needs and values, reflect the current science, and be consistent with the adopted review methods. Gaining familiarity with a deep understanding of the study field through relevant videos and discussions is of paramount importance for better retrieval of results. If we ignore this step, our study could be canceled whenever we find out a similar study published before. This means we are wasting our time to deal with a problem that has been tackled for a long time.

To do this, we can start by doing a simple search in PubMed or Google Scholar with search terms Ebola AND vaccine. While doing this step, we identify a systematic review and meta-analysis of determinant factors influencing antibody response from vaccination of Ebola vaccine in non-human primate and human [ 7 ], which is a relevant paper to read to get a deeper insight and identify gaps for better formulation of our research question or purpose. We can still conduct systematic review and meta-analysis of Ebola vaccine because we evaluate safety as a different outcome and different population (only human).

Inclusion and exclusion criteria

Eligibility criteria are based on the PICO approach, study design, and date. Exclusion criteria mostly are unrelated, duplicated, unavailable full texts, or abstract-only papers. These exclusions should be stated in advance to refrain the researcher from bias. The inclusion criteria would be articles with the target patients, investigated interventions, or the comparison between two studied interventions. Briefly, it would be articles which contain information answering our research question. But the most important is that it should be clear and sufficient information, including positive or negative, to answer the question.

For the topic we have chosen, we can make inclusion criteria: (1) any clinical trial evaluating the safety of Ebola vaccine and (2) no restriction regarding country, patient age, race, gender, publication language, and date. Exclusion criteria are as follows: (1) study of Ebola vaccine in non-human subjects or in vitro studies; (2) study with data not reliably extracted, duplicate, or overlapping data; (3) abstract-only papers as preceding papers, conference, editorial, and author response theses and books; (4) articles without available full text available; and (5) case reports, case series, and systematic review studies. The PRISMA flow diagram template that is used in SR/MA studies can be found in Fig. 2 .

PRISMA flow diagram of studies’ screening and selection

Search strategy

A standard search strategy is used in PubMed, then later it is modified according to each specific database to get the best relevant results. The basic search strategy is built based on the research question formulation (i.e., PICO or PICOS). Search strategies are constructed to include free-text terms (e.g., in the title and abstract) and any appropriate subject indexing (e.g., MeSH) expected to retrieve eligible studies, with the help of an expert in the review topic field or an information specialist. Additionally, we advise not to use terms for the Outcomes as their inclusion might hinder the database being searched to retrieve eligible studies because the used outcome is not mentioned obviously in the articles.

The improvement of the search term is made while doing a trial search and looking for another relevant term within each concept from retrieved papers. To search for a clinical trial, we can use these descriptors in PubMed: “clinical trial”[Publication Type] OR “clinical trials as topic”[MeSH terms] OR “clinical trial”[All Fields]. After some rounds of trial and refinement of search term, we formulate the final search term for PubMed as follows: (ebola OR ebola virus OR ebola virus disease OR EVD) AND (vaccine OR vaccination OR vaccinated OR immunization) AND (“clinical trial”[Publication Type] OR “clinical trials as topic”[MeSH Terms] OR “clinical trial”[All Fields]). Because the study for this topic is limited, we do not include outcome term (safety and immunogenicity) in the search term to capture more studies.

Search databases, import all results to a library, and exporting to an excel sheet

According to the AMSTAR guidelines, at least two databases have to be searched in the SR/MA [ 8 ], but as you increase the number of searched databases, you get much yield and more accurate and comprehensive results. The ordering of the databases depends mostly on the review questions; being in a study of clinical trials, you will rely mostly on Cochrane, mRCTs, or International Clinical Trials Registry Platform (ICTRP). Here, we propose 12 databases (PubMed, Scopus, Web of Science, EMBASE, GHL, VHL, Cochrane, Google Scholar, Clinical trials.gov , mRCTs, POPLINE, and SIGLE), which help to cover almost all published articles in tropical medicine and other health-related fields. Among those databases, POPLINE focuses on reproductive health. Researchers should consider to choose relevant database according to the research topic. Some databases do not support the use of Boolean or quotation; otherwise, there are some databases that have special searching way. Therefore, we need to modify the initial search terms for each database to get appreciated results; therefore, manipulation guides for each online database searches are presented in Additional file 5 : Table S2. The detailed search strategy for each database is found in Additional file 5 : Table S3. The search term that we created in PubMed needs customization based on a specific characteristic of the database. An example for Google Scholar advanced search for our topic is as follows:

With all of the words: ebola virus

With at least one of the words: vaccine vaccination vaccinated immunization

Where my words occur: in the title of the article

With all of the words: EVD

Finally, all records are collected into one Endnote library in order to delete duplicates and then to it export into an excel sheet. Using remove duplicating function with two options is mandatory. All references which have (1) the same title and author, and published in the same year, and (2) the same title and author, and published in the same journal, would be deleted. References remaining after this step should be exported to an excel file with essential information for screening. These could be the authors’ names, publication year, journal, DOI, URL link, and abstract.

Protocol writing and registration

Protocol registration at an early stage guarantees transparency in the research process and protects from duplication problems. Besides, it is considered a documented proof of team plan of action, research question, eligibility criteria, intervention/exposure, quality assessment, and pre-analysis plan. It is recommended that researchers send it to the principal investigator (PI) to revise it, then upload it to registry sites. There are many registry sites available for SR/MA like those proposed by Cochrane and Campbell collaborations; however, we recommend registering the protocol into PROSPERO as it is easier. The layout of a protocol template, according to PROSPERO, can be found in Additional file 5 : File S1.

Title and abstract screening

Decisions to select retrieved articles for further assessment are based on eligibility criteria, to minimize the chance of including non-relevant articles. According to the Cochrane guidance, two reviewers are a must to do this step, but as for beginners and junior researchers, this might be tiresome; thus, we propose based on our experience that at least three reviewers should work independently to reduce the chance of error, particularly in teams with a large number of authors to add more scrutiny and ensure proper conduct. Mostly, the quality with three reviewers would be better than two, as two only would have different opinions from each other, so they cannot decide, while the third opinion is crucial. And here are some examples of systematic reviews which we conducted following the same strategy (by a different group of researchers in our research group) and published successfully, and they feature relevant ideas to tropical medicine and disease [ 9 , 10 , 11 ].

In this step, duplications will be removed manually whenever the reviewers find them out. When there is a doubt about an article decision, the team should be inclusive rather than exclusive, until the main leader or PI makes a decision after discussion and consensus. All excluded records should be given exclusion reasons.

Full text downloading and screening

Many search engines provide links for free to access full-text articles. In case not found, we can search in some research websites as ResearchGate, which offer an option of direct full-text request from authors. Additionally, exploring archives of wanted journals, or contacting PI to purchase it if available. Similarly, 2–3 reviewers work independently to decide about included full texts according to eligibility criteria, with reporting exclusion reasons of articles. In case any disagreement has occurred, the final decision has to be made by discussion.

Manual search

One has to exhaust all possibilities to reduce bias by performing an explicit hand-searching for retrieval of reports that may have been dropped from first search [ 12 ]. We apply five methods to make manual searching: searching references from included studies/reviews, contacting authors and experts, and looking at related articles/cited articles in PubMed and Google Scholar.

We describe here three consecutive methods to increase and refine the yield of manual searching: firstly, searching reference lists of included articles; secondly, performing what is known as citation tracking in which the reviewers track all the articles that cite each one of the included articles, and this might involve electronic searching of databases; and thirdly, similar to the citation tracking, we follow all “related to” or “similar” articles. Each of the abovementioned methods can be performed by 2–3 independent reviewers, and all the possible relevant article must undergo further scrutiny against the inclusion criteria, after following the same records yielded from electronic databases, i.e., title/abstract and full-text screening.

We propose an independent reviewing by assigning each member of the teams a “tag” and a distinct method, to compile all the results at the end for comparison of differences and discussion and to maximize the retrieval and minimize the bias. Similarly, the number of included articles has to be stated before addition to the overall included records.

Data extraction and quality assessment

This step entitles data collection from included full-texts in a structured extraction excel sheet, which is previously pilot-tested for extraction using some random studies. We recommend extracting both adjusted and non-adjusted data because it gives the most allowed confounding factor to be used in the analysis by pooling them later [ 13 ]. The process of extraction should be executed by 2–3 independent reviewers. Mostly, the sheet is classified into the study and patient characteristics, outcomes, and quality assessment (QA) tool.

Data presented in graphs should be extracted by software tools such as Web plot digitizer [ 14 ]. Most of the equations that can be used in extraction prior to analysis and estimation of standard deviation (SD) from other variables is found inside Additional file 5 : File S2 with their references as Hozo et al. [ 15 ], Xiang et al. [ 16 ], and Rijkom et al. [ 17 ]. A variety of tools are available for the QA, depending on the design: ROB-2 Cochrane tool for randomized controlled trials [ 18 ] which is presented as Additional file 1 : Figure S1 and Additional file 2 : Figure S2—from a previous published article data—[ 19 ], NIH tool for observational and cross-sectional studies [ 20 ], ROBINS-I tool for non-randomize trials [ 21 ], QUADAS-2 tool for diagnostic studies, QUIPS tool for prognostic studies, CARE tool for case reports, and ToxRtool for in vivo and in vitro studies. We recommend that 2–3 reviewers independently assess the quality of the studies and add to the data extraction form before the inclusion into the analysis to reduce the risk of bias. In the NIH tool for observational studies—cohort and cross-sectional—as in this EBOLA case, to evaluate the risk of bias, reviewers should rate each of the 14 items into dichotomous variables: yes, no, or not applicable. An overall score is calculated by adding all the items scores as yes equals one, while no and NA equals zero. A score will be given for every paper to classify them as poor, fair, or good conducted studies, where a score from 0–5 was considered poor, 6–9 as fair, and 10–14 as good.

In the EBOLA case example above, authors can extract the following information: name of authors, country of patients, year of publication, study design (case report, cohort study, or clinical trial or RCT), sample size, the infected point of time after EBOLA infection, follow-up interval after vaccination time, efficacy, safety, adverse effects after vaccinations, and QA sheet (Additional file 6 : Data S1).

Data checking

Due to the expected human error and bias, we recommend a data checking step, in which every included article is compared with its counterpart in an extraction sheet by evidence photos, to detect mistakes in data. We advise assigning articles to 2–3 independent reviewers, ideally not the ones who performed the extraction of those articles. When resources are limited, each reviewer is assigned a different article than the one he extracted in the previous stage.

Statistical analysis

Investigators use different methods for combining and summarizing findings of included studies. Before analysis, there is an important step called cleaning of data in the extraction sheet, where the analyst organizes extraction sheet data in a form that can be read by analytical software. The analysis consists of 2 types namely qualitative and quantitative analysis. Qualitative analysis mostly describes data in SR studies, while quantitative analysis consists of two main types: MA and network meta-analysis (NMA). Subgroup, sensitivity, cumulative analyses, and meta-regression are appropriate for testing whether the results are consistent or not and investigating the effect of certain confounders on the outcome and finding the best predictors. Publication bias should be assessed to investigate the presence of missing studies which can affect the summary.

To illustrate basic meta-analysis, we provide an imaginary data for the research question about Ebola vaccine safety (in terms of adverse events, 14 days after injection) and immunogenicity (Ebola virus antibodies rise in geometric mean titer, 6 months after injection). Assuming that from searching and data extraction, we decided to do an analysis to evaluate Ebola vaccine “A” safety and immunogenicity. Other Ebola vaccines were not meta-analyzed because of the limited number of studies (instead, it will be included for narrative review). The imaginary data for vaccine safety meta-analysis can be accessed in Additional file 7 : Data S2. To do the meta-analysis, we can use free software, such as RevMan [ 22 ] or R package meta [ 23 ]. In this example, we will use the R package meta. The tutorial of meta package can be accessed through “General Package for Meta-Analysis” tutorial pdf [ 23 ]. The R codes and its guidance for meta-analysis done can be found in Additional file 5 : File S3.

For the analysis, we assume that the study is heterogenous in nature; therefore, we choose a random effect model. We did an analysis on the safety of Ebola vaccine A. From the data table, we can see some adverse events occurring after intramuscular injection of vaccine A to the subject of the study. Suppose that we include six studies that fulfill our inclusion criteria. We can do a meta-analysis for each of the adverse events extracted from the studies, for example, arthralgia, from the results of random effect meta-analysis using the R meta package.

From the results shown in Additional file 3 : Figure S3, we can see that the odds ratio (OR) of arthralgia is 1.06 (0.79; 1.42), p value = 0.71, which means that there is no association between the intramuscular injection of Ebola vaccine A and arthralgia, as the OR is almost one, and besides, the P value is insignificant as it is > 0.05.

In the meta-analysis, we can also visualize the results in a forest plot. It is shown in Fig. 3 an example of a forest plot from the simulated analysis.

Random effect model forest plot for comparison of vaccine A versus placebo

From the forest plot, we can see six studies (A to F) and their respective OR (95% CI). The green box represents the effect size (in this case, OR) of each study. The bigger the box means the study weighted more (i.e., bigger sample size). The blue diamond shape represents the pooled OR of the six studies. We can see the blue diamond cross the vertical line OR = 1, which indicates no significance for the association as the diamond almost equalized in both sides. We can confirm this also from the 95% confidence interval that includes one and the p value > 0.05.

For heterogeneity, we see that I 2 = 0%, which means no heterogeneity is detected; the study is relatively homogenous (it is rare in the real study). To evaluate publication bias related to the meta-analysis of adverse events of arthralgia, we can use the metabias function from the R meta package (Additional file 4 : Figure S4) and visualization using a funnel plot. The results of publication bias are demonstrated in Fig. 4 . We see that the p value associated with this test is 0.74, indicating symmetry of the funnel plot. We can confirm it by looking at the funnel plot.

Publication bias funnel plot for comparison of vaccine A versus placebo

Looking at the funnel plot, the number of studies at the left and right side of the funnel plot is the same; therefore, the plot is symmetry, indicating no publication bias detected.

Sensitivity analysis is a procedure used to discover how different values of an independent variable will influence the significance of a particular dependent variable by removing one study from MA. If all included study p values are < 0.05, hence, removing any study will not change the significant association. It is only performed when there is a significant association, so if the p value of MA done is 0.7—more than one—the sensitivity analysis is not needed for this case study example. If there are 2 studies with p value > 0.05, removing any of the two studies will result in a loss of the significance.

Double data checking

For more assurance on the quality of results, the analyzed data should be rechecked from full-text data by evidence photos, to allow an obvious check for the PI of the study.

Manuscript writing, revision, and submission to a journal

Writing based on four scientific sections: introduction, methods, results, and discussion, mostly with a conclusion. Performing a characteristic table for study and patient characteristics is a mandatory step which can be found as a template in Additional file 5 : Table S3.

After finishing the manuscript writing, characteristics table, and PRISMA flow diagram, the team should send it to the PI to revise it well and reply to his comments and, finally, choose a suitable journal for the manuscript which fits with considerable impact factor and fitting field. We need to pay attention by reading the author guidelines of journals before submitting the manuscript.

The role of evidence-based medicine in biomedical research is rapidly growing. SR/MAs are also increasing in the medical literature. This paper has sought to provide a comprehensive approach to enable reviewers to produce high-quality SR/MAs. We hope that readers could gain general knowledge about how to conduct a SR/MA and have the confidence to perform one, although this kind of study requires complex steps compared to narrative reviews.

Having the basic steps for conduction of MA, there are many advanced steps that are applied for certain specific purposes. One of these steps is meta-regression which is performed to investigate the association of any confounder and the results of the MA. Furthermore, there are other types rather than the standard MA like NMA and MA. In NMA, we investigate the difference between several comparisons when there were not enough data to enable standard meta-analysis. It uses both direct and indirect comparisons to conclude what is the best between the competitors. On the other hand, mega MA or MA of patients tend to summarize the results of independent studies by using its individual subject data. As a more detailed analysis can be done, it is useful in conducting repeated measure analysis and time-to-event analysis. Moreover, it can perform analysis of variance and multiple regression analysis; however, it requires homogenous dataset and it is time-consuming in conduct [ 24 ].

Conclusions

Systematic review/meta-analysis steps include development of research question and its validation, forming criteria, search strategy, searching databases, importing all results to a library and exporting to an excel sheet, protocol writing and registration, title and abstract screening, full-text screening, manual searching, extracting data and assessing its quality, data checking, conducting statistical analysis, double data checking, manuscript writing, revising, and submitting to a journal.

Availability of data and materials

Not applicable.

Abbreviations

Network meta-analysis

Principal investigator

Population, Intervention, Comparison, Outcome

Preferred Reporting Items for Systematic Review and Meta-analysis statement

Quality assessment

Sample, Phenomenon of Interest, Design, Evaluation, Research type

Systematic review and meta-analyses

Bello A, Wiebe N, Garg A, Tonelli M. Evidence-based decision-making 2: systematic reviews and meta-analysis. Methods Mol Biol (Clifton, NJ). 2015;1281:397–416.

Article   Google Scholar  

Khan KS, Kunz R, Kleijnen J, Antes G. Five steps to conducting a systematic review. J R Soc Med. 2003;96(3):118–21.

Rys P, Wladysiuk M, Skrzekowska-Baran I, Malecki MT. Review articles, systematic reviews and meta-analyses: which can be trusted? Polskie Archiwum Medycyny Wewnetrznej. 2009;119(3):148–56.

PubMed   Google Scholar  

Higgins JPT, Green S. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. 2011.

Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009;339:b2535.

Methley AM, Campbell S, Chew-Graham C, McNally R, Cheraghi-Sohi S. PICO, PICOS and SPIDER: a comparison study of specificity and sensitivity in three search tools for qualitative systematic reviews. BMC Health Serv Res. 2014;14:579.

Gross L, Lhomme E, Pasin C, Richert L, Thiebaut R. Ebola vaccine development: systematic review of pre-clinical and clinical studies, and meta-analysis of determinants of antibody response variability after vaccination. Int J Infect Dis. 2018;74:83–96.

Article   CAS   Google Scholar  

Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, ... Henry DA. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ. 2017;358:j4008.

Giang HTN, Banno K, Minh LHN, Trinh LT, Loc LT, Eltobgy A, et al. Dengue hemophagocytic syndrome: a systematic review and meta-analysis on epidemiology, clinical signs, outcomes, and risk factors. Rev Med Virol. 2018;28(6):e2005.

Morra ME, Altibi AMA, Iqtadar S, Minh LHN, Elawady SS, Hallab A, et al. Definitions for warning signs and signs of severe dengue according to the WHO 2009 classification: systematic review of literature. Rev Med Virol. 2018;28(4):e1979.

Morra ME, Van Thanh L, Kamel MG, Ghazy AA, Altibi AMA, Dat LM, et al. Clinical outcomes of current medical approaches for Middle East respiratory syndrome: a systematic review and meta-analysis. Rev Med Virol. 2018;28(3):e1977.

Vassar M, Atakpo P, Kash MJ. Manual search approaches used by systematic reviewers in dermatology. Journal of the Medical Library Association: JMLA. 2016;104(4):302.

Naunheim MR, Remenschneider AK, Scangas GA, Bunting GW, Deschler DG. The effect of initial tracheoesophageal voice prosthesis size on postoperative complications and voice outcomes. Ann Otol Rhinol Laryngol. 2016;125(6):478–84.

Rohatgi AJaiWa. Web Plot Digitizer. ht tp. 2014;2.

Hozo SP, Djulbegovic B, Hozo I. Estimating the mean and variance from the median, range, and the size of a sample. BMC Med Res Methodol. 2005;5(1):13.

Wan X, Wang W, Liu J, Tong T. Estimating the sample mean and standard deviation from the sample size, median, range and/or interquartile range. BMC Med Res Methodol. 2014;14(1):135.

Van Rijkom HM, Truin GJ, Van’t Hof MA. A meta-analysis of clinical studies on the caries-inhibiting effect of fluoride gel treatment. Carries Res. 1998;32(2):83–92.

Higgins JP, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928.

Tawfik GM, Tieu TM, Ghozy S, Makram OM, Samuel P, Abdelaal A, et al. Speech efficacy, safety and factors affecting lifetime of voice prostheses in patients with laryngeal cancer: a systematic review and network meta-analysis of randomized controlled trials. J Clin Oncol. 2018;36(15_suppl):e18031-e.

Wannemuehler TJ, Lobo BC, Johnson JD, Deig CR, Ting JY, Gregory RL. Vibratory stimulus reduces in vitro biofilm formation on tracheoesophageal voice prostheses. Laryngoscope. 2016;126(12):2752–7.

Sterne JAC, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355.

RevMan The Cochrane Collaboration %J Copenhagen TNCCTCC. Review Manager (RevMan). 5.0. 2008.

Schwarzer GJRn. meta: An R package for meta-analysis. 2007;7(3):40-45.

Google Scholar  

Simms LLH. Meta-analysis versus mega-analysis: is there a difference? Oral budesonide for the maintenance of remission in Crohn’s disease: Faculty of Graduate Studies, University of Western Ontario; 1998.

Download references

Acknowledgements

This study was conducted (in part) at the Joint Usage/Research Center on Tropical Disease, Institute of Tropical Medicine, Nagasaki University, Japan.

Author information

Authors and affiliations.

Faculty of Medicine, Ain Shams University, Cairo, Egypt

Gehad Mohamed Tawfik

Online research Club http://www.onlineresearchclub.org/

Gehad Mohamed Tawfik, Kadek Agus Surya Dila, Muawia Yousif Fadlelmola Mohamed, Dao Ngoc Hien Tam, Nguyen Dang Kien & Ali Mahmoud Ahmed

Pratama Giri Emas Hospital, Singaraja-Amlapura street, Giri Emas village, Sawan subdistrict, Singaraja City, Buleleng, Bali, 81171, Indonesia

Kadek Agus Surya Dila

Faculty of Medicine, University of Khartoum, Khartoum, Sudan

Muawia Yousif Fadlelmola Mohamed

Nanogen Pharmaceutical Biotechnology Joint Stock Company, Ho Chi Minh City, Vietnam

Dao Ngoc Hien Tam

Department of Obstetrics and Gynecology, Thai Binh University of Medicine and Pharmacy, Thai Binh, Vietnam

Nguyen Dang Kien

Faculty of Medicine, Al-Azhar University, Cairo, Egypt

Ali Mahmoud Ahmed

Evidence Based Medicine Research Group & Faculty of Applied Sciences, Ton Duc Thang University, Ho Chi Minh City, 70000, Vietnam

Nguyen Tien Huy

Faculty of Applied Sciences, Ton Duc Thang University, Ho Chi Minh City, 70000, Vietnam

Department of Clinical Product Development, Institute of Tropical Medicine (NEKKEN), Leading Graduate School Program, and Graduate School of Biomedical Sciences, Nagasaki University, 1-12-4 Sakamoto, Nagasaki, 852-8523, Japan

You can also search for this author in PubMed   Google Scholar

Contributions

NTH and GMT were responsible for the idea and its design. The figure was done by GMT. All authors contributed to the manuscript writing and approval of the final version.

Corresponding author

Correspondence to Nguyen Tien Huy .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Additional files

Additional file 1:.

Figure S1. Risk of bias assessment graph of included randomized controlled trials. (TIF 20 kb)

Additional file 2:

Figure S2. Risk of bias assessment summary. (TIF 69 kb)

Additional file 3:

Figure S3. Arthralgia results of random effect meta-analysis using R meta package. (TIF 20 kb)

Additional file 4:

Figure S4. Arthralgia linear regression test of funnel plot asymmetry using R meta package. (TIF 13 kb)

Additional file 5:

Table S1. PRISMA 2009 Checklist. Table S2. Manipulation guides for online database searches. Table S3. Detailed search strategy for twelve database searches. Table S4. Baseline characteristics of the patients in the included studies. File S1. PROSPERO protocol template file. File S2. Extraction equations that can be used prior to analysis to get missed variables. File S3. R codes and its guidance for meta-analysis done for comparison between EBOLA vaccine A and placebo. (DOCX 49 kb)

Additional file 6:

Data S1. Extraction and quality assessment data sheets for EBOLA case example. (XLSX 1368 kb)

Additional file 7:

Data S2. Imaginary data for EBOLA case example. (XLSX 10 kb)

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article.

Tawfik, G.M., Dila, K.A.S., Mohamed, M.Y.F. et al. A step by step guide for conducting a systematic review and meta-analysis with simulation data. Trop Med Health 47 , 46 (2019). https://doi.org/10.1186/s41182-019-0165-6

Download citation

Received : 30 January 2019

Accepted : 24 May 2019

Published : 01 August 2019

DOI : https://doi.org/10.1186/s41182-019-0165-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Tropical Medicine and Health

ISSN: 1349-4147

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

• UNC Libraries
• HSL Academic Process
• Systematic Reviews

Systematic Reviews: Home

Created by health science librarians.

• Systematic review resources

What is a Systematic Review?

A simplified process map, how can the library help, publications by hsl librarians, systematic reviews in non-health disciplines, resources for performing systematic reviews.

• Step 1: Complete Pre-Review Tasks
• Step 2: Develop a Protocol
• Step 3: Conduct Literature Searches
• Step 4: Manage Citations
• Step 5: Screen Citations
• Step 6: Assess Quality of Included Studies
• Step 7: Extract Data from Included Studies
• Step 8: Write the Review

  Check our FAQ's

   Email us

   Call (919) 962-0800

   Make an appointment with a librarian

  Request a systematic or scoping review consultation

A systematic review is a literature review that gathers all of the available evidence matching pre-specified eligibility criteria to answer a specific research question. It uses explicit, systematic methods, documented in a protocol, to minimize bias , provide reliable findings , and inform decision-making.  ¹  

There are many types of literature reviews.

Before beginning a systematic review, consider whether it is the best type of review for your question, goals, and resources. The table below compares a few different types of reviews to help you decide which is best for you. 

• Scoping Review Guide For more information about scoping reviews, refer to the UNC HSL Scoping Review Guide.

• UNC HSL's Simplified, Step-by-Step Process Map A PDF file of the HSL's Systematic Review Process Map.
• Text-Only: UNC HSL's Systematic Reviews - A Simplified, Step-by-Step Process A text-only PDF file of HSL's Systematic Review Process Map.

The average systematic review takes 1,168 hours to complete. ¹   A librarian can help you speed up the process.

Systematic reviews follow established guidelines and best practices to produce high-quality research. Librarian involvement in systematic reviews is based on two levels. In Tier 1, your research team can consult with the librarian as needed. The librarian will answer questions and give you recommendations for tools to use. In Tier 2, the librarian will be an active member of your research team and co-author on your review. Roles and expectations of librarians vary based on the level of involvement desired. Examples of these differences are outlined in the table below.

• Request a systematic or scoping review consultation

The following are systematic and scoping reviews co-authored by HSL librarians.

Only the most recent 15 results are listed. Click the website link at the bottom of the list to see all reviews co-authored by HSL librarians in PubMed

Researchers conduct systematic reviews in a variety of disciplines.  If your focus is on a topic outside of the health sciences, you may want to also consult the resources below to learn how systematic reviews may vary in your field.  You can also contact a librarian for your discipline with questions.

• EPPI-Centre methods for conducting systematic reviews The EPPI-Centre develops methods and tools for conducting systematic reviews, including reviews for education, public and social policy.

Environmental Topics

• Collaboration for Environmental Evidence (CEE) CEE seeks to promote and deliver evidence syntheses on issues of greatest concern to environmental policy and practice as a public service

Social Sciences

• Siddaway AP, Wood AM, Hedges LV. How to Do a Systematic Review: A Best Practice Guide for Conducting and Reporting Narrative Reviews, Meta-Analyses, and Meta-Syntheses. Annu Rev Psychol. 2019 Jan 4;70:747-770. doi: 10.1146/annurev-psych-010418-102803. A resource for psychology systematic reviews, which also covers qualitative meta-syntheses or meta-ethnographies
• The Campbell Collaboration

Social Work

Software engineering

• Guidelines for Performing Systematic Literature Reviews in Software Engineering The objective of this report is to propose comprehensive guidelines for systematic literature reviews appropriate for software engineering researchers, including PhD students.

Sport, Exercise, & Nutrition

• Application of systematic review methodology to the field of nutrition by Tufts Evidence-based Practice Center Publication Date: 2009
• Systematic Reviews and Meta-Analysis — Open & Free (Open Learning Initiative) The course follows guidelines and standards developed by the Campbell Collaboration, based on empirical evidence about how to produce the most comprehensive and accurate reviews of research

• Systematic Reviews by David Gough, Sandy Oliver & James Thomas Publication Date: 2020

Updating reviews

• Updating systematic reviews by University of Ottawa Evidence-based Practice Center Publication Date: 2007

Looking for our previous Systematic Review guide?

Our legacy guide was used June 2020 to August 2022

• Systematic Review Legacy Guide
• Next: Step 1: Complete Pre-Review Tasks >>
• Last Updated: May 10, 2024 5:39 PM
• URL: https://guides.lib.unc.edu/systematic-reviews

Search & Find

• E-Research by Discipline
• More Search & Find

Places & Spaces

• Places to Study
• Book a Study Room
• Printers, Scanners, & Computers
• More Places & Spaces
• Borrowing & Circulation
• Request a Title for Purchase
• Schedule Instruction Session
• More Services

Support & Guides

• Course Reserves
• Research Guides
• Citing & Writing
• More Support & Guides
• Mission Statement
• Diversity Statement
• Staff Directory
• Job Opportunities
• Give to the Libraries
• News & Exhibits
• Reckoning Initiative
• More About Us

• Search This Site
• Privacy Policy
• Accessibility
• Give Us Your Feedback
• 208 Raleigh Street CB #3916
• Chapel Hill, NC 27515-8890
• 919-962-1053

Systematic Review

What is a systematic review, types of systematic reviews, steps in a systematic review.

• Developing a search strategy
• Search techniques
• Systematically search databases
• Appraisal & synthesis
• Reporting findings
• Systematic review tools

Research Services Team

For further help email us at: [email protected]

A systematic review is commonly characterised by:

• A well-defined research question
• Transparent search terms and database selection
• Exclusion/inclusion criteria with evaluation of search findings
• A research project structure with elements such as Introduction, Method, Result, Discussion

A systematic review is considered secondary research because it uses research by others and does not involve data collection for a new research project.

Video: Conducting a systematic literature review   (3.17 mins). A quick overview and comparison with traditional literature reviews.

How is it different from a traditional literature review?

The purpose of systematic review is different from that of a traditional literature review.

A systematic review further

• involves a clearly articulated search process and selection criteria of the literature which is closely examined before being included in the review.
• uses a search and selection procedure that is transparent and can be replicated. 

In a traditional literature review, the researcher

• selects and examines studies related to the research topic.
• does not have to make visible the search and selection process and criteria.

See more detailed comparison for both types of reviews in this PDF:

• Comparison table of traditional and systematic reviews

There are four common types of reviews using systematic methods:

• Systematic literature reviews
• Rapid reviews
• Scoping reviews
• Integrative reviews

For a more detailed comparison, see this PDF:

• Comparison of the common types of reviews

A common feature of these reviews is the goal of reducing bias in the search and selection of studies.

This bias mainly refers to:

• Availability of resources
• Researcher’s degree of objectivity
• Degree of similarity in type and content of research

A common strategy for reducing bias:

• Extended time to perform a thorough search in published and ‘yet to be published’ articles
• Two or more reviewers following transparent processes of conducting searches and making selections
• Homogeneity of selected research articles

Right review

Right review is a tool to assist users selecting a review type from the 41 quantitative or qualitative knowledge synthesis methods.

References and Furtherreading 

Centre for Evidence Synthesis in Health. (2018, January 4). The steps of a systematic review [Video]. YouTube.  https://www.youtube.com/watch?v=-FQSsnaAtOU

Librarian Carrie Price. (2021, May 18). Systematic vs scoping review: What’s the difference [Video]. YouTube. https://www.youtube.com/watch?v=YVckIl8_ZCg

Munn, Z., Peters, M. D. J., Stern, C., Tufanaru, C., McArthur, A., & Aromataris, E. (2018). Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach.  BMC Medical Research Methodology , 18 , Article 143.  https://doi.org/10.1186/s12874-018-0611-x

Munn, Z., Stern, C., Aromataris, E., Lockwood, C., & Jordan, Z. (2018). What kind of systematic review should I conduct? A proposed typology and guidance for systematic reviewers in the medical and health sciences.  BMC Medical Research Methodology ,  18, Article 5.  https://doi.org/10.1186/s12874-017-0468-4

Peters, M. D. J., Godfrey, C., McInerney, P., Munn, Z., Tricco, A. C., Khalil, H. (2020). Scoping reviews. In E. Aromataris, & Z. Munn (Eds.),  JBI manual for evidence synthesis.  Joanna Briggs Institute.  https://doi.org/10.46658/JBIMES-20-12

Peters, M. D. J., Marnie, C., Tricco, A. C., Pollock, D., Munn, Z., Alexander, L., McInerney, P., Godfrey, C. M., & Khalil, H. (2020). Updated methodological guidance for the conduct of scoping reviews.  JBI Evidence Synthesis ,  18 (10), 2119– 2126.  https://doi.org/10.11124/JBIES-20-00167

Pringle, J., Mills, K., McAteer, J., Jepson, R., Hogg, E., Anand, N., & Blakemore, S. J. (2016). A systematic review of adolescent physiological development and its relationship with health-related behaviour: A protocol. Systematic Reviews, 5 , Article 3. https://doi.org/10.1186/s13643-015-0173-5

Temple University Library (2021, June 11).  Systematic reviews & other review types . Retrieved July 8, 2021, from  https://guides.temple.edu/c.php?g=78618&p=3879604

Tricco, A. C., Lillie, E., Zarin, W., O'Brien, K., Colquhoun, H., Kastner, M., Levac, D., Ng, C., Sharpe, J. P., Wilson, K., Kenny, M., Warren, R., Wilson, C., Stelfox, H. T., & Straus, S. E. (2016). A scoping review on the conduct and reporting of scoping reviews.  BMC Medical Research Methodology ,  16 (1), 15.  https://doi.org/10.1186/s12874-016-0116-4

Click on a box below to go to information on that topic.

Further reading 

Bettany-Saltikov, J. (2016). How to do a systematic literature review in nursing: A step-by-step guide (2nd ed.). Open University Press.

Cranwell, M. (2021). A mixed-methods systematic review of transitions for caregivers of people living with dementia . SAGE. 

Muka, T., Glisic, M., Milic, J., Verhoog, S., Bohlius, J. Bramer, W., Chowdhury, R., & Franco, O. H .  (2020). A 24-step guide on how to design, conduct, and successfully publish a systematic review and meta-analysis in medical research.  European Journal of Epidemiology,   35 ,   49–60.  https://doi,org/10.1007/s10654-019-00576-5

Gough, D., Oliver, S., & Thomas, J. (Eds.). (2017). An introduction to systematic reviews  (2nd ed.). SAGE. 

Gough, D., Oliver, S., & Thomas, J. (Eds.). (2018). Systematic reviews and research . SAGE. 

Biondi-Zoccai, G. (Ed.). (2016). Umbrella reviews: Evidence synthesis with overviews of reviews and meta-epidemiologic studies . Springer. https://doi.org/10.1007/978-3-319-25655-9  

Holly, C., Salmond, S., & Saimbert, M. (2021). Comprehensive Systematic Review for Advanced Practice Nursing (3rd ed.). Springer. 

Higgins, J. P. T., Thomas, J., Chandler, J., Cumpston, M., Li, T., Page, M. J., & Welch, V. A. (Eds). (2021). Cochrane Handbook for Systematic Reviews of Interventions version 6.2. Cochrane. https://training.cochrane.org/handbook . 

Petticrew, M., & Roberts, H. (2006). Systematic reviews in the social sciences: A practical guide . John Wiley & Sons. 

Tawfik, G. M., Dila, K. A. S., Mohamed, M. Y. F., Tam, D. N. H., Kien, N. D., Ahmed, A. M., & Huy, N. T. (2019). A step by step guide for conducting a systematic review and meta-analysis with simulation data. Tropical Medicine Health, 47 , Article 46. https://doi.org/10.1186/s41182-019-0165-6

Tod, D. (2019). Conducting systematic reviews in sport, exercise, and physical activity . Palgrave Macmillan. 

Toronto, C. E., & Remington, R. (Eds.). (2020). A step-by-step guide to conducting an integrative review . Springer. https://doi.org/10.1007/978-3-030-37504-1

 Zawacki-Richter, O., Kerres, M., Bedenlier, S., Bond, M., & Buntins, K. (Eds.). (2020). Systematic Reviews in Educational Research: Methodology, Perspectives and Application . Springer. https://doi.org/10.1007/978-3-658-27602-7

• Next: Planning >>
• Last Updated: May 7, 2024 9:49 AM
• URL: https://aut.ac.nz.libguides.com/systematic_reviews

• - Google Chrome

Intended for healthcare professionals

• Access provided by Google Indexer
• My email alerts
• BMA member login
• Username * Password * Forgot your log in details? Need to activate BMA Member Log In Log in via OpenAthens Log in via your institution

Search form

• Advanced search
• Search responses
• Search blogs
• Guidance on...

Guidance on terminology, application, and reporting of citation searching: the TARCiS statement

• Related content
• Peer review
• Julian Hirt , research fellow and lecturer 1 2 3 ,
• Thomas Nordhausen , research fellow 4 ,
• Thomas Fuerst , medical information specialist 5 ,
• Hannah Ewald , medical information specialist 5 ,
• Christian Appenzeller-Herzog , medical information specialist 5
• on behalf of the TARCiS study group
• 1 Pragmatic Evidence Lab, Research Centre for Clinical Neuroimmunology and Neuroscience Basel, University Hospital Basel and University of Basel, Basel, Switzerland
• 2 Department of Health, Eastern Switzerland University of Applied Sciences, St Gallen, Switzerland
• 3 Department of Clinical Research, University Hospital Basel and University of Basel, Basel, Switzerland
• 4 Institute of Health and Nursing Science, Medical Faculty, Martin Luther University Halle-Wittenberg, Halle (Saale), Germany
• 5 University Medical Library, University of Basel, 4051 Basel, Switzerland
• Correspondence to: C Appenzeller-Herzog christian.appenzeller{at}unibas.ch
• Accepted 19 March 2024

Evidence syntheses adhering to systematic literature searching techniques are a cornerstone of evidence based healthcare. Beyond term based searching in electronic databases, citation searching is a prevalent search technique to identify relevant sources of evidence. However, for decades, citation searching methodology and terminology has not been standardised. An evidence guided, four round Delphi consensus study was conducted with 27 international methodological experts in order to develop the Terminology, Application, and Reporting of Citation Searching (TARCiS) statement. TARCiS comprises 10 specific recommendations, each with a rationale and explanation on when and how to conduct and report citation searching in the context of systematic literature searches. The statement also presents four research priorities, and it is hoped that systematic review teams are encouraged to incorporate TARCiS into standardised workflows.

Synthesising scientific evidence by looking at the citation relationships of a scientific record (ie, citation searching) was the underlying objective when the Science Citation Index, the antecedent of Web of Science, was introduced in 1963. 1 Although the availability of electronic citation indexes has increased, evidence syntheses in systematic reviews do not primarily rely on citation searching for literature retrieval but rather on search methods based on text and keywords. 2 When used in systematic review workflows, citation searching traditionally constitutes a supplementary search technique that builds on an initial set of references from the primary database search (seed references). 3

Citation searching is an umbrella term that entails various methods of citation based literature retrieval ( fig 1 ). Checking references cited by seed references, also known as backward citation searching, is the most prevalent and a mandatory step when conducting Cochrane reviews. 4 In forward citation searching, systematic reviewers can also assess the eligibility of articles that cite the seed references. Backward and forward citation searching are known as direct citation searching ( fig 1 ). They can be supplemented by indirect retrieval methods—namely, by co-citing citation searching (retrieving articles that share cited references with a seed reference) and co-cited citation searching (retrieving articles that share citing references with a seed reference).

Overview of citation searching methods. Direct (dark blue boxes) and indirect (light blue boxes) citation relationships of references are shown, relative to a seed reference; arrows denote the direction of citation (ie, source A citing target B); horizontal axis denotes time (ie, the chronology in which references were published relative to the seed reference). Visual examples of cited references (accessible via backward citation searching), citing references (accessible via forward citation searching), co-citing references (accessible via co-citing citation searching), and co-cited references (accessible via co-cited citation searching) are shown. Note that the total number of the co-citing and co-cited references of a seed reference far exceeds the number shown in the light blue boxes

• Download figure
• Open in new tab
• Download powerpoint

Citation searching can contribute substantially to evidence retrieval and can show similar or even superior effectiveness and efficiency compared with text and keyword based searches. An audit of the different search methods used in a systematic review of complex evidence, for instance, revealed that 44% of all included studies were identified by backward citation searching, and 7% by forward citation searching. In comparison, initial text and keyword searches accounted for only 25% of included studies. 5 For the scoping review that collected methodological studies as a foundation for the present work, these figures were 28% and 12% for backward and forward citation searching, respectively, compared with 52% for extensive primary database searching. 6

The conduct of systematic reviews is prominently guided by standard recommendations such as those in the Cochrane handbook, 4 whereas their reporting is standardised by the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) statement. 7 In contrast and despite its application by systematic reviewers for decades, standardised methodology and terminology for citation searching is not available. Of the three aspects on when to do citation searching, how to conduct citation searching, and how to report citation searching, limited guidance exists only for the third aspect in the PRISMA extension for reporting literature searches (PRISMA-S). 8 Unsurprisingly, methodological studies show considerable heterogeneity in terms of citation searching terminology and recommended best practices. 6 Even in a sample of Cochrane reviews, 13% did not use backward citation searching despite this being a mandatory step. 9 The lack of standardisation not only impairs the transparency, reproducibility, and comparability of systematic reviews, but might also reduce article recall that could affect pooled effect estimates, guidance, and clinical decision making. On the other hand, uninformed use of citation searching in contexts where it is less useful might cause undue workloads.

We systematically collected evidence on the use, benefit, and reporting of citation searching 6 and put it through a four round, online Delphi study. Together with the Terminology, Application, and Reporting of Citation Searching (TARCiS) study group, an international panel of methodological experts, we aimed to develop consensus for recommendations on when and how to conduct citation searching, and on how to report it, including a consensus set of citation searching terms. Furthermore, we framed research priorities for future methodological development of citation searching in the context of systematic literature searches.

Summary points

The TARCiS (Terminology, Application, and Reporting of Citation Searching) statement provides guidance in which contexts citation searching is likely to be beneficial for systematic reviewers

TARCiS comprises 10 specific recommendations on when and how to conduct citation searching and how to report it in the context of systematic literature searches, and also frames four research priorities

The statement will contribute to a unified terminology, systematic application, and transparent reporting of citation searching and support those who are conducting or assessing citation searching methods

To develop the TARCiS statement, a stepwise approach comprising a scoping review of the methodological literature (step 1; reported in detail in a separate publication 6 ) and a Delphi study (step 2; reported in this publication) was chosen. The methods were prespecified in two study protocols. 10 11 The complete process is shown in figure 2 .

Flow diagram of the development process of the TARCiS (Terminology, Application, and Reporting of Citation Searching) statement. Actions and outcomes of the development phases of the TARCiS statement are shown. Appendix 1 shows more detailed reporting of consensus scores

Step 1: Scoping review

We conducted a scoping review on the terminology that describes citation searching, the methods and tools used for citation searching, and its benefit. We considered methodological studies of any design that aimed to assess the role of citation searching, compared multiple citation searching methods, or compared technical uses of citation searching within health related topics. We searched five bibliographic databases, conducted backward and forward citation searches of eligible studies and pertinent reviews, and consulted librarians and information specialists for further eligible studies. The results were summarised by descriptive statistics and narratively. The detailed methods of the scoping review have been published elsewhere. 6 10

Step 2: Delphi study

To develop consensus on recommendations and research priorities as tentatively derived from the results of step 1, 6 we performed a multistage online Delphi study. Delphi refers to a structured process where collective knowledge from an expert panel is synthesised using a series of questionnaires, each one questionnaire adapted on the basis of the responses to a previous version. 12 13 14 We recruited an international panel of individuals experienced in conducting or reporting citation searching methods. For this, we invited authors of methodological studies, as identified in step 1, 6 and methodological experts from international systematic review organisations or from our professional networks by email to participate in the Delphi study.

The Delphi study comprised four prespecified rounds. 10 11 The first round was pretested by four non-study related academic affiliates. Each round covered four to five thematic parts (appendix 2; table 1 ). Briefly, part A dealt with the terminology framework to describe citation searching methods in eight domains (for details, refer to table 4 in Hirt et al 6 ). Part B contained pre-formulated recommendations on conduct and reporting of citation searching. Each recommendation was supported by a rationale and explanation text that were also subjected to collective consensus finding. Part C covered research priorities that were also derived from the scoping review. 6 Part D contained a free text field to collect general comments from the panellists. Part E was designed to collect sociodemographic information and was limited to Delphi round 1.

Data collection through four rounds of Delphi study to develop consensus on recommendations and research priorities of the TARCiS statement

• View inline

Non-participating panellists were recorded as non-participators for a given round. Panellists who missed all rounds were recorded as non-responders. Recommendations and research priorities that had not yet reached the prespecified consensus of at least 75% were refined for the subsequent Delphi round. These refinements were based on the panellists’ comments. In rare cases, when additional valid suggestions from panellists for reformulation of rationale or explanation texts were submitted, recommendations that already reached the agreement threshold were also adapted and forwarded to the next Delphi round. For more methodological details on the Delphi study, see table 1 and the published protocols. 10 11

Deviations from the Delphi study protocol

For round 3 of the Delphi, we had originally planned to formulate one recommendation for each of the eight terminology domains ( table 1 , see also description to part A above). Depending on the votes, however, this approach might have led to the selection of inconsistent terms (eg, backward citation searching v forward citation tracking). Hence, we decided to use the terms that received the most votes in Delphi round 2 to formulate four term sets, which were consistent across all eight domains. Secondly, instead of using SosciSurvey 15 as a survey tool, 8 we switched to the Unipark/Enterprise Feedback Suite survey, 16 which provided enhanced design and functional features. Thirdly, in addition to personalised emails (person based approach), we originally intended to recruit panellists using professional mailing lists and central requests to systematic review organisations (organisation based approach). 8 However, because we had already recruited sufficient panellists using the person based approach (including individuals who were affiliated with various systematic review organisations), we waived the organisation based approach.

We identified 47 methodological studies that assessed the use, benefit, and reporting of citation searching. In 45 studies (96%), the use of citation searching showed an added value. Thirty two studies (68%) analysed the impact of citation searching in one or more previous systematic reviews. Application, terminology, and reporting of citation searching were heterogeneous. Details on the results of the scoping review can be found elsewhere. 6

Recruitment and characteristics of panellists

Of 35 experts identified and contacted, 30 declared an interest in participating and were invited to Delphi round 1. Three (10%) of the 30 panellists were non-responders. Table 2 summarises the personal and professional characteristics of the 27 participating panellists.

Characteristics of 27 panellists* participating in the Delphi study to develop consensus on recommendations and research priorities of the TARCiS statement

TARCiS statement: final recommendations, rationale and explanations, and research priorities

Items for data collection through the four Delphi rounds in parts A-E are summarised in table 1 . The Delphi study started with 41 terms describing different aspects of citation searching, eight draft recommendations with rationale texts on the conduct and reporting of citation searching, and one research priority (appendix 1). After Delphi round 4, the finalised TARCiS statement comprised 10 recommendations with rationale and explanation texts and four research priorities that reached consensus scores between 83% and 100%. Figure 2 and appendix 1 show details on content and consensus scores in rounds 1-4. An overview of all 14 TARCiS items omitting rationale and explanation texts is presented in box 1 . A terminology and reporting item checklist based on TARCiS recommendations 1 and 10 is available in appendix 3 and on the TARCiS website. 17

TARCiS statement

Recommendations on terminology, conduct, and reporting of citation searching.

The following terminology should be used to describe search methods that exploit citation relationships:

“Citation searching” as an umbrella term.

“Backward citation searching” to describe the sub-method retrieving and screening cited references.

“Reference list checking” to describe the sub-method retrieving and screening cited references by manually reviewing reference lists.

“Forward citation searching” to describe the sub-method retrieving and screening citing references.

“Co-cited citation searching” to describe the sub-method retrieving and screening co-cited references.

“Co-citing citation searching” to describe the sub-method retrieving and screening co-citing references.

“Iterative citation searching” to describe one or more repetition(s) of a search method that exploits citation relationships.

“Seed references” to describe relevant articles that are known beforehand and used as a starting point for any citation search.

For systematic search topics that are difficult to search for, backward and forward citation searching should be seriously considered as supplementary search techniques.

For systematic search topics that are easier to search for and addressed by a highly sensitive search, backward and forward citation searching are not explicitly recommended as supplementary search techniques. Reference list checking of included records can be used to confirm the sensitivity of the search strategy.

Backward and forward citation searching as supplementary search techniques should be based on all included records of the primary search (ie, all records that meet the inclusion criteria of the review after full text screening of the primary search results). Occasionally, it can be justified to deviate from this recommendation and either use further pertinent records as additional seed references or only a defined sample of the included records.

Backward citation searching should ideally be conducted by screening the titles and abstracts of the seed references as provided by a citation index. Screening titles as provided when checking reference lists of the seed references can still be performed.

Using the combined coverage of two citation indexes for citation searching to achieve more extensive coverage should be considered if access is available. This combination is especially meaningful if seed references cannot be found in one index and reference lists were not checked.

Before screening, the results of supplementary backward and forward citation searching should be deduplicated.

If citation searching finds additional eligible records, another iteration of citation searching should be considered using these records as new seed references.

Standalone citation searching should not be used for literature searches that aim at completeness of recall.

Reporting of citation searching should clearly state:

the seed references (along with a justification should the seed references differ from the set of included records from the results of the primary database search),

the directionality of searching (backward, forward, co-cited, co-citing),

the date(s) of searching (which might differ between rounds of iterative citation searching) (not applicable for reference list checking),

the number of citation searching iterations (and possibly the reason for stopping if the last iteration still retrieved additional eligible records),

all citation indexes searched (eg, Lens.org, Google Scholar, Scopus, citation indexes in Web of Science) and, if applicable, the tools that were used to access them (eg, Publish or Perish, citationchaser),

if applicable, information about the deduplication process (eg, manual/automated, the software or tool used),

the method of screening (ie, state whether the records were screened in the same way as the primary search results or, if not, describe the alternative method used), and

the number of citation searching results in the right column box of the PRISMA 2020 flow diagram for new or updated systematic reviews that included searches of databases, registers, and other sources .

Research priorities

The effectiveness, applicability, and conduct of indirect citation searching methods as supplementary search methods in systematic reviewing require further research (including retrieval of additional unique references, their relevance for the review and prioritisation of results).

Further research is needed to assess the value of citation searching. Potential research topics could be:

influence of citation searching on results and conclusions of systematic evidence syntheses,

topics or at least determinants of topics where citation searching likely/not likely has additional value, or

economic evaluation of citation searching to assess the cost and time of conducting citation searching in relation to its benefit.

Further research is needed to assess the best way to perform citation searching. Potential research topics could be:

optimal selection of seed references,

optimal use of indexes and tools and their combination to conduct citation searching,

methods and tools for deduplication of citation searching results,

subjective influences on citation searching (eg, experience of researcher, prevention of mistakes), or

reproducibility of citation searching.

Further research is needed to reproduce existing studies: Any recommendations in this Delphi that are based on only 1-2 studies require reproduction of these studies in the form of larger, prospectively planned studies that grade the evidence for each recommendation and propose additional research where the grade of evidence is weak.

The TARCiS checklist for terminology and reporting of citation searching is available for download. 17

PRISMA=Preferred Reporting Items for Systematic reviews and Meta-Analyses; TARCiS=Terminology, Application, and Reporting of Citation Searching.

Recommendation 1

Rationale and explanation supporting recommendation 1.

As compiled in a recent scoping review, 6 the reporting of citation searching methods is frequently unclear and far from being standardised. For example, “citation searching,” “snowballing,” or “co-citation searching” are sometimes used as methodological umbrella terms but also to denote a specific method such as backward or forward citation searching. 6 For clarity, standardised vocabulary is needed. The set of terms brought forward in this recommendation is consistent in itself as well as with the terminology used in PRISMA-S and PRISMA 2020 guidelines 8 18 and hence well suited for uniform reporting of citation searching.

Recommendation 2

Rationale and explanation supporting recommendation 2:.

Evidence indicates that the ability of citation searching as a supplementary search technique to find additional unique records in a systematic literature search varies between reviews. 6 Searches for particular study designs (qualitative, mixed method, observational, prognostic, or diagnostic test studies) or health science topics such as non-pharmacological, non-clinical, public health, policy making, service delivery, or alternative medicine have been linked with effective supplementary citation searching. 19 20 21 22 The underlying reasons include poor transferability to text based searching owing to poor conceptual clarity, inconsistent terminology, or vocabulary overlaps with unrelated topics. 5 The ability of citation searching to find any publication type including unpublished or grey literature or literature that is not indexed in major databases (eg, concerning a developing country) might also be relevant. 23 However, a clear categorisation of topics that are difficult to search for is currently not possible and it remains for the review authors themselves to judge whether their review topic is likely to fall into this category.

For people conducting the search who have difficulty assessing whether the topic is difficult or easier to search for, we recommend that they opt for citation searching or consult an experienced information specialist. 24 If the search strategy does not exhaustively capture the topic, backward and forward citation searching might compensate for some of the potential loss of information.

Recommendation 3

Rationale and explanation supporting recommendation 3.

Evidence indicates that the ability of citation searching as a supplementary search technique to find additional unique references in a systematic literature search varies between reviews. 6 Searches for clearly defined clinical interventions as part of PICO (participant, intervention, comparison, outcome) questions have been linked with less effective supplementary citation searching, especially when the search strategies are sensitive and conducted in several databases. However, a clear categorisation of topics that are easier to search for is currently not possible, and it remains for the review authors themselves to judge whether their review topic is likely to fall into this category.

By checking reference lists within the full texts of seed references, review authors can test the sensitivity of their primary search strategy (ie, electronic database search). 25 If no additional relevant, unique studies are found, the primary search might have been sensitive enough. If additional relevant, unique studies are found, these could indicate that the primary search was not sensitive enough.

For individuals conducting the search who have difficulty assessing whether the topic is difficult or easier to search for, we recommend that they opt for citation searching or consult an experienced information specialist. 24 If, for whatever reason, the search strategy does not exhaustively capture the topic, backward and forward citation searching could compensate for some of the potential loss of information.

Recommendation 4

Rationale and explanation supporting recommendation 4.

The more seed references used, the better the chance that citation searching finds additional relevant unique records. While using only a sample of the included records as seed references might be enough, there is currently no evidence that could help decide how many seeds are needed or how to decide which might perform better. Hence, we recommend using all the records that meet the inclusion criteria of the review after full text screening of the primary database search results.

However, review authors could deviate from this recommendation if they deal with a very small or large number of included records. A very small number of included records might not yield additional relevant records or only have limited value. In this case, review authors could use further records as seed references for citation searching (eg, systematic reviews on the topic that were flagged during the screening phase). 26 A very large number of included records could lead to too many records to screen. In this case, review authors might use a selected sample of included records as seed references for citation searching. In the event of such deviation, authors should describe their rationale and sampling method (eg, random sample).

Recommendation 5

Rationale and explanation supporting recommendation 5.

Citation searching workflows encompass two consecutive steps: retrieval of records and screening of retrieved records for eligibility. When using an electronic citation index for citation searching, retrieval and screening are usually separated. While forward citation searching requires a citation index, backward citation searching can also be performed by manually checking the reference lists of the seed references. Reference list checking is sometimes part of an established workflow, for example, during the eligibility assessment of the full text record or during data extraction. 25 Merging these two steps allows researchers to know the context in which a reference was used and that all references can be screened. However, reference list checking has three disadvantages:

The retrieval and screening phases are no longer separated, which makes reporting of the methods or results difficult and unclear

Citations from reference list checking cannot be deduplicated against each other or against the primary search results, which could add an unnecessarily high workload (see recommendation 7)

Eligibility assessments are restricted to the titles (instead of titles and abstracts) which could lead to relevant records being overlooked due to uninformative titles mentioned in vague contexts.

In recent years, online citation searching options via citation indexes or free to access citation searching tools have become more readily available leading to faster and easier procedures. 27 28 29 30 More and even better tools to facilitate this workflow are expected in the future. Combining citation searching via citation indexes with automated deduplication (free online tools available) 31 32 33 makes this recommendation feasible. A caveat is that a search in one citation index will in most cases fail to retrieve all the cited references. 34 35 Thus, references to some documents (such as websites, registry entries, or grey literature) that are less likely to be indexed in databases might only be retrievable by checking reference lists or only in some citation indexes. 3

Recommendation 6

Rationale and explanation supporting recommendation 6.

A single citation index or citation analysis tool might not cover all seed references and is likely to not find all the citing and cited literature. Citation indexes do not offer 100% coverage because some references are currently not indexed in one or several citation index(es) 36 and because of data quality problems. 37 Evidence indicates that when using more than one citation index for citation searching, the results of the different indexes can complement each other. 38 39 40 Thus, retrieval of backward and forward citation searching results from more than one citation index or citation analysis tool (eg, Lens.org via citationchaser, Scopus, citation indexes in Web of Science) followed by deduplication (see recommendation 7) can increase the sensitivity of citation searching. It is similar to the complementary effect of using multiple electronic databases for the primary database search, which is the preferred method in systematic search workflows. 4 In recent years, online citation searching options have increased and many open access tools make rapid electronic citation searching universally accessible. 27 28 29 30

Recommendation 7

Rationale and explanation supporting recommendation 7.

The concept of citation searching as a supplementary search method relies on the notion that reference lists and cited-by lists of eligible references are topically related to these references. 6 This topical relation implies a considerable degree of overlap within these lists leading to several duplicates. Furthermore, the overlap likely also extends to the results of the primary database search that was performed on the same topic. Based on these considerations and on the fact that the results of the primary database search have already been screened for eligibility, the screening load of citation searching results can be substantially cut by removing those references that have already been screened for eligibility (deduplication against the primary database search) and those references that appear as duplicates during citation searching. 34 Depending on the method of deduplication, this procedure can be done in one go.

While deduplication can be conducted manually, standard bibliographic management software and specialised tools currently provide automated deduplication solutions, allowing for easier and faster processing. 34 41 42 If citation searching leads to only very few results, omission of the deduplication step can be considered to save time and administrative effort.

Recommendation 8

Rationale and explanation supporting recommendation 8.

Citation searching methods can be conducted over one or more iterations, a process that we refer to as iterative citation searching. 43 The first iteration is based on the original seed references (see recommendation 4). If eligibility screening of the results of this first iteration leads to the inclusion of further eligible records, these records serve as new seed references for the second iteration, and so forth. Evidence indicates that conducting iterative citation searching can contribute to the identification of more eligible records. 6 43 44 45

Iterations beyond the first round of citation searching require additional time and effort and could interrupt the ongoing review process, so the decision in favour of or against further iterations should be guided by an informal cost-benefit assessment. Relevant factors to be assessed include the review topic (difficult or easier to search for), sensitivity of the primary search, aim for completeness of the literature search, and the estimated potential benefit of the iteration(s) (eg, based on the number or percentage of included records found with the previous citation searching iteration).

Review authors should report the number of iterations and possibly the reason for stopping if the last iteration still retrieved additional eligible records. Furthermore, stating “citation searching was done on all included records” can lead to confusion. Most authors might mean all records were included after full text screening of the primary search results. But strictly speaking, “all included records” also includes those records retrieved via citation searching. The second interpretation implies that iterative citation searching is required until the last iteration leads to no further identification of eligible records.

As outlined in the rationale of recommendation 7, results of citation searching iterations can be deduplicated against all previously retrieved records to reduce the screening load.

Recommendation 9

Rationale and explanation supporting recommendation 9.

We refer to standalone citation searching when any form of citation searching is used as the primary search method without extensive prior database searching. 6 This is contrary to citation searching as a supplementary search method to a primary database search. Seed references for standalone citation searching could, for example, be records from researchers’ personal collections or retrieved from less sensitive literature searches. Standalone citation searching can be based on a broad set of seed references. It can comprise backward and forward citation searching as well as indirect methods that collect co-citing and co-cited references.

When study authors have replicated published systematic reviews with standalone citation searching, they have mostly missed literature that was included in the systematic review. 27 46 47 48 Since search methods for systematic reviews and scoping reviews should aim at completeness of recall, standalone citation searching is not a suitable method for these types of literature review.

Recommendation 10

Rationale and explanation supporting recommendation 10.

Relevant guidance for researchers conducting citation searching in systematic literature searching can be found in item 5 of PRISMA-S. 8 Accordingly, required reporting items are the directionality of citation searching (examination of cited or citing references), methods and resources used for citation searching (bibliographies in full text articles or citation indexes), and the seed references that citation searching was performed on. 8 Additional information for the reporting of citation searching can be found in PRISMA-S items 1 (database name), 13 (dates of searches), and 16 (deduplication). 8 Although PRISMA-S can be seen as the minimum reporting standard for citation searching as a supplementary search technique, other important elements that emerged from our scoping review 6 need to be reported to achieve full transparency or reproducibility. These elements are listed in recommendation 10 as a supplement to PRISMA-S to comprehensively guide the reporting of supplementary citation searching in systematic literature searching.

Concerning reporting of citation searching results in the PRISMA 2020 flow diagram, 49 two variants are possible: reporting only those records that are additional to the primary search results after deduplication, or reporting all retrieved records followed by insertion of an additional box where the number of deduplicated records is reported.

Researchers should be aware that the detail of the citation searching methods do not have to be reported in the main methods of a study. Detailed search information can be provided in an appendix or an online public data repository.

Examples of good reporting

“As supplementary search methods, we performed . . . direct forward and backward CT [citation searching] of included studies and pertinent review articles that were flagged during the screening of search results (on February 10, 2021). For forward CT, we used Scopus, Web of Science [core collection as provided by the University of Basel; Editions = SCI-EXPANDED, SSCI, A&HCI, CPCI-S, CPCI-SSH, BKCI-S, BKCI-SSH, ESCI, CCR-EXPANDED, IC], and Google Scholar. For backward CT, we used Scopus and, if seed references were not indexed in Scopus, we manually extracted the seed references’ reference list. We iteratively repeated forward and backward CT on newly identified eligible references until no further eligible references or pertinent reviews could be identified (three iterations; the last iteration on May 5, 2021).” 6

“To supplement the database searches, we performed a forward (citing) and backwards (cited) citation analysis on 2 August 2022 using SpiderCite ( https://sr-accelerator.com/#/spidercite ).” 50

“Reference lists of any included studies and retrieved relevant SRs [systematic reviews] published in the last five years were checked for any eligible studies that might have been missed by the database searches.” 51

Research priority 1

Rationale and explanation supporting research priority 1.

Indirect citation searching involves the collection and screening for eligibility of records that share references in their bibliography or citations with one of the seed references (ie, co-citing or co-cited references). 10 Indirect citation searching typically retrieves a large volume of records to be screened. 46 48 Therefore, prioritisation algorithms for the screening of records and cut-off thresholds that might discriminate between potentially relevant and non-relevant records have been proposed with the aim to reduce the workload of eligibility screening. 27 47 The methodological studies that have pioneered indirect citation searching methods for health related topics have so far exclusively focused on standalone citation searching. 6 It is currently unclear whether the added workload and resources for searching and screening warrant indirect citation searching methods as supplementary search techniques in systematic reviews of any type (qualitative or quantitative studies, difficult or easier topics to search for).

Research priority 2

Research priority 3, research priority 4, tarcis recommendations and research priorities.

In keeping with our study aims, the TARCiS recommendations cover three aspects of citation searching in the context of systematic literature searches. They offer guidance regarding when to conduct, how to conduct, and how to report citation searching. The strength of each recommendation reflects the panellists’ assessment of the strength of evidence to support them.

In systematic evidence syntheses, citation searching techniques can be used to fill gaps in the results of primary database searches, but their application is not universally indicated. TARCiS recommendations 2 and 3 provide critical assistance in cost-benefit considerations (ie, whether a systematic search is likely to benefit from the use of citation searching). Systematic searchers of defined pharmaceutical interventions, for instance, might take from this guidance that they can skip citation searching because their primary database search might already allow for high sensitivity at reasonable specificity and expedite other supplementary search techniques, such as clinical trial registry searching. 52 Accordingly, TARCiS does not recommend the use of citation searching in easier-to-search-for topics, but—as formulated in research priority 2—more research is needed to more reliably discriminate between topics that are easier to search for and those that are difficult to search for.

TARCiS recommendations 4-8 comprise guidance for technical aspects of citation searching. This guidance includes the selection of seed references, use of electronic citation indexes, deduplication, and iterative citation searching. While composing these recommendations, the TARCiS study group has considered that individual workflows must be framed in line with institutional licenses for subscription only databases and software. For illustration, one such workflow that is based on the licenses as provided by the University of Basel was deposited as an online video. 53

Concerning guidance for reporting of citation searching, we developed a consensus terminology set for citation searching methods (TARCiS recommendation 1) as well as a recommendation for preferred reporting items for citation searching (TARCiS recommendation 10), along with a downloadable checklist. 17 TARCiS recommendation 10 increases the reporting standards provided by PRISMA-S 8 by dealing with the reporting of citation searching iterations, software tools that facilitate citation searching via a citation index, and the method of eligibility screening. Furthermore, TARCiS recommendation 10 standardises the reporting of citation searching results in the PRISMA 2020 flow diagram. We suggest that systematic reviewers, methodologists, journal reviewers, and editors use the TARCiS statement terminology and reporting checklist 17 (appendix 3) as an additional checklist until future work by the PRISMA-S study group produces an updated reporting guideline that renders the TARCiS checklist obsolete.

Dissemination

TARCiS is intended to be used by researchers, systematic reviewers, information specialists, librarians, editors, peer reviewers, and others who are conducting citation searching or assessing citation searching methods. To enhance dissemination among these stakeholders, we aim to provide additional open access publications in scientific and non-scientific journals relevant in the field of information retrieval and evidence syntheses.

We have launched a TARCiS website ( https://tarcis.unibas.ch/ ) and plan to disseminate the TARCiS terminology and reporting checklist 17 on various platforms, including EQUATOR. We aim to make the TARCiS statement available via the Library of Guidance for Health Scientists (LIGHTS), a living database for methods guidance 54 ; the Systematic Review Toolbox, an online catalogue of tools for evidence syntheses 55 ; and ResearchGate, a social scientific network to share and discuss publications.

We will also share the TARCiS terminology and reporting checklist 17 with editors of journals relevant in the field of information retrieval and evidence syntheses to request for inclusion in their instructions for authors and raise awareness of this topic. We hope that this effort will guide authors and peer reviewers to use TARCiS and assist their conduct, reporting, and evaluation of citation searching. We will also share the TARCiS statement with primary teaching stakeholders in evidence syntheses and systematic literature searching (eg, York Health Economics Consortium, RefHunter, Cochrane, Joanna Briggs Institute, and the Campbell Collaboration) and suggest its inclusion in future editions of their handbooks. We will present and discuss the TARCiS statement on international conferences and share our publications and presentations via relevant mailing lists and newsletters, X (formerly Twitter), and LinkedIn.

Limitations

A limitation of the TARCiS statement is that, despite the expectation and intent to recruit panellists from all parts of the world, their locations were limited to Australia, Europe, and North America. In addition, only a few panellists were recruited from countries where English was not the dominant language. Furthermore, both the evidence collected in our scoping review and the participating panellists are primarily involved with health related research. These considerations might reduce the generalisability of our recommendations and research priorities in other countries, languages, and research areas.

Conclusions

TARCiS comprises 10 specific recommendations on when and how to conduct citation searching and how to report it in the context of systematic literature searches. Furthermore, TARCiS frames four research priorities. It will contribute to a unified terminology, systematic application, and transparent reporting of citation searching and support researchers, systematic reviewers, information specialists, librarians, editors, peer reviewers, and others who are conducting or assessing citation searching methods. In addition, TARCiS might inform future methodological research on the topic. We encourage systematic review teams to incorporate TARCiS into their standardised workflows.

Ethics statements

Ethical approval.

This study is based on published information and uses surveys of topical experts and therefore did not fall under the regulations of the Swiss Human Research Act, and we did not need to apply for ethical approval according to Swiss law. Data protection and privacy issues for the survey are outlined in the main text.

Data availability statement

The survey sheets and questionnaires used for this study are included in the supplementary content. Data generated and analysed during this study (except for sociodemographic information) are available on the Open Science Framework ( https://osf.io/y7kh3 ).

Acknowledgments

We would like to dedicate this work to Cecile Janssens, who died soon after agreeing to join our Delphi panel. We thank Jill Hayden (Dalhousie University) and Claire Duddy (UK) for participating in our Delphi panel; and Christian Buhtz (Martin Luther University Halle-Wittenberg), Jasmin Eppel-Meichlinger (Karl Landsteiner University of Health Sciences), Tania Rivero (University of Berne), and Monika Wechsler (University of Basel) for participating in the pretest of the Delphi survey.

TARCiS study group: Alison Avenell (University of Aberdeen, UK), Alison Bethel (University of Exeter, UK), Andrew Booth (University of Sheffield, UK; and University of Limerick, Ireland), Christopher Carroll (University of Sheffield, UK), Justin Clark (Bond University, Australia), Julie Glanville (Glanville.info, UK ), Su Golder (University of York, UK), Elke Hausner (Institute for Quality and Efficiency in Health Care, Germany), Tanya Horsley (Royal College of Physicians and Surgeons of Canada, Canada), David Kaunelis (Canadian Agency for Drugs and Technologies in Health, Canada), Shona Kirtley (University of Oxford, UK), Irma Klerings (Donau University, Austria), Jonathan Koffel (USA), Paul Levay (National Institute for Health and Care Excellence, UK), Kathrine McCain (Drexel University, USA), Maria-Inti Metzendorf (Heinrich-Heine University Duesseldorf, Germany), David Moher (University of Ottawa, Canada), Linda Murphy (University of California at Irvine, USA), Melissa Rethlefsen (University of New Mexico, USA), Amy Riegelman (University of Minnesota, USA), Morwenna Rogers (University of Exeter, UK), Margaret J Sampson (Children’s Hospital of Eastern Ontario, Canada), Jodi Schneider (University of Illinois at Urbana-Champaign, USA), Terena Solomons (Curtin University, Australia), Alison Weightman (Cardiff University, UK)

Contributors: All authors made substantial contributions to conception and design, or acquisition of data, or analysis and interpretation of data; drafted the article or revised it critically for important intellectual content; and approved the final version to be published. JH, TN, TF, HE, and CA-H had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. JH, TN, TF, HE, and CA-H contributed to the study concept and methodology; acquisition, analysis, interpretation, validation, and visualisation of data; and critical revision of the manuscript for important intellectual content. JH, HE, and CA-H conducted the statistical analysis. JH and CA-H drafted the manuscript; provided administrative, technical, and material support; and supervised the study. The TARCiS study group authors are the Delphi panellists who were involved in Delphi rounds 1-4; they received the final manuscript draft for critical revision, important intellectual input, and approval for publication. CA-H is the guarantor for the study. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.

Funding: The authors did not receive a specific grant for this study.

Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/disclosure-of-interest/ and declare: no specific support for the submitted work. CA-H received payments to his institution for a citation searching workshop by the University of Applied Sciences Northwestern Switzerland. JH received consulting fees from Medical University Brandenburg and payments for lecturing from the University of Applied Sciences Northwestern Switzerland, Catholic University of Applied Sciences, and Netzwerk Fachbibliotheken Gesundheit. From the TARCiS study group: JS received support from Alfred P Sloan Foundation; was funded by the US National Institutes of Health, National Science Foundation, US Office of Research Integrity, United States Institute of Museum and Library Services, and University of Illinois Urbana-Champaign; received book royalties from Morgan and Claypool; received consulting fees or honorariums from the European Commission, Jump ARCHES, NSF, and the Medical Library Association; received travel support by UIUC; contributes to the CREC (Communication of Retractions, Removals, and Expressions of Concern) Working Group; has non-financial associations with Crossref, COPE (Committee on Publication Ethics), the International Association of Scientific, Technical and Medical Publishers, the National Information Standards Organisation, and the Center for Scientific Integrity (parent organisation of Retraction Watch); and declares the National Information Standards Organisation as a subawardee on her Alfred P Sloan Foundation grant G-2022-19409. JG received payments for lecturing by York Health Economics Consortium. MJS received consulting fees at the Canadian Agency for Drugs and Technologies in Health and National Academy of Medicine (formerly Institute of Medicine) and for lecturing and support for attending a meeting at Institute for Quality and Efficiency in Health Care; and has a leadership role as secretary of the Ottawa Valley Health Library Association. AW received payments to her institution for a citation analysis workshop run via York Health Economics Consortium. SK declares non-financial interests as a member of the UK EQUATOR Centre and a coauthor of the PRISMA-S reporting guideline and was funded by Cancer Research UK (grant C49297/A27294); the current work was unrelated to this funding. PL is an employee of the National Institute for Health and Care Excellence. MR received payments by the Medical Library Association and declares non-financial interests as a member of the PhD programme affiliated with BMJ Publishing Group. ABo is a co-convenor of the Cochrane Qualitative and Implementation Methods Group and has authored methodological guidance on literature searching. All the other authors have no competing interests to disclose.

Provenance and peer review: Not commissioned; externally peer reviewed.

• Varley-Campbell J ,
• Britten N ,
• ↵ Higgins J, Thomas J, Chandler J, et al. Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022): Cochrane; 2022. www.training.cochrane.org/handbook .
• Greenhalgh T ,
• Nordhausen T ,
• Appenzeller-Herzog C ,
• Bossuyt PM ,
• Rethlefsen ML ,
• Kirtley S ,
• Waffenschmidt S ,
• PRISMA-S Group
• Briscoe S ,
• ↵ Hirt J, Nordhausen T, Fuerst T, et al. Internal DELPHI Protocol. 2023. https://osf.io/4nh25 .
• Radbruch L ,
• Brearley SG
• Pawlowski SD
• Schulz KF ,
• ↵ Leiner DJ. SoSci Survey (Version 3.1.06) [Computer software]. 2019. https://www.soscisurvey.de .
• ↵ Gmb HTX. Unipark: 2023. https://www.unipark.com/en .
• ↵ Hirt J, Nordhausen T, Fuerst T, et al. TARCiS terminology and reporting item checklist. 2023. https://bit.ly/tarcispdf .
• McKenzie JE ,
• Preston L ,
• Carroll C ,
• Gardois P ,
• Paisley S ,
• Kaltenthaler E
• Lasda Bergman EM
• Haddaway NR ,
• Collins AM ,
• Coughlin D ,
• Livingston EH
• Horsley T ,
• Dingwall O ,
• Westphal A ,
• Kriston L ,
• Hölzel LP ,
• von Wolff A
• Janssens ACJW ,
• Brockman JE ,
• Grainger MJ ,
• Smalheiser NR ,
• Schneider J ,
• Torvik VI ,
• Fragnito DP ,
• Pallath A ,
• Ouzzani M ,
• Hammady H ,
• Fedorowicz Z ,
• Elmagarmid A
• ↵ Zotero Version 6. 2023. https://www.zotero.org:443 .
• Glasziou P ,
• Del Mar C ,
• Bannach-Brown A ,
• Stehlik P ,
• ↵ ClarivateAnalytics. Cited Reference Search. 2023. https://webofscience.help.clarivate.com/en-us/Content/cited-reference-search.htm .
• Falagas ME ,
• Pitsouni EI ,
• Malietzis GA ,
• Franceschini F ,
• Maisano D ,
• Mastrogiacomo L
• Ainsworth N ,
• Rodriguez-Lopez R
• Borissov N ,
• Lines RLJ ,
• Gucciardi DF ,
• ↵ Janssens AC. Updating systematic reviews and meta-analyses, the easy way: 2021. https://cecilejanssens.medium.com/updating-systematic-reviews-and-meta-analyses-the-easy-way-cbb2e23b48b9 .
• Kleijnen J ,
• Knipschild P
• Janssens AC ,
• Nightingale R ,
• Fautrel B ,
• Patterson J ,
• Hunter KE ,
• Webster AC ,
• ↵ Hirt J, Nordhausen T, Fuerst T, et al. Citation searching in multiple citation indexes. 2023. https://osf.io/jaeu5 .
• Schönenberger CM ,
• ↵ Marshall C, Sutton A, O’Keefe H, et al. The Systematic Review Toolbox: 2022. http://www.systematicreviewtools.com/ .

Log in using your username and password

• Search More Search for this keyword Advanced search
• Latest content
• Current issue
• Hosted content
• BMJ Journals More You are viewing from: Google Indexer

You are here

• Online First
• Inappropriate use of proton pump inhibitors in clinical practice globally: a systematic review and meta-analysis
• Article Text
• Article info
• Citation Tools
• Rapid Responses
• Article metrics

• http://orcid.org/0000-0002-5111-7861 Amit K Dutta 1 ,
• http://orcid.org/0000-0003-2472-3409 Vishal Sharma 2 ,
• Abhinav Jain 3 ,
• Anshuman Elhence 4 ,
• Manas K Panigrahi 5 ,
• Srikant Mohta 6 ,
• Richard Kirubakaran 7 ,
• Mathew Philip 8 ,
• http://orcid.org/0000-0003-1700-7543 Mahesh Goenka 9 ,
• Shobna Bhatia 10 ,
• http://orcid.org/0000-0002-9435-3557 Usha Dutta 2 ,
• D Nageshwar Reddy 11 ,
• Rakesh Kochhar 12 ,
• http://orcid.org/0000-0002-1305-189X Govind K Makharia 4
• 1 Gastroenterology , Christian Medical College and Hospital Vellore , Vellore , India
• 2 Gastroenterology , Post Graduate Institute of Medical Education and Research , Chandigarh , India
• 3 Gastroenterology , Gastro 1 Hospital , Ahmedabad , India
• 4 Gastroenterology and Human Nutrition , All India Institute of Medical Sciences , New Delhi , India
• 5 Gastroenterology , All India Institute of Medical Sciences - Bhubaneswar , Bhubaneswar , India
• 6 Department of Gastroenterology , Narayana Superspeciality Hospital , Kolkata , India
• 7 Center of Biostatistics and Evidence Based Medicine , Vellore , India
• 8 Lisie Hospital , Cochin , India
• 9 Apollo Gleneagles Hospital , Kolkata , India
• 10 Gastroenterology , National Institute of Medical Science , Jaipur , India
• 11 Asian Institute of Gastroenterology , Hyderabad , India
• 12 Gastroenterology , Paras Hospitals, Panchkula , Chandigarh , India
• Correspondence to Dr Amit K Dutta, Gastroenterology, Christian Medical College and Hospital Vellore, Vellore, Tamil Nadu, India; akdutta1995{at}gmail.com

https://doi.org/10.1136/gutjnl-2024-332154

Statistics from Altmetric.com

Request permissions.

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

• PROTON PUMP INHIBITION
• META-ANALYSIS

We read with interest the population-based cohort studies by Abrahami et al on proton pump inhibitors (PPI) and the risk of gastric and colon cancers. 1 2 PPI are used at all levels of healthcare and across different subspecialties for various indications. 3 4 A recent systematic review on the global trends and practices of PPI recognised 28 million PPI users from 23 countries, suggesting that 23.4% of the adults were using PPI. 5 Inappropriate use of PPI appears to be frequent, although there is a lack of compiled information on the prevalence of inappropriate overuse of PPI. Hence, we conducted a systematic review and meta-analysis on the inappropriate overuse of PPI globally.

Supplemental material

Overall, 79 studies, including 20 050 patients, reported on the inappropriate overuse of PPI and were included in this meta-analysis. The pooled proportion of inappropriate overuse of PPI was 0.60 (95% CI 0.55 to 0.65, I 2 97%, figure 1 ). The proportion of inappropriate overuse by dose was 0.17 (0.08 to 0.33) and by duration of use was 0.17 (0.07 to 0.35). Subgroup analysis was done to assess for heterogeneity ( figure 2A ). No significant differences in the pooled proportion of inappropriate overuse were noted based on the study design, setting (inpatient or outpatient), data source, human development index of the country, indication for use, sample size estimation, year of publication and study quality. However, regional differences were noted (p<0.01): Australia—40%, North America—56%, Europe—61%, Asia—62% and Africa—91% ( figure 2B ). The quality of studies was good in 27.8%, fair in 62.03% and low in 10.12%. 6

• Download figure
• Open in new tab
• Download powerpoint

Forest plot showing inappropriate overuse of proton pump inhibitors.

(A) Subgroup analysis of inappropriate overuse of proton pump inhibitors (PPI). (B) Prevalence of inappropriate overuse of PPI across different countries of the world. NA, data not available.

This is the first systematic review and meta-analysis on global prescribing inappropriateness of PPI. The results of this meta-analysis are concerning and suggest that about 60% of PPI prescriptions in clinical practice do not have a valid indication. The overuse of PPI appears to be a global problem and across all age groups including geriatric subjects (63%). Overprescription increases the patient’s cost, pill burden and risk of adverse effects. 7–9 The heterogeneity in the outcome data persisted after subgroup analysis. Hence, this may be inherent to the practice of PPI use rather than related to factors such as study design, setting or study quality.

Several factors (both physician and patient-related) may contribute to the high magnitude of PPI overuse. These include a long list of indications for use, availability of the drug ‘over the counter’, an exaggerated sense of safety, and lack of awareness about the correct indications, dose and duration of therapy. A recently published guideline makes detailed recommendations on the accepted indications for the use of PPI, including the dose and duration, and further such documents may help to promote its rational use. 3 Overall, there is a need for urgent adoption of PPI stewardship practices, as is done for antibiotics. Apart from avoiding prescription when there is no indication, effective deprescription strategies are also required. 10 We hope the result of the present systematic review and meta-analysis will create awareness about the current situation and translate into a change in clinical practice globally.

Ethics statements

Patient consent for publication.

Not applicable.

Ethics approval

• Abrahami D ,
• McDonald EG ,
• Schnitzer ME , et al
• Jearth V , et al
• Malfertheiner P ,
• Megraud F ,
• Rokkas T , et al
• Shanika LGT ,
• Reynolds A ,
• Pattison S , et al
• O’Connell D , et al
• Choudhury A ,
• Gillis KA ,
• Lees JS , et al
• Paynter S , et al
• Targownik LE ,
• Fisher DA ,

Supplementary materials

Supplementary data.

This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

• Data supplement 1

X @drvishal82

Contributors AKD: concept, study design, data acquisition and interpretation, drafting the manuscript and approval of the manuscript. VS: study design, data acquisition, analysis and interpretation, drafting the manuscript and approval of the manuscript. AJ, AE, MKP, SM: data acquisition and interpretation, critical revision of the manuscript, and approval of the manuscript. RK: study design, data analysis and interpretation, critical revision of the manuscript and approval of the manuscript. MP, MG, SB, UD, DNR, RK: data interpretation, critical revision of the manuscript and approval of the manuscript. GKM: concept, study design, data interpretation, drafting the manuscript, critical revision and approval of the manuscript.

Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

Competing interests None declared.

Provenance and peer review Not commissioned; internally peer reviewed.

Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

Read the full text or download the PDF: