what is inclusion and exclusion criteria in literature review

Systematic Reviews

• The Research Question
• Inclusion and Exclusion Criteria
• Original Studies
• Translating
• Deduplication
• Project Management Tools
• Useful Resources
• What is not a systematic review?

Inclusion/Exclusion Criteria

Inclusion criteria.

Identify the criteria that will be used to determine which research studies will be included. The inclusion and exclusion criteria must be decided before you start the review. Inclusion criteria is everything a study must have to be included. Exclusion criteria are the factors that would make a study ineligible to be included. Criteria that should be considered include:

Type of studies:  It is important to select articles with an appropriate study design for the research question. Dates for the studies and a timeline of the problem/issue being examined may need to be identified. 

Type of participants:  Identify the target population characteristics. It is important to define the target population's age, sex/gender, diagnosis, as well as any other relevant factors.

Types of intervention:  Describe the intervention being investigated. Consider whether to include interventions carried out globally or just in the United States. Eligibility criteria for interventions should include things such as the dose, delivery method, and duration of the investigated intervention. The interventions that are to be excluded may also need to be described here.

Types of outcome measures:  Outcome measures usually refer to measurable outcomes or ‘clinical changes in health’. For example, these could include body structures and functions like pain and fatigue, activities as in functional abilities, and participation or quality of life questionnaires.

Read Chapter 3 of the Cochrane Handbook

Exclusion criteria.

A balance of specific inclusion and exclusion criteria is paramount. For some systematic reviews, there may already be a large pre-existing body of literature. The search strategy may retrieve thousands of results that must be screened. Having explicit exclusion criteria from the beginning allows those conducting the screening process, an efficient workflow. For the final product there should be a section in the review dedicated to 'Characteristics of excluded studies.' It is important to summarize why studies were excluded, especially if to a reader the study would appear to be eligible for the systematic review. 

For example, a team is conducting a systematic review regarding intervention options for the treatment of opioid addiction. The research team may want to exclude studies that also involve alcohol addiction to isolate the conditions for treatment interventions solely for opioid addiction.

• << Previous: Planning and Protocols
• Next: Searching >>
• Last Updated: Oct 12, 2023 12:30 PM
• URL: https://libguides.sph.uth.tmc.edu/SystematicReviews

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base

Methodology

• Inclusion and Exclusion Criteria | Examples & Definition

Inclusion and Exclusion Criteria | Examples & Definition

Published on September 17, 2022 by Kassiani Nikolopoulou . Revised on June 22, 2023.

Inclusion and exclusion criteria determine which members of the target population can or can’t participate in a research study. Collectively, they’re known as eligibility criteria , and establishing them is critical when seeking study participants for clinical trials.

This allows researchers to study the needs of a relatively homogeneous group (e.g., people with liver disease) with precision. Examples of common inclusion and exclusion criteria are:

• Demographic characteristics: Age, gender identity, ethnicity
• Study-specific variables: Type and stage of disease, previous treatment history, presence of chronic conditions, ability to attend follow-up study appointments, technological requirements (e.g., internet access)
• Control variables : Fitness level, tobacco use, medications used

Failure to properly define inclusion and exclusion criteria can undermine your confidence that causal relationships exist between treatment and control groups, affecting the internal validity of your study and the generalizability ( external validity ) of your findings.

Table of contents

What are inclusion criteria, what are exclusion criteria, examples of inclusion and exclusion criteria, why are inclusion and exclusion criteria important, other interesting articles, frequently asked questions.

Inclusion criteria comprise the characteristics or attributes that prospective research participants must have in order to be included in the study. Common inclusion criteria can be demographic, clinical, or geographic in nature.

• 18 to 80 years of age
• Diagnosis of chronic heart failure at least 6 months before trial
• On stable doses of heart failure therapies
• Willing to return for required follow-up (posttest) visits

People who meet the inclusion criteria are then eligible to participate in the study.

Prevent plagiarism. Run a free check.

Exclusion criteria comprise characteristics used to identify potential research participants who should not be included in a study. These can also include those that lead to participants withdrawing from a research study after being initially included.

In other words, individuals who meet the inclusion criteria may also possess additional characteristics that can interfere with the outcome of the study. For this reason, they must be excluded.

Typical exclusion criteria can be:

• Ethical considerations , such as being a minor or being unable to give informed consent
• Practical considerations, such as not being able to read

If potential participants possess any additional characteristics that can affect the results, such as another medical condition or a pregnancy, these are also often grounds for exclusion.

• The patient requires valve or other cardiac surgery
• The patient is unable to carry out any physical activity without discomfort
• The patient had a stroke within three months prior to enrollment
• The patient refuses to give informed consent
• The patient is a candidate for coronary bypass surgery or something similar

People who meet one or more of the exclusion criteria must be disqualified. This means that they can’t participate in the study even if they meet the inclusion criteria.

It is important that researchers clearly define the appropriate inclusion and exclusion criteria prior to recruiting participants for their experiment or trial.

Here are some examples of effective and ineffective ways to phrase your criteria:

Inclusion criteria

Bad example: “Subjects will be included in the study if they have insomnia.”

This is too vague. How are you going to establish that participants have insomnia?

Good example: “Subjects will be included in the study if they have been diagnosed with insomnia by a physician and have had symptoms (i.e., trouble falling and/or staying asleep) for at least 3 nights a week for a minimum of 3 months.”

Here, the diagnosis and symptoms are clear. Specifying the time frame ensures that the condition (insomnia) is more likely to be stable throughout the study.

Exclusion criteria

Bad example: “Subjects will be excluded from the study if they are taking medications.”

This is too broad. There are many forms of medication, and some surely will not interfere with your study results. Excluding anyone who is using any type of medication—be it painkillers, birth control, or antidepressants—makes recruitment of study participants for your sample difficult. This, in turn, affects the feasibility of your study.

Good example: “Subjects will be excluded from the study if they are currently on any medication affecting sleep, prescription drugs, or other drugs that in the opinion of the research team may interfere with the results of the study.”

Researchers review inclusion and exclusion criteria with each potential participant to determine their eligibility.

Defining inclusion and exclusion criteria is important in any type of research that examines characteristics of a specific subset of a population . This helps researchers identify the study population in a consistent, reliable, and objective manner. As a result, study participants are more likely to have the attributes that will make it possible to robustly answer the research question .

In clinical trials, establishing inclusion and exclusion criteria minimizes the likelihood of harming participants (e.g., excluding pregnant women) and safeguards vulnerable individuals from exploitation (e.g., excluding individuals who are unable to comprehend what the research entails.) Ethical considerations like these are critical in human-based research.

The main goal of clinical trials is to prove that a medication is safe and effective when used by the target population it was designed for. Therefore, ensuring that study participants are representative of the target population is crucial to the success of the study.

By applying inclusion and exclusion criteria to recruit participants, researchers can ensure that participants are indeed representative of the target population, ensuring external validity . Relatedly, defining robust inclusion and exclusion criteria strengthens your claim that causal relationships exist between your treatment and control groups , ensuring internal validity .

Strong inclusion and exclusion criteria also help other researchers, because they can follow what you did and how you selected participants, allowing them to accurately replicate or reproduce your study.

Ethnographies and a few other types of qualitative research do not usually specify exclusion criteria. However, inclusion criteria help researchers define the community of interest—for example, users of Apple watches. In this way, they can find individuals who have attributes that can help them meet the research objectives .

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Student’s  t -distribution
• Normal distribution
• Null and Alternative Hypotheses
• Chi square tests
• Confidence interval
• Quartiles & Quantiles
• Cluster sampling
• Stratified sampling
• Data cleansing
• Reproducibility vs Replicability
• Peer review
• Prospective cohort study

Research bias

• Implicit bias
• Cognitive bias
• Placebo effect
• Hawthorne effect
• Hindsight bias
• Affect heuristic
• Social desirability bias

I nternal validity is the degree of confidence that the causal relationship you are testing is not influenced by other factors or variables .

External validity is the extent to which your results can be generalized to other contexts.

The validity of your experiment depends on your experimental design .

Inclusion and exclusion criteria are predominantly used in non-probability sampling . In purposive sampling and snowball sampling , restrictions apply as to who can be included in the sample .

Inclusion and exclusion criteria are typically presented and discussed in the methodology section of your thesis or dissertation .

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

Nikolopoulou, K. (2023, June 22). Inclusion and Exclusion Criteria | Examples & Definition. Scribbr. Retrieved March 20, 2024, from https://www.scribbr.com/methodology/inclusion-exclusion-criteria/

Is this article helpful?

Kassiani Nikolopoulou

Other students also liked, population vs. sample | definitions, differences & examples, external validity | definition, types, threats & examples, reproducibility vs replicability | difference & examples, "i thought ai proofreading was useless but..".

I've been using Scribbr for years now and I know it's a service that won't disappoint. It does a good job spotting mistakes”

Jump to navigation

Cochrane Training

Chapter 3: defining the criteria for including studies and how they will be grouped for the synthesis.

Joanne E McKenzie, Sue E Brennan, Rebecca E Ryan, Hilary J Thomson, Renea V Johnston, James Thomas

Key Points:

• The scope of a review is defined by the types of population (participants), types of interventions (and comparisons), and the types of outcomes that are of interest. The acronym PICO (population, interventions, comparators and outcomes) helps to serve as a reminder of these.
• The population, intervention and comparison components of the question, with the additional specification of types of study that will be included, form the basis of the pre-specified eligibility criteria for the review. It is rare to use outcomes as eligibility criteria: studies should be included irrespective of whether they report outcome data, but may legitimately be excluded if they do not measure outcomes of interest, or if they explicitly aim to prevent a particular outcome.
• Cochrane Reviews should include all outcomes that are likely to be meaningful and not include trivial outcomes. Critical and important outcomes should be limited in number and include adverse as well as beneficial outcomes.
• Review authors should plan at the protocol stage how the different populations, interventions, outcomes and study designs within the scope of the review will be grouped for analysis.

Cite this chapter as: McKenzie JE, Brennan SE, Ryan RE, Thomson HJ, Johnston RV, Thomas J. Chapter 3: Defining the criteria for including studies and how they will be grouped for the synthesis. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.4 (updated August 2023). Cochrane, 2023. Available from www.training.cochrane.org/handbook .

3.1 Introduction

One of the features that distinguishes a systematic review from a narrative review is that systematic review authors should pre-specify criteria for including and excluding studies in the review (eligibility criteria, see MECIR Box 3.2.a ).

When developing the protocol, one of the first steps is to determine the elements of the review question (including the population, intervention(s), comparator(s) and outcomes, or PICO elements) and how the intervention, in the specified population, produces the expected outcomes (see Chapter 2, Section 2.5.1 and Chapter 17, Section 17.2.1 ). Eligibility criteria are based on the PICO elements of the review question plus a specification of the types of studies that have addressed these questions. The population, interventions and comparators in the review question usually translate directly into eligibility criteria for the review, though this is not always a straightforward process and requires a thoughtful approach, as this chapter shows. Outcomes usually are not part of the criteria for including studies, and a Cochrane Review would typically seek all sufficiently rigorous studies (most commonly randomized trials) of a particular comparison of interventions in a particular population of participants, irrespective of the outcomes measured or reported. It should be noted that some reviews do legitimately restrict eligibility to specific outcomes. For example, the same intervention may be studied in the same population for different purposes; or a review may specifically address the adverse effects of an intervention used for several conditions (see Chapter 19 ).

Eligibility criteria do not exist in isolation, but should be specified with the synthesis of the studies they describe in mind. This will involve making plans for how to group variants of the PICO elements for synthesis. This chapter describes the processes by which the structure of the synthesis can be mapped out at the beginning of the review, and the interplay between the review question, considerations for the analysis and their operationalization in terms of eligibility criteria. Decisions about which studies to include (and exclude), and how they will be combined in the review’s synthesis, should be documented and justified in the review protocol.

A distinction between three different stages in the review at which the PICO construct might be used is helpful for understanding the decisions that need to be made. In Chapter 2, Section 2.3 , we introduced the ideas of a review PICO (on which eligibility of studies is based), the PICO for each synthesis (defining the question that each specific synthesis aims to answer) and the PICO of the included studies (what was actually investigated in the included studies). In this chapter, we focus on the review PICO and the PICO for each synthesis as a basis for specifying which studies should be included in the review and planning its syntheses. These PICOs should relate clearly and directly to the questions or hypotheses that are posed when the review is formulated (see Chapter 2 ) and will involve specifying the population in question, and a set of comparisons between the intervention groups.

An integral part of the process of setting up the review is to specify which characteristics of the interventions (e.g. individual compounds of a drug), populations (e.g. acute and chronic conditions), outcomes (e.g. different depression measurement scales) and study designs, will be grouped together. Such decisions should be made independent of knowing which studies will be included and the methods of synthesis that will be used (e.g. meta-analysis). There may be a need to modify the comparisons and even add new ones at the review stage in light of the data that are collected. For example, important variations in the intervention may be discovered only after data are collected, or modifying the comparison may facilitate the possibility of synthesis when only one or few studies meet the comparison PICO. Planning for the latter scenario at the protocol stage may lead to less post-hoc decision making ( Chapter 2, Section 2.5.3 ) and, of course, any changes made during the conduct of the review should be recorded and documented in the final report.

3.2 Articulating the review and comparison PICO

3.2.1 defining types of participants: which people and populations.

The criteria for considering types of people included in studies in a review should be sufficiently broad to encompass the likely diversity of studies and the likely scenarios in which the interventions will be used, but sufficiently narrow to ensure that a meaningful answer can be obtained when studies are considered together; they should be specified in advance (see MECIR Box 3.2.a ). As discussed in Chapter 2, Section 2.3.1 , the degree of breadth will vary, depending on the question being asked and the analytical approach to be employed. A range of evidence may inform the choice of population characteristics to examine, including theoretical considerations, evidence from other interventions that have a similar mechanism of action, and in vitro or animal studies. Consideration should be given to whether the population characteristic is at the level of the participant (e.g. age, severity of disease) or the study (e.g. care setting, geographical location), since this has implications for grouping studies and for the method of synthesis ( Chapter 10, Section 10.11.5 ). It is often helpful to consider the types of people that are of interest in three steps.

MECIR Box 3.2.a Relevant expectations for conduct of intervention reviews

First, the diseases or conditions of interest should be defined using explicit criteria for establishing their presence (or absence). Criteria that will force the unnecessary exclusion of studies should be avoided. For example, diagnostic criteria that were developed more recently – which may be viewed as the current gold standard for diagnosing the condition of interest – will not have been used in earlier studies. Expensive or recent diagnostic tests may not be available in many countries or settings, and time-consuming tests may not be practical in routine healthcare settings.

Second, the broad population and setting of interest should be defined . This involves deciding whether a specific population group is within scope, determined by factors such as age, sex, race, educational status or the presence of a particular condition such as angina or shortness of breath. Interest may focus on a particular setting such as a community, hospital, nursing home, chronic care institution, or outpatient setting. Box 3.2.a outlines some factors to consider when developing population criteria.

Whichever criteria are used for defining the population and setting of interest, it is common to encounter studies that only partially overlap with the review’s population. For example, in a review focusing on children, a cut-point of less than 16 years might be desirable, but studies may be identified with participants aged from 12 to 18. Unless the study reports separate data from the eligible section of the population (in which case data from the eligible participants can be included in the review), review authors will need a strategy for dealing with these studies (see MECIR Box 3.2.a ). This will involve balancing concerns about reduced applicability by including participants who do not meet the eligibility criteria, against the loss of data when studies are excluded. Arbitrary rules (such as including a study if more than 80% of the participants are under 16) will not be practical if detailed information is not available from the study. A less stringent rule, such as ‘the majority of participants are under 16’ may be sufficient. Although there is a risk of review authors’ biases affecting post-hoc inclusion decisions (which is why many authors endeavour to pre-specify these rules), this may be outweighed by a common-sense strategy in which eligibility decisions keep faith with the objectives of the review rather than with arbitrary rules. Difficult decisions should be documented in the review, checked with the advisory group (if available, see Chapter 1 ), and sensitivity analyses can assess the impact of these decisions on the review’s findings (see Chapter 10, Section 10.14 and MECIR Box 3.2.b ).

Box 3.2.a Factors to consider when developing criteria for ‘Types of participants’

MECIR Box 3.2.b Relevant expectations for conduct of intervention reviews

Third, there should be consideration of whether there are population characteristics that might be expected to modify the size of the intervention effects (e.g. different severities of heart failure). Identifying subpopulations may be important for implementation of the intervention. If relevant subpopulations are identified, two courses of action are possible: limiting the scope of the review to exclude certain subpopulations; or maintaining the breadth of the review and addressing subpopulations in the analysis.

Restricting the review with respect to specific population characteristics or settings should be based on a sound rationale. It is important that Cochrane Reviews are globally relevant, so the rationale for the exclusion of studies based on population characteristics should be justified. For example, focusing a review of the effectiveness of mammographic screening on women between 40 and 50 years old may be justified based on biological plausibility, previously published systematic reviews and existing controversy. On the other hand, focusing a review on a particular subgroup of people on the basis of their age, sex or ethnicity simply because of personal interests, when there is no underlying biologic or sociological justification for doing so, should be avoided, as these reviews will be less useful to decision makers and readers of the review.

Maintaining the breadth of the review may be best when it is uncertain whether there are important differences in effects among various subgroups of people, since this allows investigation of these differences (see Chapter 10, Section 10.11.5 ). Review authors may combine the results from different subpopulations in the same synthesis, examining whether a given subdivision explains variation (heterogeneity) among the intervention effects. Alternatively, the results may be synthesized in separate comparisons representing different subpopulations. Splitting by subpopulation risks there being too few studies to yield a useful synthesis (see Table 3.2.a and Chapter 2, Section 2.3.2 ). Consideration needs to be given to the subgroup analysis method, particularly for population characteristics measured at the participant level (see Chapter 10 and Chapter 26 , Fisher et al 2017). All subgroup analyses should ideally be planned a priori and stated as a secondary objective in the protocol, and not driven by the availability of data.

In practice, it may be difficult to assign included studies to defined subpopulations because of missing information about the population characteristic, variability in how the population characteristic is measured across studies (e.g. variation in the method used to define the severity of heart failure), or because the study does not wholly fall within (or report the results separately by) the defined subpopulation. The latter issue mainly applies for participant characteristics but can also arise for settings or geographic locations where these vary within studies. Review authors should consider planning for these scenarios (see example reviews Hetrick et al 2012, Safi et al 2017; Table 3.2.b , column 3).

Table 3.2.a Examples of population attributes and characteristics

3.2.2 Defining interventions and how they will be grouped

In some reviews, predefining the intervention ( MECIR Box 3.2.c ) may be straightforward. For example, in a review of the effect of a given anticoagulant on deep vein thrombosis, the intervention can be defined precisely. A more complicated definition might be required for a multi-component intervention composed of dietary advice, training and support groups to reduce rates of obesity in a given population.

The inherent complexity present when defining an intervention often comes to light when considering how it is thought to achieve its intended effect and whether the effect is likely to differ when variants of the intervention are used. In the first example, the anticoagulant warfarin is thought to reduce blood clots by blocking an enzyme that depends on vitamin K to generate clotting factors. In the second, the behavioural intervention is thought to increase individuals’ self-efficacy in their ability to prepare healthy food. In both examples, we cannot assume that all forms of the intervention will work in the same way. When defining drug interventions, such as anticoagulants, factors such as the drug preparation, route of administration, dose, duration, and frequency should be considered. For multi-component interventions (such as interventions to reduce rates of obesity), the common or core features of the interventions must be defined, so that the review authors can clearly differentiate them from other interventions not included in the review.

MECIR Box 3.2.c Relevant expectations for conduct of intervention reviews

In general, it is useful to consider exactly what is delivered, who delivers it, how it is delivered, where it is delivered, when and how much is delivered, and whether the intervention can be adapted or tailored , and to consider this for each type of intervention included in the review (see the TIDieR checklist (Hoffmann et al 2014)). As argued in Chapter 17 , separating interventions into ‘simple’ and ‘complex’ is a false dichotomy; all interventions can be complex in some ways. The critical issue for review authors is to identify the most important factors to be considered in a specific review. Box 3.2.b outlines some factors to consider when developing broad criteria for the ‘Types of interventions’ (and comparisons).

Box 3.2.b Factors to consider when developing criteria for ‘Types of interventions’

Once interventions eligible for the review have been broadly defined, decisions should be made about how variants of the intervention will be handled in the synthesis. Differences in intervention characteristics across studies occur in all reviews. If these reflect minor differences in the form of the intervention used in practice (such as small differences in the duration or content of brief alcohol counselling interventions), then an overall synthesis can provide useful information for decision makers. Where differences in intervention characteristics are more substantial (such as delivery of brief alcohol counselling by nurses versus doctors), and are expected to have a substantial impact on the size of intervention effects, these differences should be examined in the synthesis. What constitutes an important difference requires judgement, but in general differences that alter decisions about how an intervention is implemented or whether the intervention is used or not are likely to be important. In such circumstances, review authors should consider specifying separate groups (or subgroups) to examine in their synthesis.

Clearly defined intervention groups serve two main purposes in the synthesis. First, the way in which interventions are grouped for synthesis (meta-analysis or other synthesis) is likely to influence review findings. Careful planning of intervention groups makes best use of the available data, avoids decisions that are influenced by study findings (which may introduce bias), and produces a review focused on questions relevant to decision makers. Second, the intervention groups specified in a protocol provide a standardized terminology for describing the interventions throughout the review, overcoming the varied descriptions used by study authors (e.g. where different labels are used for the same intervention, or similar labels used for different techniques) (Michie et al 2013). This standardization enables comparison and synthesis of information about intervention characteristics across studies (common characteristics and differences) and provides a consistent language for reporting that supports interpretation of review findings.

Table 3.2.b   outlines a process for planning intervention groups as a basis for/precursor to synthesis, and the decision points and considerations at each step. The table is intended to guide, rather than to be prescriptive and, although it is presented as a sequence of steps, the process is likely to be iterative, and some steps may be done concurrently or in a different sequence. The process aims to minimize data-driven approaches that can arise once review authors have knowledge of the findings of the included studies. It also includes principles for developing a flexible plan that maximizes the potential to synthesize in circumstances where there are few studies, many variants of an intervention, or where the variants are difficult to anticipate. In all stages, review authors should consider how to categorize studies whose reports contain insufficient detail.

Table 3.2.b A process for planning intervention groups for synthesis

3.2.3 Defining which comparisons will be made

When articulating the PICO for each synthesis, defining the intervention groups alone is not sufficient for complete specification of the planned syntheses. The next step is to define the comparisons that will be made between the intervention groups. Setting aside for a moment more complex analyses such as network meta-analyses, which can simultaneously compare many groups ( Chapter 11 ), standard meta-analysis ( Chapter 10 ) aims to draw conclusions about the comparative effects of two groups at a time (i.e. which of two intervention groups is more effective?). These comparisons form the basis for the syntheses that will be undertaken if data are available. Cochrane Reviews sometimes include one comparison, but most often include multiple comparisons. Three commonly identified types of comparisons include the following (Davey et al 2011).

• newer generation antidepressants versus placebo (Hetrick et al 2012); and
• vertebroplasty for osteoporotic vertebral compression fractures versus placebo (sham procedure) (Buchbinder et al 2018).
• chemotherapy or targeted therapy plus best supportive care (BSC) versus BSC for palliative treatment of esophageal and gastroesophageal-junction carcinoma (Janmaat et al 2017); and
• personalized care planning versus usual care for people with long-term conditions (Coulter et al 2015).
• early (commenced at less than two weeks of age) versus late (two weeks of age or more) parenteral zinc supplementation in term and preterm infants (Taylor et al 2017);
• high intensity versus low intensity physical activity or exercise in people with hip or knee osteoarthritis (Regnaux et al 2015);
• multimedia education versus other education for consumers about prescribed and over the counter medications (Ciciriello et al 2013).

The first two types of comparisons aim to establish the effectiveness of an intervention, while the last aims to compare the effectiveness of two interventions. However, the distinction between the placebo and control is often arbitrary, since any differences in the care provided between trials with a control arm and those with a placebo arm may be unimportant , especially where ‘usual care’ is provided to both. Therefore, placebo and control groups may be determined to be similar enough to be combined for synthesis.

In reviews including multiple intervention groups, many comparisons are possible. In some of these reviews, authors seek to synthesize evidence on the comparative effectiveness of all their included interventions, including where there may be only indirect comparison of some interventions across the included studies ( Chapter 11, Section 11.2.1 ). However, in many reviews including multiple intervention groups, a limited subset of the possible comparisons will be selected. The chosen subset of comparisons should address the most important clinical and research questions. For example, if an established intervention (or dose of an intervention) is used in practice, then the synthesis would ideally compare novel or alternative interventions to this established intervention, and not, for example, to no intervention.

3.2.3.1 Dealing with co-interventions

Planning is needed for the special case where the same supplementary intervention is delivered to both the intervention and comparator groups. A supplementary intervention is an additional intervention delivered alongside the intervention of interest, such as massage in a review examining the effects of aromatherapy (i.e. aromatherapy plus massage versus massage alone). In many cases, the supplementary intervention will be unimportant and can be ignored. In other situations, the effect of the intervention of interest may differ according to whether participants receive the supplementary therapy. For example, the effect of aromatherapy among people who receive a massage may differ from the effect of the aromatherapy given alone. This will be the case if the intervention of interest interacts with the supplementary intervention leading to larger (synergistic) or smaller (dysynergistic/antagonistic) effects than the intervention of interest alone (Squires et al 2013). While qualitative interactions are rare (where the effect of the intervention is in the opposite direction when combined with the supplementary intervention), it is possible that there will be more variation in the intervention effects (heterogeneity) when supplementary interventions are involved, and it is important to plan for this. Approaches for dealing with this in the statistical synthesis may include fitting a random-effects meta-analysis model that encompasses heterogeneity ( Chapter 10, Section 10.10.4 ), or investigating whether the intervention effect is modified by the addition of the supplementary intervention through subgroup analysis ( Chapter 10, Section 10.11.2 ).

3.2.4 Selecting, prioritizing and grouping review outcomes

3.2.4.1 selecting review outcomes.

Broad outcome domains are decided at the time of setting up the review PICO (see Chapter 2 ). Once the broad domains are agreed, further specification is required to define the domains to facilitate reporting and synthesis (i.e. the PICO for comparison) (see Chapter 2, Section 2.3 ). The process for specifying and grouping outcomes largely parallels that used for specifying intervention groups.

Reporting of outcomes should rarely determine study eligibility for a review. In particular, studies should not be excluded because they do not report results of an outcome they may have measured, or provide ‘no usable data’ ( MECIR Box 3.2.d ). This is essential to avoid bias arising from selective reporting of findings by the study authors (see Chapter 13 ). However, in some circumstances, the measurement of certain outcomes may be a study eligibility criterion. This may be the case, for example, when the review addresses the potential for an intervention to prevent a particular outcome, or when the review addresses a specific purpose of an intervention that can be used in the same population for different purposes (such as hormone replacement therapy, or aspirin).

MECIR Box 3.2.d Relevant expectations for conduct of intervention reviews

In general, systematic reviews should aim to include outcomes that are likely to be meaningful to the intended users and recipients of the reviewed evidence. This may include clinicians, patients (consumers), the general public, administrators and policy makers. Outcomes may include survival (mortality), clinical events (e.g. strokes or myocardial infarction), behavioural outcomes (e.g. changes in diet, use of services), patient-reported outcomes (e.g. symptoms, quality of life), adverse events, burdens (e.g. demands on caregivers, frequency of tests, restrictions on lifestyle) and economic outcomes (e.g. cost and resource use). It is critical that outcomes used to assess adverse effects as well as outcomes used to assess beneficial effects are among those addressed by a review (see Chapter 19 ).

Outcomes that are trivial or meaningless to decision makers should not be included in Cochrane Reviews. Inclusion of outcomes that are of little or no importance risks overwhelming and potentially misleading readers. Interim or surrogate outcomes measures, such as laboratory results or radiologic results (e.g. loss of bone mineral content as a surrogate for fractures in hormone replacement therapy), while potentially helpful in explaining effects or determining intervention integrity (see Chapter 5, Section 5.3.4.1 ), can also be misleading since they may not predict clinically important outcomes accurately. Many interventions reduce the risk for a surrogate outcome but have no effect or have harmful effects on clinically relevant outcomes, and some interventions have no effect on surrogate measures but improve clinical outcomes.

Various sources can be used to develop a list of relevant outcomes, including input from consumers and advisory groups (see Chapter 2 ), the clinical experiences of the review authors, and evidence from the literature (including qualitative research about outcomes important to those affected (see Chapter 21 )). A further driver of outcome selection is consideration of outcomes used in related reviews. Harmonization of outcomes across reviews addressing related questions facilitates broader evidence synthesis questions being addressed through the use of Overviews of reviews (see Chapter V ).

Outcomes considered to be meaningful, and therefore addressed in a review, may not have been reported in the primary studies. For example, quality of life is an important outcome, perhaps the most important outcome, for people considering whether or not to use chemotherapy for advanced cancer, even if the available studies are found to report only survival (see Chapter 18 ). A further example arises with timing of the outcome measurement, where time points determined as clinically meaningful in a review are not measured in the primary studies. Including and discussing all important outcomes in a review will highlight gaps in the primary research and encourage researchers to address these gaps in future studies.

3.2.4.2 Prioritizing review outcomes

Once a full list of relevant outcomes has been compiled for the review, authors should prioritize the outcomes and select the outcomes of most relevance to the review question. The GRADE approach to assessing the certainty of evidence (see Chapter 14 ) suggests that review authors separate outcomes into those that are ‘critical’, ‘important’ and ‘not important’ for decision making.

The critical outcomes are the essential outcomes for decision making, and are those that would form the basis of a ‘Summary of findings’ table or other summary versions of the review, such as the Abstract or Plain Language Summary. ‘Summary of findings’ tables provide key information about the amount of evidence for important comparisons and outcomes, the quality of the evidence and the magnitude of effect (see Chapter 14, Section 14.1 ). There should be no more than seven outcomes included in a ‘Summary of findings’ table, and those outcomes that will be included in summaries should be specified at the protocol stage. They should generally not include surrogate or interim outcomes. They should not be chosen on the basis of any anticipated or observed magnitude of effect, or because they are likely to have been addressed in the studies to be reviewed. Box 3.2.c summarizes the principal factors to consider when selecting and prioritizing review outcomes.

Box 3.2.c Factors to consider when selecting and prioritizing review outcomes

3.2.4.3 Defining and grouping outcomes for synthesis

Table 3.2.c outlines a process for planning for the diversity in outcome measurement that may be encountered in the studies included in a review and which can complicate, and sometimes prevent, synthesis. Research has repeatedly documented inconsistency in the outcomes measured across trials in the same clinical areas (Harrison et al 2016, Williamson et al 2017). This inconsistency occurs across all aspects of outcome measurement, including the broad domains considered, the outcomes measured, the way these outcomes are labelled and defined, and the methods and timing of measurement. For example, a review of outcome measures used in 563 studies of interventions for dementia and mild cognitive impairment found that 321 unique measurement methods were used for 1278 assessments of cognitive outcomes (Harrison et al 2016). Initiatives like COMET ( Core Outcome Measures in Effectiveness Trials ) aim to encourage standardization of outcome measurement across trials (Williamson et al 2017), but these initiatives are comparatively new and review authors will inevitably encounter diversity in outcomes across studies.

The process begins by describing the scope of each outcome domain in sufficient detail to enable outcomes from included studies to be categorized ( Table 3.2.c Step 1). This step may be straightforward in areas for which core outcome sets (or equivalent systems) exist ( Table 3.2.c Step 2). The methods and timing of outcome measurement also need to be specified, giving consideration to how differences across studies will be handled ( Table 3.2.c Steps 3 and 4). Subsequent steps consider options for dealing with studies that report multiple measures within an outcome domain ( Table 3.2.c Step 5), planning how outcome domains will be used in synthesis ( Table 3.2.c Step 6), and building in contingencies to maximize potential to synthesize ( Table 3.2.c Step 7).

Table 3.2.c A process for planning outcome groups for synthesis

3.3 Determining which study designs to include

Some study designs are more appropriate than others for answering particular questions. Authors need to consider a priori what study designs are likely to provide reliable data with which to address the objectives of their review ( MECIR Box 3.3.a ). Sections 3.3.1 and 3.3.2 cover randomized and non-randomized designs for assessing treatment effects; Chapter 17, Section 17.2.5  discusses other study designs in the context of addressing intervention complexity.

MECIR Box 3.3.a Relevant expectations for conduct of intervention reviews

3.3.1 Including randomized trials

Because Cochrane Reviews address questions about the effects of health care, they focus primarily on randomized trials and randomized trials should be included if they are feasible for the interventions of interest ( MECIR Box 3.3.b ). Randomization is the only way to prevent systematic differences between baseline characteristics of participants in different intervention groups in terms of both known and unknown (or unmeasured) confounders (see Chapter 8 ), and claims about cause and effect can be based on their findings with far more confidence than almost any other type of study. For clinical interventions, deciding who receives an intervention and who does not is influenced by many factors, including prognostic factors. Empirical evidence suggests that, on average, non-randomized studies produce effect estimates that indicate more extreme benefits of the effects of health care than randomized trials. However, the extent, and even the direction, of the bias is difficult to predict. These issues are discussed at length in Chapter 24 , which provides guidance on when it might be appropriate to include non-randomized studies in a Cochrane Review.

Practical considerations also motivate the restriction of many Cochrane Reviews to randomized trials. In recent decades there has been considerable investment internationally in establishing infrastructure to index and identify randomized trials. Cochrane has contributed to these efforts, including building up and maintaining a database of randomized trials, developing search filters to aid their identification, working with MEDLINE to improve tagging and identification of randomized trials, and using machine learning and crowdsourcing to reduce author workload in identifying randomized trials ( Chapter 4, Section 4.6.6.2 ). The same scale of organizational investment has not (yet) been matched for the identification of other types of studies. Consequently, identifying and including other types of studies may require additional efforts to identify studies and to keep the review up to date, and might increase the risk that the result of the review will be influenced by publication bias. This issue and other bias-related issues that are important to consider when defining types of studies are discussed in detail in Chapter 7 and Chapter 13 .

Specific aspects of study design and conduct should be considered when defining eligibility criteria, even if the review is restricted to randomized trials. For example, whether cluster-randomized trials ( Chapter 23, Section 23.1 ) and crossover trials ( Chapter 23, Section 23.2 ) are eligible, as well as other criteria for eligibility such as use of a placebo comparison group, evaluation of outcomes blinded to allocation sequence, or a minimum period of follow-up. There will always be a trade-off between restrictive study design criteria (which might result in the inclusion of studies that are at low risk of bias, but very few in number) and more liberal design criteria (which might result in the inclusion of more studies, but at a higher risk of bias). Furthermore, excessively broad criteria might result in the inclusion of misleading evidence. If, for example, interest focuses on whether a therapy improves survival in patients with a chronic condition, it might be inappropriate to look at studies of very short duration, except to make explicit the point that they cannot address the question of interest.

MECIR Box 3.3.b Relevant expectations for conduct of intervention reviews

3.3.2 Including non-randomized studies

The decision of whether non-randomized studies (and what type) will be included is decided alongside the formulation of the review PICO. The main drivers that may lead to the inclusion of non-randomized studies include: (i) when randomized trials are unable to address the effects of the intervention on harm and long-term outcomes or in specific populations or settings; or (ii) for interventions that cannot be randomized (e.g. policy change introduced in a single or small number of jurisdictions) (see Chapter 24 ). Cochrane, in collaboration with others, has developed guidance for review authors to support their decision about when to look for and include non-randomized studies (Schünemann et al 2013).

Non-randomized designs have the commonality of not using randomization to allocate units to comparison groups, but their different design features mean that they are variable in their susceptibility to bias. Eligibility criteria should be based on explicit study design features, and not the study labels applied by the primary researchers (e.g. case-control, cohort), which are often used inconsistently (Reeves et al 2017; see Chapter 24 ).

When non-randomized studies are included, review authors should consider how the studies will be grouped and used in the synthesis. The Cochrane Non-randomized Studies Methods Group taxonomy of design features (see Chapter 24 ) may provide a basis for grouping together studies that are expected to have similar inferential strength and for providing a consistent language for describing the study design.

Once decisions have been made about grouping study designs, planning of how these will be used in the synthesis is required. Review authors need to decide whether it is useful to synthesize results from non-randomized studies and, if so, whether results from randomized trials and non-randomized studies should be included in the same synthesis (for the purpose of examining whether study design explains heterogeneity among the intervention effects), or whether the effects should be synthesized in separate comparisons (Valentine and Thompson 2013). Decisions should be made for each of the different types of non-randomized studies under consideration. Review authors might anticipate increased heterogeneity when non-randomized studies are synthesized, and adoption of a meta-analysis model that encompasses heterogeneity is wise (Valentine and Thompson 2013) (such as a random effects model, see Chapter 10, Section 10.10.4 ). For further discussion of non-randomized studies, see Chapter 24 .

3.4 Eligibility based on publication status and language

Chapter 4 contains detailed guidance on how to identify studies from a range of sources including, but not limited to, those in peer-reviewed journals. In general, a strategy to include studies reported in all types of publication will reduce bias ( Chapter 7 ). There would need to be a compelling argument for the exclusion of studies on the basis of their publication status ( MECIR Box 3.4.a ), including unpublished studies, partially published studies, and studies published in ‘grey’ literature sources. Given the additional challenge in obtaining unpublished studies, it is possible that any unpublished studies identified in a given review may be an unrepresentative subset of all the unpublished studies in existence. However, the bias this introduces is of less concern than the bias introduced by excluding all unpublished studies, given what is known about the impact of reporting biases (see Chapter 13 on bias due to missing studies, and Chapter 4, Section 4.3 for a more detailed discussion of searching for unpublished and grey literature).

Likewise, while searching for, and analysing, studies in any language can be extremely resource-intensive, review authors should consider carefully the implications for bias (and equity, see Chapter 16 ) if they restrict eligible studies to those published in one specific language (usually English). See Chapter 4, Section 4.4.5 , for further discussion of language and other restrictions while searching.

MECIR Box 3.4.a Relevant expectations for conduct of intervention reviews

3.5 Chapter information

Authors: Joanne E McKenzie, Sue E Brennan, Rebecca E Ryan, Hilary J Thomson, Renea V Johnston, James Thomas

Acknowledgements: This chapter builds on earlier versions of the Handbook . In particular, Version 5, Chapter 5 , edited by Denise O’Connor, Sally Green and Julian Higgins.

Funding: JEM is supported by an Australian National Health and Medical Research Council (NHMRC) Career Development Fellowship (1143429). SEB and RER’s positions are supported by the NHMRC Cochrane Collaboration Funding Program. HJT is funded by the UK Medical Research Council (MC_UU_12017-13 and MC_UU_12017-15) and Scottish Government Chief Scientist Office (SPHSU13 and SPHSU15). RVJ’s position is supported by the NHMRC Cochrane Collaboration Funding Program and Cabrini Institute. JT is supported by the National Institute for Health Research (NIHR) Collaboration for Leadership in Applied Health Research and Care North Thames at Barts Health NHS Trust. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health.

3.6 References

Bailey JV, Murray E, Rait G, Mercer CH, Morris RW, Peacock R, Cassell J, Nazareth I. Interactive computer-based interventions for sexual health promotion. Cochrane Database of Systematic Reviews 2010; 9 : CD006483.

Bender R, Bunce C, Clarke M, Gates S, Lange S, Pace NL, Thorlund K. Attention should be given to multiplicity issues in systematic reviews. Journal of Clinical Epidemiology 2008; 61 : 857–865.

Buchbinder R, Johnston RV, Rischin KJ, Homik J, Jones CA, Golmohammadi K, Kallmes DF. Percutaneous vertebroplasty for osteoporotic vertebral compression fracture. Cochrane Database of Systematic Reviews 2018; 4 : CD006349.

Caldwell DM, Welton NJ. Approaches for synthesising complex mental health interventions in meta-analysis. Evidence-Based Mental Health 2016; 19 : 16–21.

Chamberlain C, O’Mara-Eves A, Porter J, Coleman T, Perlen S, Thomas J, McKenzie J. Psychosocial interventions for supporting women to stop smoking in pregnancy. Cochrane Database of Systematic Reviews 2017; 2 : CD001055.

Ciciriello S, Johnston RV, Osborne RH, Wicks I, deKroo T, Clerehan R, O’Neill C, Buchbinder R. Multimedia educational interventions for consumers about prescribed and over-the-counter medications. Cochrane Database of Systematic Reviews 2013; 4 : CD008416.

Cochrane Consumers & Communication Group. Outcomes of Interest to the Cochrane Consumers & Communication Group: taxonomy. http://cccrg.cochrane.org/ .

COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) initiative. COSMIN database of systematic reviews of outcome measurement instruments. https://database.cosmin.nl/ .

Coulter A, Entwistle VA, Eccles A, Ryan S, Shepperd S, Perera R. Personalised care planning for adults with chronic or long-term health conditions. Cochrane Database of Systematic Reviews 2015; 3 : CD010523.

Davey J, Turner RM, Clarke MJ, Higgins JPT. Characteristics of meta-analyses and their component studies in the Cochrane Database of Systematic Reviews: a cross-sectional, descriptive analysis. BMC Medical Research Methodology 2011; 11 : 160.

Desroches S, Lapointe A, Ratte S, Gravel K, Legare F, Turcotte S. Interventions to enhance adherence to dietary advice for preventing and managing chronic diseases in adults. Cochrane Database of Systematic Reviews 2013; 2 : CD008722.

Deyo RA, Dworkin SF, Amtmann D, Andersson G, Borenstein D, Carragee E, Carrino J, Chou R, Cook K, DeLitto A, Goertz C, Khalsa P, Loeser J, Mackey S, Panagis J, Rainville J, Tosteson T, Turk D, Von Korff M, Weiner DK. Report of the NIH Task Force on research standards for chronic low back pain. Journal of Pain 2014; 15 : 569–585.

Dodd S, Clarke M, Becker L, Mavergames C, Fish R, Williamson PR. A taxonomy has been developed for outcomes in medical research to help improve knowledge discovery. Journal of Clinical Epidemiology 2018; 96 : 84–92.

Fisher DJ, Carpenter JR, Morris TP, Freeman SC, Tierney JF. Meta-analytical methods to identify who benefits most from treatments: daft, deluded, or deft approach? BMJ 2017; 356 : j573.

Fransen M, McConnell S, Harmer AR, Van der Esch M, Simic M, Bennell KL. Exercise for osteoarthritis of the knee. Cochrane Database of Systematic Reviews 2015; 1 : CD004376.

Guise JM, Chang C, Viswanathan M, Glick S, Treadwell J, Umscheid CA. Systematic reviews of complex multicomponent health care interventions. Report No. 14-EHC003-EF . Rockville, MD: Agency for Healthcare Research and Quality; 2014.

Harrison JK, Noel-Storr AH, Demeyere N, Reynish EL, Quinn TJ. Outcomes measures in a decade of dementia and mild cognitive impairment trials. Alzheimer’s Research and Therapy 2016; 8 : 48.

Hedges LV, Tipton E, Johnson M, C. Robust variance estimation in meta-regression with dependent effect size estimates. Research Synthesis Methods 2010; 1 : 39–65.

Hetrick SE, McKenzie JE, Cox GR, Simmons MB, Merry SN. Newer generation antidepressants for depressive disorders in children and adolescents. Cochrane Database of Systematic Reviews 2012; 11 : CD004851.

Higgins JPT, López-López JA, Becker BJ, Davies SR, Dawson S, Grimshaw JM, McGuinness LA, Moore THM, Rehfuess E, Thomas J, Caldwell DM. Synthesizing quantitative evidence in systematic reviews of complex health interventions. BMJ Global Health 2019; 4 : e000858.

Hoffmann T, Glasziou P, Barbour V, Macdonald H. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ 2014; 1687 : 1-13.

Hollands GJ, Shemilt I, Marteau TM, Jebb SA, Lewis HB, Wei Y, Higgins JPT, Ogilvie D. Portion, package or tableware size for changing selection and consumption of food, alcohol and tobacco. Cochrane Database of Systematic Reviews 2015; 9 : CD011045.

Howe TE, Shea B, Dawson LJ, Downie F, Murray A, Ross C, Harbour RT, Caldwell LM, Creed G. Exercise for preventing and treating osteoporosis in postmenopausal women. Cochrane Database of Systematic Reviews 2011; 7 : CD000333.

ICHOM. The International Consortium for Health Outcomes Measurement 2018. http://www.ichom.org/ .

IPDAS. International Patient Decision Aid Standards Collaboration (IPDAS) standards. www.ipdas.ohri.ca .

Ivers N, Jamtvedt G, Flottorp S, Young JM, Odgaard-Jensen J, French SD, O’Brien MA, Johansen M, Grimshaw J, Oxman AD. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database of Systematic Reviews 2012; 6 : CD000259.

Janmaat VT, Steyerberg EW, van der Gaast A, Mathijssen RH, Bruno MJ, Peppelenbosch MP, Kuipers EJ, Spaander MC. Palliative chemotherapy and targeted therapies for esophageal and gastroesophageal junction cancer. Cochrane Database of Systematic Reviews 2017; 11 : CD004063.

Kendrick D, Kumar A, Carpenter H, Zijlstra GAR, Skelton DA, Cook JR, Stevens Z, Belcher CM, Haworth D, Gawler SJ, Gage H, Masud T, Bowling A, Pearl M, Morris RW, Iliffe S, Delbaere K. Exercise for reducing fear of falling in older people living in the community. Cochrane Database of Systematic Reviews 2014; 11 : CD009848.

Kirkham JJ, Gargon E, Clarke M, Williamson PR. Can a core outcome set improve the quality of systematic reviews? A survey of the Co-ordinating Editors of Cochrane Review Groups. Trials 2013; 14 : 21.

Konstantopoulos S. Fixed effects and variance components estimation in three-level meta-analysis. Research Synthesis Methods 2011; 2 : 61–76.

Lamb SE, Becker C, Gillespie LD, Smith JL, Finnegan S, Potter R, Pfeiffer K. Reporting of complex interventions in clinical trials: development of a taxonomy to classify and describe fall-prevention interventions. Trials 2011; 12 : 125.

Lewin S, Hendry M, Chandler J, Oxman AD, Michie S, Shepperd S, Reeves BC, Tugwell P, Hannes K, Rehfuess EA, Welch V, Mckenzie JE, Burford B, Petkovic J, Anderson LM, Harris J, Noyes J. Assessing the complexity of interventions within systematic reviews: development, content and use of a new tool (iCAT_SR). BMC Medical Research Methodology 2017; 17 : 76.

López-López JA, Page MJ, Lipsey MW, Higgins JPT. Dealing with multiplicity of effect sizes in systematic reviews and meta-analyses. Research Synthesis Methods 2018; 9 : 336–351.

Mavridis D, Salanti G. A practical introduction to multivariate meta-analysis. Statistical Methods in Medical Research 2013; 22 : 133–158.

Michie S, van Stralen M, West R. The Behaviour Change Wheel: a new method for characterising and designing behaviour change interventions. Implementation Science 2011; 6 : 42.

Michie S, Richardson M, Johnston M, Abraham C, Francis J, Hardeman W, Eccles MP, Cane J, Wood CE. The behavior change technique taxonomy (v1) of 93 hierarchically clustered techniques: building an international consensus for the reporting of behavior change interventions. Annals of Behavioral Medicine 2013; 46 : 81–95.

Moraes VY, Lenza M, Tamaoki MJ, Faloppa F, Belloti JC. Platelet-rich therapies for musculoskeletal soft tissue injuries. Cochrane Database of Systematic Reviews 2014; 4 : CD010071.

O'Neill J, Tabish H, Welch V, Petticrew M, Pottie K, Clarke M, Evans T, Pardo Pardo J, Waters E, White H, Tugwell P. Applying an equity lens to interventions: using PROGRESS ensures consideration of socially stratifying factors to illuminate inequities in health. Journal of Clinical Epidemiology 2014; 67 : 56–64.

Pompoli A, Furukawa TA, Imai H, Tajika A, Efthimiou O, Salanti G. Psychological therapies for panic disorder with or without agoraphobia in adults: a network meta-analysis. Cochrane Database of Systematic Reviews 2016; 4 : CD011004.

Pompoli A, Furukawa TA, Efthimiou O, Imai H, Tajika A, Salanti G. Dismantling cognitive-behaviour therapy for panic disorder: a systematic review and component network meta-analysis. Psychological Medicine 2018; 48 : 1–9.

Reeves BC, Wells GA, Waddington H. Quasi-experimental study designs series-paper 5: a checklist for classifying studies evaluating the effects on health interventions – a taxonomy without labels. Journal of Clinical Epidemiology 2017; 89 : 30–42.

Regnaux J-P, Lefevre-Colau M-M, Trinquart L, Nguyen C, Boutron I, Brosseau L, Ravaud P. High-intensity versus low-intensity physical activity or exercise in people with hip or knee osteoarthritis. Cochrane Database of Systematic Reviews 2015; 10 : CD010203.

Richards SH, Anderson L, Jenkinson CE, Whalley B, Rees K, Davies P, Bennett P, Liu Z, West R, Thompson DR, Taylor RS. Psychological interventions for coronary heart disease. Cochrane Database of Systematic Reviews 2017; 4 : CD002902.

Safi S, Korang SK, Nielsen EE, Sethi NJ, Feinberg J, Gluud C, Jakobsen JC. Beta-blockers for heart failure. Cochrane Database of Systematic Reviews 2017; 12 : CD012897.

Santesso N, Carrasco-Labra A, Brignardello-Petersen R. Hip protectors for preventing hip fractures in older people. Cochrane Database of Systematic Reviews 2014; 3 : CD001255.

Shepherd E, Gomersall JC, Tieu J, Han S, Crowther CA, Middleton P. Combined diet and exercise interventions for preventing gestational diabetes mellitus. Cochrane Database of Systematic Reviews 2017; 11 : CD010443.

Squires J, Valentine J, Grimshaw J. Systematic reviews of complex interventions: framing the review question. Journal of Clinical Epidemiology 2013; 66 : 1215–1222.

Stacey D, Légaré F, Lewis K, Barry MJ, Bennett CL, Eden KB, Holmes-Rovner M, Llewellyn-Thomas H, Lyddiatt A, Thomson R, Trevena L. Decision aids for people facing health treatment or screening decisions. Cochrane Database of Systematic Reviews 2017; 4 : CD001431.

Stroke Unit Trialists Collaboration. Organised inpatient (stroke unit) care for stroke. Cochrane Database of Systematic Reviews 2013; 9 : CD000197.

Taylor AJ, Jones LJ, Osborn DA. Zinc supplementation of parenteral nutrition in newborn infants. Cochrane Database of Systematic Reviews 2017; 2 : CD012561.

Valentine JC, Thompson SG. Issues relating to confounding and meta-analysis when including non-randomized studies in systematic reviews on the effects of interventions. Research Synthesis Methods 2013; 4 : 26–35.

Vaona A, Banzi R, Kwag KH, Rigon G, Cereda D, Pecoraro V, Tramacere I, Moja L. E-learning for health professionals. Cochrane Database of Systematic Reviews 2018; 1 : CD011736.

Verheyden GSAF, Weerdesteyn V, Pickering RM, Kunkel D, Lennon S, Geurts ACH, Ashburn A. Interventions for preventing falls in people after stroke. Cochrane Database of Systematic Reviews 2013; 5 : CD008728.

Weisz JR, Kuppens S, Ng MY, Eckshtain D, Ugueto AM, Vaughn-Coaxum R, Jensen-Doss A, Hawley KM, Krumholz Marchette LS, Chu BC, Weersing VR, Fordwood SR. What five decades of research tells us about the effects of youth psychological therapy: a multilevel meta-analysis and implications for science and practice. American Psychologist 2017; 72 : 79–117.

Welch V, Petkovic J, Simeon R, Presseau J, Gagnon D, Hossain A, Pardo Pardo J, Pottie K, Rader T, Sokolovski A, Yoganathan M, Tugwell P, DesMeules M. Interactive social media interventions for health behaviour change, health outcomes, and health equity in the adult population. Cochrane Database of Systematic Reviews 2018; 2 : CD012932.

Welton NJ, Caldwell DM, Adamopoulos E, Vedhara K. Mixed treatment comparison meta-analysis of complex interventions: psychological interventions in coronary heart disease. American Journal of Epidemiology 2009; 169 : 1158–1165.

Williamson PR, Altman DG, Bagley H, Barnes KL, Blazeby JM, Brookes ST, Clarke M, Gargon E, Gorst S, Harman N, Kirkham JJ, McNair A, Prinsen CAC, Schmitt J, Terwee CB, Young B. The COMET Handbook: version 1.0. Trials 2017; 18 : 280.

For permission to re-use material from the Handbook (either academic or commercial), please see here for full details.

Evidence-Based Practice (EBP)

• The EBP Process
• Forming a Clinical Question
• Inclusion & Exclusion Criteria
• Acquiring Evidence
• Appraising the Quality of the Evidence
• Writing a Literature Review
• Finding Psychological Tests & Assessment Instruments

Selection Criteria

Inclusion and exclusion criteria are determined after formulating the research question but usually before the search is conducted (although preliminary scoping searches may need to be undertaken to determine appropriate criteria).  It may be helpful to determine the inclusion criteria and exclusion criteria for each PICO component.

Be aware that you may  introduce bias  into the final review if these are not used thoughtfully. 

Inclusion and exclusion are two sides of the same coin, so—depending on your perspective—a single database filter can be said to either include or exclude. For instance, if articles must be published within the last 3 years, that is inclusion. If articles cannot be more than 3 years old, that is exclusion. 

The most straightforward way to include or exclude results is to use database limiters (filters), usually found on the left side of the search results page.

Inclusion Criteria

Inclusion criteria are the elements of an article  that must be present  in order for it to be eligible for inclusion in a literature review. Some examples are:

• Included studies must have compared certain treatments
• Included studies must be a certain type (e.g., only Randomized Controlled Trials)
• Included studies must be located in a certain geographic area
• Included studies must have been published in the last 5 years

Exclusion Criteria

Exclusion criteria are the elements of an article that  disqualify the study from inclusion  in a literature review. Some examples are:

• Study used an observational design
• Study used a qualitative methodology
• Study was published more than 5 years ago
• Study was published in a language other than English
• << Previous: Forming a Clinical Question
• Next: Acquiring Evidence >>
• Last Updated: Nov 15, 2023 11:47 AM
• URL: https://libguides.umsl.edu/ebp

Systematic Reviews for Health Sciences and Medicine

• Systematic Reviews
• The research question
• Common search errors
• Search translation
• Managing results
• Inclusion and exclusion criteria
• Critical appraisal
• Updating a Review
• Resources by Review Stage

Inclusion and Exclusion Criteria

Inclusion and exclusion criteria set the boundaries for the systematic review.  They are determined after setting the research question usually before the search is conducted, however scoping searches may need to be undertaken to determine appropriate criteria.  Many different factors can be used as inclusion or exclusion criteria. Information about the inclusion and exclusion criteria is usually recorded as a paragraph or table within the methods section of the systematic review.   It may also be necessary to give the definitions, and source of the definition, used for particular concepts in the research question (e.g. adolescence, depression).  

Other inclusion/exclusion criteria can include the sample size, method of sampling or availability of a relevant comparison group in the study.  Where a single study is reported across multiple papers the findings from the papers may be merged or only the latest data may be included.

• << Previous: Managing results
• Next: Critical appraisal >>
• Last Updated: Mar 16, 2024 12:00 PM
• URL: https://unimelb.libguides.com/sysrev

Conducting Systematic Reviews in Sport, Exercise, and Physical Activity pp 55–66 Cite as

Inclusion and Exclusion Criteria

• David Tod 2  
• First Online: 30 August 2019

1445 Accesses

A systematic literature search can yield hundreds or thousands of records, each a potential relevant study. Sustained attention to detail is a pre-requisite for identifying relevant research. Well-constructed, clear, and explicit inclusion and exclusion criteria assist decision making consistency. In this chapter, I focus on inclusion and exclusion criteria, explain their benefits, provide guidelines on their construction and use, and illustrate with examples from sport, exercise, and physical activity research.

This is a preview of subscription content, log in via an institution .

Buying options

• Available as PDF
• Read on any device
• Instant download
• Own it forever
• Available as EPUB and PDF
• Compact, lightweight edition
• Dispatched in 3 to 5 business days
• Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Abrami, P. C., Cohen, P. A., & d’Apollonia, S. (1988). Implementation problems in meta-analysis. Review of Educational Research, 58, 151–179. https://doi.org/10.3102/00346543058002151 .

Article   Google Scholar  

Andersen, M. B. (2005). Coming full circle: From practice to research. In M. B. Andersen (Ed.), Sport psychology in practice (pp. 287–298). Champaign, IL: Human Kinetics.

Google Scholar  

Andersen, M. B., McCullagh, P., & Wilson, G. J. (2007). But what do the numbers really tell us? Arbitrary metrics and effect size reporting in sport psychology research. Journal of Sport and Exercise Psychology, 29, 664–672. https://doi.org/10.1123/jsep.29.5.664 .

Article   PubMed   Google Scholar  

Benzies, K. M., Premji, S., Hayden, K. A., & Serrett, K. (2006). State-of-the-evidence reviews: Advantages and challenges of including grey literature. Worldviews on Evidence-Based Nursing, 3, 55–61. https://doi.org/10.1111/j.1741-6787.2006.00051.x .

Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis . Chichester, UK: Wiley.

Book   Google Scholar  

Bruce, R., Chauvin, A., Trinquart, L., Ravaud, P., & Boutron, I. (2016). Impact of interventions to improve the quality of peer review of biomedical journals: A systematic review and meta-analysis. BMC Medicine, 14 , article 85. https://doi.org/10.1186/s12916-016-0631-5 .

Card, N. A. (2012). Applied meta-analysis for social science research . New York, NY: Guilford.

Centre for Reviews and Dissemination. (2009). Systematic reviews: CRD’s guidance for undertaking reviews in health care . York, UK: Author.

Cooke, A., Smith, D., & Booth, A. (2012). Beyond PICO: The SPIDER tool for qualitative evidence synthesis. Qualitative Health Research, 22, 1435–1443. https://doi.org/10.1177/1049732312452938 .

Hardy, J., Oliver, E., & Tod, D. (2009). A framework for the study and application of self-talk within sport. In S. D. Mellalieu & S. Hanton (Eds.), Advances in applied sport psychology: A review (pp. 37–74). London, UK: Routledge.

Holt, N. L., Neely, K. C., Slater, L. G., Camiré, M., Côté, J., Fraser-Thomas, J., … Tamminen, K. A. (2017). A grounded theory of positive youth development through sport based on results from a qualitative meta-study. International Review of Sport and Exercise Psychology, 10, 1–49. https://doi.org/10.1080/1750984X.2016.1180704 .

Horton, J., Vandermeer, B., Hartling, L., Tjosvold, L., Klassen, T. P., & Buscemi, N. (2010). Systematic review data extraction: Cross-sectional study showed that experience did not increase accuracy. Journal of Clinical Epidemiology, 63, 289–298. https://doi.org/10.1016/j.jclinepi.2009.04.007 .

Meline, T. (2006). Selecting studies for systematic review: Inclusion and exclusion criteria. Contemporary Issues in Communication Science and Disorders, 33, 21–27.

Munoz, S. R., & Bangdiwala, S. I. (1997). Interpretation of Kappa and B statistics measures of agreement. Journal of Applied Statistics, 24, 105–112. https://doi.org/10.1080/02664769723918 .

O’Connor, D., Green, S., & Higgins, J. P. T. (2011). Defining the review question and developing criteria for including studies. In J. P. T. Higgins & S. Green (Eds.), Cochrane handbook for systematic reviews of interventions . Version 5.1.0 [updated September 2011]: The Cochrane Collaboration. Retrieved from www.cochrane-handbook.org .

Petticrew, M., & Roberts, H. (2006). Systematic reviews in the social sciences: A practical guide . Malden, MA: Blackwell.

Roig, M., O’Brien, K., Kirk, G., Murray, R., McKinnon, P., Shadgan, B., & Reid, W. D. (2009). The effects of eccentric versus concentric resistance training on muscle strength and mass in healthy adults: A systematic review with meta-analysis. British Journal of Sports Medicine, 43 , 556–568. https://doi.org/10.1136/bjsm.2008.051417 .

Sackett, D. L., & Wennberg, J. E. (1997). Choosing the best research design for each question. British Medical Journal, 315, 1636. https://doi.org/10.1136/bmj.315.7123.1636 .

Article   PubMed   PubMed Central   Google Scholar  

Tod, D., & Edwards, C. (2015). A meta-analysis of the drive for muscularity’s relationships with exercise behaviour, disordered eating, supplement consumption, and exercise dependence. International Review of Sport and Exercise Psychology, 8, 185–203. https://doi.org/10.1080/1750984X.2015.1052089 .

Treadwell, J. R., Singh, S., Talati, R., McPheeters, M. L., & Reston, J. T. (2011). A framework for “best evidence” approaches in systematic reviews . Plymouth Meeting, PA: ECRI Institute Evidence-based Practice Center.

Walach, H., & Loef, M. (2015). Using a matrix-analytical approach to synthesizing evidence solved incompatibility problem in the hierarchy of evidence. Journal of Clinical Epidemiology, 68, 1251–1260. https://doi.org/10.1016/j.jclinepi.2015.03.027 .

Wilkinson, L., & Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54 , 594–604. https://doi.org/10.1037/0003-066x.54.8.594 .

Download references

Author information

Authors and affiliations.

School of Sport and Exercise Science, Liverpool John Moores University, Liverpool, UK

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to David Tod .

Rights and permissions

Reprints and permissions

Copyright information

© 2019 The Author(s)

About this chapter

Cite this chapter.

Tod, D. (2019). Inclusion and Exclusion Criteria. In: Conducting Systematic Reviews in Sport, Exercise, and Physical Activity. Palgrave Macmillan, Cham. https://doi.org/10.1007/978-3-030-12263-8_5

Download citation

DOI : https://doi.org/10.1007/978-3-030-12263-8_5

Published : 30 August 2019

Publisher Name : Palgrave Macmillan, Cham

Print ISBN : 978-3-030-12262-1

Online ISBN : 978-3-030-12263-8

eBook Packages : Behavioral Science and Psychology Behavioral Science and Psychology (R0)

Share this chapter

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Publish with us

Policies and ethics

• Find a journal
• Track your research

USF Libraries Hours by campus

Libraries locations.

• Libraries Hours
• Outages & Maintenance Alerts

RESEARCH TOOLS

• Subject & Course Guides
• USF Libraries Catalog
• Quicksearch All-in-one-search
• Citing Sources
• Find my Librarian

GUIDES / HOW-TO

• Tutorials & Workshops
• Finding Books and Articles
• Finding Reserves
• Checking Out & Renewing
• Reserve a Study Room
• Additional Help Topics
• star Other Services
• For Faculty
• For Graduate Students
• For Undergrads
• Requesting Books & Articles (ILL)
• Textbook Affordability (TAP)
• Library Instruction
• Laptop Checkout
• Schedule Research Help
• Geographic Information Systems
• Data Management Planning
• Copyright & Intellectual Property
• Scholarly Publishing
• Other Services

COLLECTIONS

• What are Collections?
• Special Collections
• Digital Collections
• Digital Heritage & Humanities
• Digital Commons @ USF
• Oral Histories
• Online Exhibitions
• Printing in the Library
• IT Help Desk
• Digital Media Commons (DMC)
• Writing Studio
• Office of Development
• Office for Undergraduate Research
• Directions to the Library
• Library Info & Floor Maps
• Connect From Off Campus
• Renew Materials Online
• Check UBorrow Status
• Printing Help
• Report a Problem
• About the USF Libraries

Systematic Reviews for Social Sciences

• Systematic Reviews
• Mapping Review
• Mixed Methods Review
• Rapid Review
• Scoping Review
• Develop the Question

Define Inclusion/Exclusion Criteria

• Develop A Review Protocol
• Create Search Strategies
• Select Studies
• Extract Data
• Assess the Quality of Studies
• Synthesize Data and Write the Report
• Review Tools
• Librarian Involvement
• Grey Literature

One of the features that distinguishes a systematic review from a narrative review is the pre-specification of criteria for including and excluding studies in the review (eligibility criteria). Explicit criteria, based on the review’s scope and question(s), are used to include and exclude studies. 

A large number of references (study titles and abstracts) will have been found at the searching stage of the review. A proportion of these will look as though they are relevant to the review's research questions. So, having explicit criteria against which to assess studies makes the process more efficient in terms of time.

More importantly, it also helps to avoid hidden bias by having clear consistent rules about which studies are being used to answer the review's specific research questions. 

Each study needs to be compared against same criteria. To be included in the review, a study needs to meet all inclusion criteria and not meet any exclusion criteria. Inclusion/eligibility criteria include participants, interventions and comparisons and often study design. Outcomes are usually not part of the criteria, though some reviews do legitimately restrict eligibility to specific outcomes.

For example, a systematic review include criteria may be determined using ECLIPSE.

• Expectation - identify best practices for information literacy instruction
• Client Group - Higher education students
• Location - United States
• Impact -Best practices for information literacy instruction
• Professionals - Librarians
• SErvice - Information literacy instruction

Exclusion criteria may include non-peer-reviewed articles, articles not in English, articles before a specified date, and in this case, articles about theory rather than actual practice.

• << Previous: Develop the Question
• Next: Develop A Review Protocol >>
• Last Updated: Aug 31, 2023 4:34 PM
• URL: https://guides.lib.usf.edu/systematicreviews

• Collections
• Research Help
• Teaching & Learning
• Library Home

Systematic Reviews

• Getting Started
• Additional Frameworks
• More Types of Reviews
• Timeline & Resources
• Inclusion/Exclusion Criteria
• Resources & More

Inclusion Criteria

Inclusion criteria are elements of an article that must be present in order for it to be eligible for inclusion in a review or analysis. Here are some examples:

• include studies with human subjects only
• include studies published within the last five years
• included studies must be randomized controlled studies or cohort studies
• included studies must have compared certain treatments

Exclusion Criteria

Exclusion criteria are the elements of an article that disqualify the study from inclusion in a review or analysis. Here are some examples:

• the study uses an observational design
• the study was published more than five years ago
• the study uses animal subjects

Why Have Inclusion/Exclusion Criteria?

After developing your PICO question, it is critically important for you and your team to establish a set of "rules" for selecting, then screening the articles located during your database and internet searches. These rules are called the inclusion/exclusion criteria. Be aware that you may introduce bias into the final review if these are not used thoughtfully and adhered to regardless of outcome. The purpose of using inclusions/exclusion criteria "is to minimize ambiguity and reduce the possibility of poor reproducibility". This criteria should not be too narrow or too loose. 

McDonagh M, Peterson K, Raina P, et al. Avoiding Bias in Selecting Studies. 2013 Feb 20. In: Methods Guide for Effectiveness and Comparative Effectiveness Reviews [Internet]. Rockville (MD): Agency for Healthcare Research and Quality (US); 2008-. Available from: https://www.ncbi.nlm.nih.gov/books/NBK126701/

A Visual Representation

Ghezzi-Kopel, Kate. (2019, September 16). Developing your research question. (research guide). Retrieved from https://guides.library.cornell.edu/systematic_reviews/research_question

• << Previous: Timeline & Resources
• Next: Resources & More >>
• Last Updated: Mar 18, 2024 2:33 PM
• URL: https://libguides.wvu.edu/SystematicReviews

An official website of the United States government

The .gov means it's official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you're on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings
• Browse Titles

NCBI Bookshelf. A service of the National Library of Medicine, National Institutes of Health.

Methods Guide for Effectiveness and Comparative Effectiveness Reviews [Internet]. Rockville (MD): Agency for Healthcare Research and Quality (US); 2008-.

Methods Guide for Effectiveness and Comparative Effectiveness Reviews [Internet].

Avoiding bias in selecting studies.

Marian McDonagh , PharmD, Kim Peterson , MS, Parminder Raina , PhD, Stephanie Chang , MD, MPH, and Paul Shekelle , MD, PhD, MPH.

Affiliations

Published: February 20, 2013 .

• One hypothesis-testing study and numerous case examples indicate that operational criteria guiding the selection of studies into a systematic review (SR) or meta-analysis can influence the conclusions.
• Assessments of how this source of bias can be reduced, or even the magnitude of the bias, are not available.
• In the absence of conclusive evidence about how to reduce this potential for bias, we recommend that inclusion criteria be clearly described in detail sufficient to avoid inconsistent application in study selection and that inclusion criteria be documented in a protocol.
• We propose hypothetical examples that illustrate how selection of inclusion and exclusion criteria may introduce bias.
• Experience suggests that dual review can identify inclusion criteria that are not sufficiently clear and occasions where subjective judgment may differ. Gray literature (e.g., U.S. Food and Drug Administration [FDA] documents, trial registry reports) can help identify and possibly reduce bias from publication bias or selective outcome reporting.

Much has been written about the importance of various aspects of the conduct of a SR: how to best search computerized databases; whether or not reviewers should be masked to the authors and journals and outcomes of studies being reviewed; how to assess studies for the risk of bias; and the strengths and weaknesses of various different methods of statistically combining the results. The Methods Guide for the Agency for Healthcare Research and Quality (AHRQ) Evidence-based Practice Center (EPC) Program has chapters summarizing the literature and best-practices advice on numerous such aspects of a SR. 1

We are concerned here with the potential for bias at a point upstream in the SR process—namely what is the effect of going from the initial question of interest (“what is the effect of intervention X on condition Y?”) to the operational aspects of the review (such as selecting inclusion/exclusion criteria). For example, in a recent Comparative Effectiveness Review on drugs to treat low bone density, the EPC identified nine prior meta-analyses evaluating the antifracture efficacy of alendronate compared with placebo or no treatment. 2 The meta-analyses were published between 1997 and 2009, and included between them 17 randomized controlled trials (RCTs) published between 1994 and 2004. One might expect that all the trials included in earlier meta-analyses would be included in later meta-analyses, but this is not the case. One meta-analysis published in 2002 included 10 trials, while another published in 2004 included only 5: 4 were among the 10 trials in the 2002 meta-analyses, but 1 trial (published in 1998) was not. Some of the differences in trial inclusion could be explained by whether data were included on vertebral and nonvertebral fractures; whether nonvertebral fractures were treated as a general group; whether nonvertebral fractures were split out into fractures of the hip or wrist; or whether patient populations were considered as secondary prevention or as primary prevention. These differences in which trials were included led to differences in conclusions. In one meta-analysis, 3 the conclusion was that the decrease in nonvertebral fractures was not statistically significant. In another meta-analysis 4 published 3 years earlier, the conclusion was that the beneficial effect of alendronate compared with placebo on nonvertebral fractures was statistically significant. All EPCs can tell similar stories.

Conflicting conclusions confuse decisionmakers, especially if all reviews purported to answer the same question and the differences in the applicability of the evidence are not clearly denoted. Bias results from systematic alteration from the truth. Although we do not know the exact truth, different conclusions lead readers to believe that alternate inclusion and exclusion criteria result in biased conclusions. In order to investigate the potential for this source of bias and identify methods studies that investigate how best to reduce it, we searched for studies that examined two or more SRs of the same topic, evaluating the impact of variation in study inclusion.

We found a very small number of relevant studies ( Table 1 ). 5 - 8 The most relevant example was a prospective study designed to examine reproducibility between two review groups (on different continents) commissioned to review evidence on the same question, using a common methods specification manual. 8 While the manual outlined the important features of inclusion criteria, the specific criteria used by each group are not reported. Search terms were specified a priori, and the groups were instructed to find and include all study designs, including non-English language, case series, ecological, cross-sectional, case-control, cohort, and intervention studies. Both review groups agreed on including 166 articles, but disagreed on 72 articles (Center A included 52 papers not included by Center B, and Center B included 20 papers not included by Center A). Sixty-three of the 72 discrepancies occurred in screening title and abstract; 9 of the 72 discrepancies occurred during review of full-text articles. Other similar retrospective studies also found differences in their lists of included studies and sometimes different conclusions ( Table 1 ). Although the amount of evidence is small to confirm the presence of bias, the potential for bias is possibly quite large.

Studies evaluating reasons for discrepancies in included studies among systematic reviews.

Other authors have addressed reasons for discrepant results from meta-analyses on the (seemingly) same topics. 9 , 10 Ioannidis has examined multiple such scenarios and concluded that the reasons for discrepancy are typically multifactorial, but include differing study questions and inclusion criteria as well as differences in the process of applying the criteria in study selection. He gives examples of situations where inclusion criteria for meta-analyses were apparently specified in way that would obtain results that supported the viewpoints of the authors rather than reflecting questions of clinical uncertainty. 9

As part of the EPC Methods Guide, we intend that this paper will guide EPCs when selecting studies for inclusion in an SR. Guidance is intended to reduce inconsistencies and risk of bias. Unfortunately, because there are no available studies to guide us how best to reduce this variation, what follows is based on fundamental principles of SRs and the experience of the EPC program.

Inconsistencies and bias can certainly occur during the development of key questions, which define the scope of the review and details the population(s), intervention(s), comparator(s), outcome(s), timing, and setting (PICOTS), and sometimes even the study designs or study characteristics of interest. The methods used by the EPC program at this earlier stage are discussed elsewhere. 11 Likewise, we recognize that bias can also be introduced during the searching stage, 12 or in how reviewers handle assessment of reporting biases, 13 and guidance on these methods are provided elsewhere. 11 , 12 This paper focuses on what to do with the literature once it is identified. We first describe the types of bias then stratify the guidance on addressing these biases into sections: Setting Inclusion Criteria to Avoid Bias in Selecting Studies, Study Selection Process, and Using Gray Literature to Assess and Reduce Bias.

• Types of Potential Biases in Selecting Studies

Spectrum Bias

The inclusion or exclusion of a specific population can have a dramatic impact on the conclusions for the effectiveness of a treatment. For example, while one meta-analysis found no significant benefit of the invasive treatment for coronary artery disease over conservative treatment, a subsequent meta-analysis by invasive cardiologists found significant benefit with invasive treatment when they included patients with unstable angina, a population in which invasive management is known to be more beneficial. 9

Publication bias and outcome reporting bias can have implications for the conclusions of a review. Bias in selection of studies may overlap with these biases, but methods for avoiding them are addressed in other chapters. 13 , 14

Random Error

Even when reviewers have a common understanding of the selection criteria, random error or mistakes may result from individual errors in reading and reviewing studies.

• Guidance for Setting Inclusion Criteria To Avoid Bias in Selecting Studies

Although setting inclusion criteria based on key questions may seem straightforward, the experience in the AHRQ EPC program has shown that this is often not the case. The AHRQ EPC program has an explicit process of systematic review development called Topic Refinement. Its goal is the development of inclusion criteria based on the Key Questions via a process that involves the review team and technical expert panel input.

One of the main goals in developing inclusion criteria is to minimize ambiguity. Greater ambiguity in inclusion criteria increases the possibility of poor reproducibility due to many subjective decisions regarding what to include, potentially resulting in at least random error in study selection.

The criteria should be set a priori and based on the analytic framework or conceptual model using a protocol. 15 - 17 The benefits of using a protocol specific to SRs include improving transparency and rigor of SRs, and important to this chapter, reducing bias in study selection decisions. Requirements for SR protocols for reviews conducted by EPCs are currently undergoing further development in coordination with other organizations (e.g., Institute of Medicine and PROSPERO). The protocol should be based on a standard set of elements, publicly available, ideally through a SR Registry, (e.g. PROSPERO, www.crd.york.ac.uk/prospero/ ).

However, there is a balance to be struck between making the inclusion criteria so narrow that it is unlikely that eligible evidence will be found and so loosely defined that it increases the possibility of poor reproducibility due to many subjective decisions regarding what to include. EPCs should attempt to strike this balance, but recognize that there will be times when their initial attempt is not working and changes need to be made. All eligibility criteria decisions should be reported transparently in the published SR.

Selecting PICOTS Criteria

In addition to random error from ambiguous definition of criteria, the selection of PICOTS inclusion or exclusion criteria can introduce systematic bias. A systematic review starts with a broad comprehensive search and the choice of which studies to include can directly influence the resulting conclusions. The EPC should carefully consider whether PICOTS criteria are effect modifiers and how inclusion and exclusion criteria may potentially skew the studies and thus results reported in the review.

Table 2 below suggests potential implications or biases that may result from specific hypothetical examples of inclusion and exclusion criteria.

Hypothetical examples of potential for bias based on inadequately defined PICOTs.

Inclusion criteria for the population(s) of interest should be defined in terms of relevant demographic variables, disease variables (i.e., variations in diagnostic criteria, disease stage, type, or severity), risk factors for disease, cointerventions, and coexisting conditions. 18 For example, if an SR is focusing only on adult populations, then the inclusion criteria should specify the age range of interest. Ambiguity in population inclusion criteria increases the risk that inclusion decisions could be influenced by differing viewpoints about potential relationships between particular demographic or disease factors and outcome. Table 2 illustrates one such example of how inadequate description of inclusion criteria for a heart failure population may bias the results of SR. Inclusion criteria for population subgroups of interest should also be defined with similar specificity.

Intervention and Comparators

Although the Key Questions may frame the interventions in broad terms such as “anticoagulants,” it is essential for the inclusion criteria to specify exactly which individual interventions are of interest, including their duration and intensity. Otherwise, reviewers may end up missing important interventions and thus overestimate or underestimate the effectiveness or harms of an intervention. This is particularly important in reviews of health care delivery programs that are less clearly defined. A review may examine a specific program as a whole, the component parts of a program, or the theoretical mechanism of action of a component part. Defining an intervention too narrowly may increase the confidence in effectiveness, but reduce the relevance of the finding for implementation in other settings.

To enhance readability, key questions may not always define the comparison, which may introduce both random and systematic error. Without specifying the comparator, one reviewer may compare the effectiveness of anticoagulants to compression stockings, another may compare them to early walking, and yet another may compare it to other anticoagulants. Selection of a comparison of known poor effectiveness may systematically bias the effectiveness of the intervention away from the null, whereas poor specification and thus inappropriate combination of comparisons may result in an uninterpretable result.

Regardless of the topic, SRs should focus on assessing a range of patient-centered outcomes, including both benefits and harms. The scope of included outcomes should address both effectiveness and harms on which strength of the evidence will be graded. 19 If intermediate outcomes are included they should be presented in context of how they relate to the clinically important harms and benefits (e.g., via an analytic framework) as outlined in the chapter of grading the strength of the evidence. 19 When there are a large number of outcomes included, EPCs should specify a priori which clinically important outcomes they will grade the strength of evidence. Despite the temptation to exclude studies that only report a specific outcome (e.g., mortality), EPCs should be cautious since this may augment the risk of identifying studies that have selectively published only outcomes with positive results (selective outcome reporting bias).

In order to reduce variation in study selection related to outcomes, we recommend that the inclusion criteria clearly identify and describe outcomes, outline any restrictions on measurement methods or timing of outcome measurement, and provide guidance for handling of composite outcomes. For clinical areas (such has pain and psychological functioning) that are notoriously characterized by variability in outcome measurement methods and a multitude of scales and instruments, the risk is greater for inconsistency in study selection. In these cases, it is especially important to consider how to handle this variation early in the SR process. The EPC may choose to restrict to specific measurement methods (i.e., only including studies that used measurement scales that have been published or validated), but need to consider what studies they will be eliminating and what effect this may have on the review. Study investigators that do not use the most commonly validated instruments may be systematically different from those that do. For example, investigators from different communities may use different instruments and systematic exclusion of these studies may exclude specific populations such as rural or small communities or nonacademic populations.

Lack of specificity on other aspects of outcome measurement may also bias SR conclusions. For example where study reports include multiple time points for outcome measurement, but the SR inclusion criteria are not adequately specific about the relative importance of different time points, the choice of which to include or to emphasize is left to the reviewers. This scenario could lead to important differences in conclusions depending on which outcome-time point pair are selected for inclusion, particularly in a meta-analysis. 10

Finally, it is ideal to consider individual outcome separately, rather than using composite outcomes. Composite endpoints are often difficult to interpret and may exaggerate the magnitude of treatment effect. 20 EPC reviewers should consider specifying whether composite outcomes are of interest and, if so, whether there is a need to place any restrictions on which combinations of outcomes are acceptable (e.g. those with similar importance to patients and magnitude of treatment effect). Otherwise, there may be variation in selection of studies that, for example, do not separately report mortality and cardiovascular events. EPC review teams should rely on empiric research when available to form the basis of any decisions to limit study selection based on outcomes.

Timeframe and Setting

Setting inclusion criteria for timeframe (duration of study, years of study conduct, etc.) and setting may not apply to all clinical questions. Reviewers should identify the expected time period of study that would be needed to identify effectiveness on patient-important outcomes and harms. Lack of specification for the need for long-term studies may overestimate the effect on short term outcomes, while under-reporting the effect on long term outcomes. EPCs should clearly specify any decision to limit studies based on followup duration and define a priori the most relevant time periods for the interventions, populations, and outcomes of interest. When the focus of a SR is confined to a particular setting, such as a nursing home environment or residential treatment center, the inclusion criteria should include guidance for considering eligibility of studies that include commingled or ill-defined settings. Reviewers should consider how interventions may be different in settings such as nursing homes or other long-term care settings compared with general inpatient or outpatient settings and how inclusion or exclusion of these settings may systematically bias the conclusions. The criterion for study setting may also be considered when setting the selection criteria for population.

Study Designs or Study Characteristics

Due to time, budget, or resource constraints as well as concerns about the validity and relevance of the studies, reviewers often make decisions about excluding studies based on study design features (randomization or nonallocation of treatment), study conduct (quality or risk of bias of individual study), language of publication, study size, or reporting of relevant data.

Observational studies make up the bulk of the published literature. EPCs should refer to the Methods guidance for when to include observational studies. 21 , 22 However after deciding to include observational studies, EPCs need to take special care in developing and testing criteria for determining eligibility. 4 Because of the lack of consensus on any single taxonomy for assigning labels to specific types of observational study designs, 23 EPC teams should define study designs with sufficient clarity so that their reviewers can consistently and correctly determine if a given study is eligible. Exclusion of observational studies without careful consideration about whether these studies may provide information that would not be available from RCTs (i.e., long-term outcomes or harms and representative populations) may bias the review conclusions.

Reviewers often include other study design or reporting characteristics as eligibility criteria. Reviewers may decide to restrict study inclusion based on sample size (e.g.,> 1,000 patients) or publication language (e.g., English language only). However, smaller studies or non-English studies may be systematically different from larger studies or English-language studies and limiting by these characteristics for convenience may introduce a systematic bias as well. For example, in a review of surgical and pharmaceutical interventions, studies on surgical interventions may be smaller than studies on pharmaceuticals, thus biasing a review that excludes small studies to find evidence on drugs but insufficient evidence on surgical interventions.

Typically such decisions are taken for reasons of time-efficiency. The assumption is that not employing such limits would yield a very large number of studies that would significantly increase workload without providing additional value in terms of high-quality evidence. Without empirical evidence relative to the topic area under review, it is not possible to rule out systematic bias. For example, the decision to use only English-language publications may be set because the review team does not have the ability to read other languages but the time and cost of translation are not feasible within the report timeline and budget. Studies of language restrictions in SRs have had variable results, from significant impact to very little impact, sometimes depending on the specific topic being studied. 24 - 34

The way that high risk of bias studies are handled in SRs also varies and may introduce bias. Once a study has been determined to have high risk of bias, options include outright exclusion; inclusion in evidence tables with or without inclusion in a narrative description of the evidence (possibly depending on whether the study constitutes the only evidence for a given intervention and/or outcome); or inclusion in quantitative analyses using weighting based on quality or sensitivity analysis. Including studies with a high risk of bias without appropriate weighting for their risk of bias may introduce bias in the SR. However, because assessments of risk of bias are never based entirely on empirical evidence, and are subjective by nature, outright exclusion of studies with high risk of bias may also introduce bias. Additionally, weighting in meta-analysis based on risk of bias assessments may introduce bias and has been shown to result in inconsistency. 35 EPCs should be explicit about how such studies will be handled, a priori. If studies with high risk of bias are to be excluded in any way, they should be clearly identified in the text or in an appendix. Such transparency improves the likelihood that erroneous ratings of studies with high risk of bias can be identified.

Study Selection Process

Even with clear, precise inclusion criteria, elements of subjectivity and potential for human error in study selection still exist. For example, inclusion judgments may be influenced by personal knowledge and understanding of the clinical area or study design (or lack thereof).

The study selection process is typically done in two stages; the first stage involves a preliminary assessment of only the titles and abstracts of the search results. The purpose of this step is to eliminate efficiently all obviously ineligible publications. The second stage involves a careful review of the full-text publications.

Dual review—having two reviewers independently assess citations for inclusion—is one method of reducing the risk of biased decisions on study inclusion, as is recommended in the Institute of Medicine's “What works in healthcare: standards for systematic reviews.” 36 Some form of dual review should be done at each stage to reduce the potential for random errors and bias. Reviewers compare decisions and resolve differences through discussion, consulting a third party when consensus cannot be reached. The third party should be an experienced senior reviewer. The two stages of assessment are discussed in more detail below. Dual review can help identify misunderstandings of the criteria and resolve them such that the studies included will truly fulfill the intended criteria.

At the title and abstract stage, one alternative to 100 percent dual review is to have one reviewer accept the citations that appear to meet inclusion criteria and send them on to full-text review, with a second reviewer assessing only those citations and abstracts that the first reviewer deemed ineligible. Although there is currently no empiric evidence to support this method, we speculate that the sensitivity of the process is increased although the specificity may be somewhat reduced; the tradeoff is a potentially larger pool of full-text articles to review but a lower chance of having missed an eligible study. Additionally there is a risk of reviewer bias, with the second reviewer's knowledge that the first reviewer had deemed the studies ineligible. A second reasonable alternative is to conduct dual review on a small percentage of the citations, insuring reliability of assessments before going on to have the remainder of citations assessed by a single reviewer. In this situation, we recommend that review teams start with a pilot phase, using screening forms based on the eligibility criteria, to screen a small number of studies (e.g., 10 to 20 percent), followed by discussion such that variation in interpretation of how the inclusion criteria should be applied can be resolved early on. For this calibration process we suggest pairing a methodologist with a clinical expert if possible. For the stage of reviewing of full-text articles we recommend that EPCs undertake a complete independent dual review.

Some experts assert that reviewers' knowledge of the identity of the study authors, institution, or journal, or year of publication may influence their decisions and that masking of these factors might be useful. 37 , 38 These assertions may be based on the findings of a randomized study conducted by Berlin, et al., where there was considerable disagreement between blinded and unblinded reviewers in selecting studies for meta-analysis in where reviewers were using the same inclusion criteria. 39 However, the conclusions of this study were that masking “during study selection and data extraction had neither a clinically nor a statistically significant effect on the summary odds ratio” and that masking required 1.3 hours per paper. Hence, masking of reviewers to manuscript details is not routinely recommended.

Testing of inter- or intra-rater reliability, using the kappa statistic is sometimes suggested as a necessary component of the dual review strategy. However, because the goal is to include the “right” studies and not necessarily to achieve perfect agreement, and using the usual dual review process should obviate the need for such testing, this approach is not generally recommended.

Documenting and reporting all decisions made in the study selection process at the full-text level provides transparency that is essential in allowing independent assessment of the potential for bias by readers of SRs. SRs should include the numbers of studies screened, assessed for eligibility, and included in the review, ideally in the form of a flow diagram as recommended in the Preferred Reporting Items for Systematic Review and Meta-Analyses (PRISMA) statement. 17

As a part of this transparency, SRs should include a listing of excluded studies, along with respective reasons for exclusion. The list of excluded studies is meant to document the reason that specific studies reviewed at the full-text level were excluded when a reader may reasonably think they might have been included. An example would be studies in which the population and interventions meet eligibility, but the study design or comparator does not.

Using Gray Literature To Assess and Reduce Bias

In reviewing gray literature documents, reviewers are seeking to identify unpublished studies and unpublished data supplemental to published studies. Just as excluding studies can cause systematic variation, different approaches to finding and including or using grey literature can also affect the studies included and thus the conclusions of a review. While there may be variation in definitions of gray literature in general, EPC guidance outlines the best practices for identifying gray literature from regulatory data (e.g., the FDA), manufacturers, and other unpublished information such as abstracts or trial registries (see Table 3 for descriptions). 12 At a minimum, knowledge of unpublished studies may lead the EPC to reduce their assessment of the strength of the body of evidence in the review because of the existence of grey literature may suggest evidence of publication bias. 40 There is a risk that the gray literature identified has a high risk of bias; that the reason for lack of publication was due to flaws in the study rather than negative results. In some cases, enough information may be available for the reviewers to assess study quality and include the study in the SR.

Sources of unpublished information for comparative effectiveness reviews.

A review of original protocols (i.e., registered with clinicaltrials.gov) may identify selective reporting in the published literature for outcomes in which there is a positive result. Comprehensive searches for protocols and identification of selective outcome reporting may lead a reviewer to reduce their confidence in a positive finding. EPC reviewers should be alert to the possibility that the study measured and analyzed the outcome of interest, but did not report the finding due to a negative result. Gray literature helps to provide some fuzzy information on areas that were previously a blind spot in SRs of only published literature.

Reviewing gray literature may be resource intensive, and it is not yet clear if or when the effort required is worth the potential benefit. Despite these limitations, the risk for selective and biased publication of studies makes the inclusion of gray literature a necessary component of high quality SRs until empirical evidence is available to provide further guidance. Given the complexity of gray literature and the likelihood that a given review may not be able to fully search and include all gray literature, we recommend that the review protocol define, a priori, the sources of gray literature ( Table 3 ), and the eligibility criteria applied to them. The following are our recommendations for how to approach selecting studies from gray literature documents in a way that will minimize potential bias in selection of studies:

• Identify studies for the SR using standard search techniques first and become familiar with these studies before reviewing gray literature documents.
• Assess studies in gray literature documents for eligibility in the SR using the key questions and inclusion criteria as discussed above.
• As some sources of gray literature will have overlap with published literature, for example, FDA documents and trial registries, reviewers should match studies in gray literature documents based on characteristics such as unique study identifies, sample size (by group), and study duration, to those found in published literature to remove any duplicates. This information is sometimes readily available, but often matching is difficult.
• As with assessment of other types of evidence, dual review is a good way to guard against potentially biased inclusion decisions. Reporting on the inclusion of unpublished studies or data is important to ensure transparency and to identify areas about which EPCs have less confidence that the reporting is unbiased because the included information had not been published and, therefore, had not yet been vetted through a peer review process.
• If gray literature search uncovers studies that were not included in the published literature, EPC must consider whether the studies have sufficient data and are of sufficient quality to be included in the analysis. If not, then consider whether the presence of such studies suggests that the published literature is biased and should be “downgraded” for publication bias in assessing the strength of evidence.

Because the studies in the FDA documents and trial registries are referred to by codes and because the publications of these studies may or may not also list these numbers, EPCs must often match up the studies using study characteristics (e.g., numbers of included patients, duration of study). Doing so allows reviewers to identify relevant unpublished studies or additional outcomes or and statistical analyses examined in a known study that had not been not reported in the published literature. This process, although lengthy, can help EPCs identify the full body of evidence that is relevant to the question and better identify or reduce bias in selection of studies. Comparing these documents to published manuscripts of the trials may also uncover changes in the definition the primary outcome or misrepresentation of the primary outcome. 41 Dual review of gray literature documents is recommended when assessing relevance for potential inclusion into the review.

EPCs may determine that unpublished, supplemental data from the documents in the scientific information packets (SIPs) pertaining to studies with publications may be appropriate for inclusion into their review. For example, subgroup analyses may be reported in SIPS that had not appeared in the published manuscript(s); however, EPCs do need to view these data with caution. EPC reviewers should have discussed and established a priori guidance on when to include specific types of unpublished data and how to handle such data when they are included. With respect to subgroup data or analyses, for example, the review team should define the clinically relevant subgroup populations (e.g., characterized by comorbidities and drug co-administration) during topic development and document them a priori in the inclusion criteria document. If SIPs present data on populations other than those identified as clinically relevant, then EPCs would not include them or include them only as hypothesis generating; alternatively, EPCs may consider formally amending the inclusion criteria if clinical expertise indicates that noninclusion of these subgroups was an oversight.

Our review of the literature indicates that systematic bias and random error can potentially occur in the selection of studies for SRs. Methods exist to reduce the likelihood of both problems, as described in this chapter. Some aspects of potential bias in study selection overlap with considerations to reduce bias when defining the key questions (discussed in further detail by Whitlock, et al. 11 ). Table 2 highlights some potential sources of bias that reviewers should consider when selecting inclusion and exclusion criteria. However these are only potential sources of bias and need further research to establish which may be more likely to introduce systematic bias into a review. Further, as this is likely topic specific, reviewers need to have a careful and considered approach in selecting inclusion and exclusion criteria. After selection of inclusion and exclusion criteria, reviewers should track the reasons for exclusions of studies and consider at the end whether exclusion of studies due to the reasons identified in Table 2 may have biased the study. The potential effect of excluding or combining studies on the results should be highlighted as a potential limitation in the Discussion section of the SR.

A potential source of bias that was not addressed in this paper is the assessment and management of conflict of interest for authors, funders, and others with input into the SR process, including technical experts, key informants, and peer reviewers. The possible impact of conflicts is unknown at this time, but is the subject of future research, and is addressed in the Institute of Medicine's Standards for Systematic Reviews. 15 EPCs must be aware of not only the possibility of outcome reporting bias of individual studies, but also their own presentation of outcomes and how that may be introduce bias into the interpretation of findings. While some of these issues have been touched on in this paper, they are the subject of future research as well.

EPC reviewers should explicitly consider how they handle the concept of “best evidence” in both inclusion and synthesis of studies. Even when studies technically meet all eligibility criteria, and are correctly identified for inclusion using rigorous assessment procedures, the level of contribution each eligible study will make to the body of evidence can vary importantly. Depending on the availability of the best possible evidence, EPCs may differ in the extent to which they use lower-strength evidence for a given SR.

For example, when the evidence from randomized controlled trials that directly compare interventions has no obvious gaps, then the value of lower-strength evidence from observational studies, indirect comparisons from placebo-controlled trials, and pooled analyses of only a select number of studies is lower than it would be if the EPC reviewers did encounter such gaps. Thus, when gaps exist in the best possible evidence, the value of lower-strength evidence is greater. Reviewers must rely on their expert judgment as to what constitutes a gap in the best possible evidence and to what extent to report the lower-strength evidence. Systematic bias or random error can occur when EPCs do not clearly establish decision rules for utilizing lower-strength evidence. 22

In summary, EPCs should write the key questions and inclusion criteria in a way that provides their reviewers with detail sufficient to minimize variation in interpretation. Discussion, dual review, and practice will aid in reducing potential bias by establishing consistent interpretation of the criteria. EPCs should disclose the studies evaluated at the full-text level and determined to be ineligible and provide brief reasons for those exclusions.

Reporting the steps taken to avoid bias in selecting studies, such as conducting dual review, tracing the resulting flow of studies through the review (e.g., PRISMA diagram), and reporting potentially relevant studies that were excluded (with reasons for their exclusion) in the SR is essential for transparency. Gray literature can provide evidence on publication bias and outcomes reporting bias; EPCs should use processes similar to those used with published literature in reviewing gray literature to avoid potential bias in selecting unpublished studies or data. Depending on the experience levels of the SR team members, the complexity of the clinical area, the size of the SR, and other factors, the exact approach to operationalizing the study selection process may vary somewhat from SR to SR. Below are some summary points to minimize various types of study selection bias.

• Define inclusion and exclusion criteria by PICOTS clearly and in a protocol. Reduce ambiguity as much as possible.
• Consider the risk of introducing spectrum bias when selecting populations.
• Define interventions with specificity such that they are applicable to the intended user of the review.
• Be cautious about excluding studies based on reporting of outcomes of interest.
• Dual review can help reduce random error in applying inclusion and exclusion criteria
• Examine grey literature for evidence of unpublished data or studies that may indicate the presence of publication bias or selective outcome reporting bias. Consider the risk of bias of this information before using the information in the review or to adjust the strength of evidence of the review.

This report is based on research conducted by the Oregon Health & Science University, McMaster University, and Southern California Evidence-based Practice Centers (EPCs) under contract to the Agency for Healthcare Research and Quality (AHRQ), Rockville, MD (Contract No. 290-2007-10057-I). The findings and conclusions in this document are those of the authors, who are responsible for its contents; the findings and conclusions do not necessarily represent the views of AHRQ. Therefore, no statement in this report should be construed as an official position of AHRQ or of the U.S. Department of Health and Human Services.

The information in this report is intended to help health care decisionmakers—patients and clinicians, health system leaders, and policymakers, among others—make well informed decisions and thereby improve the quality of health care services. This report is not intended to be a substitute for the application of clinical judgment. Anyone who makes decisions concerning the provision of clinical care should consider this report in the same way as any medical reference and in conjunction with all other pertinent information, i.e., in the context of available resources and circumstances presented by individual patients.

This report may be used, in whole or in part, as the basis for development of clinical practice guidelines and other quality enhancement tools, or as a basis for reimbursement and coverage policies. AHRQ or U.S. Department of Health and Human Services endorsement of such derivative products may not be stated or implied.

None of the investigators have any affiliations or financial involvement that conflicts with the material presented in this report.

Suggested Citation: McDonagh M, Peterson K, Raina P, Chang S, Shekelle P. Avoiding Bias in Selecting Studies. Methods Guide for Comparative Effectiveness Reviews. (Prepared by the Oregon Health & Science University, McMaster University, and Southern California Evidence-based Practice Centers under Contract No. 290-2007-10057-I.) AHRQ Publication No. 13-EHC045-EF. Rockville, MD: Agency for Healthcare Research and Quality; February 2013. www ​.effectivehealthcare ​.ahrq.gov/reports/final.cfm .

• Cite this Page McDonagh M, Peterson K, Raina P, et al. Avoiding Bias in Selecting Studies. 2013 Feb 20. In: Methods Guide for Effectiveness and Comparative Effectiveness Reviews [Internet]. Rockville (MD): Agency for Healthcare Research and Quality (US); 2008-.
• PDF version of this page (247K)

In this Page

Other titles in these collections.

• AHRQ Methods for Effective Health Care
• Health Services/Technology Assessment Text (HSTAT)

Related information

• PMC PubMed Central citations
• PubMed Links to PubMed

Similar articles in PubMed

• Folic acid supplementation and malaria susceptibility and severity among people taking antifolate antimalarial drugs in endemic areas. [Cochrane Database Syst Rev. 2022] Folic acid supplementation and malaria susceptibility and severity among people taking antifolate antimalarial drugs in endemic areas. Crider K, Williams J, Qi YP, Gutman J, Yeung L, Mai C, Finkelstain J, Mehta S, Pons-Duran C, Menéndez C, et al. Cochrane Database Syst Rev. 2022 Feb 1; 2(2022). Epub 2022 Feb 1.
• Review Inclusion of Nonrandomized Studies of Interventions in Systematic Reviews of Intervention Effectiveness: An Update [ 2022] Review Inclusion of Nonrandomized Studies of Interventions in Systematic Reviews of Intervention Effectiveness: An Update Saldanha IJ, Skelly AC, Ley KV, Wang Z, Berliner E, Bass EB, Devine B, Hammarlund N, Adam GP, Duan-Porter D, et al. 2022 Sep
• Review Evidence Brief: The Effectiveness Of Mandatory Computer-Based Trainings On Government Ethics, Workplace Harassment, Or Privacy And Information Security-Related Topics [ 2014] Review Evidence Brief: The Effectiveness Of Mandatory Computer-Based Trainings On Government Ethics, Workplace Harassment, Or Privacy And Information Security-Related Topics Peterson K, McCleery E. 2014 May
• Review Assessing the Risk of Bias of Individual Studies in Systematic Reviews of Health Care Interventions. [Methods Guide for Effectivenes...] Review Assessing the Risk of Bias of Individual Studies in Systematic Reviews of Health Care Interventions. Viswanathan M, Ansari MT, Berkman ND, Chang S, Hartling L, McPheeters M, Santaguida PL, Shamliyan T, Singh K, Tsertsvadze A, et al. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. 2008
• Association between pacifier use and breast-feeding, sudden infant death syndrome, infection and dental malocclusion. [JBI Libr Syst Rev. 2005] Association between pacifier use and breast-feeding, sudden infant death syndrome, infection and dental malocclusion. Callaghan A, Kendall G, Lock C, Mahony A, Payne J, Verrier L. JBI Libr Syst Rev. 2005; 3(6):1-33.

Recent Activity

• Avoiding Bias in Selecting Studies - Methods Guide for Effectiveness and Compara... Avoiding Bias in Selecting Studies - Methods Guide for Effectiveness and Comparative Effectiveness Reviews

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

Connect with NLM

National Library of Medicine 8600 Rockville Pike Bethesda, MD 20894

Web Policies FOIA HHS Vulnerability Disclosure

Help Accessibility Careers

• Introduction
• Preamble: Systematic Review: What it is and isn't
• Systematic Review Guidelines
• 1. Formulate a Research Question
• 2. Develop a Research Protocol
• 3. Conduct a Thorough Literature Search
• 4. Apply Inclusion and Exclusion Criteria
• 5. Perform Data Extraction/Abstraction
• 6. Conduct a Quality Appraisal of Included Studies
• 7. Complete Data Analysis and Compile Results
• 8. Interpret Results
• How to Appraise a Systematic Review
• Systematic Review Software and Tools
• Knowledge Synthesis Services This link opens in a new window

Systematic Review Overview : 4. Apply Inclusion and Exclusion Criteria

• Health Science Information Consortium of Toronto
• University Health Network - New
• Systematic Review Overview

Table of Contents

• Systematic Review Software

Step 4. Apply Inclusion and Exclusion Criteria

At the beginning of large systematic reviews, researchers discuss and develop a series of inclusion and exclusion criteria to fit in with their review question and/or the brief provided by whoever is funding the project.

Systematic reviews often exclude studies if they do not conform to specific study designs, are not written in English or within a certain time frame. As a researcher, you should be cautious of any bias you might introduce into the review by adding certain inclusion or exclusion criteria. For example: limiting to studies in English may miss important studies published in other languages, leading to language bias.

All decisions to include or exclude certain studies or groups of studies should be documented in the methods section of the research proposal/protocol - this way it can be demonstrated that a systematic process has been followed.

In large systematic reviews, the inclusion/exclusion criteria are applied by at least 2 reviewers to all the studies retrieved by the literature search. A strategy to resolve any disagreements between the reviewers should be outlined in the protocol, such as bringing in a third screener.

There are two levels of the screening process. The first level of screening involves scanning the titles and abstracts of the articles; those that are clearly irrelevant can be excluded.

Full text papers are obtained for the remaining articles and the criteria are applied again for the second level of screening on the full text. Those that meet the criteria are included in the review (although sometimes if too many papers are obtained, the question and criteria are refined and the process repeated). At this stage of screening, the reason for exclusion(s) must be recorded. This process is represented by the following flow diagram ( See PRISMA Flow Diagram ).

Key Points Regarding Study Selection

• Section 1.3.2. Process for Study selection (http://www.york.ac.uk/inst/crd/pdf/Systematic_Reviews.pdf, actual page #35)
• Studies should be selected in an unbiased way, based on selection criteria that flow directly from the review questions, and that have been piloted to check that they can be reliably applied.
• Study selection is a staged process involving sifting through the citations located by the search, retrieving full reports of potentially relevant citations and, from their assessment, identifying those studies that fulfill the inclusion criteria.
• Parallel independent assessments should be conducted to minimize the risk of errors of judgment. If disagreements occur between reviewers, they should be resolved according to a predefined strategy using consensus and arbitration as appropriate.
• The study selection process should be documented, detailing reasons for inclusion and exclusion.

Tips to Improve Inter-Rater Reliability / Screener Selection Accuracy

While awaiting search strategy development and final citation results:

• Provide clear and explicit inclusion and exclusion criteria, with definitions and explanations where warranted.
• Conduct thorough training for all involved.
• Provide clear guidelines which should be reviewed by all prior to starting the activity.
• Provide pilot testing or beta testing of screening tools/procedures, using samples/subsets of real data (with test inter-rater reliability calculations to determine preliminary agreement or variability).
• Optional: pilot or beta test screeners in pairs: one screener with previous experience paired with a more novice screener.
• Conduct ongoing, active surveillance/auditing of activities (can see if/when going off course)
• Provide ongoing opportunities for discussion, education, and training.
• The Screening Phase for Reviews Tutorial (5 min+) This tutorial presents information on the screening process for systematic reviews or other knowledge syntheses, and contains a variety of resources including guidelines, best practices, tips, and tools for successfully preparing to complete this important research stage.
• 1. Slavin RE. Best evidence synthesis: an intelligent alternative to meta-analysis. J Clin Epidemiol. 1995 Jan;48(1):9-18.
• 2. Eysenck HJ. Meta-analysis and its problems. BMJ. 1994 Sep 24;309(6957):789-92.
• 3. Moher D, Fortin P, Jadad AR, Juni P, Klassen T, Le Lorier J, et al. Completeness of reporting of trials published in languages other than English: implications for conduct and reporting of systematic reviews. Lancet. 1996 Feb 10;347(8998):363-6.
• 4. Vickers A, Goyal N, Harland R, Rees R. Do certain countries produce only positive results? A systematic review of controlled trials. Control Clin Trials. 1998 Apr;19(2):159-66.
• 5. Moher D, Pham B, Klassen T, Schultz KF, Berlin J, Jadad AR, et al. Does the language of publication of reports of randomized trials influence the estimates of intervention effectiveness reported in meta-analyses? Systematic Reviews: Evidence in Action,
• 6. PRISMA Statement. "The PRISMA Statement consists of a 27-item checklist and a four-phase flow diagram. It is an evolving document that is subject to change periodically as new evidence emerges. In fact, the PRISMA Statement is an update and expansion of the now-out dated QUORUM Statement. This website contains the current definitive version of the PRISMA Statement."
• << Previous: 3. Conduct a Thorough Literature Search
• Next: 5. Perform Data Extraction/Abstraction >>
• Last Updated: Feb 20, 2024 3:34 PM
• URL: https://guides.hsict.library.utoronto.ca/c.php?g=699108

We acknowledge this sacred land on which the University Health Network operates. For thousands of years it has been the traditional territory of the Huron-Wendat, the Haudenosaunee, and most recently, the Mississaugas of the Credit River. This territory was the subject of the Dish With One Spoon Wampum Belt Covenant, an agreement between the Haudenosaunee Confederacy and the Confederacy of the Ojibwe and allied nations to peaceably share and care for the resources around the Great Lakes. Today, the meeting place of Toronto is still the home to many Indigenous people from across Turtle Island and we are grateful to have the opportunity to work and learn on this territory

UHN Library and Information Services

Systematic Reviews

• Introduction to Systematic Reviews
• Systematic review
• Systematic literature review
• Scoping review
• Rapid evidence assessment / review
• Evidence and gap mapping exercise
• Meta-analysis
• Systematic Reviews in Science and Engineering
• Timescales and processes
• Question frameworks (e.g PICO)

Inclusion and exclusion criteria

• Using grey literature
• Search Strategy This link opens in a new window
• Subject heading searching (e.g MeSH)
• Database video & help guides This link opens in a new window
• Documenting your search and results
• Data management
• How the library can help
• Systematic reviews A to Z

Inclusion and exclusion criteria are a list of pre-defined characteristics to which literature must adhere to be included in a study. They are vital for the decision-making progress on what to review when undertaking a systematic review and will also help with systematic literature reviews.

You should be able to establish your inclusion/exclusion criteria during the process of defining your question. These criteria clearly demonstrate the scope of the study and provide justification for the exclusion of any information that does not meet these characteristics.

Example criteria

• Intervention, treatment, process or experience
• Reported outcomes
• Research methodology
• Participants
• Age of study
• Sample size
• Place of study
• Type of publication

E.g. stage 4 lung disease patients

E.g. whether the study's reported outcomes are relevant to your study and have been presented objectively

E.g. randomised control trial

E.g. age, sex ethnicity etc.

E.g. last 5 years

E.g. over 100 participants

E.g. UK based

E.g. primary research, peer-reviewed

E.g. community-based care

E.g. English

Precision vs Sensitivity

You should aim to be as extensive as possible when conducting searches for systematic reviews. However, it may be necessary to strike a balance between the sensitivity and precision of your search.

• Sensitivity – the number of relevant results identified divided by the total number of relevant results in existence
• Precision - the number of relevant results identified divided by the total number of results identified.

Increasing the comprehensiveness of a search will reduce its precision and will retrieve more non-relevant results. However, 

... at a conservatively-estimated reading rate of two abstracts per minute, the results of a   database search can be ‘scanread’ at the rate of 120 per hour (or approximately 1000  over an 8-hour period), so the high yield and low precision associated with systematic review searching is not as daunting as it might at first appear in comparison with the total  time to be invested in the review.  ( Cochrane Handbook for Systematic Reviews of Interventions, 2008, Section 6.4.4 )

• << Previous: Question frameworks (e.g PICO)
• Next: Where to search >>
• Last Updated: Jan 23, 2024 10:52 AM
• URL: https://plymouth.libguides.com/systematicreviews

Systematic Reviews: Inclusion and Exclusion Criteria

• What Type of Review is Right for You?
• What is in a Systematic Review
• Finding and Appraising Systematic Reviews
• Formulating Your Research Question
• Inclusion and Exclusion Criteria
• Creating a Protocol
• Results and PRISMA Flow Diagram
• Searching the Published Literature
• Searching the Gray Literature
• Methodology and Documentation
• Managing the Process
• Scoping Reviews

Defining Inclusion/Exclusion Criteria

An important part of the SR process is defining what will and will not be included in your review. 

Inclusion and exclusion criteria are developed after a research question is finalized but before a search is carried out. They determine the limits for the evidence synthesis and are typically reported in the methods section of the publication. For unfamiliar or unclear concepts, a definition may be necessary to adequately describe the criterion for readers. 

Some examples of common inclusion/exclusion criteria might be:

• Date of publication : only articles published in the last ten years
• Exposure to intervention/ or specific health condition : only people who have participated in the DASH diet
• Language of Publication* : only looking at English articles 
• Settings : Hospitals, nursing homes, schools
• Geography : specific locations such as states, countries, or specific populations

*note of caution: research is published all over the world and in multiple languages. Limiting to just English can be considered a bias to your research.

• Common Inclusion/Exclusion Criteria from the University of Melbourne

What happens if no study meets my inclusion/exclusion criteria?

Empty reviews are when no studies meet the inclusion criteria for a SR. Empty reviews are more likely to subject to publication bias, however, they are important in identifying gaps in the literature. 

• Unanswered questions implications of an empty review Slyer, Jason T. Unanswered questions, JBI Database of Systematic Reviews and Implementation Reports: June 2016 - Volume 14 - Issue 6 - p 1-2 doi: 10.11124/JBISRIR-2016-002934
• Rapid Prompting Method and Autism Spectrum Disorder: Systematic Review Exposes Lack of Evidence Schlosser, R.W., Hemsley, B., Shane, H. et al. Rapid Prompting Method and Autism Spectrum Disorder: Systematic Review Exposes Lack of Evidence. Rev J Autism Dev Disord 6, 403–412 (2019).
• << Previous: Formulating Your Research Question
• Next: Creating a Protocol >>
• Last Updated: Mar 5, 2024 3:43 PM
• URL: https://guides.lib.lsu.edu/Systematic_Reviews

Provide Website Feedback Accessibility Statement

• Library guides
• Book study rooms
• Library Workshops
• Library Account  

Library Services

• Library Services Home

Literature searching and finding evidence

• Literature searching or literature review?
• Use the PICO or PEO frameworks

Establish your Inclusion and Exclusion criteria

• Find related search terms
• Subject Heading/MeSH Searching
• Select databases to search
• Structure your search
• Search techniques
• Search key databases
• Manage results in EBSCOhost and Ovid
• Analyse your search results
• Document your search results
• Training and support

These criteria help you decide which studies will/will not be included in your work. This will help make sure your work is as unbiased, transparent and ethical as possible.

How to establish your Inclusion and Exclusion criteria

To establish your criteria you need to define each aspect of your question to clarify what you are focusing on, and consider if there are any variations you also wish to explore. This is where using frameworks like PICO help:

Example:   Alternatives to drugs for controlling headaches in children

Using the PICO structure you clarify what aspects you are most interested in. Here are some examples to consider:

The aspects of the topic you decide to focus on are the  Inclusion  criteria.

The aspects you don't wish to include are the  Exclusion  criteria.

• << Previous: Use the PICO or PEO frameworks
• Next: Find related search terms >>
• Last Updated: Mar 22, 2024 9:52 AM
• URL: https://libguides.city.ac.uk/SHS-Litsearchguide

• University of Texas Libraries

Literature Reviews

Determine inclusion and exclusion criteria.

• What is a literature review?
• Steps in the Literature Review Process
• Define your research question
• Choose databases and search
• Review Results
• Synthesize Results
• Analyze Results
• Librarian Support

Once you have a clearly defined research question, make sure you are getting precisely the right search results from searching the databases by making decisions about these items:

• Would the most recent five years be appropriate?
• Is your research from a more historical perspective?
• Where has this type of research taken place?
• Will you confine your results to the United States?
• To English speaking countries?
• Will you translate works if needed?
• Is there a particular methodology, or population that you are focused on?
• Are there adjacent fields in which this type of research has been conducted that you would like to include?
• Is there a controversy or debate in your research field that you want to highlight
• Are you creating a historical overview? Is this background reading for your research?
• Is there new technology that can shed light on an old problem or an old technology that can be used in a new way?
• Last Updated: Oct 26, 2022 2:49 PM
• URL: https://guides.lib.utexas.edu/literaturereviews

Systematic Reviews

• The Research Question
• Basic Service
• Full Service
• Inclusion and Exclusion Criteria
• Translating
• Deduplication
• Project Management Tools
• Useful Resources
• Other Review Types

Inclusion Criteria

Read chapter 3 of the cochrane handbook.

Identify the criteria that will be used to determine which research studies will be included. The inclusion and exclusion criteria must be decided before you start the review. Inclusion criteria is everything a study must have to be included. Exclusion criteria are the factors that would make a study ineligible to be included. Criteria that should be considered include:

Type of studies:  It is important to select articles with an appropriate study design for the research question. Dates for the studies and a timeline of the problem/issue being examined may need to be identified. 

Type of participants:  Identify the target population characteristics. It is important to define the target population's age, sex/gender, diagnosis, as well as any other relevant factors.

Types of intervention:  Describe the intervention being investigated. Consider whether to include interventions carried out globally or just in the United States. Eligibility criteria for interventions should include things such as the dose, delivery method, and duration of the investigated intervention. The interventions that are to be excluded may also need to be described here.

Types of outcome measures:  Outcome measures usually refer to measurable outcomes or ‘clinical changes in health’. For example, these could include body structures and functions like pain and fatigue, activities as in functional abilities, and participation or quality of life questionnaires.

Exclusion Criteria

A balance of specific inclusion and exclusion criteria is paramount. For some systematic reviews, there may already be a large pre-existing body of literature. The search strategy may retrieve thousands of results that must be screened. Having explicit exclusion criteria from the beginning allows those conducting the screening process, an efficient workflow. For the final product there should be a section in the review dedicated to 'Characteristics of excluded studies.' It is important to summarize why studies were excluded, especially if to a reader the study would appear to be eligible for the systematic review. 

For example, a team is conducting a systematic review regarding intervention options for the treatment of opioid addiction. The research team may want to exclude studies that also involve alcohol addiction to isolate the conditions for treatment interventions solely for opioid addiction. 

Exercise for Developing Inclusion/Exclusion

Before developing your inclusion/exclusion criteria, please read Chapter Three of the Cochrane Handbook that reviews considerations for developing this criteria.

You must have a selection of relevant articles (a max of 5). Review the articles and make a bullet point list for each study of why that study would be either included or excluded from the review. This exercise can help jump start your predefined inclusion and exclusion criteria. This should be done before you start the review. 

Types of Study Design

There are different study types used for the evidence base in systematic reviews. Below are some definitions of the different study types that may be used. 

• Randomized controlled trials (RCT) A group of patients is randomized into an experimental group and a control group to test the efficacy of a treatment/intervention. 
• Cohort study Involves the identification of two groups (cohorts) of patients, one which did receive the exposure of interest, and one which did not, and following these cohorts forward for the outcome of interest.
• Case-control study Involves identifying patients who have the outcome of interest (cases) and control patients without the same outcome, and looking to see if they had the exposure of interest. Just like cohort studies, case-control studies are observational.
• Cross-sectional study Typically involves the surveying of a randomly selected group to find out their opinions or facts. These studies can answer questions such as how common a particular disease is, but cause and treatment of the disease cannot be gleaned.
• Qualitative study Collects information on patients with diseases and those close to them. Requires specialized tools for analyzation and interpretation. These studies typically want to access a person's experience .
• Meta-analysis A statistical analysis, which can either be a study in itself or a component of another study type. It uses quantitative methods to summarize the results of scientific studies. 

For further reading

Check out the Institute for Quality and Efficiency in Health Care (IQWiG) article What types of studies are there?  offered through the National Center for Biotechnology Information, U.S. National Library of Medicine.

• << Previous: Planning and Protocols
• Next: Searching >>
• Last Updated: Mar 18, 2024 1:40 PM
• URL: https://libguides.library.tmc.edu/Systematic_Reviews

Literature Reviews: Systematic, Scoping, Integrative

• Introduction to Literature Reviews
• Creating a Search Strategy

Using Limits and Inclusion/Exclusion Criteria

Filters and Limits

Inclusion and Exclusion Criteria

Limits and Inclusion Criteria

• Review Protocols
• Elements of a Systematic Review
• Review Tools and Applications

Ask A Librarian

Once you have some search results, you will need to decide which articles you will actually use in your literature review. This can be done using filters/limits in the databases, applying inclusion/exclusion criteria, and appraising the articles.

Filters and limits (the name varies by database) are tools the database provides to help you narrow your search results. Different databases offer different filters, but these are some of the more common ones you'll find.

• Publication year
• Language of the article
• Age of study subjects
• Study design

Your search terms and the filters/limits you apply are generally not enough to narrow your results to the most relevant and highest quality studies for your project. The final step to selecting these studies is to apply your inclusion and exclusion criteria. Basically, these are the reasons why you keep (include) or reject (exclude) articles as you look through the results, reading titles and abstracts (and sometimes the whole article)

Examples of types of Inclusion/Exclusion Criteria

• PICO(T) elements - if one of the main elements of your topic does not match those of the study, you may need to exclude it
• Age - if you can't use a filter/limit to exclude studies that do not focus on the age group you require, you may need to exclude those studies yourself.
• Setting - i.e. home, acute care, assisted living facility
• Study Design - sometimes a filter/limit doesn't exist for the study design you're interested in; in that case you'll need to look through articles to find that detail yourself.
• Number of subjects - do you have a minimum study group size? 
• Study drop-out rate  

• << Previous: Creating a Search Strategy
• Next: Review Protocols >>
• Last Updated: Mar 13, 2024 7:58 AM
• URL: https://libguides.massgeneral.org/reviews

Selecting Studies for Systematic Review: Inclusion and Exclusion Criteria

Automate every stage of your literature review to produce evidence-based research faster and more accurately.

One factor that can help make sure your review is of high quality, is the effective and appropriate selection of inclusion and exclusion criteria to be used in the systematic review. This vital factor has an impact on what is discussed and debated in your review. In this article, we explore what these criteria actually are, how you establish them, and why they are critical to the final publication.

What Are Inclusion And Exclusion Criteria?

Before identifying what criteria to include and what to exclude, it is always beneficial to establish what exactly is meant by both of those terms. In short, inclusion criteria are the characteristics that define the population eligible for a study, or that define the studies that will be eligible for inclusion in a systematic review. In contrast, exclusion criteria are a set of characteristics of studies that will not be included in the review. These inclusion and exclusion criteria help to evaluate the merit of the review in question.

How Do You Choose Inclusion And Exclusion Criteria?

Generally speaking, it is best to choose the inclusion criteria of your systematic review after you have determined what research question you are hoping to answer. When you choose the criteria to both to include and (importantly) exclude studies, your research question sets boundaries to it, so that the eligibility criteria are neither too broad nor too narrow.

Learn More About DistillerSR

(Article continues below)

Why are Inclusion And Exclusion Criteria Important In Systematic Review?

When you define what your inclusion criteria are, and outline what your exclusion criteria are, you are improving the chances of your systematic review producing reliable results. By identifying what can and cannot be included in your review, you reduce any ambiguity in your findings. When you reduce ambiguity, your findings are far more persuasive and less subject to debate, or criticism. As a result, you are ensuring that all the time you spent on your systematic review was worthwhile and any costs were worth the expense.

Appropriate choice of inclusion and exclusion criteria is also essential to prevent any inherent selection bias. Again, such biases can result in your final review losing its persuasive power, as your results are not considered reliable. An inherent bias can mean your findings and results are liable to a ‘data mining’ style scenario, where your research question cannot be answered fairly. It could mean that your research does not take into account key variables that may have better informed your final discussion.

In short, inclusion and exclusion criteria selection is not only important it is essential. By making effective selections, you guarantee that your research’s validity cannot be questioned.

Selecting Studies for Systematic Reviews

Conducting a systematic review can be a time-consuming process to generate evidence around specific areas of research. As a result, it may be tempting for some authors to make quick selections with regard to inclusion and exclusion criteria. This lapse in judgment could prove detrimental to the review process.  Exclusion and inclusion criteria optimize the scope of your research and provide the necessary merit to your research question.

3 Reasons to Connect

Writing your thesis and conducting a literature review

• Writing your thesis
• Your literature review
• Defining a research question
• Choosing where to search
• Search strings
• Limiters and filters

Developing inclusion/exclusion criteria

• Managing your search results
• Screening, evaluating and recording
• Snowballing and grey literature
• Further information and resources

A feature of the systematic literature review is using pre-specified criteria to include/exclude studies. Through searching the literature and formulating your review questions, for example by using PICO, PEO , etc., you will be able to define the specific attributes that research studies must have to be eligible for inclusion in your review, along with other attributes that will exclude them. These attributes will form your inclusion and exclusion criteria, which you will use to assess the relevance and quality of the studies to be included in your final analysis.

Examples of inclusion/exclusion criteria could be:

• Language, e.g., only include articles published in English.
• Timeframe, e.g., papers published after a certain date.
• Geographic location, e.g., UK only.
• Format, e.g., peer reviewed journal articles.
• Type of research, e.g., case studies, empirical papers, qualitative research.

To justify their use, you will need to provide a rationale for each of your inclusion/exclusion criteria.

You will find examples of inclusion/exclusion criteria in research theses on CERES, Cranfield’s repository. Simply keyword search for “systematic review”.

• << Previous: Running your search and reporting search terms
• Next: Managing your search results >>
• Last Updated: Feb 21, 2024 2:01 PM
• URL: https://library.cranfield.ac.uk/writing-your-thesis

• Open access
• Published: 22 March 2024

Systematic review of empiric studies on lockdowns, workplace closures, and other non-pharmaceutical interventions in non-healthcare workplaces during the initial year of the COVID-19 pandemic: benefits and selected unintended consequences

• Faruque Ahmed 1 ,
• Livvy Shafer 1 , 2 ,
• Pallavi Malla 1 , 2 ,
• Roderick Hopkins 1 , 3 ,
• Sarah Moreland 1 , 2 ,
• Nicole Zviedrite 1 &
• Amra Uzicanin 1  

BMC Public Health volume  24 , Article number:  884 ( 2024 ) Cite this article

Metrics details

We conducted a systematic review aimed to evaluate the effects of non-pharmaceutical interventions within non-healthcare workplaces and community-level workplace closures and lockdowns on COVID-19 morbidity and mortality, selected mental disorders, and employment outcomes in workers or the general population.

The inclusion criteria included randomized controlled trials and non-randomized studies of interventions. The exclusion criteria included modeling studies. Electronic searches were conducted using MEDLINE, Embase, and other databases from January 1, 2020, through May 11, 2021. Risk of bias was assessed using the Risk of Bias in Non-Randomized Studies of Interventions (ROBINS-I) tool. Meta-analysis and sign tests were performed.

A total of 60 observational studies met the inclusion criteria. There were 40 studies on COVID-19 outcomes, 15 on anxiety and depression symptoms, and five on unemployment and labor force participation. There was a paucity of studies on physical distancing, physical barriers, and symptom and temperature screening within workplaces. The sign test indicated that lockdown reduced COVID-19 incidence or case growth rate (23 studies, p  < 0.001), reproduction number (11 studies, p  < 0.001), and COVID-19 mortality or death growth rate (seven studies, p  < 0.05) in the general population. Lockdown did not have any effect on anxiety symptoms (pooled standardized mean difference = -0.02, 95% CI: -0.06, 0.02). Lockdown had a small effect on increasing depression symptoms (pooled standardized mean difference = 0.16, 95% CI: 0.10, 0.21), but publication bias could account for the observed effect. Lockdown increased unemployment (pooled mean difference = 4.48 percentage points, 95% CI: 1.79, 7.17) and decreased labor force participation (pooled mean difference = -2.46 percentage points, 95% CI: -3.16, -1.77). The risk of bias for most of the studies on COVID-19 or employment outcomes was moderate or serious. The risk of bias for the studies on anxiety or depression symptoms was serious or critical.

Conclusions

Empiric studies indicated that lockdown reduced the impact of COVID-19, but that it had notable unwanted effects. There is a pronounced paucity of studies on the effect of interventions within still-open workplaces. It is important for countries that implement lockdown in future pandemics to consider strategies to mitigate these unintended consequences.

Systematic review registration

PROSPERO registration # CRD42020182660.

Peer Review reports

Coronavirus disease (COVID-19) is an infectious disease caused by Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) that emerged in December 2019. COVID-19 has caused a global pandemic that resulted in long-term health problems as well as millions of deaths around the world [ 1 ]. The World Health Organization Director-General indicated that all countries must strike a fine balance between protecting health and minimizing economic and social disruption [ 2 ]. Several community-level containment and closure policies were implemented by government authorities to reduce the transmission of SARS-CoV-2 and avert overwhelming of healthcare systems. These policies included cancellation of public events, restrictions on gathering sizes, restrictions on internal movement and international travel, closure of public transport systems, school closures, closures of non-essential businesses, and lockdowns [ 3 ]. Governments provided fiscal support to varying extents to reduce financial hardship due to the COVID-19 pandemic and the interventions to reduce SARS-CoV-2 transmission [ 1 , 4 , 5 ].

About two-thirds of the global population over 15 years of age participate in the labor force [ 6 ]. SARS-CoV-2 transmission can occur in workplaces through respiratory droplets and aerosols generated by pre-symptomatic, asymptomatic, or symptomatic persons and through fomites [ 7 , 8 ]. In 2020, employers were encouraged to implement several measures to prevent and reduce the transmission of SARS-CoV-2 within the workplace, including use of face masks or coverings, physical distancing to increase physical space between people and decrease the frequency of face-to-face contact (including teleworking), symptom and temperature screening, flexible leave policies to facilitate self-isolation of sick workers, environmental cleaning and disinfection, and engineering controls to improve air quality (Additional file 1 : Appendix Table S1) [ 9 , 10 , 11 ]. These measures could be used by essential businesses that were not subject to government-mandated closures and by all businesses when lockdowns were not in effect.

Research has primarily focused on preventing or reducing SARS-CoV-2 infection in healthcare workers, with non-healthcare workers receiving less attention [ 12 ]. A Cochrane systematic review on interventions in non-healthcare workplaces examined the effect of interventions introduced by researchers [ 12 ]. The review identified one study that met their inclusion criteria, which was a cluster-randomized non-inferiority trial that assigned staff working in schools to standard isolation after contact with a SARS-CoV-2-infected person or to daily COVID-19 testing and staying at work if the test was negative. Because randomizing employers or geographic regions to workplace-related non-pharmaceutical interventions (NPIs) may not be feasible or ethical during an outbreak, observational studies may provide the best available evidence. We conducted a systematic review to assess the benefits and unintended consequences of NPIs in non-healthcare workplaces that included observational studies. The objectives of our review were to evaluate the effects of NPIs within non-healthcare workplaces and community-level workplace closures and lockdowns, compared to no intervention, on the following outcomes in workers or the general population: 1) COVID-19 morbidity and mortality, 2) selected mental disorders, and 3) employment outcomes.

We registered our systematic review protocol on PROSPERO (ID # CRD42020182660) [ 13 ]. We reported the review according to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) statement (Appendix Table S 2 ) [ 14 ].

Protocol amendments

We amended our original protocol to exclude studies on severe acute respiratory syndrome (SARS) and Middle East respiratory syndrome (MERS). We included lockdown that affects workplaces and selected mental disorders. We excluded the following interventions: staying home when ill, respiratory etiquette, and cleaning and disinfection of frequently touched surfaces and objects. We excluded qualitative and modeling studies. We examined the references of relevant systematic reviews to identify studies that met our inclusion criteria instead of performing a systematic review of systematic reviews.

Literature search strategy and study selection

Electronic searches of the published and grey literature were conducted using MEDLINE, Embase, PsycINFO, Cumulative Index to Nursing and Allied Health Literature (CINAHL), Scopus, Cochrane Library, NIOSHTIC-2, and EconLit to identify studies published in English from January 1, 2020, through May 11, 2021. The search strategy is provided in Appendix Table S 3 . Additional studies were identified through authors’ knowledge and examination of references of included studies and previous systematic reviews.

Inclusion and exclusion criteria

The inclusion criteria included randomized controlled trials and non-randomized studies (cohort, case–control, before-after, controlled before-after, interrupted time series). Cohort studies include both inception cohorts and retrospective cohorts. Controlled before-after studies commonly present a ‘difference in differences’ analysis, where before-after differences in the outcome are compared between the intervention and comparator groups. Before-after and controlled before-after studies can include measurements on the same individual before and after the intervention, or on different individuals at each time point. Interrupted time series studies are those with at least three measurement times before the intervention and at least three measurement times after the intervention. More details about the study designs are available elsewhere [ 15 ].

The population of interest was persons working in non-healthcare settings, with no restrictions regarding age, sex, or race/ethnicity. We included the following NPIs within non-healthcare workplaces: 1) Physical distancing (e.g., increased use of telework, email, and teleconferences; increasing physical space between employees; modifying schedules for on-site work; staggered work hours; limiting customers in indoor spaces, including capacity restrictions and outdoor dining; increasing physical space between employees and customers, including delivering services remotely, drive-through service, curbside pick-up, or delivery); 2) Physical barriers (e.g., plexiglass partitions between workstations or at other points of close, frequent contact); 3) Symptom and/or temperature screening to prevent potentially infectious persons entering the job site. We also included community-level initial business closures (e.g., closing of restaurants, bars, and entertainment venues), closures of workplaces with exceptions for essential workers, and lockdowns. Lockdowns represent government mandates to stay home except for essential work or necessities and often include several but not necessarily all of the following in a geographic area: closure of non-essential businesses, restaurants and entertainment facilities; closure of schools and universities; prohibition of indoor and outdoor gatherings; restrictions on non-essential travel [ 16 , 17 ]. Lockdowns are also called stay-at-home or shelter-in-place orders [ 18 ]. Persons may telework, if feasible, during workplace closures and lockdowns.

We assessed both public health benefits and selected unintended consequences of an intervention. The beneficiaries may be workers or the general population (including both working and non-working persons of any age). The benefits examined were reduction of COVID-19 morbidity and mortality: COVID-19 incidence, case growth rate, reproductive number, epidemic doubling time, COVID-19 mortality, death growth rate. COVID-19 incidence is defined as the number of new cases per 100,000 population, and COVID-19 mortality represents the number of COVID-19-attributed deaths per 100,000 population over a specified time period; the case or death growth rate is the percent increase/decrease in daily incidence of cases or deaths, respectively [ 19 ]. The reproduction number is the average number of secondary cases each current case would produce, and the epidemic doubling time is the number of days required for the daily incidence to double [ 19 ].

The unintended consequences assessed were anxiety and depression symptoms in workers or the general adult population (including both working and non-working persons), and unemployment and labor force participation rates in persons ages 16 years and older. Anxiety is characterized by excessive fear and worry and related behavioral disturbances [ 20 ], and depression is characterized by persistent sadness and a lack of interest or pleasure in previously rewarding or enjoyable activities [ 21 ]. The labor force participation rate is the number of people who are either employed or actively looking for work as a percentage of the civilian noninstitutional population aged 16 years and older [ 22 ]. The unemployment rate is the number of employed people as a percentage of the number of people who are employed or actively looking for work. People who are not actively looking for work are excluded from the denominator for computing the unemployment rate.

The exclusion criteria included the following: 1) Studies on SARS, MERS, influenza, influenza-like illness, or other diseases; 2) Editorials, commentaries, narrative reviews, as well as case series, cross-sectional, qualitative, and modeling studies; 3) Studies in healthcare, long-term care, nursing home, school, or university settings; 4) Studies on children, family members of healthcare workers or patients, or studies in animals; 5) Studies on hand hygiene, respiratory hygiene (including face mask or covering), generic physical distancing with no specific mention of workplace physical distancing, environmental cleaning and disinfection, isolation, quarantine, postponing work-related travel, or building engineering controls (e.g., ventilation, avoiding air recirculation, particle filtration, ultraviolet germicidal irradiation); 6) Studies that lacked a "no intervention" comparator; 7) Studies on mobility, workplace social contact rates, air pollution, access to health care (e.g., visits to physicians, cancer screening), mental disorders other than anxiety or depression (e.g., post-traumatic stress disorder), or employment outcomes other than unemployment and labor force participation (e.g., reduced work hours); 8) Publications in languages other than English.

Data extraction and risk of bias assessment

Seven reviewers independently performed title and abstract screening, full text reviews, and data extraction using Covidence software, with each record reviewed by two persons [ 23 ]. The variables for which data were extracted included the following: country, population, source of outcome data, sample size, period of data collection, intervention, comparator, outcomes (COVID-19 incidence or case growth rate, epidemic doubling time, reproduction number, COVID-19 mortality or death growth rate, anxiety symptoms, depression symptoms, unemployment, labor force participation), study design, and funding source. Any disagreements were resolved through discussion or by a third reviewer. All risk of bias assessments were reviewed by a senior author. Study investigators were not contacted.

We did not identify any eligible randomized controlled trial. The quality of observational studies was assessed using the Risk of Bias in Non-Randomized Studies of Interventions (ROBINS-I) tool, which assesses the risk of bias of non-randomized studies compared to a well-performed randomized trial [ 15 , 24 ]. Our effect of interest was assignment to intervention as opposed to adherence to intervention. The ROBINS-I tool has seven bias domains: confounding, selection of participants into the study, classification of interventions, deviations from intended interventions, missing data, measurement of outcome, and selection of the reported result. To assess confounding for COVID-19 outcomes, we examined whether studies adjusted for population characteristics (age structure, population size) and for social contact or proxies for social contact at baseline (e.g., mobility, population density, occupation, socioeconomic variables such as income or education) [ 25 ]. For anxiety and depression outcomes, we assessed whether studies adjusted for age, sex, marital status, and socioeconomic status [ 26 ]. For employment outcomes, we assessed adjustment for age, sex, and education [ 27 ].

The risk of bias judgment for each ROBINS-I domain is classified as follows: low (study is comparable to a well-performed randomized trial), moderate (study appears to provide sound evidence for a non-randomized study but cannot be considered comparable to a well-performed randomized trial), serious (study has one or more important problems), and critical (study is too problematic to provide any useful evidence on the effect of the intervention). It is rare for a non- randomized study to be judged as low risk of confounding because of the potential for residual or unmeasured confounding. Before-after studies are usually judged to have at least serious risk of bias because it is not possible to determine whether pre-post changes are due to the intervention rather than other factors. A particular level of risk of bias for a specific domain will mean that the overall risk of bias across domains for a study is at least this severe for the outcome being assessed.

Data synthesis

A study could include more than one intervention or more than one outcome. Because studies used several instruments to measure anxiety and depression symptoms, we computed the standardized mean difference (mean difference in each study divided by that study’s standard deviation) to enable comparison across studies [ 28 ]. We conducted random-effects meta-analysis to compute pooled effect sizes for anxiety, depression, unemployment, and labor force participation using the Comprehensive Meta-Analysis software [ 29 ]. We created funnel plots if there were at least 10 studies and used the Trim and Fill adjustment to estimate the true effect size if there was publication bias [ 28 ]. We could not perform meta-analysis of studies on COVID-19 morbidity and mortality because these studies rarely reported sample sizes; we performed the sign test where a non-significant p -value (two-sided) supports the null hypothesis that the mean effect across studies is zero [ 28 ].

Search of the databases yielded 15,529 studies. After screening titles and abstracts, we reviewed the full text of 853 studies for eligibility (Fig.  1 ). Among these studies, we excluded 806 that did not meet the inclusion criteria. The percent agreement between reviewers was 95% for title and abstract screening and 87% for full-text reviews. We identified 47 observational studies through database searching and 13 via other sources (i.e., examination of references of previous systematic reviews and authors’ knowledge), yielding a total of 60 observational studies that met the inclusion criteria. Forty studies reported on COVID-19 morbidity and mortality outcomes (Appendix Table S 4 ) [ 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 , 62 , 63 , 64 , 65 , 66 , 67 , 68 , 69 , 70 ], 15 assessed the effect on anxiety and depression symptoms (Appendix Table S 5 ) [ 71 , 72 , 73 , 74 , 75 , 76 , 77 , 78 , 79 , 80 , 81 , 82 , 83 , 84 , 85 ], and five assessed the effect on unemployment and labor force participation (Appendix Table S 6 ) [ 5 , 86 , 87 , 88 , 89 ]. The studies were based on data from the first year of the pandemic, mostly covering the period March to July 2020. The domain-specific and overall risk of bias for each study are shown in Appendix Tables S 7 -S 9 . Studies that were excluded from the review are listed in Appendix Table S 10 .

Systematic review of the effects of non-pharmaceutical interventions in non-healthcare workplaces, January 1, 2020–May 11, 2021

Of the 40 studies on COVID-19 morbidity and mortality, 16 were based on data from the USA, and 13 studies analyzed data from multiple countries, ranging from 2 to 202 countries (Appendix Table S 4 ). Other studies included data from countries in Europe (Spain, Italy, Germany), Asia (India, China), Africa (South Africa), and Australia. The median study period over which outcome data were collected was 10 weeks (interquartile range: 8 weeks, 17 weeks). The overall risk of bias was moderate for 25 studies, serious for 14 studies, and critical for one study (Appendix Table S 7 ). All studies had at least a moderate risk of confounding, and most studies had a low risk of bias for the other six domains. Although studies on physical distancing (teleworking) [ 41 ] and physical barriers [ 45 ] reported a significant decrease in COVID-19 incidence in workers, and studies on initial business closures (i.e., restaurant or entertainment business closures) reported a significant decrease in COVID-19 case growth rate and epidemic doubling time in the general population [ 35 , 67 ], the sign tests were not significant (Table  1 ). Studies on workplace closures reported a decrease in COVID-19 incidence or case growth rate (six of seven studies) and reproduction number (four studies) in the general population, but the sign tests were not significant (Table  1 ). Studies showed that lockdown significantly decreased COVID-19 incidence or case growth rate (23 studies, p  < 0.001), reproduction number (11 studies, p  < 0.001), and COVID-19 mortality or death growth rate (seven studies, p  < 0.05) in the general population (Table  1 ). The 23 studies on the effect of lockdown on COVID-19 incidence or case growth rate reported a variety of effect measures, with seven studies reporting percentage decrease in daily case growth rate (median: 6 percentage decrease) [ 33 , 34 , 35 , 42 , 50 , 56 , 65 ], and six studies reporting the growth rate before and after lockdown (median growth rate before lockdown: 18.0 percentage increase; median growth rate after lockdown: 3.8 percentage increase) [ 32 , 60 , 61 , 66 , 68 , 70 ].

Among the 15 studies on anxiety and depression symptoms, 10 were conducted in European countries (Spain, Italy, Germany, Ireland, United Kingdom) and two were conducted in the USA (Appendix Table S 5 ). All studies reported on the effect of lockdown, with the median interval between the baseline and follow-up outcome measurements being 6 weeks. Several instruments were used for assessing anxiety symptoms, including the Generalized Anxiety Disorder Scale [ 76 , 77 , 78 , 79 , 82 ], the Brief Symptom Inventory [ 74 , 81 ], and the Depression Anxiety and Stress Scale [ 83 , 84 ]. Instruments for assessing depression symptoms included the Patients Health Questionnaire [ 71 , 76 , 77 , 78 , 79 , 82 ], the Brief Symptom Inventory [ 74 , 81 ], and the Depression Anxiety and Stress Scale [ 83 , 84 ]. The overall risk of bias was serious for five studies and critical for 10 studies (Appendix Table S 8 ). This was mainly because of risk of bias in the confounding, selection of participants, and missing data domains. Fourteen studies reported the effect of lockdown on anxiety and/or depression symptoms in the general adult population and one study reported the effect on depression symptoms in workers. For the effect of lockdown on anxiety symptoms, the pooled standardized mean difference was -0.02 (95% CI: -0.06, 0.02) (Fig.  2 a). For the effect of lockdown on depression symptoms, the pooled standardized mean difference was 0.16 (95% CI: 0.10, 0.21) (Fig.  2 b). The funnel plot for depression symptoms showed some asymmetry in the distribution of studies about the pooled standardized mean difference (Fig.  3 b), and the Trim and Fill adjustment indicated that publication bias could account for the observed effect (adjusted pooled standardized mean difference = 0.001, 95% CI: -0.04, 0.02).

Forest plots of the effect of lockdown on anxiety and depression symptoms, January 1, 2020–May 11, 2021 a . a I 2 for heterogeneity for studies on anxiety symptoms = 94% (Q test p  < 0.001). I 2 for heterogeneity for studies on depression symptoms = 98% (Q test p  < 0.001)

Funnel plots of the effect of lockdown on anxiety and depression symptoms, January 1, 2020–May 11, 2021 a . a The graph on the left shows studies on anxiety symptoms, and that on the right shows studies on depression symptoms. The Trim and Fill adjusted pooled standardized mean difference for depression symptoms = 0.001 (95% CI: -0.04, 0.02)

Among the five studies on unemployment and labor force participation, three were from the USA, one from Mexico, and one from Australia (Appendix Table S 6 ). The median interval between the baseline and follow-up outcome measurements was 3 months. The overall risk of bias was moderate in two studies and serious in three studies (Appendix Table S 9 ). All studies had a moderate or serious risk of confounding, and one study had a serious risk of bias because of missing data. The studies showed that lockdown increased unemployment (pooled mean difference = 4.48 percentage points, 95% CI: 1.79, 7.17) (Fig.  4 a) and decreased labor force participation (pooled mean difference = -2.46 percentage points, 95% CI: -3.16, -1.77) (Fig.  4 b).

Forest plots of the effect of lockdown on unemployment and labor force participation, January 1, 2020–May 11, 2021 a . a I 2 for heterogeneity for studies on unemployment = 92% (Q test p  < 0.001). I 2 for heterogeneity for studies on labor force participation = 0% (Q test p  = 0.69)

Empiric studies showed that lockdown reduced COVID-19 incidence or case growth rate, reproduction number, and COVID-19 mortality or death growth rate in the general population during the initial year of COVID-19 pandemic. We found few studies on the effect of NPIs other than lockdown on COVID-19 morbidity and mortality outcomes. Lockdown increased unemployment and decreased labor force participation, but no effect was observed on anxiety symptoms. Lockdown had a small effect on increasing depression symptoms, but publication bias could account for the observed effect. The risk of bias for most of the studies on COVID-19 and employment outcomes was moderate or serious, and that for the studies on anxiety and depression symptoms was serious or critical.

Non-pharmaceutical measures can reduce SARS-CoV-2 transmission by reducing the likelihood of transmission per contact and by reducing contacts between infectious and healthy persons [ 90 ]. Studies published in 2023 found that employed adults who had telework experience before illness onset were less likely to work onsite while ill during COVID-19 and other acute respiratory illnesses than persons without telework experience, suggesting that telework may reduce workplace virus exposure [ 91 , 92 ]. Systematic reviews that assessed the effect of physical distancing and screening in non-workplace settings or on other respiratory viruses provide indirect evidence for the effect of these measures on COVID-19 illness in non-healthcare workplaces. A systematic review assessed the effectiveness of physical distancing measures in non-healthcare workplaces on influenza attack rates [ 93 ]. One review included studies of physical distancing on COVID-19 illness in settings other than workplaces (e.g., ≥ 3 vs. ≥ 6 feet distancing policies in schools; frequency of close contact with a primary case in a household) [ 94 ]. A Cochrane rapid review assessed the effect of symptom/exposure-based or test-based screening of international travelers for SARS-CoV-2 at borders before or after travel [ 95 ]. Systematic reviews of modeling studies on the effect of NPIs within non-healthcare workplaces on COVID-19 illness are needed because modeling studies fill in gaps of information when decisions must be made and there is limited information [ 96 , 97 ].

Recent systematic reviews of empiric studies have assessed the effect of workplace closures and lockdowns [ 18 , 94 , 98 ]. Two of these reviews included cross-sectional studies [ 94 , 98 ]. We excluded cross-sectional studies because it is difficult to assess cause-and-effect relationships from such studies [ 99 ]. The previous reviews reported that workplace closures and lockdowns reduced COVID-19 incidence, case growth rate, reproduction number, COVID-19 mortality, and death growth rate in the general population [ 18 , 94 , 98 ]. Lockdowns have been shown to reduce population mobility, with increased time at home, reductions in visits to shops and workplaces, and decline in use of public transport [ 17 ].

Our systematic review did not find conclusive evidence that lockdown increased anxiety and depression symptoms. A previous rapid review of studies published from January 2020 to June 2020 reported small effects of lockdown on anxiety and depression symptoms [ 100 ]. Among the 11 empiric studies on anxiety and depression symptoms included in the review, four were conducted in college or university students and thus not directly relevant to our systematic review. Another review estimated that the global prevalence of anxiety and depression symptoms increased during the COVID-19 pandemic compared to the pre-pandemic period [ 101 ]. The authors attributed the increase in anxiety and depression symptoms to the combined effects of the spread of SARS-CoV-2 and the interventions, including lockdown, school and workplace closures, decreased public transport, and reduction of social interactions. Several risk factors for anxiety and depression during lockdown have been reported. Risk factors for anxiety include loneliness and history of mental health issues, while higher level of resilience and spiritual well-being are associated with lower anxiety [ 77 , 78 ]. Risk factors for depression include loneliness, detachment, negative affect, history of mental health issues, concerns about changes at work and running out of money, and unemployment [ 71 , 77 , 84 ]. On the other hand, protective factors associated with depression include more resilient coping style, higher level of resilience, spiritual well-being, and moderate-to-vigorous physical activity [ 75 , 77 , 78 , 84 ].

Our systematic review showed lockdown increased unemployment and decreased labor force participation. Lockdown can directly lead to layoffs because of business closures, cancelation of events, and reduced economic activities. However, in the absence of lockdown, employment can be affected by individuals’ refraining from activities outside their household to reduce their risk of infection, which can lead to decreased consumer spending and business revenues [ 5 , 88 ]. We did not identify any previous systematic reviews of the effect of lockdown on unemployment and labor force participation.

Findings of our systematic review should be considered in context of at least seven limitations. First, some studies on the effects of workplace closures and lockdowns on COVID-19 outcomes used quasi-experimental designs (controlled before after, interrupted time series) that can allow for causal inferences without randomized trials [ 102 , 103 ], but it is unclear if the assumptions required to ensure valid causal inference were met. The findings therefore need to be interpreted as showing an association. Second, the included studies often did not describe in detail the interventions that were assessed, which may make it difficult to compare findings across studies. Third, many NPIs were implemented together or within a short time, and so the independent effects of interventions may be difficult to determine [ 104 ], particularly for studies that did not have a concurrent control group. Fourth, the number of COVID-19 cases could have been underestimated to a greater degree during the early phase of the pandemic because of limited availability of COVID-19 tests. However, the underestimation would likely bias the effect of an intervention toward the null [ 105 ]. Fifth, several studies on the effect of lockdown on anxiety and depression symptoms collected baseline data after the start of lockdown, and so the magnitude of the effect may be under-estimated. In addition, anxiety and depression were assessed using screening questionnaires that identified probable cases, and the findings may not be extrapolated to diagnosed cases of anxiety and depression [ 101 ]. Sixth, although our electronic search identified grey literature (e.g., working papers, medRxiv preprints) [ 36 , 40 , 51 , 86 , 88 ], we did not specifically search preprint databases or dissertations and theses databases. Finally, we limited studies to English when we performed the electronic searches and screened articles, and thus the findings may not be generalizable to studies published in other languages.

However, this systematic review also has several strengths. We assessed both desired effects (i.e., public health benefits) and secondary (unintended / unwanted) effects of NPIs during the initial year of COVID-19 pandemic. Additionally, we used several electronic databases to search for studies and examined the references of previous systematic reviews, which increased the comprehensiveness of the literature search. Next, our review was based on empiric studies that provide direct evidence of effectiveness in real-world settings.

The COVID-19 pandemic had unequal effects on the population, with people who could work remotely faring better in terms of health and socioeconomic wellbeing than persons who were required to work in-person, such as those in goods production or essential industries [ 1 ]. Minority and low-income vulnerable persons are over-represented in high-risk essential industries [ 1 , 45 , 106 ]. COVID-19 death rates in the U.S. have been estimated to be substantially higher in Hispanics and non-Hispanic Blacks compared to non-Hispanic Whites [ 107 , 108 ]. Compared to people working in non-essential sectors, those working in essential sectors (particularly in agriculture, emergency, manufacturing, facilities, and transportation or logistics) were found to have higher COVID-19 deaths [ 109 , 110 ]. It is important to deploy effective science based NPIs to reduce health inequities and decrease overall disease transmission, especially in industries where work cannot be performed remotely.

Our systematic review showed that several empiric studies assessed the effect of lockdowns, but there is a paucity of studies on the effects of other interventions undertaken in many workplace settings, including temperature/symptom screening, use of different barrier protections including some which were not previously proposed as an NPI or tested (e.g., plexiglass or curtain partitions), and physical distancing measures within the workplace. With the availability of COVID-19 vaccines and effective therapeutics that reduce hospitalizations and deaths [ 1 ], as well as the desire to avoid detrimental effects on daily life and the economy, the use of workplace closures and lockdowns abated after the initial year of the pandemic in most countries. However, because SARS-CoV-2 remains endemic and because it evolved into variants which can evade immunity acquired through prior infection or vaccination and transmit more efficiently [ 111 ], use of less disruptive NPIs including better ventilation, face masks, and some variations of physical distancing within the workplace may still have relevance. Addressing the gaps in the evidence base on the effects of NPIs pertaining to workplaces is therefore important for informing ongoing prevention strategies as well as future pandemic preparedness.

There was scarce direct evidence on the benefits of symptom and/or temperature screening, physical barriers, and physical distancing measures to reduce COVID-19 illness within workplaces that are open. While the use of these interventions is less likely to be perceived as disruptive for work process than lockdowns, they are not likely to be effective in reducing the transmission of an airborne virus like SARS-CoV-2 that can be readily spread in indoor settings by asymptomatic or pre-symptomatic individuals. There was evidence to indicate that lockdown helped reduce COVID-19 morbidity and mortality in the general population, but it increased unemployment and reduced labor force participation. It is important for countries that implement lockdown in future outbreaks of emerging infectious diseases or pandemics to consider strategies to mitigate these unintended consequences.

Availability of data and materials

All data generated or analyzed during this study are included in this published study and its Additional file 1 .

Abbreviations

Cumulative Index to Nursing and Allied Health Literature

Non-pharmaceutical intervention

Preferred Reporting Items for Systematic Reviews and Meta-Analysis

Risk of Bias in Non-Randomized Studies of Interventions

Severe Acute Respiratory Syndrome Coronavirus 2

Standard deviation

Standard error

Sachs JD, Karim SSA, Aknin L, Allen J, Brosbol K, Colombo F, Barron GC, Espinosa MF, Gaspar V, Gaviria A, et al. The lancet commission on lessons for the future from the COVID-19 pandemic. Lancet. 2022;400(10359):1224–80.

Article   PubMed   PubMed Central   Google Scholar  

World Health Organization: WHO Director-General's opening remarks at the media briefing on COVID-19 - 11 March 2020. https://www.who.int/director-general/speeches/detail/who-director-general-s-opening-remarks-at-the-media-briefing-on-covid-19---11-march-2020 . Accessed 29 Dec 2023.

Hale T, Angrist N, Goldszmidt R, Kira B, Petherick A, Phillips T, Webster S, Cameron-Blake E, Hallas L, Majumdar S, Tatlow H. A global panel database of pandemic policies (Oxford COVID-19 Government Response Tracker). Nat Hum Behav. 2021;5(4):529–38.

Article   PubMed   Google Scholar  

Nicola M, Alsafi Z, Sohrabi C, Kerwan A, Al-Jabir A, Iosifidis C, Agha M, Agha R. The socio-economic implications of the coronavirus pandemic (COVID-19): a review. Int J Surg. 2020;78:185–93.

Hoehn-Velasco L, Silverio-Murillo A, Balmori de la Miyar JR: The long downturn: The impact of the great lockdown on formal employment. J Econ Bus 2021;115(May-June):105983.

The World Bank: Labor force participation rate, total (% of total population age 15+). https://data.worldbank.org/indicator/SL.TLF.CACT.NE.ZS?name_desc=false . Accessed 6 Dec 2022.

Duval D, Palmer JC, Tudge I, Pearce-Smith N, O’Connell E, Bennett A, Clark R. Long distance airborne transmission of SARS-CoV-2: rapid systematic review. BMJ. 2022;377:e068743.

Jimenez JL, Marr LC, Randall K, Ewing ET, Tufekci Z, Greenhalgh T, Tellier R, Tang JW, Li Y, Morawska L, et al. What were the historical reasons for the resistance to recognizing airborne transmission during the COVID-19 pandemic? Indoor Air. 2022;32(8):e13070.

Article   CAS   PubMed   PubMed Central   Google Scholar  

World Health Organization: Considerations for public health and social measures in the workplace in the context of COVID-19. https://www.who.int/publications/i/item/considerations-for-public-health-and-social-measures-in-the-workplace-in-the-context-of-covid-19 . Accessed 26 Jan 2023.

European Centre for Disease Prevention and Control: Guidelines for the implementation of non-pharmaceutical interventions against COVID-19. 24 September 2020. https://www.ecdc.europa.eu/sites/default/files/documents/covid-19-guidelines-non-pharmaceutical-interventions-september-2020.pdf . Accessed 26 Jan 2023.

Centers for Disease Control and Prevention: Guidance for Businesses and Employers Responding to Coronavirus Disease 2019 (COVID-19). https://public4.pagefreezer.com/browse/CDC%20Covid%20Pages/11-05-2022T12:30/https://www.cdc.gov/coronavirus/2019-ncov/community/guidance-business-response.html . Accessed 26 Jan 2023.

Pizarro AB, Persad E, Durao S, Nussbaumer-Streit B, Engela-Volker JS, McElvenny D, Rhodes S, Stocking K, Fletcher T, Martin C, et al. Workplace interventions to reduce the risk of SARS-CoV-2 infection outside of healthcare settings. Cochrane Database Syst Rev. 2022;5(5):CD015112.

PubMed   Google Scholar  

National Institute for Health Research: PROSPERO. International prospective register of systematic reviews. https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42020182660 . Accessed 26 Jan 2023.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, Shamseer L, Tetzlaff JM, Akl EA, Brennan SE, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2021;372:n71.

Higgins JP TJ, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors): Cochrane Handbook for systematic reviews of interventions version 6.3 (updated February 2022). Cochrane, 2022. Chapters 12, 25. https://training.cochrane.org/handbook/current/ . Accessed 26 Jan 2023.

Fuss FK, Weizman Y, Tan AM. COVID-19 pandemic: How effective are preventive control measures and is a complete lockdown justified? A comparison of countries and states. COVID. 2022;2:18–46.

Article   CAS   Google Scholar  

Tully MA, McMaw L, Adlakha D, Blair N, McAneney J, McAneney H, Carmichael C, Cunningham C, Armstrong NC, Smith L. The effect of different COVID-19 public health restrictions on mobility: a systematic review. PLoS ONE. 2021;16(12):e0260919.

Mendez-Brito A, El Bcheraoui C, Pozo-Martin F. Systematic review of empirical studies comparing the effectiveness of non-pharmaceutical interventions against COVID-19. J Infect. 2021;83(3):281–93.

World Health Organization: Considerations for implementing and adjusting public health and social measures in the context of COVID-19. https://www.who.int/publications/i/item/considerations-in-adjusting-public-health-and-social-measures-in-the-context-of-covid-19-interim-guidance . Accessed 26 Jan 2023.

World Health Organization: Mental disorders. https://www.who.int/news-room/fact-sheets/detail/mental-disorders . Accessed 26 Jan 2023.

World Health Organization: Depression. https://www.who.int/health-topics/depression#tab=tab_1 . Accessed 26 Jan 2023.

U.S. Bureau of Labor Statistics: Labor Force Statistics from the Current Population Survey. https://www.bls.gov/cps/definitions.htm . Accessed 26 Jan 2023.

Covidence systematic review software VHI, Melbourne, Australia: https://www.covidence.org . Accessed 26 Jan 2023.

Sterne JA HJ, Elbers RG, Reeves BC: Risk Of Bias In Non-randomized Studies of Interventions (ROBINS-I): detailed guidance. https://www.riskofbias.info/welcome/home/current-version-of-robins-i/robins-i-detailed-guidance-2016 . Accessed 26 Jan 2023.

Barcelo MA, Saez M. Methodological limitations in studies assessing the effects of environmental and socioeconomic variables on the spread of COVID-19: a systematic review. Environ Sci Eur. 2021;33(1):108.

Akhtar-Danesh N, Landeen J. Relation between depression and sociodemographic factors. Int J Ment Health Syst. 2007;1(1):4.

Leonardi M, Guido D, Quintas R, Silvaggi F, Guastafierro E, Martinuzzi A, Chatterji S, Koskinen S, Tobiasz-Adamczyk B, Haro JM et al: Factors related to unemployment in Europe. A cross-sectional study from the COURAGE survey in Finland, Poland and Spain. Int J Environ Res Public Health. 2018;15(4):1–21.

Borenstein M, Hedges LV, Higgins JPT, Rothstein HR. Introduction to meta-analysis. 1st ed. Chichester: Wiley; 2009.

Book   Google Scholar  

Comprehensive meta analysis: Version 4.0.000. NJ, USA. https://www.meta-analysis.com .

Alfano V, Ercolano S. The efficacy of lockdown against COVID-19: a cross-country panel analysis. Appl Health Econ Health Policy. 2020;18(4):509–17.

Askitas N, Tatsiramos K, Verheyden B. Estimating worldwide effects of non-pharmaceutical interventions on COVID-19 incidence and population mobility patterns using a multiple-event study. Sci Rep. 2021;11(1):1972.

Castillo RC, Staguhn ED, Weston-Farber E. The effect of state-level stay-at-home orders on COVID-19 infection rates. Am J Infect Control. 2020;48(8):958–60.

Chae SH, Park HJ. Effectiveness of penalties for lockdown violations during the covid-19 pandemic in Germany. Am J Public Health. 2020;110(12):1844–9.

Cobb JS, Seale MA. Examining the effect of social distancing on the compound growth rate of COVID-19 at the county level (United States) using statistical analyses and a random forest machine learning model. Public Health. 2020;185:27–9.

Article   CAS   PubMed   Google Scholar  

Courtemanche C, Garuccio J, Le A, Pinkston J, Yelowitz A. Strong social distancing measures in the United States reduced the COVID-19 growth rate. Health Aff. 2020;39(7):1237–46.

Article   Google Scholar  

Deb P, Furceri D, Ostry JD, Tawk N: The effect of containment measures on the COVID-19 pandemic. International Monetary Fund, Working Paper No. 2020/159. https://www.imf.org/en/Publications/WP/Issues/2020/08/07/The-Effect-of-Containment-Measures-on-the-COVID-19-Pandemic-49572#:~:text=We%20examine%20this%20question%20using,is%20significant%20heterogeneity%20across%20countries . Accessed 6 Jun 2022.

Dreher N, Spiera Z, McAuley FM, Kuohn L, Durbin JR, Marayati NF, Ali M, Li AY, Hannah TC, Gometz A, et al. Policy interventions, social distancing, and SARS-CoV-2 transmission in the United States: a retrospective state-level analysis. Am J Med Sci. 2021;361(5):575–84.

Duhon J, Bragazzi N, Kong JD. The impact of non-pharmaceutical interventions, demographic, social, and climatic factors on the initial growth rate of COVID-19: a cross-country study. Sci Total Environ. 2021;760:144325.

Ebrahim S, Ashworth H, Noah C, Kadambi A, Toumi A, Chhatwal J. Reduction of COVID-19 incidence and nonpharmacologic interventions: analysis using a US county-level policy data set. J Med Internet Res. 2020;22(12):e24614.

Esra R, Jamieson L, Fox MP, Letswalo D, Ngcobo N, Mngadi S, Estill J, Meyer-Rath G, Keiser O: Evaluating the impact of non-pharmaceutical interventions for SARS-CoV-2 on a global scale. MedRxiv 2020.

Fisher KA, Olson SM, Tenforde MW, Feldstein LR, Lindsell CJ, Shapiro NI, Files DC, Gibbs KW, Erickson HL, Prekker ME, et al. Telework before illness onset among symptomatic adults aged >/=18 years with and without COVID-19 in 11 outpatient health care facilities - United States, July 2020. MMWR Morb Mortal Wkly Rep. 2020;69(44):1648–53.

Gokmen Y, Baskici C, Ercil Y. Effects of non-pharmaceutical interventions against COVID-19: a cross-country analysis. Int J Health Plan Manage. 2021;05:05.

Google Scholar  

Guzzetta G, Riccardo F, Marziano V, Poletti P, Trentini F, Bella A, Andrianou X, Del Manso M, Fabiani M, Bellino S, et al. Impact of a nationwide lockdown on SARS-CoV-2 transmissibility, Italy. Emerg Infect Dis. 2021;27(1):01.

Haug N, Geyrhofer L, Londei A, Dervic E, Desvars-Larrive A, Loreto V, Pinior B, Thurner S, Klimek P. Ranking the effectiveness of worldwide COVID-19 government interventions. Nat Hum Behav. 2020;4(12):1303–12.

Herstein JJ, Degarege A, Stover D, Austin C, Schwedhelm MM, Lawler JV, Lowe JJ, Ramos AK, Donahue M. Characteristics of SARS-CoV-2 transmission among meat processing workers in Nebraska, USA, and effectiveness of risk mitigation measures. Emerg Infect Dis. 2021;27(4):1032–8.

Islam N, Sharp SJ, Chowell G, Shabnam S, Kawachi I, Lacey B, Massaro JM, D’Agostino RB Sr, White M. Physical distancing interventions and incidence of coronavirus disease 2019: natural experiment in 149 countries. BMJ. 2020;370:m2743.

Koh WC, Naing L, Wong J. Estimating the impact of physical distancing measures in containing COVID-19: an empirical analysis. Int J Infect Dis. 2020;100:42–9.

Lau H, Khosrawipour V, Kocbach P, Mikolajczyk A, Schubert J, Bania J, Khosrawipour T. The positive impact of lockdown in Wuhan on containing the COVID-19 outbreak in China. J Travel Med. 2021;27(3):1–7.

Li Y, Campbell H, Kulkarni D, Harpur A, Nundy M, Wang X, Nair H, Usher Network for Covid-Evidence Reviews group. The temporal association of introducing and lifting non-pharmaceutical interventions with the time-varying reproduction number (R) of SARS-CoV-2: a modelling study across 131 countries. Lancet Infect Dis. 2021;21(2):193–202.

Li Y, Li M, Rice M, Zhang H, Sha D, Li M, Su Y, Yang C. The impact of policy measures on human mobility, COVID-19 cases, and mortality in the US: a spatiotemporal perspective. Int J Environ Res Public Health. 2021;18(3):996.

Lin Z, Meissner CM: Health vs. Wealth? Public health policies and the economy during covid-19. In. National Bureau of Economic Research, Inc, NBER Working Papers: 27099; 2020.

Liu Y, Morgenstern C, Kelly J, Lowe R, Cmmid Covid- Working G, Jit M. The impact of non-pharmaceutical interventions on SARS-CoV-2 transmission across 130 countries and territories. BMC Med. 2021;19(1):40.

Lyu W, Wehby GL. Comparison of estimated rates of coronavirus disease 2019 (COVID-19) in border counties in Iowa without a stay-at-home order and border counties in Illinois with a stay-at-home order. JAMA Netw Open. 2020;3(5):e2011102–e2011102.

Lyu W, Wehby GL. Shelter-in-place orders reduced covid-19 mortality and reduced the rate of growth in hospitalizations. Health Aff. 2020;39(9):1615–23.

Padalabalanarayanan S, Hanumanthu VS, Sen B. Association of state stay-at-home orders and state-level African American population with COVID-19 case rates. JAMA Netw Open. 2020;3(10):e2026010–e2026010.

Saez M, Tobias A, Varga D, Barcelo MA. Effectiveness of the measures to flatten the epidemic curve of COVID-19 The case of Spain. Sci Total Environ. 2020;727:138761.

Salvatore M, Basu D, Ray D, Kleinsasser M, Purkayastha S, Bhattacharyya R, Mukherjee B. Comprehensive public health evaluation of lockdown as a non-pharmaceutical intervention on COVID-19 spread in India: national trends masking state-level variations. BMJ Open. 2020;10(12):e041778.

Santamaria L, Hortal J. Chasing the ghost of infection past: identifying thresholds of change during the COVID-19 infection in Spain. Epidemiol Infect. 2020;148:e282.

Santamaria L, Hortal J. COVID-19 effective reproduction number dropped during Spain’s nationwide dropdown, then spiked at lower-incidence regions. Sci Total Environ. 2021;751:142257.

Saul A, Scott N, Crabb BS, Majumdar SS, Coghlan B, Hellard ME. Impact of Victoria’s Stage 3 lockdown on COVID-19 case numbers. Med J Aust. 2020;213(11):494-496.e491.

Schroder M, Bossert A, Kersting M, Aeffner S, Coetzee J, Timme M, Schluter J. COVID-19 in South Africa: outbreak despite interventions. Sci Rep. 2021;11(1):4956.

Silva L, Figueiredo Filho D, Fernandes A. The effect of lockdown on the COVID-19 epidemic in Brazil: evidence from an interrupted time series design. Cad Saude Publica. 2020;36(10):e00213920.

Singh BB, Lowerison M, Lewinson RT, Vallerand IA, Deardon R, Gill JPS, Singh B, Barkema HW. Public health interventions slowed but did not halt the spread of COVID-19 in India. Transbound Emerg Dis. 2020;04:04.

Singh S, Shaikh M, Hauck K, Miraldo M. Impacts of introducing and lifting nonpharmaceutical interventions on COVID-19 daily growth rate and compliance in the United States. Proc Natl Acad Sci USA. 2021;118(12):23.

Thayer WM, Hasan MZ, Sankhla P, Gupta S. An interrupted time series analysis of the lockdown policies in India: a national-level analysis of COVID-19 incidence. Health Policy Plan. 2021;26:26.

Tobias A. Evaluation of the lockdowns for the SARS-CoV-2 epidemic in Italy and Spain after one month follow up. Sci Total Environ. 2020;725:138539.

White ER, Hebert-Dufresne L. State-level variation of initial COVID-19 dynamics in the United States. PLoS ONE [Electronic Resource]. 2020;15(10):e0240648.

Wong CKH, Wong JYH, Tang EHM, Au CH, Lau KTK, Wai AKC. Impact of national containment measures on decelerating the increase in daily new cases of COVID-19 in 54 countries and 4 epicenters of the pandemic: comparative observational study. J Med Internet Res. 2020;22(7):e19904.

Xu J, Hussain S, Lu G, Zheng K, Wei S, Bao W, Zhang L. Associations of stay-at-home order and face-masking recommendation with trends in daily new cases and deaths of laboratory-confirmed COVID-19 in the United States. Explor Res Hypothesis Med. 2020;5(3):77–86.

Zhang X, Warner ME. COVID-19 Policy Differences across US States: Shutdowns, Reopening, and Mask Mandates. Int J Environ Res Publ Health [Electronic Resource]. 2020;17(24):18.

Badellino H, Gobbo ME, Torres E, Aschieri ME, Biotti M, Alvarez V, Gigante C, Cachiarelli M. “It’s the economy, stupid”: Lessons of a longitudinal study of depression in Argentina. Int J Soc Psych. 2022;68(2):384–91.

Barone Gibbs B, Kline CE, Huber KA, Paley JL, Perera S. Covid-19 shelter-at-home and work, lifestyle and well-being in desk workers. Occup Med (Oxford). 2021;71(2):86–94.

Canet-Juric L, Andres ML, Del Valle M, Lopez-Morales H, Poo F, Galli JI, Yerro M, Urquijo S. A longitudinal study on the emotional impact cause by the COVID-19 pandemic quarantine on general population. Front Psychol. 2020;11:565688.

Castellini G, Rossi E, Cassioli E, Sanfilippo G, Innocenti M, Gironi V, Silvestri C, Voller F, Ricca V. A longitudinal observation of general psychopathology before the COVID-19 outbreak and during lockdown in Italy. J Psychosom Res. 2021;141:110328.

Cecchini JA, Carriedo A, Fernandez-Rio J, Mendez-Gimenez A, Gonzalez C, Sanchez-Martinez B, Rodriguez-Gonzalez P. A longitudinal study on depressive symptoms and physical activity during the Spanish lockdown. Int J Clin Health Psychol. 2021;21(1):100200.

Fancourt D, Steptoe A, Bu F. Trajectories of anxiety and depressive symptoms during enforced isolation due to COVID-19 in England: a longitudinal observational study. Lancet Psychiatry. 2021;8(2):141–9.

Gonzalez-Sanguino C, Ausin B, Castellanos MA, Saiz J, Munoz M. Mental health consequences of the Covid-19 outbreak in Spain. A longitudinal study of the alarm situation and return to the new normality. Prog Neuropsychopharmacol Biol Psychiatry. 2021;107:110219.

Gopal A, Sharma AJ, Subramanyam MA. Dynamics of psychological responses to COVID-19 in India: a longitudinal study. PLoS ONE [Electronic Resource]. 2020;15(10):e0240650.

Hyland P, Shevlin M, Murphy J, McBride O, Fox R, Bondjers K, Karatzias T, Bentall RP, Martinez A, Vallieres F. A longitudinal assessment of depression and anxiety in the Republic of Ireland before and during the COVID-19 pandemic. Psychiatry Res. 2021;300:113905.

Le K, Nguyen M. The psychological consequences of COVID-19 lockdowns. Int Rev Appl Econ. 2021;35(2):147–63.

Mergel E, Schutzwohl M. A longitudinal study on the COVID-19 pandemic and its divergent effects on social participation and mental health across different study groups with and without mental disorders. Soc Psychiatry Psychiatr Epidemiol. 2021;10:10.

O’Connor RC, Wetherall K, Cleare S, McClelland H, Melson AJ, Niedzwiedz CL, O’Carroll RE, O’Connor DB, Platt S, Scowcroft E, et al. Mental health and well-being during the COVID-19 pandemic: longitudinal analyses of adults in the UK COVID-19 mental health & wellbeing study. Brit J Psych. 2021;218:326–33.

Ozamiz-Etxebarria N, Idoiaga Mondragon N, Dosil Santamaria M, Picaza Gorrotxategi M. Psychological symptoms during the two stages of lockdown in response to the COVID-19 outbreak: an investigation in a sample of citizens in Northern Spain. Front Psychol. 2020;11:1491.

Roma P, Monaro M, Colasanti M, Ricci E, Biondi S, Di Domenico A, Verrocchio MC, Napoli C, Ferracuti S, Mazza C. A 2-month follow-up study of psychological distress among Italian people during the COVID-19 lockdown. Int J Environ Res Publ Health [Electronic Resource]. 2020;17(21):05.

Somma A, Krueger RF, Markon KE, Gialdi G, Colanino M, Ferlito D, Liotta C, Frau C, Fossati A. A longitudinal study on clinically relevant self-reported depression, anxiety and acute stress features among Italian community-dwelling adults during the COVID-19 related lockdown: Evidence of a predictive role for baseline dysfunctional personality dimensions. J Affect Disord. 2021;282:364–71.

Beland L-P, Brodeur A, Wright T: COVID-19, stay-at-home orders and employment: evidence from CPS Data. In. Carleton University, Department of Economics, Carleton Economic Papers: 20–04. 2020: 111 pages.

Churchill B. COVID-19 and the immediate impact on young people and employment in Australia: a gendered analysis. Gend Work Organ. 2020;31:31.

Coibion O, Gorodnichenko Y, Weber M: The cost of the covid-19 crisis: lockdowns, macroeconomic expectations, and consumer spending. In. National Bureau of Economic Research, Inc, NBER Working Papers: 27141; 2020.

Robinson E, Daly M. Explaining the rise and fall of psychological distress during the COVID-19 crisis in the United States: longitudinal evidence from the understanding America study. Br J Health Psychol. 2021;26(2):570–87.

Delamater PL, Street EJ, Leslie TF, Yang YT, Jacobsen KH. Complexity of the basic reproduction number (R(0)). Emerg Infect Dis. 2019;25(1):1–4.

Shafer L, Ahmed F, Kim S, Wernli KJ, Jackson ML, Nowalk MP, Bear T, Zimmerman RK, Martin ET, Monto AS, et al. Relationship between telework experience and presenteeism during COVID-19 pandemic, United States, March-November 2020. Emerg Infect Dis. 2023;29(2):278–85.

Ahmed F, Nowalk MP, Zimmerman RK, Bear T, Grijalva CG, Talbot HK, Florea A, Tartof SY, Gaglani M, Smith M, et al. Work attendance with acute respiratory illness before and during COVID-19 pandemic, United States, 2018–2022. Emerg Infect Dis. 2023;29(12):2442–50.

Ahmed F, Zviedrite N, Uzicanin A. Effectiveness of workplace social distancing measures in reducing influenza transmission: a systematic review. BMC Public Health. 2018;18(1):518.

Talic S, Shah S, Wild H, Gasevic D, Maharaj A, Ademi Z, Li X, Xu W, Mesa-Eguiagaray I, Rostron J, et al. Effectiveness of public health measures in reducing the incidence of covid-19, SARS-CoV-2 transmission, and covid-19 mortality: systematic review and meta-analysis. BMJ. 2021;375:e068302.

Burns J, Movsisyan A, Stratil JM, Biallas RL, Coenen M, Emmert-Fees KM, Geffert K, Hoffmann S, Horstick O, Laxy M, et al. International travel-related control measures to contain the COVID-19 pandemic: a rapid review. Cochr Database Syst Rev. 2021;3(3):CD013717.

Fischer LS, Santibanez S, Hatchett RJ, Jernigan DB, Meyers LA, Thorpe PG, Meltzer MI. CDC grand rounds: modeling and public health decision-making. MMWR Morb Mortal Wkly Rep. 2016;65(48):1374–7.

Holmdahl I, Buckee C. Wrong but useful - what covid-19 epidemiologic models can and cannot tell us. N Engl J Med. 2020;383(4):303–5.

Iezadi S, Gholipour K, Azami-Aghdash S, Ghiasi A, Rezapour A, Pourasghari H, Pashazadeh F. Effectiveness of non-pharmaceutical public health interventions against COVID-19: a systematic review and meta-analysis. PLoS ONE. 2021;16(11):e0260371.

Setia MS. Methodology series module 3: cross-sectional studies. Indian J Dermatol. 2016;61(3):261–4.

Prati G, Mancini AD. The psychological impact of COVID-19 pandemic lockdowns: a review and meta-analysis of longitudinal studies and natural experiments. Psychol Med. 2021;51(2):201–11.

Collaborators C-M. Global prevalence and burden of depressive and anxiety disorders in 204 countries and territories in 2020 due to the COVID-19 pandemic. Lancet. 2021;398(10312):1700–12.

Barnighausen T, Tugwell P, Rottingen JA, Shemilt I, Rockers P, Geldsetzer P, Lavis J, Grimshaw J, Daniels K, Brown A, et al. Quasi-experimental study designs series-paper 4: uses and value. J Clin Epidemiol. 2017;89:21–9.

Barnighausen T, Oldenburg C, Tugwell P, Bommer C, Ebert C, Barreto M, Djimeu E, Haber N, Waddington H, Rockers P, et al. Quasi-experimental study designs series-paper 7: assessing the assumptions. J Clin Epidemiol. 2017;89:53–66.

Ayouni I, Maatoug J, Dhouib W, Zammit N, Fredj SB, Ghammam R, Ghannem H. Effective public health measures to mitigate the spread of COVID-19: a systematic review. BMC Public Health. 2021;21(1):1015.

Caristia S, Ferranti M, Skrami E, Raffetti E, Pierannunzio D, Palladino R, Carle F, Saracci R, Badaloni C, Barone-Adesi F, et al. Effect of national and local lockdowns on the control of COVID-19 pandemic: a rapid review. Epidemiologia e Prevenzione. 2020;44(5-6 Suppl 2):60–8.

Dyal JW, Grant MP, Broadwater K, Bjork A, Waltenburg MA, Gibbins JD, Hale C, Silver M, Fischer M, Steinberg J, et al. COVID-19 among workers in meat and poultry processing facilities - 19 states, April 2020. MMWR Morb Mortal Wkly Rep. 2020;69(18):557–61.

Cronin CJ, Evans WN. Excess mortality from COVID and non-COVID causes in minority populations. Proc Natl Acad Sci USA. 2021;118(39):1–5.

Cummings KJ, Beckman J, Frederick M, Harrison R, Nguyen A, Snyder R, Chan E, Gibb K, Rodriguez A, Wong J, et al. Disparities in COVID-19 fatalities among working Californians. PLoS ONE. 2022;17(3):e0266058.

Chen YH, Riley AR, Duchowny KA, Aschmann HE, Chen R, Kiang MV, Mooney AC, Stokes AC, Glymour MM, Bibbins-Domingo K. COVID-19 mortality and excess mortality among working-age residents in California, USA, by occupational sector: a longitudinal cohort analysis of mortality surveillance data. Lancet Public Health. 2022;7(9):e744–53.

Chen YH, Stokes AC, Aschmann HE, Chen R, DeVost S, Kiang MV, Koliwad S, Riley AR, Glymour MM, Bibbins-Domingo K. Excess natural-cause deaths in California by cause and setting: March 2020 through February 2021. PNAS Nexus. 2022;1(3):pgac079.

Andrews N, Stowe J, Kirsebom F, Toffa S, Rickeard T, Gallagher E, Gower C, Kall M, Groves N, O’Connell AM, et al. Covid-19 vaccine effectiveness against the omicron (B.1.1.529) variant. N Engl J Med. 2022;386(16):1532–46.

Download references

Acknowledgements

The authors thank Joanna Taliano, librarian at the Stephen B. Thacker CDC Library, for her expert contribution and assistance in developing the search strategy and conducting electronic database searches. We appreciate Jeffrey Hodis’ contributions in screening articles and conducting full-text reviews.

The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention.

Author information

Authors and affiliations.

Division of Global Migration Health, Centers for Disease Control and Prevention, 1600 Clifton Road NE, Mailstop V18-2, Atlanta, GA, 30329-4027, USA

Faruque Ahmed, Livvy Shafer, Pallavi Malla, Roderick Hopkins, Sarah Moreland, Nicole Zviedrite & Amra Uzicanin

Oak Ridge Institute for Science and Education, Oak Ridge, TN, USA

Livvy Shafer, Pallavi Malla & Sarah Moreland

Cherokee Nation Operational Solutions, Tulsa, OK, USA

Roderick Hopkins

You can also search for this author in PubMed   Google Scholar

Contributions

F.A. participated in all steps of the research and was a major contributor in writing the manuscript. L.S. and P.M. participated in screening records, full-text reviews, data extraction, and risk of bias assessment. R.H. participated in screening records, full-text reviews, data extraction, risk of bias assessment, and drafting the manuscript. S.M. and N.Z. participated in screening records and full-text reviews. A.U. participated in conceptualizing the systematic review, resolving questions about the review, and revising the manuscript. F.A. prepared the figures. All authors reviewed the manuscript.

Corresponding author

Correspondence to Faruque Ahmed .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: table s1..

Guidance for the implementation of non-pharmaceutical interventions to prevent the transmission of COVID-19 in non-healthcare workplaces, 2020. Table S2. PRISMA 2020 checklist. Table S3. Search strategy, January 1, 2020–May 11, 2021. Table S4. Characteristics and results of studies assessing effect of physical distancing, physical barriers, workplace closures, and lockdowns, January 1, 2020-May 11, 2021: COVID-19 morbidity and mortality outcomes. Table S5. Characteristics and results of studies assessing effect of lockdowns, January 1, 2020-May 11, 2021: Anxiety and depression symptoms. Table S6. Characteristics and results of studies assessing effect of lockdowns, January 1, 2020-May 11, 2021: Unemployment and labor force participation. Table S7. Risk of bias assessment for studies, January 1, 2020–May 11, 2021: COVID-19 morbidity and mortality outcomes. Table S8. Risk of bias assessment for studies, January 1, 2020–May 11, 2021: Anxiety and depression symptoms. Table S9. Risk of bias assessment for studies, January 1, 2020–May 11, 2021: Unemployment and labor force participation outcomes. Table S10. Studies excluded from the review and reasons for exclusion, January 1, 2020–May 11, 2021.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Ahmed, F., Shafer, L., Malla, P. et al. Systematic review of empiric studies on lockdowns, workplace closures, and other non-pharmaceutical interventions in non-healthcare workplaces during the initial year of the COVID-19 pandemic: benefits and selected unintended consequences. BMC Public Health 24 , 884 (2024). https://doi.org/10.1186/s12889-024-18377-1

Download citation

Received : 05 April 2023

Accepted : 17 March 2024

Published : 22 March 2024

DOI : https://doi.org/10.1186/s12889-024-18377-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Community mitigation
• Non-pharmaceutical
• Novel coronavirus
• Social distancing
• Systematic review

BMC Public Health

ISSN: 1471-2458

• Submission enquiries: [email protected]
• General enquiries: [email protected]

REVIEW article

Cognitive, behavioral and psychiatric symptoms in patients with spinal cord injury: a scoping review.

• 1 Department of Clinical and Experimental Medicine, University of Messina, Messina, Italy
• 2 IRCCS Centro Neurolesi Bonino-Pulejo, Messina, Italy

Spinal Cord Injury (SCI) is a condition where the spinal cord is damaged and experiences partial or complete loss of motor and/or sensory function, which is typically less than normal. After SCI, patients may exhibit more severe psychiatric symptoms and experience cognitive impairments, including reduced speed and attention processing capacity, as well as difficulties with executive function and episodic memory retention. Among the behavioral and psychiatric symptoms, depression, anxiety, substance use disorder, and posttraumatic stress disorder are the most common. This review aims to investigate the cognitive, behavioral, or psychiatric symptoms of the patient with SCI and their influence on the rehabilitation process. Studies were identified from an online search of PubMed, Web of Science, Cochrane Library, and Embase databases. Studies published between 2013-2023 were selected. This review has been registered on OSF (n) 3KB2U. We have found that patients with SCI are at high risk of cognitive impairment and experience a wide range of difficulties, including tasks based on processing speed and executive function. This clinical population may experience adjustment disorders with depression and anxiety, as well as other psychiatric symptoms such as fatigue, stress, and suicidal ideation. This review has demonstrated that SCI patients may experience psychiatric symptoms and cognitive impairments that affect their functioning. At the same time, these patients may be more prone to various adjustment and mood disorders. Moreover, these two aspects may interact with each other, causing a range of symptoms, increasing the risk of hospitalization, and delaying the rehabilitation process.

1 Introduction

Spinal cord injury (SCI) is a devastating neurological condition that causes physical dependence, morbidity, and psychological stress and occurs when the spinal cord is damaged, usually below the level of injury, leading to partial or complete loss of motor and/or sensory function ( 1 ). Over the last 30 years, the global prevalence has increased from 236 to 1,298 cases per million population, whereas the global incidence of SCI is estimated to have increased from 250,000 to 500,000 cases per year ( 2 ). Before the 1940s, only 10% to 20% of people survived more than a few weeks after injury ( 3 ). Technological advances have significantly improved this condition, with 90% of people now surviving for more than a year after injury and around 50% surviving for 40 years after injury ( 4 ). SCI has a severe impact on physical well-being, leading to decreased physical functioning and increased pain ( 5 , 6 ). Maladaptive emotional and/or behavioral responses to identifiable psychosocial stressors, such as SCI or other stress-related events, are hallmarks of adjustment disorder, which results in maladjustment following an event that is disproportionate to the stressor. They are characterized by stress responses that deviate from socially or culturally expected responses to the stressor and/or cause significant distress and impairment in daily functioning ( 7 ). Mood is defined as a broad and persistent emotional tone that persists internally and affects almost all aspects of a person’s behavior in the external world. Mood disorders are expressed by significant emotional disturbances (severe lows, called depression, or highs, called hypomania or mania). These include bipolar disorder, cyclical hypomania, hypomania, major depressive disorder, mood dysphoria, persistent depressive disorder, and premenstrual dysphoric mood disorder. These are common mental disorders leading to increased morbidity and mortality, and mood disorders are broadly divided into bipolar and depressive disorders ( 8 ).

These disorders may be present in SCI patients who experience increased depression, stress, and anxiety following an injury ( 9 , 10 ). Resulting psychiatric symptoms are common in this clinical population ( 11 – 13 ) reflecting significant changes in general well-being after injury ( 14 ). Furthermore, up to 50% of patients with SCI appear to have cognitive impairment ( 15 – 18 ). Common cognitive problems include decreased processing speed and attention, and difficulties in episodic memory and executive function ( 19 – 27 ). According to a recent report, the risk of cognitive impairment in patients with SCI is approximately 13 times higher than in healthy individuals. Moreover, cognitive impairment is a strong predictor of inadequate social participation after hospital discharge, with an 8.4:1 probability of being unable to work ( 28 ). Impairments in attention, concentration, memory, problem-solving, and reasoning tend to be most common, but there are significant individual differences in cognitive functioning after SCI and a wide range in performance depending on the severity and pattern of affected areas ( 29 ). Traumatic brain injury (TBI) comorbidity is often cited as a cause of cognitive impairment in people with SCI, as external forces causing trauma to the spinal cord ( 30 , 31 ) sometimes result in TBI. However, research also shows that a significant number of people with traumatic or non-traumatic SCI and no history of TBI have neuropsychological dysfunction ( 32 , 33 ). Studies in human and animal models suggest that cognitive impairment in patients with SCI may be a consequence of pathophysiological factors such as cortical remodeling and atrophy, neuroinflammation, hypoxia, vascular dysfunction, and accelerated aging ( 34 – 36 ). Furthermore, treatment-related factors (including medications prescribed during acute and chronic treatment), sequelae of SCI (e.g., sleep disorders and chronic pain), and pre-onset conditions (e.g., psychiatric disorders and learning difficulties) may contribute to the development of cognitive impairment after SCI ( 37 ). Patients with moderate to severe cognitive impairment have been reported to have difficulty learning skills during rehabilitation, experience more disturbances in sleep and appetite, require higher levels of care, and have reduced functional independence ( 38 , 39 ). One tool that can be used to create a baseline cognitive profile for people with SCI is the Repeatable Battery for Assessment of Neuropsychological Status (RBANS), which is not suitable for the cognitive assessment of people with SCI with upper motor dysfunction, as it includes several subtests of this domain ( 40 , 41 ). Primary and Secondary Disability Rating Scales (ADAPSS) are also available to cover important stressors specific to the SCI population ( 42 ). To detect anxiety and depression, the Hospital Anxiety and Depression Scale (HADS) can also be used ( 43 ). A summary of these three tools is displayed in Supplementary Table 1 .

Depression is a common secondary complication after SCI ( 44 ) and is associated with poorer health status ( 45 ), reduced functioning ( 46 , 47 ), and increased mortality ( 48 ). SCI patients with probable major depression are known to be more likely to report a history of other psychiatric disorders than those without probable major depression. Depression is often associated with diagnoses of other psychiatric disorders such as anxiety disorders and substance use disorders ( 49 ). Anxiety is a problem for adults with acquired SCI, with 45% of injured individuals reporting experiencing excessive worry, fear, or panic ( 50 ) and a high risk of experiencing disorders such as generalized anxiety disorder. The traumatic nature of SCI ongoing fear of life-threatening secondary outcomes, or pre-injury psychological morbidity can cause increased distress ( 51 – 53 ). Instead, post-injury psychological distress can be so intense and unbearable that patients may seek relief through the consumption of substances such as alcohol, tobacco, and cannabis ( 54 , 55 ). For alcohol, consumption patterns are similar to the general population in terms of gender and age, but the rate of “risky consumption” is higher in the SCI population ( 56 , 57 ). Tobacco is the second most consumed substance by this clinical population ( 58 ). Its prevalence ranges between 19-40% of injured patients. Tobacco use, in this population is often associated with harmful alcohol consumption. Logically, this harms an individual’s health and increases the likelihood of suffering from other medical complications ( 59 – 61 ). Another psychiatric condition that may develop in this population is post-traumatic stress disorder (PTSD) ( 62 ). Research has found that quadriplegia is associated with a reduced risk of PTSD, while paraplegia is associated with an increased risk ( 63 ) Moreover, combat veterans are more likely to suffer from PTSD and experience more severe symptoms than non-combat zone veterans ( 64 ). A summary of cognitive and adjustment disorders in patients with SCI is shown in Figure 1 .

Figure 1 A summary of cognitive, behavioral, and psychiatric symptoms, in patients with SCI.

All these symptoms and dysfunctional behaviors must be treated. Specialized medical care is essential in the early management of SCI to ensure survival and prevent unnecessary complications. However, once the acute phase has passed, medicine cannot provide a cure, and everyone must learn to live with the disability that presents itself in their environment and work on their cognitive limitations or dysfunctional behaviors and emotions.

This scoping review aims to update what is known in this field about the cognitive, behavioral, or psychiatric symptoms in patients with SCI and their influence on the rehabilitation process.

2 Materials and methods

2.1 search strategy.

A literature search was conducted via PubMed, Web of Science, Cochrane Library, and Embase, and it was carried out for articles using the following search keyword terms: (All Fields: “Spinal Cord Injury”) AND (All Fields: “Cognitive Symptoms”); (All Fields: “Spinal Cord Injury”) AND (All Fields: “Psychiatric Symptoms”) with 2013-2023 search time range. We adopted the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow diagram to describe the sequence of steps (identification, screening, eligibility, and inclusion) for the collection and determination of qualified studies as shown in Figure 2 . Titles and abstracts were independently scanned and retrieved from database searches. The suitability of the article was then assessed according to the defined inclusion criteria. Ultimately, we received all titles and abstracts that met the criteria for inclusion in the full text. To avoid bias, several expert teams worked together, selected the articles, and analyzed the data independently, and discussed any discrepancies with each other. Disagreements between reviewers were resolved by consensus.

Figure 2 PRISMA 2020 flow diagram of evaluated studies.

This review has been registered on OSF (n) 3KB2U.

2.2 PICO evaluation

We defined our combination of search terms using a PICO (population, intervention, comparison, outcome) model. The population was limited to patients with moderate to severe SCI; the intervention included all studies, rehabilitation approaches, and assessment tools to measure and understand cognitive, behavioral, or psychiatric symptoms, and their influence on the rehabilitation process; the comparison was evaluated considering the different cognitive, behavioral and psychiatric symptoms in patients with SCI both before and during a psychological and motor rehabilitation process; and the result included any improvements in the identification of cognitive, behavioral and psychiatric symptoms as well as their evaluation, resolution or attenuation during the rehabilitation process.

2.3 Inclusion criteria

A study was included if it described or investigated the cognitive, behavioral, or psychiatric symptoms of a patient with SCI and their influence on the rehabilitation process. The review included only articles written in English. Studies describing or investigating the functional assessment of these patients were also included. We only included studies conducted in human populations that met the following criteria: (i) original or protocol studies of any type and (ii) articles that presented some cognitive, behavioral, or psychiatric symptoms of the patient with SCI and their influence on the rehabilitation process.

2.4 Exclusion criteria

A study was excluded if there was a lack of data or information about the description of the cognitive, behavioral, or psychiatric symptoms of the patient with SCI and their influence on the rehabilitation process. Systematic, integrated, or narrative reviews were also excluded, but reference lists were reviewed and included as necessary. Articles with cognitive and psychiatric symptoms purely due to TBI or other existing neurological conditions, such as Dementia, PD, or MS were also excluded. All articles written in languages other than English were finally excluded.

In total, 1397 articles were found: 353 articles were removed due to duplication after screening; 17 articles were excluded because they were not published in English; 860 articles were excluded based on title and abstract screening. Finally, 153 articles were removed based on screening for inadequate study designs and untraceable articles ( Figure 2 ). Fourteen research articles met the inclusion criteria and were therefore included in the review. A survey of these studies is shown in Supplementary Table 2 .

The articles described in this review investigated the cognitive, behavioral, or psychiatric symptoms of the patient with SCI and their influence on the rehabilitation process. The cognitive symptoms of patients with SCI were analyzed in four articles ( 12 , 41 , 65 , 66 ). The behavioral and psychiatric symptoms of these clinical populations are described in ten articles ( 67 – 76 ).

3.1 Cognitive symptoms in patients with SCI

Functional diagnosis of SCI patients involves the identification of cognitive symptoms that are relevant to their psychological functional status during rehabilitation. According to one study, these patients who resided in the community were found to be at a greater risk of mild cognitive impairment and faced certain challenges, including issues with processing speed, executive function-based tasks, and episodic memory tests. Furthermore, individuals with quadriplegia exhibit lower scores on cognitive tests of processing speed and executive function than those without paraplegic impairment ( 12 ). The second research discovered variations in cognition, age, and quality of life (QoL) between individuals with and without SCI. Those participants who did not have SCI had more favorable associations with QoL and cognition, while those with SCI did less poorly. Positive emotions/well-being and resilience were observed in those with SCI, along with a higher QoL. However, there was no difference in cognitive functioning between SCI and non-injured participants. This implies that individuals with SCI can adjust their QoL by decreasing the significance placed on mobility and cognitive impairment ( 65 ).The psychological effect of cognitive decline during acute rehabilitation is still evident in a prospective observational study of over 89 patients, which can have implications for the mental health of those with SCI. Through the use of RBANS and profile analysis, psychometric results indicated three groups that could be distinguished from each other by cognitive function. The majority of individuals (class 1 [54%]) did not experience any cognitive impairment in any domain. On the other hand, investigations revealed a population with late-onset memory impairment (class 2 [26%]) and cognitive impairment in different areas (Class 3 [20%]). Education, smoking, and drug use were strongly linked to cognitive impairment and classroom conditions. Those who had not been educated in high school had a history of drug use, smoked, and experienced more post-concussion symptoms were at heightened risk for enrollment in class 3 than in the first year of the program ( 41 ). A further study revealed that there was no overall correlation between neuropsychological test performance and symptom measures. Despite this, self-cognition was affected by anxiety and tiredness. Individuals who had cognitive impairment in one or more domains were found to have worse cognitive functioning when anxious. Those with cognitive impairment in at least one cognitive area reported lower working memory scores due to anxiety, while those with fatigue reported impaired delayed memory performance. Poor performance was also correlated with delayed recall, and cognitive performance showed no correlation with other depressive or fatigue symptoms ( 66 ).These first results demonstrate that the environment, together with age and education, in which people with this pathology live can negatively compromise cognitive functioning, increasing the probability of deterioration or further risks; however, they can implement processes of adaptation in their QoL by decreasing the importance or meaning attributed to their cognitive deficit. Levels of anxiety and fatigue can also influence cognitive performance so attention to these two aspects is essential to prevent or reduce other possible risk factors.

3.2 Behavioral and psychiatric symptoms in patients with SCI

Patients with SCI conditions may experience behavioral and psychiatric symptoms, which can lead to other disorders and clinical pathologies. In one study, was found that depression scores decreased 3 years after injury. Compared to the 1-year post-injury period, depression scores decreased in the 3 years after injury (T2) and 1 year (T1). This result suggests that the risk of depression in SCI patients increases with age, and the balance between cognitive and somatic symptoms of illness may be influenced by high autonomic fluctuations ( 67 ). A cross-sectional study found that individuals with SCI exhibit symptoms of depression and anxiety, which indicate lower levels of independence in exercise, personal hygiene, bowel control, and social interactions. A negative correlation was discovered with the Language subtest of the Montreal Cognitive Assessment Scale (MoCA) while the degree, nature, and duration of the injury were not correlated with alcohol or illicit drug use. The primary predictor of depression was the presence of anxiety, and SCI-related factors were not significant. Finally, characteristics of the Functional Independence Measure (FIM) and cognitive aspects of the MoCA scale were found to be the main predictors of depressive symptoms ( 68 ).A significant correlation was found between lower depressive symptoms and greater general self-efficacy and meaning in life. Through appraisal and coping strategies, it was observed that there were significant direct effects on higher life satisfaction and significant negative effects in individuals with depressive symptoms.The effect of general self-efficacy on depressive symptoms was fully mediated by appraisal and coping strategies ( 69 ). A study conducted in 2007 revealed that out of the 41,213 veterans surveyed, 2,615 had been diagnosed with SCI and depression, while 70% were also found to have other mental illnesses, with trauma Post-stress disorder and other anxiety disorders being the most prevalent. Veterans who suffered from SCI and depression were prone to frequenting more medical facilities and receiving additional medication than those who did not suffer from any depression ( 70 ). Another article highlights the connection between daily fatigue in these clinical populations, which results in heightened depressive symptoms reduced cognitive function, and a significantly lower level of social interaction during the event. Despite being socially engaged, anxiety and pain did not change significantly on that day. In comparison, when taking into account all symptoms, there was no correlation between daily changes in pain intensity or anxiety and social interaction on the same day. By altering the model to reflect age, gender, education level, injury category number, and time since the injury occurred, these effects were discovered ( 71 ). A multicenter study has revealed medically and psychologically related outcomes to pain. The presence of depression and anxiety symptoms at high levels is a reliable predictor of various pain factors, including gender, injury-related traits, and secondary comorbidities. Most patients with SCI endorsed lower levels of depression and anxiety on average. Moderate to severe depression was seen in 9.1% (suicidal ideation in 3.2%). Moderate to severe anxiety was seen in 8.0%. In addition, 28.3% were currently or in the past treated for or diagnosed with depression (5.9% for suicidal ideation and 3.2% for suicide attempts), while anxiety rates were slightly lower (22.5%). Only a small proportion of the sample (9.6%) reported receiving psychological treatment for mental health problems in the past year ( 72 ). A longitudinal study was conducted on 21 community-dwelling adults with chronic spinal cord injury at ages T1 (2004) and T2 (2009). Most were married, men, and completely paralyzed. Although some participants at T2 reported clinically significant psychiatric symptoms, none of the remaining participants at T1 met the eligibility criteria for T2. A 30-year-old woman with complete paralysis and non-traumatic SCI was found to be extremely stressed and depressed (18 years later), while a 64-year-old man suffered from a traumatic stress disorder and paralyzing limb weakness and was severely depressed (7 years after the injury) ( 73 ). In another article, 33% (n=21) of a sample of 63 SCI patients had suicidal ideation in the last two weeks; 71.4% (n=15) of suicidal ideation patients had depression; 52.4% (n=11) of patients with suicidal ideation were diagnosed with full-blown PTSD and 52.4% (n=11) of suicidal ideation patients had depression. Resilience was significantly lower in patients with depression and SI. While depression predicts suicidal ideation in traumatic SCI, resilience is a protective factor against SI ( 74 ). In a cross-sectional survey study, it was demonstrated that pain and fatigue were independently associated with depression, but only pain was associated with physical functioning. Furthermore, depression was more severe in middle-aged participants than in younger or older participants. Physical functioning decreased with increasing age and severity of injury ( 75 ). In one last comparative study of 37 patients with SCI, 89.2% (n. 33) of the patients had pain and 27.0% (n. 10) reported very severe pain; 9% had a history of psychiatric treatment for insomnia, depression, or anxiety after SCI but were currently receiving psychiatric treatment. The results found were that resilience can reduce the negative effects of pain. In addition, resilience independently contributed to reduced depression and posttraumatic growth ( 76 ). These results indicate that among the psychiatric symptoms that these patients may manifest are depression and anxiety. Regarding the first psychiatric pathology, we can state that its symptoms increase with age leading to autonomic fluctuations between cognitive/somatic deficits and lower levels in various aspects of functional and daily independence. However, the use of functional assessment and coping strategies leads to greater satisfaction. Patients with SCI are also more vulnerable to anxiety, daily fatigue, and pain, leading to a greater likelihood of developing depressive symptoms, including suicidal ideation, with high levels of stress, even years later. A greater focus on identifying and measuring the intensity and degree of these psychiatric symptoms becomes essential to prevent worse consequences.

4 Discussion

Our scoping review aimed to update what is known in this field about the cognitive, behavioral, or psychiatric symptoms in patients with SCI and their influence on the rehabilitation process. The studies included in this review have demonstrated that patients with SCI are at high risk of cognitive impairment and experience a wide range of difficulties, including tasks based on processing speed and executive function. Despite these difficulties, however, they appear to be able to readjust their QoL through positive affect and flexibility, minimizing the importance of mobility and cognitive impairment ( 12 , 65 ). The effects of cognitive decline in acute rehabilitation have been shown to persist after hospital discharge and may affect the psychological well-being of this clinical population. Factors such as education, smoking, drug use, and post-concussion symptoms may also influence cognitive impairment. Patients with SCI with anxiety show worsening cognitive functioning, leading to impaired working memory functioning and delayed memory functioning ( 41 , 66 ). Among behavioral and psychiatric symptoms, some articles suggest that people with SCI may experience depression, and their risk of developing it, may increase over time. Anxiety symptoms can also be a precursor to depression, and these adjustment disorders can harm independence in many aspects of life, such as exercise, personal hygiene, and social interaction ( 67 – 69 ). Routine fatigue, pain, depression, and anxiety may be associated with poorer social participation. Despite these findings, depressive symptoms are mediated by appraisal and coping strategies, and general self-efficacy and high life purpose are among the protective factors ( 70 , 71 ). In addition, symptoms of intense anxiety and depression are consistent predictors of various aspects of pain and stress ( 72 , 73 ). When depression is present in these patients, suicidal ideation occurs as a psychiatric symptom, and resilience may be a protective factor against suicidal ideation. Finally, pain and fatigue are independently associated with depression in patients with SCI ( 74 – 76 ).

Scientific literature supports the idea that psychiatric and cognitive symptoms can affect the rehabilitation process. Cognitive impairment has a significant negative impact on functional outcomes after SCI, with little functional gain during rehabilitation ( 77 ). In addition, these individuals, with cognitive impairment, have been reported to have increased aggressive behaviors and a higher risk of rehospitalization ( 34 ). SCI patients may exhibit different patterns of cognitive performance depending on the degree of injury, and overall performance may differ significantly from healthy controls. Furthermore, these patients may have pre-illness difficulties that affect cognition (and increase the risk of SCI), such as learning disabilities, and substance abuse ( 78 , 79 ). The literature highlights the fact that cognitive and psychiatric symptoms are linked, in fact, some consequences and complications of SCI, such as sleep-disordered breathing/sleep apnea, mental disorders, chronic pain, medication side effects, fatigue, and decreased physical activity, may affect cognition ( 80 – 84 ). For example, up to 50% of patients with high-grade SCI have sleep apnea and its severity may be directly related to cognitive impairment ( 85 , 86 ). Several studies have shown that there is a significant negative correlation between the severity of depression and cognitive ability and that cognitive ability is a strong predictor of psychological impairment after hospital discharge ( 87 ). Looking at this issue from a rehabilitation perspective, four possible treatments have been explored to address cognitive impairment: 1) drug therapy ( 82 , 84 , 88 ), 2) percutaneous tibial nerve stimulation ( 89 ), 3) dietary therapy and supplements ( 90 ), and 4) inpatient rehabilitation ( 91 , 92 ). Evidence on the effects of drug therapy dietary modification and supplement interventions on cognition in SCI is sparse and inconclusive ( 93 ), while percutaneous tibial nerve stimulation is a safe treatment for cognitive impairment after SCI ( 89 ). Combining several inpatient rehabilitation treatments has positive but heterogeneous effects on cognition.

From a psychiatric perspective, SCI patients experiencing psychological distress also show strong associations with measures of mood: both acceptability and self-efficacy are significantly negatively associated with depression, and the latter is also associated with anxiety, all of which may influence negative rehabilitation ( 94 ). The relationship between SCI and depressive symptoms is moderated by the quality of social support, the degree of conflict within the family, and the cognitive mediation of events. Major depressive episodes are more likely to occur when several psychosocial variables lead to prolonged feelings of hopelessness and helplessness. Frequent stressful events following an injury are likely to increase the patient’s feelings of helplessness and hopelessness. The impact of stressful events is therefore mediated by patients’ cognitive appraisals of their coping resources and their characteristic patterns of response to threatening events. Healthcare providers should assess the patient’s psychiatric and emotional problems and utilize the wide range of resources that may be available to the patient. Treatment professionals may need to assess the patient’s coping skills to anticipate adjustment problems. By facilitating access to patients’ adaptive personality traits, intellectual abilities, and social outlets, treatment can reduce the risk of psychiatric symptoms such as loss of interest, apathy, loneliness, loss of motivation, and depression ( 95 ). Cognitive therapy can also help reduce negative attitudes underlying depressed mood ( 96 ). To guarantee functional psychological rehabilitation in this patient population, the situation should be considered as a ‘non-isolated’ process involving the medical team, family, and close friends. All these people can play an important role in alleviating cognitive and psychological symptoms. Therefore, the goals of the rehabilitation team should include facilitating the family’s adaptation to changing circumstances ( 97 ). Furthermore, by understanding the influential variables associated with disability acceptance, rehabilitation educators and researchers can design professional training and research programs that fully consider the psychological, social, and occupational consequences of living with SCI ( 98 ).

This scoping review had several strengths. It is based on evidence from longitudinal observational populations and cross-sectional studies with large sample sizes. It includes an analysis of the cognitive impairments and psychiatric symptoms as well as some instruments to detect them and their disability. We have also identified data gaps in many areas, hopefully providing information for future research. This review has contributed to highlighting some cognitive, behavioral, and psychiatric aspects and symptoms of patients with SCI, often overshadowed compared to medical and biological problems from the point of view of the literature, through a selection of studies carried out over the last 10 years (from 2013 to 2023) and therefore based on recent data. Compared to past literature, this review allows us to infer that the psychological functioning (both cognitive and emotional) of the patient with SCI must be analyzed and monitored over time to avoid the evolution of a worse clinical picture characterized by various cognitive deficits (in information processing speed, working memory) or psychiatric symptoms (depression, anxiety, fatigue, stress). Particular attention must be paid to the living environment (together with age) and their behavioral patterns as these are two variables that can increase or decrease exposure to further risks (education, smoking, drug use, post-concussion symptoms) and connote different diagnosis and rehabilitation paths. Unlike previous literature, this study highlighted various potential environmental and subjective risk factors and identified functional coping mechanisms and resilience as protective factors for managing clinical symptoms, particularly psychiatric symptoms. Furthermore, its positive impact was also underscored.

The main limitation of the present study is the few papers that meet the inclusion criteria, as we included only fourteen articles that explored cognitive, behavioral, and psychiatric symptoms, and only four of them focused on the cognitive aspect. This, besides the heterogenous methodology and samples, prevents us from drawing robust evidence on this important topic. Four databases were also used, and the articles were restricted by date, so it is possible that important evidence was omitted. Furthermore, the sample size varies a lot: some are large, some are small, and the parameters measured are different. Clinicians in the rehabilitation field should recognize the potential for cognitive impairment, screen for such impairments, and provide proactive interventions for this population. Much information is needed to clarify the meaning of cognitive impairment to effectively improve cognitive impairment and maximize functional independence in the SCI population. Future research could also examine protective factors that reduce the likelihood of developing adjustment and mood disorders such as depression and anxiety after SCI, such as seeking social support rather than isolation and avoidance, and a tendency to engage in difficult emotional experiences.

In conclusion, this review shows that SCI patients may experience psychiatric symptoms and cognitive impairments that affect their functioning. At the same time, these patients may be more prone to various adjustment and mood disorders such as depression, stress, and anxiety. Moreover, these two aspects may interact with each other, causing a range of symptoms, increasing the risk of hospitalization, and delaying the rehabilitation process. Given the few studies included in our work, the conclusions that can currently be drawn are preliminary and the current evidence requires further investigations. Researchers should continue to study these clinical symptoms and disorders and their role in the rehabilitation process and develop practical interventions to be implemented as early as possible after a traumatic or non-traumatic SCI.

Author contributions

AC: Data curation, Investigation, Methodology, Writing – original draft, Writing – review & editing. DC: Data curation, Formal analysis, Resources, Visualization, Writing – original draft, Writing – review & editing. RDL: Conceptualization, Formal analysis, Visualization, Writing – original draft, Writing – review & editing. AQ: Funding acquisition, Project administration, Supervision, Writing – original draft, Writing – review & editing. FC: Data curation, Visualization, Writing – original draft, Writing – review & editing. RC: Conceptualization, Methodology, Supervision, Writing – original draft, Writing – review & editing.

The author(s) declare financial support was received for the research, authorship, and/or publication of this article. This study was supported by Current Research Funds 2024, Ministry of Health, Italy.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyt.2024.1369714/full#supplementary-material

1. American Spinal Injury Association. (2015). Available online at: http://www.asia-spinalinjury.org .

Google Scholar

2. Khorasanizadeh M, Yousefifard M, Eskian M, Lu Y, Chalangari M, Harrop JS, et al. Neurological recovery following traumatic spinal cord injury: a systematic review and meta-analysis. J Neurosurg Spine . (2019) 15:1–17. doi: 10.3171/2018.10.SPINE18802

CrossRef Full Text | Google Scholar

3. Silver JR. History of treatment of spinal injuries. Postgraduate Medical Journal . Oxford Academic (2005) 81:108–14. doi: 10.1136/pgmj.2004.0199922

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Middleton JW, Dayton A, Walsh J, Rutkowski SB, Leong G, Duong S. Life expectancy after spinal cord injury: a 50-year study. Spinal Cord . (2012) 50:803–11. doi: 10.1038/sc.2012.55

5. Burke D, Lennon O, Fullen BM. Quality of life after spinal cord injury: The impact of pain. Eur J Pain . (2018) 22:1662–72. doi: 10.1002/ejp.1248

6. Gaspar R, Padula N, Freitas TB, de Oliveira JPJ, Torriani-Pasin C. Physical exercise for individuals with spinal cord injury: systematic review based on the international classification of functioning, disability, and health. J Sport Rehabil . (2019) 28:505–16. doi: 10.1123/jsr.2017-0185

7. O’Donnell ML, Agathos JA, Metcalf O, Gibson K, Lau W. Adjustment disorder: current developments and future directions. Int J Environ Res Public Health . (2019) 16:2537. doi: 10.3390/ijerph16142537

8. Spijker J, Claes S. Stemmingsstoornissen in de DSM-5 [Mood disorders in the DSM-5]. TijdschrPsychiatr . (2014) 56:173–6.

9. Siddall PJ, Middleton JW. Spinal cord injury-induced pain: mechanisms and treatments. Pain Manage . (2015) 5:493–507. doi: 10.2217/pmt.15.47

10. Lim SW, Shiue YL, Ho CH, Yu SC, Kao PH, Wang JJ, et al. Anxiety and depression in patients with traumatic spinal cord injury: A nationwide population-based cohort study. PloS One . (2017) 12:e0169623. doi: 10.1371/journal.pone.0169623

11. Placeres AF, Fiorati RC. Assessment instruments and depression rates in people with spinal cord injury: a systematic review. Rev Esc Enferm USP . (2018) 52:e03388. doi: 10.1590/S1980-220X2017037303388

12. Cohen ML, Tulsky DS, Holdnack JA, Carlozzi NE, Wong A, Magasi S, et al. Cognition among community-dwelling individuals with spinal cord injury. Rehabil Psychol . (2017) 62:425–34. doi: 10.1037/rep0000140

13. Macciocchi SN, Seel RT, Thompson N. The impact of mild traumatic brain injury on cognitive functioning following co-occurring spinal cord injury. Arch Clin Neuropsychol . (2013) 28:684–91. doi: 10.1093/arclin/act049

14. Sachdeva R, Gao F, Chan CCH, Krassioukov AV. Cognitive function after spinal cord injury: A systematic review. Neurology . (2018) 91:611–21. doi: 10.1212/WNL.0000000000006244

15. Whalley Hammell K. Quality of life after spinal cord injury: a meta-synthesis of qualitative findings. Spinal Cord . (2007) 45:124–39. doi: 10.1038/sj.sc.3101992

16. Craig A, Guest R, Tran Y, Middleton J. Cognitive impairment and mood states after spinal cord injury. J Neurotrauma . (2017) 34:1156–63. doi: 10.1089/neu.2016.4632

17. Davidoff GN, Roth EJ, Richards JS. Cognitive deficits in spinal cord injury: epidemiology and outcome. Arch Phys Med Rehabil . (1992) 73:275–84.

PubMed Abstract | Google Scholar

18. Dowler RN, Harrington DL, Haaland KY, Swanda RM, Fee F, Fiedler K. Profiles of cognitive functioning in chronic spinal cord injury and the role of moderating variables. J Int Neuropsychol Soc . (1997) 3:464–72.

19. Murray RF, Asghari A, Egorov DD, Rutkowski SB, Siddall PJ, Soden RJ, et al. Impact of spinal cord injury on self-perceived pre- and postmorbid cognitive, emotional and physical functioning. Spinal Cord . (2007) 45:429–36. doi: 10.1038/sj.sc.3102022

20. Richards JS, Brown L, Hagglund K, Bua G, Reeder K. Spinal cord injury and concomitant traumatic brain injury. Results of a longitudinal investigation. Am J Phys Med Rehabil . (1988) 67:211–6. doi: 10.1097/00002060-198810000-00005

21. Bradbury CL, Wodchis WP, Mikulis DJ, Pano EG, Hitzig SL, McGillivray CF, et al. Traumatic brain injury in patients with traumatic spinal cord injury: clinical and economic consequences. Arch Phys Med Rehabil . (2008) 89:S77–84. doi: 10.1016/j.apmr.2008.07.008

22. Davidoff G, Morris J, Roth E, Bleiberg J. Cognitive dysfunction and mild closed head injury in traumatic spinal cord injury. Arch Phys Med Rehabil . (1985) 66:489–91.

23. Dowler RN, O’Brien SA, Haaland KY, Harrington DL, Feel F, Fiedler K. Neuropsychological functioning following a spinal cord injury. Appl Neuropsychol . (1995) 2:124–9. doi: 10.1080/09084282.1995.9645349

24. Hess MJ, Zhan EH, Foo DK, Yalla SV. Bladder cancer in patients with spinal cord injury. J Spinal Cord Med . (2003) 26:335–8. doi: 10.1080/10790268.2003.11753702

25. Lazzaro I, Tran Y, Wijesuriya N, Craig A. Central correlates of impaired information processing in people with spinal cord injury. J Clin Neurophysiol . (2013) 30:59–65. doi: 10.1097/WNP.0b013e31827edb0c

26. Roth E, Davidoff G, Thomas P, Doljanac R, Dijkers M, Berent S, et al. A controlled study of neuropsychological deficits in acute spinal cord injury patients. Paraplegia . (1989) 27:480–9. doi: 10.1038/sc.1989.75

27. Wilmot CB, Cope DN, Hall KM, Acker M. Occult head injury: its incidence in spinal cord injury. Arch Phys Med Rehabil . (1985) 66:227–31. doi: 10.1016/0003-9993(85)90148-0

28. Craig A, Nicholson Perry K, Guest R, Tran Y, Middleton J. Adjustment following chronic spinal cord injury: Determining factors that contribute to social participation. Br J Health Psychol . (2015) 20:807–23. doi: 10.1111/bjhp.12143

29. Strubreither W, Hackbusch B, Hermann-Gruber M, Stahr G, Jonas HP. Neuropsychological aspects of the rehabilitation of patients with paralysis from a spinal injury who also have a brain injury. Spinal Cord . (1997) 35:487–92. doi: 10.1038/sj.sc.3100495

30. Iida H, Tachibana S, Kitahara T, Horiike S, Ohwada T, Fujii K. Association of head trauma with cervical spine injury, spinal cord injury, or both. J Trauma . (1999) 46:450–2. doi: 10.1097/00005373-199903000-00018

31. Michael DB, Guyot DR, Darmody WR. Coincidence of head and cervical spine injury. J Neurotrauma . (1989) 6:177–89. doi: 10.1089/neu.1989.6.177

32. Pasipanodya EC, Dirlikov B, Castillo K, Shem KL. Cognitive profilesamong individuals with spinal cord injuries: predictors and relations with psychological well-being. Arch Phys Med Rehabil . (2020). doi: 10.1016/j.apmr.2020.06.022

33. Rahn K, Slusher B, Kaplin A. Cognitive impairment in multiple sclerosis: a forgotten disability remembered. Cerebrum . (2012) 2012:14.

34. Chiaravalloti ND, Weber E, Wylie G, Dyson-Hudson T, Wecht JM. Patterns of cognitive deficits in persons with spinal cord injury as compared with both age-matched and older individuals without spinal cord injury. J Spinal Cord Med . (2020) 43:88–97. doi: 10.1080/10790268.2018.1543103

35. Wu J, Stoica BA, Luo T, Sabirzhanov B, Zhao Z, Guanciale K, et al. Isolated spinal cord contusion in rats induces chronic brain neuroinflammation, neurodegeneration, and cognitive impairment. Involvement of cell cycle activation. Cell Cycle . (2014) 13:2446–58. doi: 10.4161/cc.29420

36. Jegede AB, Rosado-Rivera D, Bauman WA, Cardozo CP, Sano M, Moyer JM, et al. Cognitive performance in hypotensive persons with spinal cord injury. Clin Auton Res . (2010) 20:3–9. doi: 10.1007/s10286-009-0036-z

37. Warren AM, Pullins J, Elliott TR. Concomitant cognitive impairment in persons with spinal cord injuries in rehabilitation settings . Neuropsychol Inpatient Rehabil Environ Hauppauge NY Nova Sci Publ Inc (2008) p. 79–98.

38. Avluk OC, Gurcay E, Gurcay AG, Karaahmet OZ, Tamkan U, Cakci A. Effects of chronic pain on function, depression, and sleep among patients with traumatic spinal cord injury. Ann Saudi Med . (2014) 34:211–6. doi: 10.5144/0256-4947.2014.211

39. Richards JS, Osuna FJ, Jaworski TM, Novack TA, Leli DA, Boll TJ. The effectiveness of different methods of defining traumatic brain injury in predicting postdischarge adjustment in a spinal cord injury population. Arch Phys Med Rehabil . (1991) 72:275–9.

40. Randolph C, Tierney MC, Mohr E, Chase TN. The Repeatable Battery for the Assessment of Neuropsychological Status (RBANS): preliminary clinical validity. J Clin Exp Neuropsychol . (1998) 20:310–9. doi: 10.1076/jcen.20.3.310.823

41. Pasipanodya EC, Dirlikov B, Castillo K, Shem KL. Cognitive profiles among individuals with spinal cord injuries: predictors and relations with psychological well-being. Arch Phys Med Rehabil . (2021) 102:431–9. doi: 10.1016/j.apmr.2020.06.022

42. Deane KC, Chlan KM, Vogel LC, Zebracki K. Use of Appraisals of DisAbility Primary and Secondary Scale-Short Form (ADAPSS-sf) in individuals with pediatric-onset spinal cord injury. Spinal Cord . (2020) 58:290–7. doi: 10.1038/s41393-019-0375-0

43. Sivertsen HE, Helvik AS, Gjøra L, Haugan G. Psychometric validation of the Hospital Anxiety and Depression Scale (HADS) in community-dwelling older adults. BMC Psychiatry . (2023) 23:903. doi: 10.1186/s12888-023-05407-2

44. Craig A, Tran Y, Middleton J. Psychological morbidity and spinal cord injury: a systematic review. Spinal Cord . (2009) 47:108–14. doi: 10.1038/sc.2008.115

45. Herrick S, Elliott TR, Crow F. Self-appraised problem-solving skills and the prediction of secondary complications among persons with spinal cord injuries. J Clin Psychol Med Settings . (1994) 1:269–83. doi: 10.1007/BF01989628

46. Fuhrer MJ, Rintala DH, Hart KA, Clearman R, Young ME. Depressive symptomatology in persons with spinal cord injury who reside in the community. Arch Phys Med Rehabil . (1993) 74:255–60.

47. Umlauf R, Frank RG. A cluster-analytic description of patient subgroups in the rehabilitation setting. Rehabil Psychol . (1983) 28:157–67.

48. Krause JS, Carter RE, Pickelsimer EE, Wilson D. A prospective study of health and risk of mortality after spinal cord injury. Arch Phys Med Rehabil . (2008) 89:1482–91. doi: 10.1016/j.apmr.2007.11.062

49. Banerjea R, Findley PA, Smith B, Findley T, Sambamoorthi U. Co-occurring medical and mental illness and substance use disorders among veteran clinic users with spinal cord injury patients with complexities. Spinal Cord . (2009) 47:789–95. doi: 10.1038/sc.2009.42

50. Mitchell MC, Burns NR, Dorstyn DS. Screening for depression and anxiety in spinal cord injury with DASS-21. Spinal Cord . (2008) 46:547–51. doi: 10.1038/sj.sc.3102154

51. Agar E, Kennedy P, King NS. The role of negative cognitive appraisals in PTSD symptoms following spinal cord injuries. BehavCognPsychother . (2006) 34:437–52.

52. Chung MC, Preveza E, Papandreou K, Prevezas N. The relationship between posttraumatic stress disorder following spinal cord injury and locus of control. J Affect Disord . (2006) 93:229–32. doi: 10.1016/j.jad.2006.02.021

53. Tuszynski MH, Steeves JD, Fawcett JW, Lammertse D, Kalichman M, Rask C, et al. Guidelines for the conduct of clinical trials for spinal cord injury as developed by the ICCP Panel: clinical trial inclusion/exclusion criteria and ethics. Spinal Cord . (2007) 45:222–31. doi: 10.1038/sj.sc.3102009

54. Smedema SM, Ebener D. Substance abuse and psychosocial adaptation to physical disability: analysis of the literature and future directions. DisabilRehabil . (2010) 32:1311–9. doi: 10.3109/09638280903514721

55. Treischmann RB. Spinal Cord Injuries: Psychological, social and vocational rehabilitation . 2nd ed. New York: Demos Medical Publishing (1988).

56. Tate DG, Forchheimer MB, Krause JS, Meade MA, Bombardier CH. Patterns of alcohol and substance use and abuse in persons with spinal cord injury: risk factors and correlates. Arch Phys Med Rehabil . (2004) 85:1837–47. doi: 10.1016/j.apmr.2004.02.022

57. Turner AP, Bombardier CH, Rimmele CT. A typology of alcohol use patterns among persons with recent traumatic brain injury or spinal cord injury: implications for treatment matching. Arch Phys Med Rehabil . (2003) 84:358–64. doi: 10.1053/apmr.2003.50107

58. Weaver FM, Smith B, LaVela SL, Evans CT, Ullrich P, Miskevics S, et al. Smoking behavior and delivery of evidence-based care for veterans with spinal cord injuries and disorders. J Spinal Cord Med . (2011) 34:35–45. doi: 10.1179/107902610X12911165975061

59. de Groot S, Post MW, Snoek GJ, Schuitemaker M, van der Woude LH. Longitudinal association between lifestyle and coronary heart disease risk factors among individuals with spinal cord injury. Spinal Cord . (2013) 51:314–8. doi: 10.1038/sc.2012.153

60. Hwang M, Chlan KM, Vogel LC, Zebracki K. Substance use in young adults with pediatric-onset spinal cord injury. Spinal Cord . (2012) 50:497–501. doi: 10.1038/sc.2012.8

61. Krause JS, Saunders LL. Risk of hospitalizations after spinal cord injury: relationship with biographical, injury, educational, and behavioral factors. Spinal Cord . (2009) 47:692–7. doi: 10.1038/sc.2009.16

62. Radnitz CL, Schlein IS, Walczak S, Broderick CP, Binks M, Tirch DD, et al. The prevalence of posttraumatic stress disorder in veterans with spinal cord injury. Sci Psychosoc Process . (1995) 8:145–9.

63. Radnitz CL, Hsu L, Willard J, Perez-Strumolo L, Festa J, Lillian LB, et al. Posttraumatic stress disorder in veterans with spinal cord injury: trauma-related risk factors. J Trauma Stress . (1998) 11:505–20. doi: 10.1023/A:1024404729251

64. Radnitz CL, Schlein IS, Hsu L. The effect of prior trauma exposure on the development of PTSD following spinal cord injury. J Anxiety Disord . (2000) 14:313–24. doi: 10.1016/s0887-6185(00)00025-6

65. Dudley-Javoroski S, Lee J, Shields RK. Cognitive function, quality of life, and aging: relationships in individuals with and without spinal cord injury. Physiother Theory Pract . (2022) 38:36–45. doi: 10.1080/09593985.2020.1712755

66. Carlozzi NE, Graves CM, Troost JP, Ehde DM, Miner JA, Kratz AL. Association of physical and mental symptoms with cognition in people with spinal cord injury. Rehabil Psychol . (2021) 66:532–40. doi: 10.1037/rep0000416

67. Singh V, Mitra S. Autonomic variability, depression and the disability paradox in spinal cord injury. Spinal Cord Ser Cases . (2022) 8:76. doi: 10.1038/s41394-022-00542-6

68. Hara ACP, Aching NC, Marques LM, Fregni F, Battisttella LR, Simis M. Clinical and demographic predictors of symptoms of depression and anxiety in patients with spinal cord injury. Spinal Cord . (2022) 60:1123–9. doi: 10.1038/s41393-022-00831-9

69. Peter C, Müller R, Post MW, van Leeuwen CM, Werner CS, Geyh S, et al. Depression in spinal cord injury: assessing the role of psychological resources. Rehabil Psychol . (2015) 60:67–80. doi: 10.1037/rep0000021

70. Ullrich PM, Smith BM, Blow FC, Valenstein M, Weaver FM. Depression, healthcare utilization, and comorbid psychiatric disorders after spinal cord injury. J Spinal Cord Med . (2014) 37:40–5. doi: 10.1179/2045772313Y.0000000137

71. Kuzu D, Troost JP, Carlozzi NE, Ehde DM, Molton IR, Kratz AL. How do fluctuations in pain, fatigue, anxiety, depressed mood, and perceived cognitive function relate to same-day social participation in individuals with spinal cord injury? Arch Phys Med Rehabil . (2022) 103:385–93. doi: 10.1016/j.apmr.2021.07.809

72. Murray CB, Zebracki K, Chlan KM, Moss AC, Vogel LC. Medical and psychological factors related to pain in adults with pediatric-onset spinal cord injury: a biopsychosocial model. Spinal Cord . (2017) 55:405–10. doi: 10.1038/sc.2016.137

73. Migliorini C, Callaway L, New P. Preliminary investigation into subjective well-being, mental health, resilience, and spinal cord injury. J Spinal Cord Med . (2013) 36:660–5. doi: 10.1179/2045772313Y.0000000100

74. UstaSağlam NG, Aksoy Poyraz C, Doğan D, Erhan B. Suicidal ideation, post-traumatic stress disorder, and depression in traumatic spinal cord injury: What resilience tells us. J Spinal Cord Med . (2023) 46:309–16. doi: 10.1080/10790268.2022.2039856

75. Alschuler KN, Jensen MP, Sullivan-Singh SJ, Borson S, Smith AE, Molton IR. The association of age, pain, and fatigue with physical functioning and depressive symptoms in persons with spinal cord injury. J Spinal Cord Med . (2013) 36:483–91. doi: 10.1179/2045772312Y.0000000072

76. Min JA, Lee CU, Hwang SI, Shin JI, Lee BS, Han SH, et al. The moderation of resilience on the negative effect of pain on depression and post-traumatic growth in individuals with spinal cord injury. DisabilRehabil . (2014) 36:1196–202. doi: 10.3109/09638288.2013.834985

77. Chiaravalloti ND, Weber E, Wylie G, Dyson-Hudson T, Wecht JM. The impact of level of injury on patterns of cognitive dysfunction in individuals with spinal cord injury. J Spinal Cord Med . (2020) 43:633–41. doi: 10.1080/10790268.2019.1696076

78. Molina B, Segura A, Serrano JP, Alonso FJ, Molina L, Pérez-Borrego YA, et al. Cognitive performance of people with traumatic spinal cord injury: a cross-sectional study comparing people with subacute and chronic injuries. Spinal Cord . (2018) 56:796–805. doi: 10.1038/s41393-018-0076-0

79. Mollayeva T, Pacheco N, D’Souza A, Colantonio A. The course and prognostic factors of cognitive status after central nervous system trauma: a systematic review protocol. BMJ Open . (2017) 7:e017165. doi: 10.1136/bmjopen-2017-017165

80. Wecht JM, Rosado-Rivera D, Jegede A, Cirnigliaro CM, Jensen MA, Kirshblum S, et al. Systemic and cerebral hemodynamics during cognitive testing. Clin Auton Res . (2012) 22:25–33. doi: 10.1007/s10286-011-0139-1

81. Nightingale TE, Zheng MMZ, Sachdeva R, Phillips AA, Krassioukov AV. Diverse cognitive impairment after spinal cord injury is associated with orthostatic hypotension symptom burden. PhysiolBehav . (2020) 213:112742. doi: 10.1016/j.physbeh.2019.112742

82. Shem K, Barncord S, Flavin K, Mohan M. Adverse cognitive effect of gabapentin in individuals with spinal cord injury: preliminary findings. Spinal Cord Ser Cases . (2018) 4:9. doi: 10.1038/s41394-018-0038-y

83. Schembri R, Spong J, Graco M, Berlowitz DJ. COSAQ study team. Neuropsychological Function in Patients With Acute Tetraplegia and Sleep Disordered Breathing. Sleep . (2017) 40. doi: 10.1093/sleep/zsw037

84. Krebs J, Scheel-Sailer A, Oertli R, Pannek J. The effects of antimuscarinic treatment on the cognition of spinal cord injured individuals with neurogenic lower urinary tract dysfunction: a prospective controlled before-and-after study. Spinal Cord . (2018) 56:22–7. doi: 10.1038/sc.2017.94

85. Sankari A, Vaughan S, Bascom A, Martin JL, Badr MS. Sleep-disordered breathing and spinal cord injury: A state-of-the-art review. Chest . (2019) 155:438–45. doi: 10.1016/j.chest.2018.10.002

86. Shnek ZM, Foley FW, LaRocca NG, Gordon WA, DeLuca J, Schwartzman HG, et al. Helplessness, self-efficacy, cognitive distortions, and depression in multiple sclerosis and spinal cord injury. Ann Behav Med . (1997) 19:287–94. doi: 10.1007/BF02892293

87. Craig A, Tran Y, Wijesuriya N, Middleton J. Fatigue and tiredness in people with spinal cord injury. J Psychosom Res . (2012) 73:205–10. doi: 10.1016/j.jpsychores.2012.07.005

88. NorouziJavidan A, Sabour H, Latifi S, Abrishamkar M, Soltani Z, Shidfar F, et al. Does consumption of polyunsaturated fatty acids influence on neurorehabilitation in traumatic spinal cord-injured individuals? A double-blinded clinical trial. Spinal Cord . (2014) 52:378–82. doi: 10.1038/sc.2014.30

89. Stampas A, Korupolu R, Zhu L, Smith CP, Gustafson K. Safety, feasibility, and efficacy of transcutaneous tibial nerve stimulation in acute spinal cord injury neurogenic bladder: A randomized control pilot trial. Neuromodulation . (2019) 22:716–22. doi: 10.1111/ner.12855

90. Allison DJ, Josse AR, Gabriel DA, Klentrou P, Ditor DS. Targeting inflammation to influence cognitive function following spinal cord injury: a randomized clinical trial. Spinal Cord . (2017) 55:26–32. doi: 10.1038/sc.2016.96

91. Zonfrillo MR, Durbin DR, Winston FK, Zhang X, Stineman MG. Residual cognitive disability after completion of inpatient rehabilitation among injured children. J Pediatr . (2014) 164:130–5. doi: 10.1016/j.jpeds.2013.09.022

92. Hartmann A, Kegelmeyer D, Kloos A. Use of an errorless learning approach in a person with concomitant traumatic spinal cord injury and brain injury: A case report. J Neurol Phys Ther . (2018) 42:102–9. doi: 10.1097/NPT.0000000000000218

93. Pacheco N, Mollayeva S, Jacob B, Colantonio A, Mollayeva T. Interventions and cognitive functioning in adults with traumatic spinal cord injuries: a systematic review and meta-analysis. DisabilRehabil . (2021) 43:903–19. doi: 10.1080/09638288.2019.1644380

94. Nicholson Perry K, Nicholas MK, Middleton J. Spinal cord injury-related pain in rehabilitation: a cross-sectional study of relationships with cognitions, mood and physical function. Eur J Pain . (2009) 13:511–7. doi: 10.1016/j.ejpain.2008.06.003

95. Overholser JC, Schubert DS. Depression in patients with spinal cord injuries: A synthesis of cognitive and somatic processes. Curr Psychol . (1993) 12:172–83.

96. Overholser JC. Cognitive-behavioral treatment of depression, Part III: Reducing cognitive biases. J Contemp Psychother . (1995) 25:311–29.

97. Judd FK, Brown DJ. The psychosocial approach to rehabilitation of the spinal cord injured patient. Paraplegia . (1988) 26:419–24. doi: 10.1038/sc.1988.65

98. Woodrich F, Patterson JB. Variables related to acceptance of disability in persons with spinal cord injuries. J Rehabil . (1983) 49:26–30.

Keywords: spinal cord injury, cognitive symptoms, psychiatric symptoms, neurorehabilitation, mental health

Citation: Calderone A, Cardile D, De Luca R, Quartarone A, Corallo F and Calabrò RS (2024) Cognitive, behavioral and psychiatric symptoms in patients with spinal cord injury: a scoping review. Front. Psychiatry 15:1369714. doi: 10.3389/fpsyt.2024.1369714

Received: 12 January 2024; Accepted: 12 March 2024; Published: 20 March 2024.

Reviewed by:

Copyright © 2024 Calderone, Cardile, De Luca, Quartarone, Corallo and Calabrò. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Davide Cardile, [email protected]

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.