peer review in research paper

When you choose to publish with PLOS, your research makes an impact. Make your work accessible to all, without restrictions, and accelerate scientific discovery with options like preprints and published peer review that make your work more Open.

• PLOS Biology
• PLOS Climate
• PLOS Complex Systems
• PLOS Computational Biology
• PLOS Digital Health
• PLOS Genetics
• PLOS Global Public Health
• PLOS Medicine
• PLOS Mental Health
• PLOS Neglected Tropical Diseases
• PLOS Pathogens
• PLOS Sustainability and Transformation
• PLOS Collections

How to Write a Peer Review

When you write a peer review for a manuscript, what should you include in your comments? What should you leave out? And how should the review be formatted?

This guide provides quick tips for writing and organizing your reviewer report.

Review Outline

Use an outline for your reviewer report so it’s easy for the editors and author to follow. This will also help you keep your comments organized.

Think about structuring your review like an inverted pyramid. Put the most important information at the top, followed by details and examples in the center, and any additional points at the very bottom.

Here’s how your outline might look:

1. Summary of the research and your overall impression

In your own words, summarize what the manuscript claims to report. This shows the editor how you interpreted the manuscript and will highlight any major differences in perspective between you and the other reviewers. Give an overview of the manuscript’s strengths and weaknesses. Think about this as your “take-home” message for the editors. End this section with your recommended course of action.

2. Discussion of specific areas for improvement

It’s helpful to divide this section into two parts: one for major issues and one for minor issues. Within each section, you can talk about the biggest issues first or go systematically figure-by-figure or claim-by-claim. Number each item so that your points are easy to follow (this will also make it easier for the authors to respond to each point). Refer to specific lines, pages, sections, or figure and table numbers so the authors (and editors) know exactly what you’re talking about.

Major vs. minor issues

What’s the difference between a major and minor issue? Major issues should consist of the essential points the authors need to address before the manuscript can proceed. Make sure you focus on what is  fundamental for the current study . In other words, it’s not helpful to recommend additional work that would be considered the “next step” in the study. Minor issues are still important but typically will not affect the overall conclusions of the manuscript. Here are some examples of what would might go in the “minor” category:

• Missing references (but depending on what is missing, this could also be a major issue)
• Technical clarifications (e.g., the authors should clarify how a reagent works)
• Data presentation (e.g., the authors should present p-values differently)
• Typos, spelling, grammar, and phrasing issues

3. Any other points

Confidential comments for the editors.

Some journals have a space for reviewers to enter confidential comments about the manuscript. Use this space to mention concerns about the submission that you’d want the editors to consider before sharing your feedback with the authors, such as concerns about ethical guidelines or language quality. Any serious issues should be raised directly and immediately with the journal as well.

This section is also where you will disclose any potentially competing interests, and mention whether you’re willing to look at a revised version of the manuscript.

Do not use this space to critique the manuscript, since comments entered here will not be passed along to the authors.  If you’re not sure what should go in the confidential comments, read the reviewer instructions or check with the journal first before submitting your review. If you are reviewing for a journal that does not offer a space for confidential comments, consider writing to the editorial office directly with your concerns.

Get this outline in a template

Giving Feedback

Giving feedback is hard. Giving effective feedback can be even more challenging. Remember that your ultimate goal is to discuss what the authors would need to do in order to qualify for publication. The point is not to nitpick every piece of the manuscript. Your focus should be on providing constructive and critical feedback that the authors can use to improve their study.

If you’ve ever had your own work reviewed, you already know that it’s not always easy to receive feedback. Follow the golden rule: Write the type of review you’d want to receive if you were the author. Even if you decide not to identify yourself in the review, you should write comments that you would be comfortable signing your name to.

In your comments, use phrases like “ the authors’ discussion of X” instead of “ your discussion of X .” This will depersonalize the feedback and keep the focus on the manuscript instead of the authors.

General guidelines for effective feedback

• Justify your recommendation with concrete evidence and specific examples.
• Be specific so the authors know what they need to do to improve.
• Be thorough. This might be the only time you read the manuscript.
• Be professional and respectful. The authors will be reading these comments too.
• Remember to say what you liked about the manuscript!

Don’t

• Recommend additional experiments or  unnecessary elements that are out of scope for the study or for the journal criteria.
• Tell the authors exactly how to revise their manuscript—you don’t need to do their work for them.
• Use the review to promote your own research or hypotheses.
• Focus on typos and grammar. If the manuscript needs significant editing for language and writing quality, just mention this in your comments.
• Submit your review without proofreading it and checking everything one more time.

Before and After: Sample Reviewer Comments

Keeping in mind the guidelines above, how do you put your thoughts into words? Here are some sample “before” and “after” reviewer comments

✗ Before

“The authors appear to have no idea what they are talking about. I don’t think they have read any of the literature on this topic.”

✓ After

“The study fails to address how the findings relate to previous research in this area. The authors should rewrite their Introduction and Discussion to reference the related literature, especially recently published work such as Darwin et al.”

“The writing is so bad, it is practically unreadable. I could barely bring myself to finish it.”

“While the study appears to be sound, the language is unclear, making it difficult to follow. I advise the authors work with a writing coach or copyeditor to improve the flow and readability of the text.”

“It’s obvious that this type of experiment should have been included. I have no idea why the authors didn’t use it. This is a big mistake.”

“The authors are off to a good start, however, this study requires additional experiments, particularly [type of experiment]. Alternatively, the authors should include more information that clarifies and justifies their choice of methods.”

Suggested Language for Tricky Situations

You might find yourself in a situation where you’re not sure how to explain the problem or provide feedback in a constructive and respectful way. Here is some suggested language for common issues you might experience.

What you think : The manuscript is fatally flawed. What you could say: “The study does not appear to be sound” or “the authors have missed something crucial”.

What you think : You don’t completely understand the manuscript. What you could say : “The authors should clarify the following sections to avoid confusion…”

What you think : The technical details don’t make sense. What you could say : “The technical details should be expanded and clarified to ensure that readers understand exactly what the researchers studied.”

What you think: The writing is terrible. What you could say : “The authors should revise the language to improve readability.”

What you think : The authors have over-interpreted the findings. What you could say : “The authors aim to demonstrate [XYZ], however, the data does not fully support this conclusion. Specifically…”

What does a good review look like?

Check out the peer review examples at F1000 Research to see how other reviewers write up their reports and give constructive feedback to authors.

Time to Submit the Review!

Be sure you turn in your report on time. Need an extension? Tell the journal so that they know what to expect. If you need a lot of extra time, the journal might need to contact other reviewers or notify the author about the delay.

Tip: Building a relationship with an editor

You’ll be more likely to be asked to review again if you provide high-quality feedback and if you turn in the review on time. Especially if it’s your first review for a journal, it’s important to show that you are reliable. Prove yourself once and you’ll get asked to review again!

• Getting started as a reviewer
• Responding to an invitation
• Reading a manuscript
• Writing a peer review

The contents of the Peer Review Center are also available as a live, interactive training session, complete with slides, talking points, and activities. …

The contents of the Writing Center are also available as a live, interactive training session, complete with slides, talking points, and activities. …

There’s a lot to consider when deciding where to submit your work. Learn how to choose a journal that will help your study reach its audience, while reflecting your values as a researcher…

Peer review templates, expert examples and free training courses

Joanna Wilkinson

Learning how to write a constructive peer review is an essential step in helping to safeguard the quality and integrity of published literature. Read on for resources that will get you on the right track, including peer review templates, example reports and the Web of Science™ Academy: our free, online course that teaches you the core competencies of peer review through practical experience ( try it today ).

How to write a peer review

Understanding the principles, forms and functions of peer review will enable you to write solid, actionable review reports. It will form the basis for a comprehensive and well-structured review, and help you comment on the quality, rigor and significance of the research paper. It will also help you identify potential breaches of normal ethical practice.

This may sound daunting but it doesn’t need to be. There are plenty of peer review templates, resources and experts out there to help you, including:

Peer review training courses and in-person workshops

• Peer review templates ( found in our Web of Science Academy )
• Expert examples of peer review reports
• Co-reviewing (sharing the task of peer reviewing with a senior researcher)

Other peer review resources, blogs, and guidelines

We’ll go through each one of these in turn below, but first: a quick word on why learning peer review is so important.

Why learn to peer review?

Peer reviewers and editors are gatekeepers of the research literature used to document and communicate human discovery. Reviewers, therefore, need a sound understanding of their role and obligations to ensure the integrity of this process. This also helps them maintain quality research, and to help protect the public from flawed and misleading research findings.

Learning to peer review is also an important step in improving your own professional development.

You’ll become a better writer and a more successful published author in learning to review. It gives you a critical vantage point and you’ll begin to understand what editors are looking for. It will also help you keep abreast of new research and best-practice methods in your field.

We strongly encourage you to learn the core concepts of peer review by joining a course or workshop. You can attend in-person workshops to learn from and network with experienced reviewers and editors. As an example, Sense about Science offers peer review workshops every year. To learn more about what might be in store at one of these, researcher Laura Chatland shares her experience at one of the workshops in London.

There are also plenty of free, online courses available, including courses in the Web of Science Academy such as ‘Reviewing in the Sciences’, ‘Reviewing in the Humanities’ and ‘An introduction to peer review’

The Web of Science Academy also supports co-reviewing with a mentor to teach peer review through practical experience. You learn by writing reviews of preprints, published papers, or even ‘real’ unpublished manuscripts with guidance from your mentor. You can work with one of our community mentors or your own PhD supervisor or postdoc advisor, or even a senior colleague in your department.

Go to the Web of Science Academy

Peer review templates

Peer review templates are helpful to use as you work your way through a manuscript. As part of our free Web of Science Academy courses, you’ll gain exclusive access to comprehensive guidelines and a peer review report. It offers points to consider for all aspects of the manuscript, including the abstract, methods and results sections. It also teaches you how to structure your review and will get you thinking about the overall strengths and impact of the paper at hand.

• Web of Science Academy template (requires joining one of the free courses)
• PLoS’s review template
• Wiley’s peer review guide (not a template as such, but a thorough guide with questions to consider in the first and second reading of the manuscript)

Beyond following a template, it’s worth asking your editor or checking the journal’s peer review management system. That way, you’ll learn whether you need to follow a formal or specific peer review structure for that particular journal. If no such formal approach exists, try asking the editor for examples of other reviews performed for the journal. This will give you a solid understanding of what they expect from you.

Peer review examples

Understand what a constructive peer review looks like by learning from the experts.

Here’s a sample of pre and post-publication peer reviews displayed on Web of Science publication records to help guide you through your first few reviews. Some of these are transparent peer reviews , which means the entire process is open and visible — from initial review and response through to revision and final publication decision. You may wish to scroll to the bottom of these pages so you can first read the initial reviews, and make your way up the page to read the editor and author’s responses.

• Pre-publication peer review: Patterns and mechanisms in instances of endosymbiont-induced parthenogenesis
• Pre-publication peer review: Can Ciprofloxacin be Used for Precision Treatment of Gonorrhea in Public STD Clinics? Assessment of Ciprofloxacin Susceptibility and an Opportunity for Point-of-Care Testing
• Transparent peer review: Towards a standard model of musical improvisation
• Transparent peer review: Complex mosaic of sexual dichromatism and monochromatism in Pacific robins results from both gains and losses of elaborate coloration
• Post-publication peer review: Brain state monitoring for the future prediction of migraine attacks
• Web of Science Academy peer review: Students’ Perception on Training in Writing Research Article for Publication

F1000 has also put together a nice list of expert reviewer comments pertaining to the various aspects of a review report.

Co-reviewing

Co-reviewing (sharing peer review assignments with senior researchers) is one of the best ways to learn peer review. It gives researchers a hands-on, practical understanding of the process.

In an article in The Scientist , the team at Future of Research argues that co-reviewing can be a valuable learning experience for peer review, as long as it’s done properly and with transparency. The reason there’s a need to call out how co-reviewing works is because it does have its downsides. The practice can leave early-career researchers unaware of the core concepts of peer review. This can make it hard to later join an editor’s reviewer pool if they haven’t received adequate recognition for their share of the review work. (If you are asked to write a peer review on behalf of a senior colleague or researcher, get recognition for your efforts by asking your senior colleague to verify the collaborative co-review on your Web of Science researcher profiles).

The Web of Science Academy course ‘Co-reviewing with a mentor’ is uniquely practical in this sense. You will gain experience in peer review by practicing on real papers and working with a mentor to get feedback on how their peer review can be improved. Students submit their peer review report as their course assignment and after internal evaluation receive a course certificate, an Academy graduate badge on their Web of Science researcher profile and is put in front of top editors in their field through the Reviewer Locator at Clarivate.

Here are some external peer review resources found around the web:

• Peer Review Resources from Sense about Science
• Peer Review: The Nuts and Bolts by Sense about Science
• How to review journal manuscripts by R. M. Rosenfeld for Otolaryngology – Head and Neck Surgery
• Ethical guidelines for peer review from COPE
• An Instructional Guide for Peer Reviewers of Biomedical Manuscripts by Callaham, Schriger & Cooper for Annals of Emergency Medicine (requires Flash or Adobe)
• EQUATOR Network’s reporting guidelines for health researchers

And finally, we’ve written a number of blogs about handy peer review tips. Check out some of our top picks:

• How to Write a Peer Review: 12 things you need to know
• Want To Peer Review? Top 10 Tips To Get Noticed By Editors
• Review a manuscript like a pro: 6 tips from a Web of Science Academy supervisor
• How to write a structured reviewer report: 5 tips from an early-career researcher

Want to learn more? Become a master of peer review and connect with top journal editors. The Web of Science Academy – your free online hub of courses designed by expert reviewers, editors and Nobel Prize winners. Find out more today.

Related posts

Reimagining research impact: introducing web of science research intelligence.

Beyond discovery: AI and the future of the Web of Science

Clarivate welcomes the Barcelona Declaration on Open Research Information

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 12 November 2021

Demystifying the process of scholarly peer-review: an autoethnographic investigation of feedback literacy of two award-winning peer reviewers

• Sin Wang Chong   ORCID: orcid.org/0000-0002-4519-0544 1 &
• Shannon Mason 2  

Humanities and Social Sciences Communications volume  8 , Article number:  266 ( 2021 ) Cite this article

2798 Accesses

6 Citations

18 Altmetric

Metrics details

• Language and linguistics

A Correction to this article was published on 26 November 2021

This article has been updated

Peer reviewers serve a vital role in assessing the value of published scholarship and improving the quality of submitted manuscripts. To provide more appropriate and systematic support to peer reviewers, especially those new to the role, this study documents the feedback practices and experiences of two award-winning peer reviewers in the field of education. Adopting a conceptual framework of feedback literacy and an autoethnographic-ecological lens, findings shed light on how the two authors design opportunities for feedback uptake, navigate responsibilities, reflect on their feedback experiences, and understand journal standards. Informed by ecological systems theory, the reflective narratives reveal how they unravel the five layers of contextual influences on their feedback practices as peer reviewers (micro, meso, exo, macro, chrono). Implications related to peer reviewer support are discussed and future research directions are proposed.

Similar content being viewed by others

The transformative power of values-enacted scholarship

What matters in the cultivation of student feedback literacy: exploring university efl teachers’ perceptions and practices, why english exploring chinese early career returnee academics’ motivations for writing and publishing in english, introduction.

The peer-review process is the longstanding method by which research quality is assured. On the one hand, it aims to assess the quality of a manuscript, with the desired outcome being (in theory if not always in practice) that only research that has been conducted according to methodological and ethical principles be published in reputable journals and other dissemination outlets (Starck, 2017 ). On the other hand, it is seen as an opportunity to improve the quality of manuscripts, as peers identify errors and areas of weakness, and offer suggestions for improvement (Kelly et al., 2014 ). Whether or not peer review is actually successful in these areas is open to considerable debate, but in any case it is the “critical juncture where scientific work is accepted for publication or rejected” (Heesen and Bright, 2020 , p. 2). In contemporary academia, where higher education systems across the world are contending with decreasing levels of public funding, there is increasing pressure on researchers to be ‘productive’, which is largely measured by the number of papers published, and of funding grants awarded (Kandiko, 2010 ), both of which involve peer review.

Researchers are generally invited to review manuscripts once they have established themselves in their disciplinary field through publication of their own research. This means that for early career researchers (ECRs), their first exposure to the peer-review process is generally as an author. These early experiences influence the ways ECRs themselves conduct peer review. However, negative experiences can have a profound and lasting impact on researchers’ professional identity. This appears to be particularly true when feedback is perceived to be unfair, with feedback tone largely shaping author experience (Horn, 2016 ). In most fields, reviewers remain anonymous to ensure freedom to give honest and critical feedback, although there are concerns that a lack of accountability can result in ‘bad’ and ‘rude’ reviews (Mavrogenis et al., 2020 ). Such reviews can negatively impact all researchers, but disproportionately impact underrepresented researchers (Silbiger and Stubler, 2019 ). Regardless of career phase, no one is served well by unprofessional reviews, which contribute to the ongoing problem of bullying and toxicity prevalent in academia, with serious implications on the health and well-being of researchers (Keashly and Neuman, 2010 ).

Because of its position as the central process through which research is vetted and refined, peer review should play a similarly central role in researcher training, although it rarely features. In surveying almost 3000 researchers, Warne ( 2016 ) found that support for reviewers was mostly received “in the form of journal guidelines or informally as advice from supervisors or colleagues” (p. 41), with very few engaging in formal training. Among more than 1600 reviewers of 41 nursing journals, only one third received any form of support (Freda et al., 2009 ), with participants across both of these studies calling for further training. In light of the lack of widespread formal training, most researchers learn ‘on the job’, and little is known about how researchers develop their knowledge and skills in providing effective assessment feedback to their peers. In this study, we undertake such an investigation, by drawing on our first-hand experiences. Through a collaborative and reflective process, we look to identify the forms and forces of our feedback literacy development, and seek to answer specifically the following research questions:

What are the exhibited features of peer reviewer feedback literacy?

What are the forces at work that affect the development of feedback literacy?

Literature review

Conceptualisation of feedback literacy.

The notion of feedback literacy originates from the research base of new literacy studies, which examines ‘literacies’ from a sociocultural perspective (Gee, 1999 ; Street, 1997 ). In the educational context, one of the most notable types of literacy is assessment literacy (Stiggins, 1999 ). Traditionally, assessment literacy is perceived as one of the indispensable qualities of a successful educator, which refers to the skills and knowledge for teachers “to deal with the new world of assessment” (Fulcher, 2012 , p. 115). Following this line of teacher-oriented assessment literacy, recent attempts have been made to develop more subject-specific assessment literacy constructs (e.g., Levi and Inbar-Lourie, 2019 ). Given the rise of student-centred approaches and formative assessment in higher education, researchers began to make the case for students to be ‘assessment literate’; comprising of such knowledge and skills as understanding of assessment standards, the relationship between assessment and learning, peer assessment, and self-assessment skills (Price et al., 2012 ). Feedback literacy, as argued by Winstone and Carless ( 2019 ), is essentially a subset of assessment literacy because “part of learning through assessment is using feedback to calibrate evaluative judgement” (p. 24). The notion of feedback literacy was first extensively discussed by Sutton ( 2012 ) and more recently by Carless and Boud ( 2018 ). Focusing on students’ feedback literacy, Sutton ( 2012 ) conceptualised feedback literacy as a three-dimensional construct—an epistemological dimension (what do I know about feedback?), an ontological dimension (How capable am I to understand feedback?), and a practical dimension (How can I engage with feedback?). In close alignment with Sutton’s construct, the seminal conceptual paper by Carless and Boud ( 2018 ) further illustrated the four distinctive abilities of feedback literate students: the abilities to (1) understand the formative role of feedback, (2) make informed and accurate evaluative judgement against standards, (3) manage emotions especially in the face of critical and harsh feedback, and (4) take action based on feedback. Since the publication of Carless and Boud ( 2018 ), student and teacher feedback literacy has been in the limelight of assessment research in higher education (e.g., Chong 2021b ; Carless and Winstone 2020 ). These conceptual contributions expand the notion of feedback literacy to consider not only the manifestations of various forms of effective student engagement with feedback but also the confluence of contexts and individual differences of students in developing students’ feedback literacy by drawing upon various theoretical perspectives (e.g., ecological systems theory; sociomaterial perspective) and disciplines (e.g., business and human resource management). Others address practicalities of feedback literacy; for example, how teachers and students can work in synergy to develop feedback literacy (Carless and Winstone, 2020 ) and ways to maximise student engagement with feedback at a curricular level (Malecka et al., 2020). In addition to conceptualisation, advancement of the notion of feedback literacy is evident in the recent proliferation of primary studies. The majority of these studies are conducted in the field of higher education, focusing mostly on student feedback literacy in classrooms (e.g., Molloy et al., 2019 ; Winstone et al., 2019 ) and in the workplace (Noble et al., 2020 ), with a handful focused on teacher feedback literacy (e.g., Xu and Carless 2016 ). Some studies focusing on student feedback literacy adopt a qualitative case study research design to delve into individual students’ experience of engaging with various forms of feedback. For example, Han and Xu ( 2019 ) analysed the profiles of feedback literacy of two Chinese undergraduate students. Findings uncovered students’ resistance to engagement with feedback, which relates to the misalignment between the cognitive, social, and affective components of individual students’ feedback literacy profiles. Others reported interventions designed to facilitate students’ uptake of feedback, focusing on their effectiveness and students’ perceptions. Specifically, affordances and constraints of educational technology such as electronic feedback portfolio (Chong, 2019 ; Winstone et al., 2019 ) are investigated. Of particular interest is a recent study by Noble et al. ( 2020 ), which looked into student feedback literacy in the workplace by probing into the perceptions of a group of Australian healthcare students towards a feedback literacy training programme conducted prior to their placement. There is, however, a dearth of primary research in other areas where elicitation, process, and enactment of feedback are vital; for instance, academics’ feedback literacy. In the ‘publish or perish’ culture of higher education, academics, especially ECRs, face immense pressure to publish in top-tiered journals in their fields and face the daunting peer-review process, while juggling other teaching and administrative responsibilities (Hollywood et al., 2019 ; Tynan and Garbett 2007 ). Taking up the role of authors and reviewers, researchers have to possess the capacity and disposition to engage meaningfully with feedback provided by peer reviewers and to provide constructive comments to authors. Similar to students, researchers have to learn how to manage their emotions in the face of critical feedback, to understand the formative values of feedback, and to make informed judgements about the quality of feedback (Gravett et al., 2019 ). At the same time, feedback literacy of academics also resembles that of teachers. When considering the kind of feedback given to authors, academics who serve as peer reviewers have to (1) design opportunities for feedback uptake, (2) maintain a professional and supportive relationship with authors, and (3) take into account the practical dimension of giving feedback (e.g., how to strike a balance between quality of feedback and time constraints due to multiple commitments) (Carless and Winstone 2020 ). To address the above, one of the aims of the present study is to expand the application of feedback literacy as a useful analytical lens to areas outside the classroom, that is, scholarly peer-review activities in academia, by presenting, analysing, and synthesising the personal experiences of the authors as successful peer reviewers for academic journals.

Conceptual framework

We adopt a feedback literacy of peer reviewers framework (Chong 2021a ) as an analytical lens to analyse, systemise, and synthesise our own experiences and practices as scholarly peer reviewers (Fig. 1 ). This two-tier framework includes a dimension on the manifestation of feedback literacy, which categorises five features of feedback literacy of peer reviewers, informed by student and teacher feedback literacy frameworks by Carless and Boud ( 2018 ) and Carless and Winstone ( 2020 ). When engaging in scholarly peer review, reviewers are expected to be able to provide constructive and formative feedback, which authors can act on in their revisions ( engineer feedback uptake ). Besides, peer reviewers who are usually full-time researchers or academics lead hectic professional lives; thus, when writing reviewers’ reports, it is important for them to consider practically and realistically the time they can invest and how their various degrees of commitment may have an impact on the feedback they provide ( navigate responsibilities ). Furthermore, peer reviewers should consider the emotional and relational influences their feedback exert on the authors. It is crucial for feedback to be not only informative but also supportive and professional (Chong, 2018 ) ( maintain relationships ). Equally important, it is imperative for peer reviewers to critically reflect on their own experience in the scholarly peer-review process, including their experience of receiving and giving feedback to academic peers, as well as the ways authors and editors respond to their feedback ( reflect on feedback experienc e). Lastly, acting as gatekeepers of journals to assess the quality of manuscripts, peer reviewers have to demonstrate an accurate understanding of the journals’ aims, remit, guidelines and standards, and reflect those in their written assessments of submitted manuscripts ( understand standards ). Situated in the context of scholarly peer review, this collaborative autoethnographic study conceptualises feedback literacy not only as a set of abilities but also orientations (London and Smither, 2002 ; Steelman and Wolfeld, 2016 ), which refers to academics’ tendency, beliefs, and habits in relation to engaging with feedback (London and Smither, 2002 ). According to Cheung ( 2000 ), orientations are influenced by a plethora of factors, namely experiences, cultures, and politics. It is important to understand feedback literacy as orientations because it takes into account that feedback is a convoluted process and is influenced by a plethora of contextual and personal factors. Informed by ecological systems theory (Bronfenbrenner, 1986 ; Neal and Neal, 2013 ) and synthesising existing feedback literacy models (Carless and Boud, 2018 ; Carless and Winstone, 2020 ; Chong, 2021a , 2021b ), we consider feedback literacy as a malleable, situated, and emergent construct, which is influenced by the interplay of various networked layers of ecological systems (Neal and Neal, 2013 ) (Fig. 1 ). Also important is that conceptualising feedback literacy as orientations avoids dichotomisation (feedback literate vs. feedback illiterate), emphasises the developmental nature of feedback literacy, and better captures the multifaceted manifestations of feedback engagement.

The outer ring of the figure shows the components of feedback literacy while the inner ring concerns the layers of contexts (ecosystems) which influence the manifestation of feedback literacy of peer reviewers.

Echoing recent conceptual papers on feedback literacy which emphasises the indispensable role of contexts (Chong 2021b ; Boud and Dawson, 2021 ; Gravett et al., 2019 ), our conceptual framework includes an underlying dimension of networked ecological systems (micro, meso, exo, macro, and chrono), which portrays the contextual forces shaping our feedback orientations. Informed by the networked ecological system theory of Neal and Neal ( 2013 ), we postulate that there are five systems of contextual influence, which affect the feedback experience and development of feedback literacy of peer reviewers. The five ecological systems refer to ‘settings’, which is defined by Bronfenbrenner ( 1986 ) as “place[s] where people can readily engage in social interactions” (p. 22). Even though Bronfenbrenner’s ( 1986 ) somewhat dated definition of ‘place’ is limited to ‘physical space’, we believe that ‘places’ should be more broadly defined in the 21st century to encompass physical and virtual, recent and dated, closed and distanced locations where people engage; as for ‘interactions’, from a sociocultural perspective, we understand that ‘interactions’ can include not only social, but also cognitive and emotional exchanges (Vygotsky, 1978 ). Microsystem refers to a setting where people, including the focal individual, interact. Mesosystem , on the other hand, means the interactions between people from different settings and the influence they exert on the focal individual. An exosystem , similar to a microsystem, is understood as a single setting but this setting excludes the focal individual but it is likely that participants in this setting would interact with the focal individual. The remaining two systems, macrosystem and chronosystem, refer not only to ‘settings’ but ‘forces that shape the patterns of social interactions that define settings’ (Neal and Neal, 2013 , p. 729). Macrosystem is “the set of social patterns that govern the formation and dissolution of… interactions… and thus the relationship among ecological systems” (ibid). Some examples of macrosystems given by Neal and Neal ( 2013 ) include political and cultural systems. Finally, chronosystem is “the observation that patterns of social interactions between individuals change over time, and that such changes impact on the focal individual” (ibid, p. 729). Figure 2 illustrates this networked ecological systems theory using a hypothetical example of an early career researcher who is involved in scholarly peer review for Journal A; at the same time, they are completing a PhD and are working as a faculty member at a university.

This is a hypothetical example of an early career researcher who is involved in scholarly peer review for Journal A.

From the reviewed literature on the construct of feedback literacy, the investigation of feedback literacy as a personal, situated, and unfolding process is best done through an autoethnographic lens, which underscores critical self-reflection. Autoethnography refers to “an approach to research and writing that seeks to describe and systematically analyse (graphy) personal experience (auto) in order to understand cultural experience (ethno)” (Ellis et al., 2011 , p. 273). Autoethnography stems from research in the field of anthropology and is later introduced to the fields of education by Ellis and Bochner ( 1996 ). In higher education research, autoethnographic studies are conducted to illuminate on topics related to identity and teaching practices (e.g., Abedi Asante and Abubakari, 2020 ; Hains-Wesson and Young 2016 ; Kumar, 2020 ). In this article, a collaborative approach to autoethnography is adopted. Based on Chang et al. ( 2013 ), Lapadat ( 2017 ) defines collaborative autoethnography (CAE) as follows:

… an autobiographic qualitative research method that combines the autobiographic study of self with ethnographic analysis of the sociocultural milieu within which the researchers are situated, and in which the collaborating researchers interact dialogically to analyse and interpret the collection of autobiographic data. (p. 598)

CAE is not only a product but a worldview and process (Wall, 2006 ). CAE is a discrete view about the world and research, which straddles between paradigmatic boundaries of scientific and literary studies. Similar to traditional scientific research, CAE advocates systematicity in the research process and consideration is given to such crucial research issues as reliability, validity, generalisability, and ethics (Lapadat, 2017 ). In closer alignment with studies on humanities and literature, the goal of CAE is not to uncover irrefutable universal truths and generate theories; instead, researchers of CAE are interested in co-constructing and analysing their own personal narratives or ‘stories’ to enrich and/or challenge mainstream beliefs and ideas, embracing diverse rather than canonical ways of behaviour, experience, and thinking (Ellis et al., 2011 ). Regarding the role of researchers, CAE researchers openly acknowledge the influence (and also vulnerability) of researchers throughout the research process and interpret this juxtaposition of identities between researchers and participants of research as conducive to offering an insider’s perspective to illustrate sociocultural phenomena (Sughrua, 2019 ). For our CAE on the scholarly peer-review experiences of two ECRs, the purpose is to reconstruct, analyse, and publicise our lived experience as peer reviewers and how multiple forces (i.e., ecological systems) interact to shape our identity, experience, and feedback practice. As a research process, CAE is a collaborative and dynamic reflective journey towards self-discovery, resulting in narratives, which connect with and add to the existing literature base in a personalised manner (Ellis et al., 2011 ). The collaborators should go beyond personal reflection to engage in dialogues to identify similarities and differences in experiences to throw new light on sociocultural phenomena (Merga et al., 2018 ). The iterative process of self- and collective reflections takes place when CAE researchers write about their own “remembered moments perceived to have significantly impacted the trajectory of a person’s life” and read each other’s stories (Ellis et al., 2011 , p. 275). These ‘moments’ or vignettes are usually written retrospectively, selectively, and systematically to shed light on facets of personal experience (Hughes et al., 2012 ). In addition to personal stories, some autoethnographies and CAEs utilise multiple data sources (e.g., reflective essays, diaries, photographs, interviews with co-researchers) and various ways of expressions (e.g., metaphors) to achieve some sort of triangulation and to present evidence in a ‘systematic’ yet evocative manner (Kumar, 2020 ). One could easily notice that overarching methodological principles are discussed in lieu of a set of rigid and linear steps because the process of reconstructing experience through storytelling can be messy and emergent, and certain degree of flexibility is necessary. However, autoethnographic studies, like other primary studies, address core research issues including reliability (reader’s judgement of the credibility of the narrator), validity (reader’s judgement that the narratives are believable), and generalisability (resemblance between the reader’s experience and the narrative, or enlightenment of the reader regarding unfamiliar cultural practices) (Ellis et al., 2011 ). Ethical issues also need to be considered. For example, authors are expected to be honest in reporting their experiences; to protect the privacy of the people who ‘participated’ in our stories, pseudonyms need to be used (Wilkinson, 2019 ). For the current study, we follow the suggested CAE process outlined by Chang et al. ( 2013 ), which includes four stages: deciding on topic and method , collecting materials , making meaning , and writing . When deciding on the topic, we decided to focus on our experience as scholarly peer reviewers because doing peer review and having our work reviewed are an indispensable part of our academic lives. The next is to collect relevant autoethnographic materials. In this study, we follow Kumar ( 2020 ) to focus on multiple data sources: (1) reflective essays which were written separately through ‘recalling’, which is referred to by Chang et al. ( 2013 ) as ‘a free-spirited way of bringing out memories about critical events, people, place, behaviours, talks, thoughts, perspectives, opinions, and emotions pertaining to the research topic’ (p. 113), and (2) discussion meetings. In our reflective essays, we included written records of reflection and excerpts of feedback in our peer-review reports. Following material collection is meaning making. CAE, as opposed to autoethnography, emphasises the importance of engaging in dialogues with collaborators and through this process we identify similarities and differences in our experiences (Sughrua, 2019 ). To do so, we exchanged our reflective essays; we read each other’s reflections and added questions or comments on the margins. Then, we met online twice to share our experiences and exchange views regarding the two reflective essays we wrote. Both meetings lasted for approximately 90 min, were audio-recorded and transcribed. After each meeting, we coded our stories and experiences with reference to the two dimensions of the ecological framework of feedback literacy (Fig. 1 ). With regards to coding our data, we followed the model of Miles and Huberman ( 1994 ), which comprises four stages: data reduction (abstracting data), data display (visualising data in tabular form), conclusion-drawing, and verification. The coding and writing processes were done collaboratively on Google Docs and care was taken to address the aforesaid ethical (e.g., honesty, privacy) and methodological issues (e.g., validity, reliability, generalisability). As a CAE study, the participants are the researchers themselves, that is, the two authors of this paper. We acknowledge that research data are collected from human subjects (from the two authors), such data are collected in accordance with the standards and guidelines of the School Research Ethics Committee at the School of Social Sciences, Education and Social Work, Queen’s University Belfast (Ref: 005_2021). Despite our different experiences in our unique training and employment contexts, we share some common characteristics, both being ECRs (<5 years post-PhD), working in the field of education, active in the scholarly publication process as both authors and peer reviewers. Importantly for this study, we were both recipients of Reviewer of the Year Award 2019 awarded jointly by the journal, Higher Education Research & Development and the publisher , Taylor & Francis. This award in recognition of the quality of our reviewing efforts, as determined by the editorial board of a prestigious higher education journal, provided a strong impetus for this study, providing an opportunity to reflect on our own experiences and practices. The extent of our peer-review activities during our early career leading up to the time of data collection is summarised in Table 1 .

Findings and discussion

Analysis of the four individual essays (E1 and E2 for each participant) and transcripts of the two subsequent discussions (D1 and D2) resulted in the identification of multiple descriptive codes and in turn a number of overarching themes (Supplementary Appendix 1). Our reporting of these themes is guided by our conceptual framework, where we first focus on the five manifestations of feedback literacy to highlight the experiences that contribute to our growth as effective and confident peer reviewers. Then, we report on the five ecological systems to unravel how each contextual layer develops our feedback literacy as peer reviewers. (Note that the discussion of the chronosystem has been necessarily incorporated into each of the four others dimensions: microsystem , mesosystem , exosystem , and macrosystem in order to demonstrate temporal changes). In particular, similarities and differences will be underscored, and connections with manifested feedback beliefs and behaviours will be made. We include quotes from both Author 1 (A1) and Author 2 (A2), in order to illustrate our findings, and to show the richness and depth of the data collected (Corden and Sainsbury, 2006 ). Transcribed quotes may be lightly edited while retaining meaning, for example through the removal of fillers and repetitions, which is generally accepted practice to ensure readability ( ibid ).

Manifestations of feedback literacy

Engineering feedback uptake.

The two authors have a strong sense of the purpose of peer review as promoting not only research quality, but the growth of researchers. One way that we engineer author uptake is to ensure that feedback is ‘clear’ (A2,E1), ‘explicit’ (A2,E1), ‘specific’ (A1,E1), and importantly ‘actionable… to ensure that authors can act on this feedback so that their manuscripts can be improved and ultimately accepted for publication’ (A1,E1). In less than favourable author outcomes, we ensure that there is reference to the role of the feedback in promoting the development of the manuscript, which A1 refers to as ‘promotion of a growth mindset’ (A1,E1). For example, after requesting a second round of major revisions, A2 ‘acknowledged the frustration that the author might have felt on getting further revisions by noting how much improvement was made to the paper, but also making clear the justification for sending it off for more work’ (A2,E1). We both note that we tend to write longer reviews when a rejection is the recommended outcome, as our ultimate goal is to aid in the development of a manuscript.

Rejections doesn’t mean a paper is beyond repair. It can still be fixed and improved; a rejection simply means that the fix may be too extensive even for multiple review cycles. It is crucial to let the authors whose manuscripts are rejected know that they can still act on the feedback to improve their work; they should not give up on their own work. I think this message is especially important to first-time authors or early career researchers. (A1,E1)

In promoting a growth mindset and in providing actionable feedback, we hope to ‘show the authors that I’m not targeting them, but their work’ (A1,D1). We particularly draw on our own experiences as ECRs, with first-hand understanding that ‘everyone takes it personally when they get rejected. Yeah. Moreover, it is hard to separate (yourself from the paper)’ (A2,D1).

Navigating responsibilities

As with most academics, the two authors have multiple pressures on their time, and there ‘isn’t much formal recognition or reward’ (A1,E1) and ‘little extrinsic incentive for me to review’ (A2,E1). Nevertheless we both view our roles as peer reviewers as ‘an important part of the process’ (A2,E1), ‘a modest way for me to give back to the academic community’ (A1,E1). Through peer review we have built a sense of ‘identity as an academic’ (A1,D1), through ‘being a member of the academic community’ (A2,D1). While A1 commits to ‘review as many papers as possible’ (A1,E1) and A2 will usually accept offers to review, there are still limits on our time and therefore we consider the topic and methods employed when deciding whether or not to accept an invitation, as well as the journal itself, as we feel we can review more efficiently for journals with which we are more familiar. A1 and A2 have different processes for conducting their review that are most efficient for their own situations. For A1, the process begins with reading the whole manuscript in one go, adding notes to the pdf document along the way, which he then reviews, and makes a tentative decision, including ‘a few reasons why I have come to this decision’ (A1,E1). After waiting at least one day, he reviews all of the notes and begins writing the report, which is divided into the sections of the paper. He notes it ‘usually takes me 30–45 min to write a report. I then proofread this report and submit it to the system. So it usually takes me no more than three hours to complete a review’ (A1,E1). For A2, the process for reviewing and structuring the report is quite different, with a need to ‘just find small but regular opportunities to work on the review’ (A2,E1). As was the case during her Ph.D, which involved juggling research and raising two babies, ‘I’ve trained myself to be able to do things in bits’ (A2,D1). So A2 also begins by reading the paper once through, although generally without making initial comments. The next phase involves going through the paper at various points in time whenever possible, and at the same time building up the report, making the report structurally slightly different to that of A1.

What my reviews look like are bullet points, basically. And they’re not really in a particular order. They generally… follow the flow (of the paper). But I mean, I might think of something, looking at the methods and realise, hey, you haven’t defined this concept in the literature review so I’ll just add you haven’t done this. And so I will usually preface (the review)… Here’s a list of suggestions. Some of them are minor, some of them are serious, but they’re in no particular order. (A1,D1)

As such, both reviewers engage in personalised strategies to make more effective use of their time. Both A1 and A2 give explicit but not exhaustive examples of an area of concern, and they also pose questions for the author to consider, in both cases placing the onus back on the author to take action. As A1 notes, ‘I’m not going to do a summary of that reference for you. I’m just going to include that there. If you’d like you can check it out’ (A1,D1). For A2, a lack of adequate reporting of the methods employed in a study makes it difficult to proceed, and in such cases will not invest further time, sending it back to the editor, because ‘I can’t even comment on the findings… I can’t go on. I’m not gonna waste my time’ (A2,D1). In cases where the authors may be ‘on the fence’ about a particular review, they will use the confidential comments to the editor to help work through difficult cases as ‘they are obviously very experienced reviewers’ (A1,D1). Delegating tasks to the expertise of the editorial teams when appropriate also ensures time is used more prudently.

Maintaining relationships

Except in a few cases where A2 has reviewed for journals with a single-blind model, the vast majority of the reviews that we have completed have been double-blind. This means that we are unaware of the identity of the author/s, and we are unknown to them. However, ‘even with blind-reviews I tend to think of it as a conversation with a person’ (A2,E1). A1 talks about the need to have respect for the author and their expertise and effort ‘regardless of the quality of the submission (which can be in some cases subjective)’ (A1,E1). A2 writes similarly about the ‘privilege’ and ‘responsibility’ of being able to review manuscripts that authors ‘have put so much time and energy into possibly over an extended period’ (A2,E1). In this way it is possible to develop a sort of relationship with an author even without knowing their identity. In trying to articulate the nature of that relationship (which we struggle to do so definitively), we note that it is more than just a reviewer, and A2 reflected on a recent review, which went through a number of rounds of resubmission where ‘it felt like we were developing a relationship, more like a mentor than a reviewer’ (A2,E1).

I consider this role as a peer reviewer more than giving helpful and actionable feedback; I would like to be a supporter and critical friend to the authors, even though in most cases I don’t even know who they are or what career stage they are at (A1,E1).

In any case, as A1 notes, ‘we don’t even need to know who that person is because we know that people like encouragement’ (A1,D1), and we are very conscious of the emotional impact that feedback can have on authors, and the inherent power imbalance in the relationship. For this reason, A1 is ‘cautious about the way I write so that I don’t accidentally make the authors the target of my feedback’. As A2 notes ‘I don’t want authors feeling depressed after reading a review’ (A2,E1). While we note that we try to deliver our feedback with ‘respect’ (A1,E1; A1,E2; A2,D1) ‘empathy’ (A1,E1), and ‘kindness’ (A2,D1), we both noted that we do not ‘sugar coat’ our feedback and A1 describes himself as ‘harsh’ and ‘critical’ (A1,E1) while A2 describes herself as ‘pretty direct’ (A2,E1). In our discussion, we tried to delve into this seeming contradiction:… the encouragement, hopefully is to the researcher, but the directness it should be, I hope, is related directly to whatever it is, the methods or the reporting or the scope of the literature review. It’s something specific about the manuscript itself. And I know myself, being an ECR and being reviewed, that it’s hard to separate yourself from your work… And I want to make it really explicit. If it’s critical, it’s not about the person. It’s about the work, you know, the weakness of the work, but not the person. (A2,D1)

A1 explains that at times his initial report may be highly critical, and at times he will ‘sit back and rethink… With empathy, I will write feedback, which is more constructive’ (A1,E1). However, he adds that ‘I will never try to overrate a piece or sugar-coat my comments just to sound “friendly”’ (A1,E1), with the ultimate goal being to uphold academic rigour. Thus, honesty is seen as the best strategy to maintain a strong, professional relationship with reviewers. Another strategy employed by A2 is showing explicit commitment to the review process. One way this is communicated is by prefacing a review with a summary of the paper, not only ‘to confirm with the author that I am interpreting the findings in the way that they intended, but also importantly to show that I have engaged with the paper’ (A2,E1). Further, if the recommendation is for a further round of review, she will state directly to the authors ‘that I would be happy to review a revised manuscript’ (A2,E1).

Reflecting on feedback experience

As ECRs we have engaged in the scholarly publishing process initially as authors, subsequently as reviewers, and most recently as Associate Editors. Insights gained in each of these roles have influenced our feedback practices, and have interacted to ‘develop a more holistic understanding of the whole review process’ (A1,E1).

We reflect on our experiences as authors beginning in our doctoral candidatures, with reviews that ranged from ‘the most helpful to the most cynical’ (A1,E1). A2 reflected on two particular experiences both of which resulted in rejection, one being ‘snarky’ and ‘unprofessional’ with ‘no substance’, the other providing ‘strong encouragement … the focus was clearly on the paper and not me personally’ (A2,E1). It was this experience that showed the divergence between the tone and content of review despite the same outcome, and as result A2 committed to being ‘ the amazing one’. A1 also drew from a negative experience noting that ‘I remember the least useful feedback as much as I do with the most constructive one’ (A1,E1). This was particularly the case when a reviewer made apparently politically-motivated judgements that A1 ‘felt very uncomfortable with’ and flagged with the editor (A1,E1). Through these experiences both authors wrote in their essays about the need to focus on the work and not on the individual, with an understanding that a review ‘can have a really serious impact’ (A2,D1) on an author.

It is important to note that neither authors have been involved in any formal or informal training on how to conduct peer review, although A1 expresses appreciation of the regular practice of one journal for which he reviews, where ‘the editor would write an email to the reviewers giving feedback on the feedback we have given’ (A1,E1). For A2, an important source of learning is in comparing her reviews with that of others who have reviewed the same manuscript, the norm for some journals being to send all reports to all reviewers along with the final decision.

I’m always interested to see how [my] review compares with others. Have I given the same recommendation? Have I identified the same areas of weakness? Have I formatted my review in the same way? How does the tone of delivery differ? I generally find that I give a similar if not the same response to other reviews, and I’m happy to see that I often pick up the same issues with methodology. (A2,E1)

For A2 there is comfort in seeing reviews that are similar to others, although we both draw on experiences where our recommendation diverged from others, with a source of assurance being the ultimate decision of the editor.

So it’s like, I don’t think it can be published and that [other] reviewer thinks it’s excellent. So usually, what the editor would do in this instance is invite the third one. Right, yeah. But then this editor told me… that they decided to go with my decision to reject because they find that my comments are more convincing. (A1,D1)

A2 also was surprised to read another report of the same manuscript she reviewed, that raised similar concerns and gave the same recommendation for major revisions, but noted the ‘wording is soooo snarky. What need?’ (A2,E1). In one case that A1 detailed in our first discussion, significant but improbable changes made to the methodology section of a resubmitted paper caused him to question the honesty of the reporting, making him ‘uncomfortable’ and as a result reported his concerns to the editor. In this case the review took some time to craft, trying to balance the ‘fine line between catering for the emotion [of the author], right, and upholding the academic standards’ (A1,D1). While he conceded initially his report was ‘kind of too harsh… later I think I rephrased it a little bit, I kind of softened (it)’.

While the role of Associate Editor is very new to A2 and thus was yet unable to comment, for A1 the ‘opportunity to read various kinds of comments given by reviewers’ (A1,E1) is viewed favourably. This includes not only how reviewers structure their feedback, but also how they use the confidential comments to the editors to express their thoughts more openly, providing important insights into the process that are largely hidden.

Understanding standards

While our reviewing practices are informed more broadly ‘according to more general academic standards of the study itself, and the clarity and fullness of the reporting’ (A2,E1), we look in the first instance to advice and guidelines from journals to develop an understanding of journal-specific standards, although A2 notes that a lack of review guidelines for one of the earliest journals she reviewed led her to ‘searching Google for standard criteria’ (A2,E1). However, our development in this area seems to come from developing a familiarity with a journal, particularly through engagement with the journal as an author.

In addition to reading the scope and instructions for authors to obtain such basic information as readership, length of submissions, citation style, the best way for me to understand the requirements and preferences of the journals is my own experience as an author. I review for journals which I have published in and for those which I have not. I always find it easier to make a judgement about whether the manuscripts I review meet the journal’s standards if I have published there before. (A1,E1)

Indeed, it seems that journal familiarity is connected closely to our confidence in reviewing, and while both authors ‘review for journals which I have published in and for those which I have not’ (A1,E1), A2 states that she is reluctant to ‘readily accept an offer to review for a journal that I’m not familiar with’, and A1 takes extra time to ‘do more preparatory work before I begin reading the manuscript and writing the review’ when reviewing for an unfamiliar journal.

Ecological systems

Microsystem.

Three microsystems exert influence on A1’s and A2’s development of feedback literacy: university, journal community, and Twitter.

In regards to the university, we are full-time academics in research-intensive universities in the UK and Japan where expectations for academics include publishing research in high-impact journals ‘which is vital to promotion’ (A1,E2). It is especially true in A2’s context where the national higher education agenda is to increase world rankings of universities. Thus, ‘there is little value placed on peer review, as it is not directly related to the broader agenda’ (A2,E2). When considering his recent relocation to the UK together with the current pandemic, A1 navigated his responsibilities within the university context and decided to allocate more time to his university-related responsibilities, especially providing learning and pastoral support to his students, who are mostly international students. Besides, A2 observed that there is a dearth of institution-wide support on conducting peer review although ‘there are a lot of training opportunities related to how to write academic papers in English, how to present at international conferences, how to write grant applications’, etc. (A2,E2). As a result, she ‘struggled for a couple of years’ because of the lack of institutional support for her development as a peer reviewer’ (A2,D2); but this helplessness also motivated her to seek her own ways to learn how to give feedback, such as ‘seeing through glimpses of other reviews, how others approach it, in terms of length, structure, tone, foci etc.’ (A2,E2). A1 shares the same view that no training is available at his institution to support his development as a peer reviewer. However, his postgraduate supervision experiences enabled him to reflect on how his feedback can benefit researchers. In our second online discussion, A1 shared that he held individual advising sessions with some postgraduate students, which made him realise that it is important for feedback to serve the function to inspire rather than to ‘give them right answers’ (A1,D2).

Because of the lack of formal training provided by universities, both authors searched for other professional communities to help us develop our expertise in giving feedback as peer reviewers, with journal communities being the next microsystem. We found that international journals provide valuable opportunities for us to understand more about the whole peer-review process, in particular the role of feedback. For A1, the training which he received from the editor-in-chief when he took up the associate editorship of a language education journal two years ago was particularly useful. A1 benefited greatly from meetings with the editor who walked him through every stage in the review process and provided ‘hands-on experience on how to handle delicate scenarios’ (A1,E2). Since then, A1 has had plenty of opportunities to oversee various stages of peer review and read a large number of reviewers’ reports which helped him gain ‘a holistic understanding of the peer-review process’ (A1,E2) and gradually made him become more cognizant of how he wants to give feedback. Although there was no explicit instruction on the technical aspect of giving feedback, A1 found that being an associate editor has developed his ‘consciousness’ and ‘awareness’ of giving feedback as a peer reviewer (A1,D2). Further, he felt that his editorial experiences provided him the awareness to constantly refine and improve his ways of giving feedback, especially ways to make his feedback ‘more structured, evidence-based, and objective’ (A1,E2). Despite not reflecting from the perspective of an editor, A2 recalled her experience as an author who received in-depth and constructive feedback from a reviewer, which really impacted the way she viewed the whole review process. She understood from this experience that even though the paper under review may not be particularly strong, peer reviewers should always aim to provide formative feedback which helps the authors to improve their work. These positive experiences of the two authors are impactful on the ways they give feedback as peer reviewers. In addition, close engagement with a specific journal has helped A2 to develop a sense of belonging, making it ‘much more than a journal, but also a way to become part of an academic community’ (A2,E2). With such a sense of belonging, it is more likely for her to be ‘pulled towards that journal than others’ when she can only review a limited number of manuscripts (A2,D2).

Another professional community in which we are both involved is Twitter. We regard Twitter as a platform for self-learning, reflection, and inspiration. We perceive Twitter as a space where we get to learn from others’ peer-review experiences and disciplinary practices. For example, A1 found the tweets on peer-review informative ‘because they are written by different stakeholders in the process—the authors, editors, reviewers’ and offer ‘different perspectives and sometimes different versions of the same story’ (A1,E2). A2 recalled a tweet she came across about the ‘infamous Reviewer 2’ and how she learned to not make the same mistakes (A2,D2). Reading other people’s experiences helps us reconsider our own feedback practices and, more broadly, the whole peer-review system because we ‘get a glimpse of the do’s and don’ts for peer reviewers’ (A1,E2).

Further to our three common microsystems, A2 also draws on a unique microsystem, that of her former profession as a teacher, which shapes her feedback practices in three ways. First, in her four years of teacher training, a lot of emphasis was placed on assessment and feedback such as ‘error correction’; this understanding related to giving feedback to students and was solidified through ‘learning on the job’ (A2,D2). Second, A2 acknowledges that as a teacher, she has a passion to ‘guide others in their knowledge and skill development… and continue this in our review practices’ (A2,E2). Finally, her teaching experience prepared her to consider the authors’ emotional responses in her peer-review feedback practices, constantly ‘thinking there’s a person there who’s going to be shattered getting a rejection’ (A2,D2).

Mesosystem considers the confluence of our interactions in various microsystems. Particularly, we experienced a lack of support from our institutions, which pushed us to seek alternative paths to acquire the art of giving feedback. This has made us realise the importance of self-learning in developing feedback literacy as peer reviewers, especially in how to develop constructive and actionable feedback. Both authors self-learn how to give feedback by reading others’ feedback. A1 felt ‘fortunate to be involved in journal editing and Twitter’ because he gets ‘a glimpse of how other peer reviewers give feedback to authors’ (A1,E2). A2, on the other hand, learned through her correspondences with a journal editor who made her stop ‘looking for every word’ and move away from ‘over proofreading and over editing’ (A2,D2).

Focusing on the chronosystem, it is noticed that both authors adjusted how they give feedback over time because of the aggregated influence of their microsystems. What stands out is that they have become more strategic in giving feedback. One way this is achieved is through focusing their comments on the arguments of the manuscripts instead of burning the midnight oil with error-correcting.

Exosystem concerns the environment where the focal individuals do not have direct interactions with the people in it but have access to information about. In his case, A1’s understanding of advising techniques promoted by a self-access language learning centre is conducive to the cultivation of his feedback literacy. Although A1 is not a part of the language advising team, he has a working relationship with the director. A1 was especially impressed by the learner-centeredness of an advising process:

The primary duty of the language advisor is not to be confused with that of a language teacher. Language teachers may teach a lecture on a linguistic feature or correct errors on an essay, but language advisors focus on designing activities and engaging students in dialogues to help them reflect on their own learning needs… The advisors may also suggest useful resources to the students which cater to their needs. In short, language advisors work in partnership with the students to help them improve their language while language teachers are often perceived as more authoritative figures (A1, E2).

His understanding of advising has affected how A1 provides feedback as a peer reviewer in a number of ways. First, A1 places much more emphasis on humanising his feedback, for example, by considering ‘ways to work in partnership with the authors and making this “partnership mindset” explicit to the authors through writing’ (A1,E2). One way to operationalise this ‘partnership mindset’ in peer review is to ‘ask a lot of questions’ and provide ‘multiple suggestions’ for the authors to choose from (A1,E2). Furthermore, his knowledge of the difference between feedback as giving advice and feedback as instruction has led him to include feedback, which points authors to additional resources. Below is a feedback point A1 gave in one of his reviews:

The description of the data analysis process was very brief. While we are not aiming at validity and reliability in qualitative studies, it is important for qualitative researchers to describe in detail how the data collected were analysed (e.g. iterative coding, inductive/deductive coding, thematic analysis) in order to ascertain that the findings were credible and trustworthy. See Johnny Saldaña’s ‘The Coding Manual for Qualitative Researchers’.

Another exosystem that we have knowledge about is formal peer-review training courses provided by publishers. These online courses are usually run asynchronously. Even though we did not enrol in these courses, our interest in peer review has led us to skim the content of these courses. Both of us questioned the value of formal peer-review training in developing feedback literacy of peer reviewers. For example, A2 felt that opportunities to review are more important because they ‘put you in that position where you have responsibility and have to think critically about how you are going to respond’ (A2,D2). To A1, formal peer-review training mostly focuses on developing peer reviewers’ ‘understanding of the whole mechanism’ but not providing ‘training on how to give feedback… For example, do you always ask a question without giving the answers you know? What is a good suggestion?’ (A1,D2).

Macrosystem

The two authors have diverse sociocultural experiences because of their family backgrounds and work contexts. When reflecting on their sociocultural experiences, A1 focused on his upbringing in Hong Kong where both of his parents are school teachers and his professional experience as a language teacher in secondary and tertiary education in Hong Kong while A2 discussed her experience of working in academia in Japan as an anglophone.

Observing his parents’ interactions with their students in schools, A1 was immersed in an Asian educational discourse characterised by ‘mutual respect and all sorts of formality’ (A1,E2). After he finished university, A1 became a school teacher and then a university lecturer (equivalent to a teaching fellow in the UK), getting immersed continuously in the etiquette of educational discourse in Hong Kong. Because of this, A1 knows that being professional means to be ‘formal and objective’ and there is a constant expectation to ‘treat people with respect’ (A1,E2). At the same time, his parents are unlike typical Asian parents; they are ‘more open-minded’, which made him more willing to listen and ‘consider different perspectives’ (A1,D2). Additionally, social hierarchy also impacted his approach to giving feedback as a peer reviewer. A1 started his career as a school teacher and then a university lecturer in Hong Kong with no formal research training. After obtaining his BA and MA, it is not until recently that A1 obtained his PhD by Prior Publication. Perhaps because of his background as a frontline teacher, A1 did not regard himself as ‘a formally trained researcher’ and perceived himself as not ‘elite enough to give feedback to other researchers’ (A1,E2). Both his childhood and his self-perceived identity have led to the formation of two feedback strategies: asking questions and providing a structured report mimicking the sections in the manuscript. A1 frequently asks questions in his reports ‘in a bid to offset some of the responsibilities to the authors’ (A1,E2). A1 struggles to decide whether to address authors using second- or third-person pronouns. A1 consistently uses third-person pronouns in his feedback because he wants to sound ‘very formal’ (A1,D2). However, A1 shared that he has recently started using second-person pronouns to make his feedback more interactive.

A2, on the other hand, pondered upon her sociocultural experiences as a school teacher in Australia, her position as an anglophone in a Japanese university, and her status as first-generation high school graduate. Reflecting on her career as a school teacher, A2 shared that her students had high expectations on her feedback:

So if you give feedback that seems unfair, you know … they’ll turn around and say, ‘What are you talking about’? They’re going to react back if your feedback is not clear. I think a lot of them [the students] appreciate the honesty. (A2,D2)

A2 acknowledges that her identity as a native English speaker has given her the advantage to publish extensively in international journals because of her high level of English proficiency and her access to ‘data from the US and from Australia which are more marketable’ (A2,D2). At the same time, as a native English speaker, she has empathy for her Japanese colleagues who struggle to write proficiently in English and some who even ‘pay thousands of dollars to have their work translated’ (A2,D2). Therefore, when giving feedback as a peer reviewer, she tries not to make a judgement on an author’s English proficiency and will not reject a paper based on the standard of English alone. Finally, as a first-generation scholar without any previous connections to academia, she struggles with belonging and self-confidence. As a result she notes that it usually takes her a long time to complete a review because she would like to be sure what she is saying is ‘right or constructive and is not on the wrong track’ (A2,D2).

Implications and future directions

In investigating the manifestations of the authors’ feedback literacy development, and the ecological systems in which this development occurs, this study unpacks the various sources of influence behind our feedback behaviours as two relatively new but highly commended peer reviewers. The findings show that our feedback literacy development is highly personalised and contextualised, and the sources of influence are diverse and interconnected, albeit largely informal. Our peer-review practices are influenced by our experiences within academia, but influences are much broader and begin much earlier. Peer-review skills were enhanced through direct experience not only in peer review but also in other activities related to the peer-review process, and as such more hands-on, on-site feedback training for peer reviewers may be more appropriate than knowledge-based training. The authors gain valuable insights from seeing the reviews of others, and as this is often not possible until scholars take on more senior roles within journals, co-reviewing is a potential way for ECRs to gain experience (McDowell et al., 2019 ). We draw practical and moral support from various communities, particularly online to promote “intellectual candour”, which refers to honest expressions of vulnerability for learning and trust building (Molloy and Bearman, 2019 , p. 32); in response to this finding we have developed an online community of practice, specifically as a space for discussing issues related to peer review (a Twitter account called “Scholarly Peers”). Importantly, our review practices are a product not only of how we review, but why we review, and as such training should not focus solely on the mechanics of review, but extend to its role within academia, and its impact not only on the quality of scholarship, but on the growth of researchers.

The significance of this study is its insider perspective, and the multifaceted framework that allows the capturing of the complexity of factors that influence individual feedback literacy development of two recognised peer reviewers. It must be stressed that the findings of this study are highly idiosyncratic, focusing on the experiences of only two peer reviewers and the educational research discipline. While the research design is such that it is not an attempt to describe a ‘typical’ or ‘expected’ experience, the scope of the study is a limitation, and future research could be expanded to studies of larger cohorts in order to identify broader trends. In this study, we have not included the reviewer reports themselves, and these reports provide a potentially rich source of data, which will be a focus in our continued investigation in this area. Further research could also investigate the role that peer-review training courses play in the feedback literacy development and practices of new and experienced peer reviewers. Since journal peer review is a communication process, it is equally important to investigate authors’ perspectives and experiences, especially pertaining to how authors interpret reviewers’ feedback based on the ways that it is written.

Data availability

Because of the sensitive nature of the data these are not made available.

Change history

26 november 2021.

A Correction to this paper has been published: https://doi.org/10.1057/s41599-021-00996-3

Abedi Asante L, Abubakari Z (2020) Pursuing PhD by publication in geography: a collaborative autoethnography of two African doctoral researchers. J Geogr High Educ 45(1):87–107. https://doi.org/10.1080/03098265.2020.1803817

Article   Google Scholar  

Boud D, Dawson P (2021). What feedback literate teachers do: An empirically-derived competency framework. Assess Eval High Educ. Advanced online publication. https://doi.org/10.1080/02602938.2021.1910928

Bronfenbrenner U (1986) Ecology of the family as a context for human development. Res Perspect Dev Psychol 22:723–742. https://doi.org/10.1037/0012-1649.22.6.723

Carless D, Boud D (2018) The development of student feedback literacy: enabling uptake of feedback. Assess Eval High Educ 43(8):1315–1325. https://doi.org/10.1080/02602938.2018.1463354

Carless D, Winstone N (2020) Teacher feedback literacy and its interplay with student feedback literacy. Teach High Educ, 1–14. https://doi.org/10.1080/13562517.2020.1782372

Chang H, Ngunjiri FW, Hernandez KC (2013) Collaborative autoethnography. Left Coast Press

Cheung D (2000) Measuring teachers’ meta-orientations to curriculum: application of hierarchical confirmatory factor analysis. The J Exp Educ 68(2):149–165. https://doi.org/10.1080/00220970009598500

Chong SW (2021a) Improving peer-review by developing peer reviewers’ feedback literacy. Learn Publ 34(3):461–467. https://doi.org/10.1002/leap.1378

Chong SW (2021b) Reconsidering student feedback literacy from an ecological perspective. Assess Eval High Educ 46(1):92–104. https://doi.org/10.1080/02602938.2020.1730765

Chong SW (2019) College students’ perception of e-feedback: a grounded theory perspective. Assess Eval High Educ 44(7):1090–1105. https://doi.org/10.1080/02602938.2019.1572067

Chong SW (2018) Interpersonal aspect of written feedback: a community college students’ perspective. Res Post-Compul Educ 23(4):499–519. https://doi.org/10.1080/13596748.2018.1526906

Corden A, Sainsbury R (2006) Using verbatim quotations in reporting qualitative social research: the views of research users. University of York Social Policy Research Unit

Ellis C, Adams TE, Bochner AP (2011) Autoethnography: An Overview. Historical Soc Res, 12:273–290

Ellis C, Bochner A (1996) Composing ethnography: Alternative forms of qualitative writing. Sage

Freda MC, Kearney MH, Baggs JG, Broome ME, Dougherty M (2009) Peer reviewer training and editor support: results from an international survey of nursing peer reviewers. J Profession Nurs 25(2):101–108. https://doi.org/10.1016/j.profnurs.2008.08.007

Fulcher G (2012) Assessment literacy for the language classroom. Lang Assess Quart 9(2):113–132. https://doi.org/10.1080/15434303.2011.642041

Gee JP (1999) Reading and the new literacy studies: reframing the national academy of sciences report on reading. J Liter Res 3(3):355–374. https://doi.org/10.1080/10862969909548052

Gravett K, Kinchin IM, Winstone NE, Balloo K, Heron M, Hosein A, Lygo-Baker S, Medland E (2019) The development of academics’ feedback literacy: experiences of learning from critical feedback via scholarly peer review. Assess Eval High Educ 45(5):651–665. https://doi.org/10.1080/02602938.2019.1686749

Hains-Wesson R, Young K (2016) A collaborative autoethnography study to inform the teaching of reflective practice in STEM. High Educ Res Dev 36(2):297–310. https://doi.org/10.1080/07294360.2016.1196653

Han Y, Xu Y (2019) Student feedback literacy and engagement with feedback: a case study of Chinese undergraduate students. Teach High Educ, https://doi.org/10.1080/13562517.2019.1648410

Heesen R, Bright LK (2020) Is Peer Review a Good Idea? Br J Philos Sci, https://doi.org/10.1093/bjps/axz029

Hollywood A, McCarthy D, Spencely C, Winstone N (2019) ‘Overwhelmed at first’: the experience of career development in early career academics. J Furth High Educ 44(7):998–1012. https://doi.org/10.1080/0309877X.2019.1636213

Horn SA (2016) The social and psychological costs of peer review: stress and coping with manuscript rejection. J Manage Inquiry 25(1):11–26. https://doi.org/10.1177/1056492615586597

Hughes S, Pennington JL, Makris S (2012) Translating Autoethnography Across the AERA Standards: Toward Understanding Autoethnographic Scholarship as Empirical Research. Educ Researcher, 41(6):209–219

Kandiko CB(2010) Neoliberalism in higher education: a comparative approach. Int J Art Sci 3(14):153–175. http://www.openaccesslibrary.org/images/BGS220_Camille_B._Kandiko.pdf

Keashly L, Neuman JH (2010) Faculty experiences with bullying in higher education-causes, consequences, and management. Adm Theory Prax 32(1):48–70. https://doi.org/10.2753/ATP1084-1806320103

Kelly J, Sadegieh T, Adeli K (2014) Peer review in scientific publications: benefits, critiques, & a survival guide. J Int Fed Clin Chem Labor Med 25(3):227–243. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4975196/

Google Scholar  

Kumar KL (2020) Understanding and expressing academic identity through systematic autoethnography. High Educ Res Dev, https://doi.org/10.1080/07294360.2020.1799950

Lapadat JC (2017) Ethics in autoethnography and collaborative autoethnography. Qual Inquiry 23(8):589–603. https://doi.org/10.1177/1077800417704462

Levi T, Inbar-Lourie O (2019) Assessment literacy or language assessment literacy: learning from the teachers. Lang Assess Quarter 17(2):168–182. https://doi.org/10.1080/15434303.2019.1692347

London MS, Smither JW (2002) Feedback orientation, feedback culture, and the longitudinal performance management process. Hum Res Manage Rev 12(1):81–100. https://doi.org/10.1016/S1053-4822(01)00043-2

Malecka B, Boud D, Carless D (2020) Eliciting, processing and enacting feedback: mechanisms for embedding student feedback literacy within the curriculum. Teach High Educ, 1–15. https://doi.org/10.1080/13562517.2020.1754784

Mavrogenis AF, Quaile A, Scarlat MM (2020) The good, the bad and the rude peer-review. Int Orthopaed 44(3):413–415. https://doi.org/10.1007/s00264-020-04504-1

McDowell GS, Knutsen JD, Graham JM, Oelker SK, Lijek RS (2019) Co-reviewing and ghostwriting by early-career researchers in the peer review of manuscripts. ELife 8:e48425. https://doi.org/10.7554/eLife.48425

Article   CAS   PubMed   PubMed Central   Google Scholar  

Merga MK, Mason S, Morris JE (2018) Early career experiences of navigating journal article publication: lessons learned using an autoethnographic approach. Learn Publ 31(4):381–389. https://doi.org/10.1002/leap.1192

Miles MB, Huberman AM (1994) Qualitative data analysis: An expanded sourcebook (2nd edn.). Sage

Molloy E, Bearman M (2019) Embracing the tension between vulnerability and credibility: ‘Intellectual candour’ in health professions education. Med Educ 53(1):32–41. https://doi.org/10.1111/medu.13649

Article   PubMed   Google Scholar  

Molloy E, Boud D, Henderson M (2019) Developing a learning-centred framework for feedback literacy. Assess Eval High Educ 45(4):527–540. https://doi.org/10.1080/02602938.2019.1667955

Neal JW, Neal ZP (2013) Nested or networked? Future directions for ecological systems theory. Soc Dev 22(4):722–737. https://doi.org/10.1111/sode.12018

Noble C, Billett S, Armit L, Collier L, Hilder J, Sly C, Molloy E (2020) “It’s yours to take”: generating learner feedback literacy in the workplace. Adv Health Sci Educ Theory Pract 25(1):55–74. https://doi.org/10.1007/s10459-019-09905-5

Price M, Rust C, O’Donovan B, Handley K, Bryant R (2012) Assessment literacy: the foundation for improving student learning. Oxford Centre for Staff and Learning Development

Silbiger NJ, Stubler AD (2019) Unprofessional peer reviews disproportionately harm underrepresented groups in STEM. PeerJ 7:e8247. https://doi.org/10.7717/peerj.8247

Article   PubMed   PubMed Central   Google Scholar  

Starck JM (2017) Scientific peer review: guidelines for informative peer review. Springer Spektrum

Steelman LA, Wolfeld L (2016) The manager as coach: the role of feedback orientation. J Busi Psychol 33(1):41–53. https://doi.org/10.1007/s10869-016-9473-6

Stiggins RJ (1999) Evaluating classroom assessment training in teacher education programs. Educ Meas: Issue Pract 18(1):23–27. https://doi.org/10.1111/j.1745-3992.1999.tb00004.x

Street B (1997) The implications of the ‘new literacy studies’ for literacy Education. Engl Educ 31(3):45–59. https://doi.org/10.1111/j.1754-8845.1997.tb00133.x

Sughrua WM (2019) A nomenclature for critical autoethnography in the arena of disciplinary atomization. Cult Stud Crit Methodol 19(6):429–465. https://doi.org/10.1177/1532708619863459

Sutton P (2012) Conceptualizing feedback literacy: knowing, being, and acting. Innov Educ Teach Int 49(1):31–40. https://doi.org/10.1080/14703297.2012.647781

Article   MathSciNet   Google Scholar  

Tynan BR, Garbett DL (2007) Negotiating the university research culture: collaborative voices of new academics. High Educ Res Dev 26(4):411–424. https://doi.org/10.1080/07294360701658617

Vygotsky LS (1978) Mind in society: The development of higher psychological processes. Harvard University Press

Wall S (2006) An autoethnography on learning about autoethnography. Int J Qual Methods 5(2):146–160. https://doi.org/10.1177/160940690600500205

Article   ADS   MathSciNet   Google Scholar  

Warne V (2016) Rewarding reviewers-sense or sensibility? A Wiley study explained. Learn Publ 29:41–40. https://doi.org/10.1002/leap.1002

Wilkinson S (2019) The story of Samantha: the teaching performances and inauthenticities of an early career human geography lecturer. High Educ Res Dev 38(2):398–410. https://doi.org/10.1080/07294360.2018.1517731

Winstone N, Carless D (2019) Designing effective feedback processes in higher education: a learning-focused approach. Routledge

Winstone NE, Mathlin G, Nash RA (2019) Building feedback literacy: students’ perceptions of the developing engagement with feedback toolkit. Front Educ 4:1–11. https://doi.org/10.3389/feduc.2019.00039

Xu Y, Carless D (2016) ‘Only true friends could be cruelly honest’: cognitive scaffolding and social-affective support in teacher feedback literacy. Assess Eval High Educ 42(7):1082–1094. https://doi.org/10.1080/02602938.2016.1226759

Download references

Author information

Authors and affiliations.

Queen’s University Belfast, Belfast, UK

Sin Wang Chong

Nagasaki University, Nagasaki, Japan

Shannon Mason

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Sin Wang Chong .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Ethical approval

We acknowledge that research data are collected from human subjects (from the two authors), such data are collected in accordance with the standards and guidelines of the School Research Ethics Committee at the School of Social Sciences, Education and Social Work, Queen’s University Belfast (Ref: 005_2021).

Informed consent

Since the participants are the two authors, there is no informed consent form.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplemental material file #1, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Chong, S.W., Mason, S. Demystifying the process of scholarly peer-review: an autoethnographic investigation of feedback literacy of two award-winning peer reviewers. Humanit Soc Sci Commun 8 , 266 (2021). https://doi.org/10.1057/s41599-021-00951-2

Download citation

Received : 02 August 2021

Accepted : 12 October 2021

Published : 12 November 2021

DOI : https://doi.org/10.1057/s41599-021-00951-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

• Science Notes Posts
• Contact Science Notes
• Todd Helmenstine Biography
• Anne Helmenstine Biography
• Free Printable Periodic Tables (PDF and PNG)
• Periodic Table Wallpapers
• Interactive Periodic Table
• Periodic Table Posters
• How to Grow Crystals
• Chemistry Projects
• Fire and Flames Projects
• Holiday Science
• Chemistry Problems With Answers
• Physics Problems
• Unit Conversion Example Problems
• Chemistry Worksheets
• Biology Worksheets
• Periodic Table Worksheets
• Physical Science Worksheets
• Science Lab Worksheets
• My Amazon Books

Understanding Peer Review in Science

Peer review is an essential element of the scientific publishing process that helps ensure that research articles are evaluated, critiqued, and improved before release into the academic community. Take a look at the significance of peer review in scientific publications, the typical steps of the process, and and how to approach peer review if you are asked to assess a manuscript.

What Is Peer Review?

Peer review is the evaluation of work by peers, who are people with comparable experience and competency. Peers assess each others’ work in educational settings, in professional settings, and in the publishing world. The goal of peer review is improving quality, defining and maintaining standards, and helping people learn from one another.

In the context of scientific publication, peer review helps editors determine which submissions merit publication and improves the quality of manuscripts prior to their final release.

Types of Peer Review for Manuscripts

There are three main types of peer review:

• Single-blind review: The reviewers know the identities of the authors, but the authors do not know the identities of the reviewers.
• Double-blind review: Both the authors and reviewers remain anonymous to each other.
• Open peer review: The identities of both the authors and reviewers are disclosed, promoting transparency and collaboration.

There are advantages and disadvantages of each method. Anonymous reviews reduce bias but reduce collaboration, while open reviews are more transparent, but increase bias.

Key Elements of Peer Review

Proper selection of a peer group improves the outcome of the process:

• Expertise : Reviewers should possess adequate knowledge and experience in the relevant field to provide constructive feedback.
• Objectivity : Reviewers assess the manuscript impartially and without personal bias.
• Confidentiality : The peer review process maintains confidentiality to protect intellectual property and encourage honest feedback.
• Timeliness : Reviewers provide feedback within a reasonable timeframe to ensure timely publication.

Steps of the Peer Review Process

The typical peer review process for scientific publications involves the following steps:

• Submission : Authors submit their manuscript to a journal that aligns with their research topic.
• Editorial assessment : The journal editor examines the manuscript and determines whether or not it is suitable for publication. If it is not, the manuscript is rejected.
• Peer review : If it is suitable, the editor sends the article to peer reviewers who are experts in the relevant field.
• Reviewer feedback : Reviewers provide feedback, critique, and suggestions for improvement.
• Revision and resubmission : Authors address the feedback and make necessary revisions before resubmitting the manuscript.
• Final decision : The editor makes a final decision on whether to accept or reject the manuscript based on the revised version and reviewer comments.
• Publication : If accepted, the manuscript undergoes copyediting and formatting before being published in the journal.

Pros and Cons

While the goal of peer review is improving the quality of published research, the process isn’t without its drawbacks.

• Quality assurance : Peer review helps ensure the quality and reliability of published research.
• Error detection : The process identifies errors and flaws that the authors may have overlooked.
• Credibility : The scientific community generally considers peer-reviewed articles to be more credible.
• Professional development : Reviewers can learn from the work of others and enhance their own knowledge and understanding.
• Time-consuming : The peer review process can be lengthy, delaying the publication of potentially valuable research.
• Bias : Personal biases of reviews impact their evaluation of the manuscript.
• Inconsistency : Different reviewers may provide conflicting feedback, making it challenging for authors to address all concerns.
• Limited effectiveness : Peer review does not always detect significant errors or misconduct.
• Poaching : Some reviewers take an idea from a submission and gain publication before the authors of the original research.

Steps for Conducting Peer Review of an Article

Generally, an editor provides guidance when you are asked to provide peer review of a manuscript. Here are typical steps of the process.

• Accept the right assignment: Accept invitations to review articles that align with your area of expertise to ensure you can provide well-informed feedback.
• Manage your time: Allocate sufficient time to thoroughly read and evaluate the manuscript, while adhering to the journal’s deadline for providing feedback.
• Read the manuscript multiple times: First, read the manuscript for an overall understanding of the research. Then, read it more closely to assess the details, methodology, results, and conclusions.
• Evaluate the structure and organization: Check if the manuscript follows the journal’s guidelines and is structured logically, with clear headings, subheadings, and a coherent flow of information.
• Assess the quality of the research: Evaluate the research question, study design, methodology, data collection, analysis, and interpretation. Consider whether the methods are appropriate, the results are valid, and the conclusions are supported by the data.
• Examine the originality and relevance: Determine if the research offers new insights, builds on existing knowledge, and is relevant to the field.
• Check for clarity and consistency: Review the manuscript for clarity of writing, consistent terminology, and proper formatting of figures, tables, and references.
• Identify ethical issues: Look for potential ethical concerns, such as plagiarism, data fabrication, or conflicts of interest.
• Provide constructive feedback: Offer specific, actionable, and objective suggestions for improvement, highlighting both the strengths and weaknesses of the manuscript. Don’t be mean.
• Organize your review: Structure your review with an overview of your evaluation, followed by detailed comments and suggestions organized by section (e.g., introduction, methods, results, discussion, and conclusion).
• Be professional and respectful: Maintain a respectful tone in your feedback, avoiding personal criticism or derogatory language.
• Proofread your review: Before submitting your review, proofread it for typos, grammar, and clarity.
• Couzin-Frankel J (September 2013). “Biomedical publishing. Secretive and subjective, peer review proves resistant to study”. Science . 341 (6152): 1331. doi: 10.1126/science.341.6152.1331
• Lee, Carole J.; Sugimoto, Cassidy R.; Zhang, Guo; Cronin, Blaise (2013). “Bias in peer review”. Journal of the American Society for Information Science and Technology. 64 (1): 2–17. doi: 10.1002/asi.22784
• Slavov, Nikolai (2015). “Making the most of peer review”. eLife . 4: e12708. doi: 10.7554/eLife.12708
• Spier, Ray (2002). “The history of the peer-review process”. Trends in Biotechnology . 20 (8): 357–8. doi: 10.1016/S0167-7799(02)01985-6
• Squazzoni, Flaminio; Brezis, Elise; Marušić, Ana (2017). “Scientometrics of peer review”. Scientometrics . 113 (1): 501–502. doi: 10.1007/s11192-017-2518-4

Related Posts

Disclaimer » Advertising

• HealthyChildren.org

• Previous Article
• Next Article

What is the Purpose of Peer Review?

What makes a good peer reviewer, how do you decide whether to review a paper, how do you complete a peer review, limitations of peer review, conclusions, research methods: how to perform an effective peer review.

• Split-Screen
• Article contents
• Figures & tables
• Supplementary Data
• Peer Review
• CME Quiz Close Quiz
• Open the PDF for in another window
• Get Permissions
• Cite Icon Cite
• Search Site

Elise Peterson Lu , Brett G. Fischer , Melissa A. Plesac , Andrew P.J. Olson; Research Methods: How to Perform an Effective Peer Review. Hosp Pediatr November 2022; 12 (11): e409–e413. https://doi.org/10.1542/hpeds.2022-006764

Download citation file:

• Ris (Zotero)
• Reference Manager

Scientific peer review has existed for centuries and is a cornerstone of the scientific publication process. Because the number of scientific publications has rapidly increased over the past decades, so has the number of peer reviews and peer reviewers. In this paper, drawing on the relevant medical literature and our collective experience as peer reviewers, we provide a user guide to the peer review process, including discussion of the purpose and limitations of peer review, the qualities of a good peer reviewer, and a step-by-step process of how to conduct an effective peer review.

Peer review has been a part of scientific publications since 1665, when the Philosophical Transactions of the Royal Society became the first publication to formalize a system of expert review. 1 , 2   It became an institutionalized part of science in the latter half of the 20 th century and is now the standard in scientific research publications. 3   In 2012, there were more than 28 000 scholarly peer-reviewed journals and more than 3 million peer reviewed articles are now published annually. 3 , 4   However, even with this volume, most peer reviewers learn to review “on the (unpaid) job” and no standard training system exists to ensure quality and consistency. 5   Expectations and format vary between journals and most, but not all, provide basic instructions for reviewers. In this paper, we provide a general introduction to the peer review process and identify common strategies for success as well as pitfalls to avoid.

Modern peer review serves 2 primary purposes: (1) as “a screen before the diffusion of new knowledge” 6   and (2) as a method to improve the quality of published work. 1 , 5  

As screeners, peer reviewers evaluate the quality, validity, relevance, and significance of research before publication to maintain the credibility of the publications they serve and their fields of study. 1 , 2 , 7   Although peer reviewers are not the final decision makers on publication (that role belongs to the editor), their recommendations affect editorial decisions and thoughtful comments influence an article’s fate. 6 , 8  

As advisors and evaluators of manuscripts, reviewers have an opportunity and responsibility to give authors an outside expert’s perspective on their work. 9   They provide feedback that can improve methodology, enhance rigor, improve clarity, and redefine the scope of articles. 5 , 8 , 10   This often happens even if a paper is not ultimately accepted at the reviewer’s journal because peer reviewers’ comments are incorporated into revised drafts that are submitted to another journal. In a 2019 survey of authors, reviewers, and editors, 83% said that peer review helps science communication and 90% of authors reported that peer review improved their last paper. 11  

Expertise: Peer reviewers should be up to date with current literature, practice guidelines, and methodology within their subject area. However, academic rank and seniority do not define expertise and are not actually correlated with performance in peer review. 13  

Professionalism: Reviewers should be reliable and objective, aware of their own biases, and respectful of the confidentiality of the peer review process.

Critical skill : Reviewers should be organized, thorough, and detailed in their critique with the goal of improving the manuscript under their review, regardless of disposition. They should provide constructive comments that are specific and addressable, referencing literature when possible. A peer reviewer should leave a paper better than he or she found it.

Is the manuscript within your area of expertise? Generally, if you are asked to review a paper, it is because an editor felt that you were a qualified expert. In a 2019 survey, 74% of requested reviews were within the reviewer’s area of expertise. 11   This, of course, does not mean that you must be widely published in the area, only that you have enough expertise and comfort with the topic to critique and add to the paper.

Do you have any biases that may affect your review? Are there elements of the methodology, content area, or theory with which you disagree? Some disagreements between authors and reviewers are common, expected, and even helpful. However, if a reviewer fundamentally disagrees with an author’s premise such that he or she cannot be constructive, the review invitation should be declined.

Do you have the time? The average review for a clinical journal takes 5 to 6 hours, though many take longer depending on the complexity of the research and the experience of the reviewer. 1 , 14   Journals vary on the requested timeline for return of reviews, though it is usually 1 to 4 weeks. Peer review is often the longest part of the publication process and delays contribute to slower dissemination of important work and decreased author satisfaction. 15   Be mindful of your schedule and only accept a review invitation if you can reasonably return the review in the requested time.

Once you have determined that you are the right person and decided to take on the review, reply to the inviting e-mail or click the associated link to accept (or decline) the invitation. Journal editors invite a limited number of reviewers at a time and wait for responses before inviting others. A common complaint among journal editors surveyed was that reviewers would often take days to weeks to respond to requests, or not respond at all, making it difficult to find appropriate reviewers and prolonging an already long process. 5  

Now that you have decided to take on the review, it is best of have a systematic way of both evaluating the manuscript and writing the review. Various suggestions exist in the literature, but we will describe our standard procedure for review, incorporating specific do’s and don’ts summarized in Table 1 .

Dos and Don’ts of Peer Review

First, read the manuscript once without making notes or forming opinions to get a sense of the paper as whole. Assess the overall tone and flow and define what the authors identify as the main point of their work. Does the work overall make sense? Do the authors tell the story effectively?

Next, read the manuscript again with an eye toward review, taking notes and formulating thoughts on strengths and weaknesses. Consider the methodology and identify the specific type of research described. Refer to the corresponding reporting guideline if applicable (CONSORT for randomized control trials, STROBE for observational studies, PRISMA for systematic reviews). Reporting guidelines often include a checklist, flow diagram, or structured text giving a minimum list of information needed in a manuscript based on the type of research done. 16   This allows the reviewer to formulate a more nuanced and specific assessment of the manuscript.

Next, review the main findings, the significance of the work, and what contribution it makes to the field. Examine the presentation and flow of the manuscript but do not copy edit the text. At this point, you should start to write your review. Some journals provide a format for their reviews, but often it is up to the reviewer. In surveys of journal editors and reviewers, a review organized by manuscript section was the most favored, 5 , 6   so that is what we will describe here.

As you write your review, consider starting with a brief summary of the work that identifies the main topic, explains the basic approach, and describes the findings and conclusions. 12 , 17   Though not universally included in all reviews, we have found this step to be helpful in ensuring that the work is conveyed clearly enough for the reviewer to summarize it. Include brief notes on the significance of the work and what it adds to current knowledge. Critique the presentation of the work: is it clearly written? Is its length appropriate? List any major concerns with the work overall, such as major methodological flaws or inaccurate conclusions that should disqualify it from publication, though do not comment directly on disposition. Then perform your review by section:

Abstract : Is it consistent with the rest of the paper? Does it adequately describe the major points?

Introduction : This section should provide adequate background to explain the need for the study. Generally, classic or highly relevant studies should be cited, but citations do not have to be exhaustive. The research question and hypothesis should be clearly stated.

Methods: Evaluate both the methods themselves and the way in which they are explained. Does the methodology used meet the needs of the questions proposed? Is there sufficient detail to explain what the authors did and, if not, what needs to be added? For clinical research, examine the inclusion/exclusion criteria, control populations, and possible sources of bias. Reporting guidelines can be particularly helpful in determining the appropriateness of the methods and how they are reported.

Some journals will expect an evaluation of the statistics used, whereas others will have a separate statistician evaluate, and the reviewers are generally not expected to have an exhaustive knowledge of statistical methods. Clarify expectations if needed and, if you do not feel qualified to evaluate the statistics, make this clear in your review.

Results: Evaluate the presentation of the results. Is information given in sufficient detail to assess credibility? Are the results consistent with the methodology reported? Are the figures and tables consistent with the text, easy to interpret, and relevant to the work? Make note of data that could be better detailed in figures or tables, rather than included in the text. Make note of inappropriate interpretation in the results section (this should be in discussion) or rehashing of methods.

Discussion: Evaluate the authors’ interpretation of their results, how they address limitations, and the implications of their work. How does the work contribute to the field, and do the authors adequately describe those contributions? Make note of overinterpretation or conclusions not supported by the data.

The length of your review often correlates with your opinion of the quality of the work. If an article has major flaws that you think preclude publication, write a brief review that focuses on the big picture. Articles that may not be accepted but still represent quality work merit longer reviews aimed at helping the author improve the work for resubmission elsewhere.

Generally, do not include your recommendation on disposition in the body of the review itself. Acceptance or rejection is ultimately determined by the editor and including your recommendation in your comments to the authors can be confusing. A journal editor’s decision on acceptance or rejection may depend on more factors than just the quality of the work, including the subject area, journal priorities, other contemporaneous submissions, and page constraints.

Many submission sites include a separate question asking whether to accept, accept with major revision, or reject. If this specific format is not included, then add your recommendation in the “confidential notes to the editor.” Your recommendation should be consistent with the content of your review: don’t give a glowing review but recommend rejection or harshly criticize a manuscript but recommend publication. Last, regardless of your ultimate recommendation on disposition, it is imperative to use respectful and professional language and tone in your written review.

Although peer review is often described as the “gatekeeper” of science and characterized as a quality control measure, peer review is not ideally designed to detect fundamental errors, plagiarism, or fraud. In multiple studies, peer reviewers detected only 20% to 33% of intentionally inserted errors in scientific manuscripts. 18 , 19   Plagiarism similarly is not detected in peer review, largely because of the huge volume of literature available to plagiarize. Most journals now use computer software to identify plagiarism before a manuscript goes to peer review. Finally, outright fraud often goes undetected in peer review. Reviewers start from a position of respect for the authors and trust the data they are given barring obvious inconsistencies. Ultimately, reviewers are “gatekeepers, not detectives.” 7  

Peer review is also limited by bias. Even with the best of intentions, reviewers bring biases including but not limited to prestige bias, affiliation bias, nationality bias, language bias, gender bias, content bias, confirmation bias, bias against interdisciplinary research, publication bias, conservatism, and bias of conflict of interest. 3 , 4 , 6   For example, peer reviewers score methodology higher and are more likely to recommend publication when prestigious author names or institutions are visible. 20   Although bias can be mitigated both by the reviewer and by the journal, it cannot be eliminated. Reviewers should be mindful of their own biases while performing reviews and work to actively mitigate them. For example, if English language editing is necessary, state this with specific examples rather than suggesting the authors seek editing by a “native English speaker.”

Peer review is an essential, though imperfect, part of the forward movement of science. Peer review can function as both a gatekeeper to protect the published record of science and a mechanism to improve research at the level of individual manuscripts. Here, we have described our strategy, summarized in Table 2 , for performing a thorough peer review, with a focus on organization, objectivity, and constructiveness. By using a systematized strategy to evaluate manuscripts and an organized format for writing reviews, you can provide a relatively objective perspective in editorial decision-making. By providing specific and constructive feedback to authors, you contribute to the quality of the published literature.

Take-home Points

FUNDING: No external funding.

CONFLICT OF INTEREST DISCLOSURES: The authors have indicated they have no potential conflicts of interest to disclose.

Dr Lu performed the literature review and wrote the manuscript. Dr Fischer assisted in the literature review and reviewed and edited the manuscript. Dr Plesac provided background information on the process of peer review, reviewed and edited the manuscript, and completed revisions. Dr Olson provided background information and practical advice, critically reviewed and revised the manuscript, and approved the final manuscript.

Advertising Disclaimer »

Citing articles via

Email alerts.

Affiliations

• Editorial Board
• Editorial Policies
• Pediatrics On Call
• Online ISSN 2154-1671
• Print ISSN 2154-1663
• Pediatrics Open Science
• Hospital Pediatrics
• Pediatrics in Review
• AAP Grand Rounds
• Latest News
• Pediatric Care Online
• Red Book Online
• Pediatric Patient Education
• AAP Toolkits
• AAP Pediatric Coding Newsletter

First 1,000 Days Knowledge Center

Institutions/librarians, group practices, licensing/permissions, integrations, advertising.

• Privacy Statement | Accessibility Statement | Terms of Use | Support Center | Contact Us
• © Copyright American Academy of Pediatrics

This Feature Is Available To Subscribers Only

Sign In or Create an Account

• Search Search
• CN (Chinese)
• DE (German)
• ES (Spanish)
• FR (Français)
• JP (Japanese)
• Open Research
• Booksellers
• Peer Reviewers
• Springer Nature Group ↗
• Publish an article
• Roles and responsibilities
• Signing your contract
• Writing your manuscript
• Submitting your manuscript
• Producing your book
• Promoting your book
• Submit your book idea
• Manuscript guidelines
• Book author services
• Publish a book
• Publish conference proceedings

How to peer review

Author tutorials 

For science to progress, research methods and findings need to be closely examined and verified, and from them a decision on the best direction for future research is made. After a study has gone through peer review and is accepted for publication, scientists and the public can be confident that the study has met certain standards, and that the results can be trusted.

What you will get from this course

When you have completed this course and the included quizzes, you will have gained the skills needed to evaluate another researcher’s manuscript in a way that will help a journal Editor make a decision about publication. Additionally, having successfully completed the quizzes will let you demonstrate that competence to the wider research community

Topics covered

How the peer review process works.

Journals use peer review to both validate the research reported in submitted manuscripts, and sometimes to help inform their decisions about whether or not to publish that article in their journal. 

If the Editor does not immediately reject the manuscript (a “desk rejection”), then the editor will send the manuscript to two or more experts in the field to review it. The experts—called peer reviewers—will then prepare a report that assesses the manuscript, and return it to the editor. After reading the peer reviewer's report, the editor will decide to do one of three things: reject the manuscript, accept the manuscript, or ask the authors to revise and resubmit the manuscript after responding to the peer reviewers’ feedback. If the authors resubmit the manuscript, editors will sometimes ask the same peer reviewers to look over the manuscript again to see if their concerns have been addressed. This is called re-review.

Some of the problems that peer reviewers may find in a manuscript include errors in the study’s methods or analysis that raise questions about the findings, or sections that need clearer explanations so that the manuscript is easily understood. From a journal editor’s point of view, comments on the importance and novelty of a manuscript, and if it will interest the journal’s audience, are particularly useful in helping them to decide which manuscripts to publish.

Will the authors know I am a reviewer? Will I know who the authors are? 

Traditionally, peer review worked in a way we now call “closed,” where the editor and the reviewers knew who the authors were, but the authors did not know who the reviewers were. In recent years, however, many journals have begun to develop other approaches to peer review. These include:

• Closed peer review — where the reviewers are aware of the authors’ identities but the authors’ are never informed of the reviewers’ identities.
• Double-blind peer review —where neither author nor reviewer is aware of each other’s identities.
• Open peer review —where authors and reviewers are aware of each other’s identity. In some journals with open peer review the reviewers’ reports are published alongside the article.

The type of peer review used by a journal should be clearly stated in the invitation to review letter you receive and policy pages on the journal website. If, after checking the journal website, you are unsure of the type of peer review used or would like clarification on the journal’s policy you should contact the journal’s editors.

Why serve as a peer reviewer?

As your career advances, you are likely to be asked to serve as a peer reviewer.

As well as supporting the advancement of science, and providing guidance on how the author can improve their paper, there are also some benefits of peer reviewing to you as a researcher:

• Serving as a peer reviewer looks good on your CV as it shows that your expertise is recognized by other scientists. (See the supplemental material about the Web of Science Reviewer Recognition Service to learn more about getting credit for the reviews you do. Also see the supplemental material about ORCiD iDs to learn how to connect your reviews to your unique ORCiD iD.) 
• You will get to read some of the latest science in your field well before it is in the public domain.
• The critical thinking skills needed during peer review will help you in your own research and writing.

Who does peer review benefit?

When performed correctly peer review helps improve the clarity, robustness and reproducibility of research.

When peer reviewing, it is helpful to think from the point of view of three different groups of people:

• Authors . Try to review the manuscript as you would like others to review your work. When you point out problems in a manuscript, do so in a way that will help the authors to improve the manuscript. Even if you recommend to the editor that the manuscript be rejected, your suggested revisions could help the authors prepare the manuscript for submission to a different journal. 
• Journal editors . Comment on the importance and novelty of the study. Editors will use your comments to assess whether the manuscript is of the right level of impact for the journal. Your comments and opinions on the paper are much more important that a simple recommendation; editors need to know why you think a paper should be published or rejected as your reasoning will help inform their decision.
• Readers . Identify areas that need clarification to make sure other readers can easily understand the manuscript. As a reviewer, you can also save readers’ time and frustration by helping to keep unimportant or error filled research out of the published literature.

Writing a thorough, thoughtful review usually takes several hours or more. But by taking the time to be a good reviewer, you will be providing a service to the scientific community.  

Accepting an invitation to review

Editors invite you to review as they believe that you are an expert in a certain area. They would have judged this from your previous publication record or posters and/or sessions you have contributed to at conferences. You may find that the number of invitations to review increases as you progress in your career.

There are several questions to consider before you accept an invitation to review a paper.

• Are you qualified? The editor has asked you to review the manuscript because he or she believes you are familiar with the specific topic or research method used in the paper. It will usually be okay if you can review some, but not all, aspects of a manuscript. Take as an example, if the study focused on a certain physiological process in an animal model you conduct your research on but used a technique that you have never used. In this case, simply review the parts of the manuscript that are in your area of expertise, and tell the editor which parts you cannot review. However, if the manuscript is too far outside your area, you should decline to review it.
• Do you have time? If you know you will not be able to review the manuscript by the deadline, then you should not accept the invitation. Sending in a review long after the deadline will delay the publication process and frustrate the editor and authors. Keep in mind that reviewing manuscripts, like research and teaching, is a valuable contribution to science, and is worth making time for whenever possible.
• The reported results could cause you to make or lose money, e.g., the authors are developing a drug that could compete with a drug you are working on.
• The manuscript concerns a controversial question that you have strong feelings about (either agreeing or disagreeing with the authors).
• You have strong positive or negative feelings about one of the authors, e.g., a former teacher who you admire greatly.
• You have published papers or collaborated with one of the co-authors in recent years.

If you are not sure if you have a conflict of interest, discuss your circumstances with the editor.

Along with avoiding a conflict of interest, there are several other ethical guidelines to keep in mind as you review the manuscript. Manuscripts under review are highly confidential, so you should not discuss the manuscript – or even mention its existence – to others. One exception is if you would like to consult with a colleague about your review; in this case, you will need to ask the editor’s permission. It is normally okay to ask one of your students or postdocs to help with the review. However, you should let the editor know that you are being helped, and tell your assistant about the need for confidentiality. In some cases case, when the journal operates an open peer review policy they will allow the student or postdoc to co-sign the report with you should they wish.

It is very unethical to use information in the manuscript to make business decisions, such as buying or selling stock. Also, you should never plagiarize the content or ideas in the manuscript.

Next: Evaluating manuscripts

For further support

We hope that with this tutorial you have a clearer idea of how the peer review process works and feel confident in becoming a peer reviewer.

If you feel that you would like some further support with writing, reviewing, and publishing, Springer Nature offer some services which may be of help.

• Nature Research Editing Service offers high quality  English language and scientific editing. During language editing , Editors will improve the English in your manuscript to ensure the meaning is clear and identify problems that require your review. With Scientific Editing experienced development editors will improve the scientific presentation of your research in your manuscript and cover letter, if supplied. They will also provide you with a report containing feedback on the most important issues identified during the edit, as well as journal recommendations.
• Our affiliates American Journal Experts also provide English language editing* as well as other author services that may support you in preparing your manuscript.
• We provide both online and face-to-face training for researchers on all aspects of the manuscript writing process.

* Please note, using an editing service is neither a requirement nor a guarantee of acceptance for publication. 

Test your knowledge

Take the Quiz!

Stay up to date

Here to foster information exchange with the library community

Connect with us on LinkedIn and stay up to date with news and development.

• Tools & Services
• Account Development
• Sales and account contacts
• Professional
• Press office
• Locations & Contact

We are a world leading research, educational and professional publisher. Visit our main website for more information.

• © 2024 Springer Nature
• General terms and conditions
• Your US State Privacy Rights
• Your Privacy Choices / Manage Cookies
• Accessibility
• Legal notice
• Help us to improve this site, send feedback.

Explainer: what is peer review?

Professor of Organisational Behaviour, Cass Business School, City, University of London

Novak Druce Research Fellow, University of Oxford

Disclosure statement

Thomas Roulet does not work for, consult to, own shares in or receive funding from any company or organisation that would benefit from this article, and has no relevant affiliations.

Andre Spicer does not work for, consult, own shares in or receive funding from any company or organisation that would benefit from this article, and has disclosed no relevant affiliations beyond their academic appointment.

City, University of London provides funding as a founding partner of The Conversation UK.

University of Oxford provides funding as a member of The Conversation UK.

View all partners

We’ve all heard the phrase “peer review” as giving credence to research and scholarly papers, but what does it actually mean? How does it work?

Peer review is one of the gold standards of science. It’s a process where scientists (“peers”) evaluate the quality of other scientists’ work. By doing this, they aim to ensure the work is rigorous, coherent, uses past research and adds to what we already knew.

Most scientific journals, conferences and grant applications have some sort of peer review system. In most cases it is “double blind” peer review. This means evaluators do not know the author(s), and the author(s) do not know the identity of the evaluators. The intention behind this system is to ensure evaluation is not biased.

The more prestigious the journal, conference, or grant, the more demanding will be the review process, and the more likely the rejection. This prestige is why these papers tend to be more read and more cited.

The process in details

The peer review process for journals involves at least three stages.

1. The desk evaluation stage

When a paper is submitted to a journal, it receives an initial evaluation by the chief editor, or an associate editor with relevant expertise.

At this stage, either can “desk reject” the paper: that is, reject the paper without sending it to blind referees. Generally, papers are desk rejected if the paper doesn’t fit the scope of the journal or there is a fundamental flaw which makes it unfit for publication.

In this case, the rejecting editors might write a letter summarising his or her concerns. Some journals, such as the British Medical Journal , desk reject up to two-thirds or more of the papers.

2. The blind review

If the editorial team judges there are no fundamental flaws, they send it for review to blind referees. The number of reviewers depends on the field: in finance there might be only one reviewer, while journals in other fields of social sciences might ask up to four reviewers. Those reviewers are selected by the editor on the basis of their expert knowledge and their absence of a link with the authors.

Reviewers will decide whether to reject the paper, to accept it as it is (which rarely happens) or to ask for the paper to be revised. This means the author needs to change the paper in line with the reviewers’ concerns.

Usually the reviews deal with the validity and rigour of the empirical method, and the importance and originality of the findings (what is called the “contribution” to the existing literature). The editor collects those comments, weights them, takes a decision, and writes a letter summarising the reviewers’ and his or her own concerns.

It can therefore happen that despite hostility on the part of the reviewers, the editor could offer the paper a subsequent round of revision. In the best journals in the social sciences, 10% to 20% of the papers are offered a “revise-and-resubmit” after the first round.

3. The revisions – if you are lucky enough

If the paper has not been rejected after this first round of review, it is sent back to the author(s) for a revision. The process is repeated as many times as necessary for the editor to reach a consensus point on whether to accept or reject the paper. In some cases this can last for several years.

Ultimately, less than 10% of the submitted papers are accepted in the best journals in the social sciences. The renowned journal Nature publishes around 7% of the submitted papers.

Strengths and weaknesses of the peer review process

The peer review process is seen as the gold standard in science because it ensures the rigour, novelty, and consistency of academic outputs. Typically, through rounds of review, flawed ideas are eliminated and good ideas are strengthened and improved. Peer reviewing also ensures that science is relatively independent.

Because scientific ideas are judged by other scientists, the crucial yardstick is scientific standards. If other people from outside of the field were involved in judging ideas, other criteria such as political or economic gain might be used to select ideas. Peer reviewing is also seen as a crucial way of removing personalities and bias from the process of judging knowledge.

Despite the undoubted strengths, the peer review process as we know it has been criticised . It involves a number of social interactions that might create biases – for example, authors might be identified by reviewers if they are in the same field, and desk rejections are not blind.

It might also favour incremental (adding to past research) rather than innovative (new) research. Finally, reviewers are human after all and can make mistakes, misunderstand elements, or miss errors.

Are there any alternatives?

Defenders of the peer review system say although there are flaws, we’re yet to find a better system to evaluate research. However, a number of innovations have been introduced in the academic review system to improve its objectivity and efficiency.

Some new open-access journals (such as PLOS ONE ) publish papers with very little evaluation (they check the work is not deeply flawed methodologically). The focus there is on the post-publication peer review system: all readers can comment and criticise the paper.

Some journals such as Nature, have made part of the review process public (“open” review), offering a hybrid system in which peer review plays a role of primary gate keepers, but the public community of scholars judge in parallel (or afterwards in some other journals) the value of the research.

Another idea is to have a set of reviewers rating the paper each time it is revised. In this case, authors will be able to choose whether they want to invest more time in a revision to obtain a better rating, and get their work publicly recognised.

• Peer review

Lecturer / Senior Lecturer - Marketing

Communications and Engagement Officer, Corporate Finance Property and Sustainability

Assistant Editor - 1 year cadetship

Executive Dean, Faculty of Health

Lecturer/Senior Lecturer, Earth System Science (School of Science)

Page Content

Overview of the review report format, the first read-through, first read considerations, spotting potential major flaws, concluding the first reading, rejection after the first reading, before starting the second read-through, doing the second read-through, the second read-through: section by section guidance, how to structure your report, on presentation and style, criticisms & confidential comments to editors, the recommendation, when recommending rejection, additional resources, step by step guide to reviewing a manuscript.

When you receive an invitation to peer review, you should be sent a copy of the paper's abstract to help you decide whether you wish to do the review. Try to respond to invitations promptly - it will prevent delays. It is also important at this stage to declare any potential Conflict of Interest.

The structure of the review report varies between journals. Some follow an informal structure, while others have a more formal approach.

" Number your comments!!! " (Jonathon Halbesleben, former Editor of Journal of Occupational and Organizational Psychology)

Informal Structure

Many journals don't provide criteria for reviews beyond asking for your 'analysis of merits'. In this case, you may wish to familiarize yourself with examples of other reviews done for the journal, which the editor should be able to provide or, as you gain experience, rely on your own evolving style.

Formal Structure

Other journals require a more formal approach. Sometimes they will ask you to address specific questions in your review via a questionnaire. Or they might want you to rate the manuscript on various attributes using a scorecard. Often you can't see these until you log in to submit your review. So when you agree to the work, it's worth checking for any journal-specific guidelines and requirements. If there are formal guidelines, let them direct the structure of your review.

In Both Cases

Whether specifically required by the reporting format or not, you should expect to compile comments to authors and possibly confidential ones to editors only.

Following the invitation to review, when you'll have received the article abstract, you should already understand the aims, key data and conclusions of the manuscript. If you don't, make a note now that you need to feedback on how to improve those sections.

The first read-through is a skim-read. It will help you form an initial impression of the paper and get a sense of whether your eventual recommendation will be to accept or reject the paper.

Keep a pen and paper handy when skim-reading.

Try to bear in mind the following questions - they'll help you form your overall impression:

• What is the main question addressed by the research? Is it relevant and interesting?
• How original is the topic? What does it add to the subject area compared with other published material?
• Is the paper well written? Is the text clear and easy to read?
• Are the conclusions consistent with the evidence and arguments presented? Do they address the main question posed?
• If the author is disagreeing significantly with the current academic consensus, do they have a substantial case? If not, what would be required to make their case credible?
• If the paper includes tables or figures, what do they add to the paper? Do they aid understanding or are they superfluous?

While you should read the whole paper, making the right choice of what to read first can save time by flagging major problems early on.

Editors say, " Specific recommendations for remedying flaws are VERY welcome ."

Examples of possibly major flaws include:

• Drawing a conclusion that is contradicted by the author's own statistical or qualitative evidence
• The use of a discredited method
• Ignoring a process that is known to have a strong influence on the area under study

If experimental design features prominently in the paper, first check that the methodology is sound - if not, this is likely to be a major flaw.

You might examine:

• The sampling in analytical papers
• The sufficient use of control experiments
• The precision of process data
• The regularity of sampling in time-dependent studies
• The validity of questions, the use of a detailed methodology and the data analysis being done systematically (in qualitative research)
• That qualitative research extends beyond the author's opinions, with sufficient descriptive elements and appropriate quotes from interviews or focus groups

Major Flaws in Information

If methodology is less of an issue, it's often a good idea to look at the data tables, figures or images first. Especially in science research, it's all about the information gathered. If there are critical flaws in this, it's very likely the manuscript will need to be rejected. Such issues include:

• Insufficient data
• Unclear data tables
• Contradictory data that either are not self-consistent or disagree with the conclusions
• Confirmatory data that adds little, if anything, to current understanding - unless strong arguments for such repetition are made

If you find a major problem, note your reasoning and clear supporting evidence (including citations).

After the initial read and using your notes, including those of any major flaws you found, draft the first two paragraphs of your review - the first summarizing the research question addressed and the second the contribution of the work. If the journal has a prescribed reporting format, this draft will still help you compose your thoughts.

The First Paragraph

This should state the main question addressed by the research and summarize the goals, approaches, and conclusions of the paper. It should:

• Help the editor properly contextualize the research and add weight to your judgement
• Show the author what key messages are conveyed to the reader, so they can be sure they are achieving what they set out to do
• Focus on successful aspects of the paper so the author gets a sense of what they've done well

The Second Paragraph

This should provide a conceptual overview of the contribution of the research. So consider:

• Is the paper's premise interesting and important?
• Are the methods used appropriate?
• Do the data support the conclusions?

After drafting these two paragraphs, you should be in a position to decide whether this manuscript is seriously flawed and should be rejected (see the next section). Or whether it is publishable in principle and merits a detailed, careful read through.

Even if you are coming to the opinion that an article has serious flaws, make sure you read the whole paper. This is very important because you may find some really positive aspects that can be communicated to the author. This could help them with future submissions.

A full read-through will also make sure that any initial concerns are indeed correct and fair. After all, you need the context of the whole paper before deciding to reject. If you still intend to recommend rejection, see the section "When recommending rejection."

Once the paper has passed your first read and you've decided the article is publishable in principle, one purpose of the second, detailed read-through is to help prepare the manuscript for publication. You may still decide to recommend rejection following a second reading.

" Offer clear suggestions for how the authors can address the concerns raised. In other words, if you're going to raise a problem, provide a solution ." (Jonathon Halbesleben, Editor of Journal of Occupational and Organizational Psychology)

Preparation

To save time and simplify the review:

• Don't rely solely upon inserting comments on the manuscript document - make separate notes
• Try to group similar concerns or praise together
• If using a review program to note directly onto the manuscript, still try grouping the concerns and praise in separate notes - it helps later
• Note line numbers of text upon which your notes are based - this helps you find items again and also aids those reading your review

Now that you have completed your preparations, you're ready to spend an hour or so reading carefully through the manuscript.

As you're reading through the manuscript for a second time, you'll need to keep in mind the argument's construction, the clarity of the language and content.

With regard to the argument’s construction, you should identify:

• Any places where the meaning is unclear or ambiguous
• Any factual errors
• Any invalid arguments

You may also wish to consider:

• Does the title properly reflect the subject of the paper?
• Does the abstract provide an accessible summary of the paper?
• Do the keywords accurately reflect the content?
• Is the paper an appropriate length?
• Are the key messages short, accurate and clear?

Not every submission is well written. Part of your role is to make sure that the text’s meaning is clear.

Editors say, " If a manuscript has many English language and editing issues, please do not try and fix it. If it is too bad, note that in your review and it should be up to the authors to have the manuscript edited ."

If the article is difficult to understand, you should have rejected it already. However, if the language is poor but you understand the core message, see if you can suggest improvements to fix the problem:

• Are there certain aspects that could be communicated better, such as parts of the discussion?
• Should the authors consider resubmitting to the same journal after language improvements?
• Would you consider looking at the paper again once these issues are dealt with?

On Grammar and Punctuation

Your primary role is judging the research content. Don't spend time polishing grammar or spelling. Editors will make sure that the text is at a high standard before publication. However, if you spot grammatical errors that affect clarity of meaning, then it's important to highlight these. Expect to suggest such amendments - it's rare for a manuscript to pass review with no corrections.

A 2010 study of nursing journals found that 79% of recommendations by reviewers were influenced by grammar and writing style (Shattel, et al., 2010).

1. The Introduction

A well-written introduction:

• Sets out the argument
• Summarizes recent research related to the topic
• Highlights gaps in current understanding or conflicts in current knowledge
• Establishes the originality of the research aims by demonstrating the need for investigations in the topic area
• Gives a clear idea of the target readership, why the research was carried out and the novelty and topicality of the manuscript

Originality and Topicality

Originality and topicality can only be established in the light of recent authoritative research. For example, it's impossible to argue that there is a conflict in current understanding by referencing articles that are 10 years old.

Authors may make the case that a topic hasn't been investigated in several years and that new research is required. This point is only valid if researchers can point to recent developments in data gathering techniques or to research in indirectly related fields that suggest the topic needs revisiting. Clearly, authors can only do this by referencing recent literature. Obviously, where older research is seminal or where aspects of the methodology rely upon it, then it is perfectly appropriate for authors to cite some older papers.

Editors say, "Is the report providing new information; is it novel or just confirmatory of well-known outcomes ?"

It's common for the introduction to end by stating the research aims. By this point you should already have a good impression of them - if the explicit aims come as a surprise, then the introduction needs improvement.

2. Materials and Methods

Academic research should be replicable, repeatable and robust - and follow best practice.

Replicable Research

This makes sufficient use of:

• Control experiments
• Repeated analyses
• Repeated experiments

These are used to make sure observed trends are not due to chance and that the same experiment could be repeated by other researchers - and result in the same outcome. Statistical analyses will not be sound if methods are not replicable. Where research is not replicable, the paper should be recommended for rejection.

Repeatable Methods

These give enough detail so that other researchers are able to carry out the same research. For example, equipment used or sampling methods should all be described in detail so that others could follow the same steps. Where methods are not detailed enough, it's usual to ask for the methods section to be revised.

Robust Research

This has enough data points to make sure the data are reliable. If there are insufficient data, it might be appropriate to recommend revision. You should also consider whether there is any in-built bias not nullified by the control experiments.

Best Practice

During these checks you should keep in mind best practice:

• Standard guidelines were followed (e.g. the CONSORT Statement for reporting randomized trials)
• The health and safety of all participants in the study was not compromised
• Ethical standards were maintained

If the research fails to reach relevant best practice standards, it's usual to recommend rejection. What's more, you don't then need to read any further.

3. Results and Discussion

This section should tell a coherent story - What happened? What was discovered or confirmed?

Certain patterns of good reporting need to be followed by the author:

• They should start by describing in simple terms what the data show
• They should make reference to statistical analyses, such as significance or goodness of fit
• Once described, they should evaluate the trends observed and explain the significance of the results to wider understanding. This can only be done by referencing published research
• The outcome should be a critical analysis of the data collected

Discussion should always, at some point, gather all the information together into a single whole. Authors should describe and discuss the overall story formed. If there are gaps or inconsistencies in the story, they should address these and suggest ways future research might confirm the findings or take the research forward.

4. Conclusions

This section is usually no more than a few paragraphs and may be presented as part of the results and discussion, or in a separate section. The conclusions should reflect upon the aims - whether they were achieved or not - and, just like the aims, should not be surprising. If the conclusions are not evidence-based, it's appropriate to ask for them to be re-written.

5. Information Gathered: Images, Graphs and Data Tables

If you find yourself looking at a piece of information from which you cannot discern a story, then you should ask for improvements in presentation. This could be an issue with titles, labels, statistical notation or image quality.

Where information is clear, you should check that:

• The results seem plausible, in case there is an error in data gathering
• The trends you can see support the paper's discussion and conclusions
• There are sufficient data. For example, in studies carried out over time are there sufficient data points to support the trends described by the author?

You should also check whether images have been edited or manipulated to emphasize the story they tell. This may be appropriate but only if authors report on how the image has been edited (e.g. by highlighting certain parts of an image). Where you feel that an image has been edited or manipulated without explanation, you should highlight this in a confidential comment to the editor in your report.

6. List of References

You will need to check referencing for accuracy, adequacy and balance.

Where a cited article is central to the author's argument, you should check the accuracy and format of the reference - and bear in mind different subject areas may use citations differently. Otherwise, it's the editor’s role to exhaustively check the reference section for accuracy and format.

You should consider if the referencing is adequate:

• Are important parts of the argument poorly supported?
• Are there published studies that show similar or dissimilar trends that should be discussed?
• If a manuscript only uses half the citations typical in its field, this may be an indicator that referencing should be improved - but don't be guided solely by quantity
• References should be relevant, recent and readily retrievable

Check for a well-balanced list of references that is:

• Helpful to the reader
• Fair to competing authors
• Not over-reliant on self-citation
• Gives due recognition to the initial discoveries and related work that led to the work under assessment

You should be able to evaluate whether the article meets the criteria for balanced referencing without looking up every reference.

7. Plagiarism

By now you will have a deep understanding of the paper's content - and you may have some concerns about plagiarism.

Identified Concern

If you find - or already knew of - a very similar paper, this may be because the author overlooked it in their own literature search. Or it may be because it is very recent or published in a journal slightly outside their usual field.

You may feel you can advise the author how to emphasize the novel aspects of their own study, so as to better differentiate it from similar research. If so, you may ask the author to discuss their aims and results, or modify their conclusions, in light of the similar article. Of course, the research similarities may be so great that they render the work unoriginal and you have no choice but to recommend rejection.

"It's very helpful when a reviewer can point out recent similar publications on the same topic by other groups, or that the authors have already published some data elsewhere ." (Editor feedback)

Suspected Concern

If you suspect plagiarism, including self-plagiarism, but cannot recall or locate exactly what is being plagiarized, notify the editor of your suspicion and ask for guidance.

Most editors have access to software that can check for plagiarism.

Editors are not out to police every paper, but when plagiarism is discovered during peer review it can be properly addressed ahead of publication. If plagiarism is discovered only after publication, the consequences are worse for both authors and readers, because a retraction may be necessary.

For detailed guidelines see COPE's Ethical guidelines for reviewers and Wiley's Best Practice Guidelines on Publishing Ethics .

8. Search Engine Optimization (SEO)

After the detailed read-through, you will be in a position to advise whether the title, abstract and key words are optimized for search purposes. In order to be effective, good SEO terms will reflect the aims of the research.

A clear title and abstract will improve the paper's search engine rankings and will influence whether the user finds and then decides to navigate to the main article. The title should contain the relevant SEO terms early on. This has a major effect on the impact of a paper, since it helps it appear in search results. A poor abstract can then lose the reader's interest and undo the benefit of an effective title - whilst the paper's abstract may appear in search results, the potential reader may go no further.

So ask yourself, while the abstract may have seemed adequate during earlier checks, does it:

• Do justice to the manuscript in this context?
• Highlight important findings sufficiently?
• Present the most interesting data?

Editors say, " Does the Abstract highlight the important findings of the study ?"

If there is a formal report format, remember to follow it. This will often comprise a range of questions followed by comment sections. Try to answer all the questions. They are there because the editor felt that they are important. If you're following an informal report format you could structure your report in three sections: summary, major issues, minor issues.

• Give positive feedback first. Authors are more likely to read your review if you do so. But don't overdo it if you will be recommending rejection
• Briefly summarize what the paper is about and what the findings are
• Try to put the findings of the paper into the context of the existing literature and current knowledge
• Indicate the significance of the work and if it is novel or mainly confirmatory
• Indicate the work's strengths, its quality and completeness
• State any major flaws or weaknesses and note any special considerations. For example, if previously held theories are being overlooked

Major Issues

• Are there any major flaws? State what they are and what the severity of their impact is on the paper
• Has similar work already been published without the authors acknowledging this?
• Are the authors presenting findings that challenge current thinking? Is the evidence they present strong enough to prove their case? Have they cited all the relevant work that would contradict their thinking and addressed it appropriately?
• If major revisions are required, try to indicate clearly what they are
• Are there any major presentational problems? Are figures & tables, language and manuscript structure all clear enough for you to accurately assess the work?
• Are there any ethical issues? If you are unsure it may be better to disclose these in the confidential comments section

Minor Issues

• Are there places where meaning is ambiguous? How can this be corrected?
• Are the correct references cited? If not, which should be cited instead/also? Are citations excessive, limited, or biased?
• Are there any factual, numerical or unit errors? If so, what are they?
• Are all tables and figures appropriate, sufficient, and correctly labelled? If not, say which are not

Your review should ultimately help the author improve their article. So be polite, honest and clear. You should also try to be objective and constructive, not subjective and destructive.

You should also:

• Write clearly and so you can be understood by people whose first language is not English
• Avoid complex or unusual words, especially ones that would even confuse native speakers
• Number your points and refer to page and line numbers in the manuscript when making specific comments
• If you have been asked to only comment on specific parts or aspects of the manuscript, you should indicate clearly which these are
• Treat the author's work the way you would like your own to be treated

Most journals give reviewers the option to provide some confidential comments to editors. Often this is where editors will want reviewers to state their recommendation - see the next section - but otherwise this area is best reserved for communicating malpractice such as suspected plagiarism, fraud, unattributed work, unethical procedures, duplicate publication, bias or other conflicts of interest.

However, this doesn't give reviewers permission to 'backstab' the author. Authors can't see this feedback and are unable to give their side of the story unless the editor asks them to. So in the spirit of fairness, write comments to editors as though authors might read them too.

Reviewers should check the preferences of individual journals as to where they want review decisions to be stated. In particular, bear in mind that some journals will not want the recommendation included in any comments to authors, as this can cause editors difficulty later - see Section 11 for more advice about working with editors.

You will normally be asked to indicate your recommendation (e.g. accept, reject, revise and resubmit, etc.) from a fixed-choice list and then to enter your comments into a separate text box.

Recommending Acceptance

If you're recommending acceptance, give details outlining why, and if there are any areas that could be improved. Don't just give a short, cursory remark such as 'great, accept'. See Improving the Manuscript

Recommending Revision

Where improvements are needed, a recommendation for major or minor revision is typical. You may also choose to state whether you opt in or out of the post-revision review too. If recommending revision, state specific changes you feel need to be made. The author can then reply to each point in turn.

Some journals offer the option to recommend rejection with the possibility of resubmission – this is most relevant where substantial, major revision is necessary.

What can reviewers do to help? " Be clear in their comments to the author (or editor) which points are absolutely critical if the paper is given an opportunity for revisio n." (Jonathon Halbesleben, Editor of Journal of Occupational and Organizational Psychology)

Recommending Rejection

If recommending rejection or major revision, state this clearly in your review (and see the next section, 'When recommending rejection').

Where manuscripts have serious flaws you should not spend any time polishing the review you've drafted or give detailed advice on presentation.

Editors say, " If a reviewer suggests a rejection, but her/his comments are not detailed or helpful, it does not help the editor in making a decision ."

In your recommendations for the author, you should:

• Give constructive feedback describing ways that they could improve the research
• Keep the focus on the research and not the author. This is an extremely important part of your job as a reviewer
• Avoid making critical confidential comments to the editor while being polite and encouraging to the author - the latter may not understand why their manuscript has been rejected. Also, they won't get feedback on how to improve their research and it could trigger an appeal

Remember to give constructive criticism even if recommending rejection. This helps developing researchers improve their work and explains to the editor why you felt the manuscript should not be published.

" When the comments seem really positive, but the recommendation is rejection…it puts the editor in a tough position of having to reject a paper when the comments make it sound like a great paper ." (Jonathon Halbesleben, Editor of Journal of Occupational and Organizational Psychology)

Visit our Wiley Author Learning and Training Channel for expert advice on peer review.

Watch the video, Ethical considerations of Peer Review

How to Write and Publish a Research Paper for a Peer-Reviewed Journal

• Open access
• Published: 30 April 2020
• Volume 36 , pages 909–913, ( 2021 )

Cite this article

You have full access to this open access article

• Clara Busse   ORCID: orcid.org/0000-0002-0178-1000 1 &
• Ella August   ORCID: orcid.org/0000-0001-5151-1036 1 , 2  

271k Accesses

15 Citations

719 Altmetric

Explore all metrics

Communicating research findings is an essential step in the research process. Often, peer-reviewed journals are the forum for such communication, yet many researchers are never taught how to write a publishable scientific paper. In this article, we explain the basic structure of a scientific paper and describe the information that should be included in each section. We also identify common pitfalls for each section and recommend strategies to avoid them. Further, we give advice about target journal selection and authorship. In the online resource 1 , we provide an example of a high-quality scientific paper, with annotations identifying the elements we describe in this article.

Similar content being viewed by others

Plagiarism in research

How metric-based performance evaluation systems fuel the growth of questionable publications?

Systematic Reviews in Educational Research: Methodology, Perspectives and Application

Avoid common mistakes on your manuscript.

Introduction

Writing a scientific paper is an important component of the research process, yet researchers often receive little formal training in scientific writing. This is especially true in low-resource settings. In this article, we explain why choosing a target journal is important, give advice about authorship, provide a basic structure for writing each section of a scientific paper, and describe common pitfalls and recommendations for each section. In the online resource 1 , we also include an annotated journal article that identifies the key elements and writing approaches that we detail here. Before you begin your research, make sure you have ethical clearance from all relevant ethical review boards.

Select a Target Journal Early in the Writing Process

We recommend that you select a “target journal” early in the writing process; a “target journal” is the journal to which you plan to submit your paper. Each journal has a set of core readers and you should tailor your writing to this readership. For example, if you plan to submit a manuscript about vaping during pregnancy to a pregnancy-focused journal, you will need to explain what vaping is because readers of this journal may not have a background in this topic. However, if you were to submit that same article to a tobacco journal, you would not need to provide as much background information about vaping.

Information about a journal’s core readership can be found on its website, usually in a section called “About this journal” or something similar. For example, the Journal of Cancer Education presents such information on the “Aims and Scope” page of its website, which can be found here: https://www.springer.com/journal/13187/aims-and-scope .

Peer reviewer guidelines from your target journal are an additional resource that can help you tailor your writing to the journal and provide additional advice about crafting an effective article [ 1 ]. These are not always available, but it is worth a quick web search to find out.

Identify Author Roles Early in the Process

Early in the writing process, identify authors, determine the order of authors, and discuss the responsibilities of each author. Standard author responsibilities have been identified by The International Committee of Medical Journal Editors (ICMJE) [ 2 ]. To set clear expectations about each team member’s responsibilities and prevent errors in communication, we also suggest outlining more detailed roles, such as who will draft each section of the manuscript, write the abstract, submit the paper electronically, serve as corresponding author, and write the cover letter. It is best to formalize this agreement in writing after discussing it, circulating the document to the author team for approval. We suggest creating a title page on which all authors are listed in the agreed-upon order. It may be necessary to adjust authorship roles and order during the development of the paper. If a new author order is agreed upon, be sure to update the title page in the manuscript draft.

In the case where multiple papers will result from a single study, authors should discuss who will author each paper. Additionally, authors should agree on a deadline for each paper and the lead author should take responsibility for producing an initial draft by this deadline.

Structure of the Introduction Section

The introduction section should be approximately three to five paragraphs in length. Look at examples from your target journal to decide the appropriate length. This section should include the elements shown in Fig.  1 . Begin with a general context, narrowing to the specific focus of the paper. Include five main elements: why your research is important, what is already known about the topic, the “gap” or what is not yet known about the topic, why it is important to learn the new information that your research adds, and the specific research aim(s) that your paper addresses. Your research aim should address the gap you identified. Be sure to add enough background information to enable readers to understand your study. Table 1 provides common introduction section pitfalls and recommendations for addressing them.

The main elements of the introduction section of an original research article. Often, the elements overlap

Methods Section

The purpose of the methods section is twofold: to explain how the study was done in enough detail to enable its replication and to provide enough contextual detail to enable readers to understand and interpret the results. In general, the essential elements of a methods section are the following: a description of the setting and participants, the study design and timing, the recruitment and sampling, the data collection process, the dataset, the dependent and independent variables, the covariates, the analytic approach for each research objective, and the ethical approval. The hallmark of an exemplary methods section is the justification of why each method was used. Table 2 provides common methods section pitfalls and recommendations for addressing them.

Results Section

The focus of the results section should be associations, or lack thereof, rather than statistical tests. Two considerations should guide your writing here. First, the results should present answers to each part of the research aim. Second, return to the methods section to ensure that the analysis and variables for each result have been explained.

Begin the results section by describing the number of participants in the final sample and details such as the number who were approached to participate, the proportion who were eligible and who enrolled, and the number of participants who dropped out. The next part of the results should describe the participant characteristics. After that, you may organize your results by the aim or by putting the most exciting results first. Do not forget to report your non-significant associations. These are still findings.

Tables and figures capture the reader’s attention and efficiently communicate your main findings [ 3 ]. Each table and figure should have a clear message and should complement, rather than repeat, the text. Tables and figures should communicate all salient details necessary for a reader to understand the findings without consulting the text. Include information on comparisons and tests, as well as information about the sample and timing of the study in the title, legend, or in a footnote. Note that figures are often more visually interesting than tables, so if it is feasible to make a figure, make a figure. To avoid confusing the reader, either avoid abbreviations in tables and figures, or define them in a footnote. Note that there should not be citations in the results section and you should not interpret results here. Table 3 provides common results section pitfalls and recommendations for addressing them.

Discussion Section

Opposite the introduction section, the discussion should take the form of a right-side-up triangle beginning with interpretation of your results and moving to general implications (Fig.  2 ). This section typically begins with a restatement of the main findings, which can usually be accomplished with a few carefully-crafted sentences.

Major elements of the discussion section of an original research article. Often, the elements overlap

Next, interpret the meaning or explain the significance of your results, lifting the reader’s gaze from the study’s specific findings to more general applications. Then, compare these study findings with other research. Are these findings in agreement or disagreement with those from other studies? Does this study impart additional nuance to well-accepted theories? Situate your findings within the broader context of scientific literature, then explain the pathways or mechanisms that might give rise to, or explain, the results.

Journals vary in their approach to strengths and limitations sections: some are embedded paragraphs within the discussion section, while some mandate separate section headings. Keep in mind that every study has strengths and limitations. Candidly reporting yours helps readers to correctly interpret your research findings.

The next element of the discussion is a summary of the potential impacts and applications of the research. Should these results be used to optimally design an intervention? Does the work have implications for clinical protocols or public policy? These considerations will help the reader to further grasp the possible impacts of the presented work.

Finally, the discussion should conclude with specific suggestions for future work. Here, you have an opportunity to illuminate specific gaps in the literature that compel further study. Avoid the phrase “future research is necessary” because the recommendation is too general to be helpful to readers. Instead, provide substantive and specific recommendations for future studies. Table 4 provides common discussion section pitfalls and recommendations for addressing them.

Follow the Journal’s Author Guidelines

After you select a target journal, identify the journal’s author guidelines to guide the formatting of your manuscript and references. Author guidelines will often (but not always) include instructions for titles, cover letters, and other components of a manuscript submission. Read the guidelines carefully. If you do not follow the guidelines, your article will be sent back to you.

Finally, do not submit your paper to more than one journal at a time. Even if this is not explicitly stated in the author guidelines of your target journal, it is considered inappropriate and unprofessional.

Your title should invite readers to continue reading beyond the first page [ 4 , 5 ]. It should be informative and interesting. Consider describing the independent and dependent variables, the population and setting, the study design, the timing, and even the main result in your title. Because the focus of the paper can change as you write and revise, we recommend you wait until you have finished writing your paper before composing the title.

Be sure that the title is useful for potential readers searching for your topic. The keywords you select should complement those in your title to maximize the likelihood that a researcher will find your paper through a database search. Avoid using abbreviations in your title unless they are very well known, such as SNP, because it is more likely that someone will use a complete word rather than an abbreviation as a search term to help readers find your paper.

After you have written a complete draft, use the checklist (Fig. 3 ) below to guide your revisions and editing. Additional resources are available on writing the abstract and citing references [ 5 ]. When you feel that your work is ready, ask a trusted colleague or two to read the work and provide informal feedback. The box below provides a checklist that summarizes the key points offered in this article.

Checklist for manuscript quality

Data Availability

Michalek AM (2014) Down the rabbit hole…advice to reviewers. J Cancer Educ 29:4–5

Article   Google Scholar  

International Committee of Medical Journal Editors. Defining the role of authors and contributors: who is an author? http://www.icmje.org/recommendations/browse/roles-and-responsibilities/defining-the-role-of-authosrs-and-contributors.html . Accessed 15 January, 2020

Vetto JT (2014) Short and sweet: a short course on concise medical writing. J Cancer Educ 29(1):194–195

Brett M, Kording K (2017) Ten simple rules for structuring papers. PLoS ComputBiol. https://doi.org/10.1371/journal.pcbi.1005619

Lang TA (2017) Writing a better research article. J Public Health Emerg. https://doi.org/10.21037/jphe.2017.11.06

Download references

Acknowledgments

Ella August is grateful to the Sustainable Sciences Institute for mentoring her in training researchers on writing and publishing their research.

Code Availability

Not applicable.

Author information

Authors and affiliations.

Department of Maternal and Child Health, University of North Carolina Gillings School of Global Public Health, 135 Dauer Dr, 27599, Chapel Hill, NC, USA

Clara Busse & Ella August

Department of Epidemiology, University of Michigan School of Public Health, 1415 Washington Heights, Ann Arbor, MI, 48109-2029, USA

Ella August

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Ella August .

Ethics declarations

Conflicts of interests.

The authors declare that they have no conflict of interest.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

(PDF 362 kb)

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Busse, C., August, E. How to Write and Publish a Research Paper for a Peer-Reviewed Journal. J Canc Educ 36 , 909–913 (2021). https://doi.org/10.1007/s13187-020-01751-z

Download citation

Published : 30 April 2020

Issue Date : October 2021

DOI : https://doi.org/10.1007/s13187-020-01751-z

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Manuscripts
• Scientific writing
• Find a journal
• Publish with us
• Track your research

• Open access
• Published: 09 May 2024

A systematic review of telemedicine for neuromuscular diseases: components and determinants of practice

• Deniz Senyel 1 , 2 ,
• Katja Senn 1 ,
• James Boyd 2 &
• Klaus Nagels 1  

BMC Digital Health volume  2 , Article number:  17 ( 2024 ) Cite this article

78 Accesses

Metrics details

Introduction

Neuromuscular diseases (NMDs) entail a group of mostly inherited genetic disorders with heterogeneous phenotypes impacting muscles, the central or peripheral nervous system. They can lead to severe disabilities and shortened lifespans. Despite their severity, NMDs often lack in public awareness and appropriate medical and social support. Telemedicine can improve patients’ and caregivers’ lives by enhancing continuity of and access to care. The first aim of this systematic review was to summarise the status quo of telemedicine services for patients with NMDs. Secondly, barriers and facilitators of the respective implementation processes should be analysed.

The databases PubMed, Web of Science and CENTRAL by Cochrane were searched in May 2022. To be truly explorative, any original evidence from any setting was included. Two independent researchers completed the screening process. Data was extracted and analysed using the taxonomy of Bashshur et al. (2011) and the Consolidated Framework for Implementation Research (CFIR).

Fifty-seven original papers were included in the systematic review. The results showed a high representation of teleconsultations and remote monitoring studies. Teleconsultations replaced in person appointments and telemonitoring mostly focused on ventilation. Physical therapy, pulmonology, neurology, and psychology were the most represented medical specialties. We found barriers and facilitators relating to implementation mainly referred to the intervention and the individuals involved. Technical errors and inaccessibility due to a lack of technical devices or the patient’s disability were stated as hindrances. A positive mindset of users as well as patient empowerment were necessary for the adoption of new technology. Technophobia or uncertainty around technology negatively impacted the implementation process.

This systematic review provides an overview of the current use of telemedicine in patients with NMDs. The distribution of telemedicine interventions between the defined domains was very heterogenous. Previous research has neglected to fully describe the implementation process of telemedicine for NMDs.

The evidence shows that telemedicine can benefit patients with NMDs in a multitude of ways. Therefore, health policies should endorse and incentivise the uptake of telemedicine by institutions and health care workers. Further research needs to be conducted to confirm the current evidence and close existing research gaps.

Peer Review reports

Neuromuscular diseases (NMDs) are a heterogeneous group of disorders, that affect the nerves controlling muscles, leading to muscle weakness, wasting, and other related symptoms [ 1 ]. NMDs are often hereditary and have been linked to 500 different affected genes [ 2 , 3 ]. Most NMDs are classified as rare diseases. The prevalence of NMDs can vary widely and, even for common diagnostic groups, the prevalence ranges between 0.1 to 60 per 100,000 [ 4 ]. The onset, cause, and course of the disease vary widely between disorders [ 5 ]. While each individual's experience is unique, there are common disability-related challenges faced by patients with NMDs. Acknowledging these commonalities and addressing the unique needs of each person are essential for providing comprehensive care and support to individuals and their families living with NMDs. NMDs are highly complex diseases defined by a degenerative course and progressive muscle weakness as the main symptom. Their impact extends beyond the musculoskeletal system, affecting various organs and systems throughout the body, such as eyes, lungs or the brain [ 1 , 2 ]. As a result, patients suffer from a reduced quality of life and a significant disease burden [ 2 , 6 ]. Multidisciplinary care is often considered the optimal approach for providing holistic treatment and symptomatic management for individuals with NMDs [ 7 , 8 , 9 , 10 , 11 ]. The needs of patients during disease progression are ever changing based on disease stage, symptom burden, and personal priorities. General practitioners, specialists, and allied health professionals each bring unique expertise to the care team, allowing for comprehensive, patient-centred care that adapts to changing needs and priorities throughout the course of the disease and ensures continuity and quality of care [ 1 , 12 , 13 ]. Recognising and supporting caregivers is crucial in the care of NMD patients. Most NMD patients receive informal care, often provided by their partner or family members. The caregiver burden increases with the progression of the patient’s disease. In severe cases, it can lead to psychological distress and burnout, a state of physical and emotional exhaustion [ 14 , 15 , 16 , 17 ].

Mobile health apps, teleconsultation and telemonitoring have been proven to be useful tools in the management and treatment of chronic diseases such as diabetes, heart failure, asthma, chronic obstructive pulmonary disease, and cancer. They have the potential to increase treatment adherence, support self-management, and promote continuity of care [ 18 , 19 , 20 ]. They have the potential to reduce hospital admissions, decrease mortality rates, and lessen health services usage [ 21 , 22 , 23 , 24 ]. The research focus in telemedicine for NMDs varies between disorders. A recent systematic review by Helleman et al. showed telemedicine for ALS patients to be a useful option for remote monitoring, consultations, and follow-ups [ 25 ]. From a patient’s perspective it can be time- and cost-saving while reducing stress and fatigue. While telemedicine has demonstrated its value in certain NMDs like ALS, its usage in the care of other NMDs have not been as extensively studied or described.

This systematic review aims to identify telemedicine interventions for patients with NMDs and analyse the barriers and facilitators of the implementation process associated with telemedicine for NMD patients. The taxonomy by Bashshur et al. will be used to standardise terminology and make it easier to categorise and study the various telehealth interventions and services [ 26 ]. The term “Telemedicine” will be used as an umbrella term to encompass a broad range of remote healthcare services and technologies. This is done to avoid the potential ambiguities and unclarities that can arise from newer terms like "e-health" or "telehealth". This review will provide an overview of the status quo and will offer recommendations for future innovations.

This systematic review followed the PRISMA [ 27 ] checklist. The study protocol was registered on PROSPERO (ID: CRD42022325481).

Databases and search strategy

For the literature search PubMed, Web of Science, and the Cochrane database CENTRAL were used as sources. If full text could not be found, the authors were contacted. The final search was conducted in May of 2022.

The search strategy consisted of two major themes: Firstly, synonyms for NMDs and secondly, synonyms and subcategories for telemedicine. The full search strings can be found in the supplementary file 1 .

Study selection

The study selection was conducted by two reviewers KS and DS. The following inclusion criteria were applied: Studies from any country with any healthcare and insurance system were eligible to maximise the diversity and inclusivity of the evidence base. No restrictions regarding cultural or socio-economic context were made to be truly explorative. Articles were eligible for inclusion if their study population consisted of patients with one or more types of NMDs. Since a single comprehensive list of all NMDs could not be found, the list of NMDs by the Muscular Dystrophy Association (MDA) was used as a reference [ 28 ]. If a disease could not be found under the listed disorders, the International Classification of Diseases (ICD) was consulted [ 29 ]. No limitations regarding sex, age, race, or nationality were made. All types of telemedicine were eligible for inclusion. The taxonomy by Bashshur et al. was used as a guiding definition [ 26 ]. Bashshur uses telemedicine in his paper as the original term for ICT in healthcare. The domains include the following components:

Telehealth : Health behavior & education; Health & disease epidemiology; Environmental/Industrial health; Health management & policy.

E-health : Electronic health record; Health information; Clinical decision support system; Physician order entry.

M-health : Clinical support; Health worker support; Remote data collection; Helplines.

Interventions could be implemented on a national, communal, or institutional level. The users could include patients, caregivers, and healthcare workers. Only primary research was included. Due to the explorative nature of the systematic review, no major restrictions regarding study types were made. Only articles written in English or German were included. Due to the rapid pace of technological progress, only studies from the last ten years were considered. This ensured that the telemedicine interventions were not out-of-date or obsolete.

Studies were excluded if no specific diagnostic group was mentioned. Further reviews, study protocols and commentaries were excluded.

Data extraction and analysis

The data extraction and analysis were done by DS. From the included studies the following data points were extracted: authors, year of publication, country, included NMDs, intervention type and analysed outcomes. Additionally, barriers and facilitators of the implementation process were collected. The Consolidated Framework for Implementation Research (CFIR) was used to guide the extraction process [ 30 ]. The CFIR is an established framework for the analysis of implementation processes. Based on this structure, a detailed coding manual with operationalised definitions for each construct was created. This manual served as a reference guide to ensure that the extraction and coding process was systematic and reproducible.

The data synthesis was done narratively. Since no effect measures were used, a quantitative analysis was not applicable. Firstly, the types of telemedicine interventions were clustered according to the domains described by Bashshur et al., to gain a comprehensive understanding of the current landscape of telemedicine applications [ 26 ]. Secondly, the CFIR was used to label quotes on implementation barriers and facilitators [ 30 ].

No meta-analysis was conducted as there are no quantitative outcomes to analyse. Further, the heterogeneity of the studies was not assessed. Due to the broad inclusion criteria, a high heterogeneity could be expected. Since the focus of this systematic review lies on the intervention types, rather than on their effectiveness, subgroup analyses were not performed. Equally no sensitivity analyses were conducted. The focus of the systematic review was not to summarise evidence regarding a specific intervention, it was an exploration of the current telemedicine options for patients with NMDs.

Risk of bias

The study protocol stated a risk of bias assessment using the RoB 2 and ROBINS-I tools [ 31 , 32 ]. This was later changed to the JBI’s critical appraisal tools as they offered a wider selection of checklists [ 33 ]. No meta-bias was analyzed since the outcomes of the studies were not a point of interest.

Included studies

Figure  1 depicts the study selection process for the systematic review, including a total of 57 reports. These included four report pairs with interlinked content. Ando et al. published two papers on the Intervention Careportal in 2019 and 2021 [ 34 , 35 ]. Hobson et al. conducted one study with results disseminated across two publications [ 36 , 37 ]. Martinet et al. conducted two studies utilising the same intervention but with distinct comparison groups and study populations [ 38 , 39 ]. Lastly, Sobierajska-Rek et al. and Wasilewska et al. published two articles addressing different subsections of one main study [ 40 , 41 ]. Studies excluded during the full text screening process can be found in supplementary file 2 .

Flow diagram of the identified studies (Source: own depiction)

Study characteristics

Table 1 presents an overview of the study characteristics. A total of 25 studies were carried out using a cross-sectional design [ 34 , 35 , 40 , 42 , 43 , 44 , 45 , 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 , 61 , 62 , 63 ]. Additionally, the review included two case series [ 64 , 65 ] and one case–control study [ 66 ]. Among the studies, 16 adopted a cohort study design [ 41 , 67 , 68 , 69 , 70 , 71 , 72 , 73 , 74 , 75 , 76 , 77 , 78 , 79 , 80 , 81 ], while ten employed an experimental design [ 36 , 37 , 38 , 39 , 82 , 83 , 84 , 85 , 86 , 87 ]. The remaining three reports were method papers [ 88 , 89 , 90 ]. Geographically, the majority of the studies took place in Europe [ 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 47 , 48 , 49 , 53 , 58 , 59 , 60 , 62 , 65 , 66 , 68 , 69 , 72 , 73 , 74 , 76 , 80 , 81 , 82 , 84 , 87 , 88 , 89 , 90 ] and the USA [ 44 , 45 , 46 , 51 , 52 , 54 , 56 , 57 , 63 , 67 , 71 , 77 , 78 , 79 , 85 , 86 ]. Two studies were conducted in Canada [ 50 , 64 ] and one in each of the following countries: Japan [ 70 ], India [ 75 ], Brazil [ 83 ], and Australia [ 55 ]. One study included participants from around the globe [ 61 ].

A total of twenty-nine studies exclusively focussed on ALS patients [ 34 , 35 , 36 , 37 , 42 , 46 , 48 , 51 , 52 , 55 , 56 , 57 , 62 , 69 , 70 , 71 , 72 , 73 , 76 , 77 , 78 , 79 , 80 , 81 , 85 , 86 , 87 , 88 , 89 , 90 ], while another four studies included ALS patients alongside other NMD diagnostic groups [ 53 , 54 , 58 , 60 ]. The study outcomes assessed in these studies varied widely. Clinical outcomes, such as physical and cognitive function, as well as mental health, were often used. Further, user satisfaction and utilisation measurements were applied to evaluate interventions. For patient registry studies, epidemiological statistics, including prevalence, were commonly employed as outcome measures.

While the primary focus has been on exploring the availability of telemedicine interventions for patients with NMDS, it is crucial not to overlook the evaluation of individual study quality and the potential impact of bias. In summary, most studies demonstrated a low risk of bias and employed sound methods and procedures. However, certain limitations, such as the lack of comparison groups, insufficient follow up time, and some inadequate reporting, should be noted. Visual depictions and the complete analysis can be found in supplementary material 3 . Three reports were not assessed as they only presented a method paper without empirical results [ 88 , 89 , 90 ].

Telemedicine domains of included interventions

In the following sections the telemedicine interventions included in the analysis will be examined, guided by the taxonomy by Bashshur et al [ 26 ]. According to their definition, telemedicine comprises of three major domains: telehealth, e-health, and m-health . Eight studies were categorised under the telehealth domain, encompassing all traditional public health areas. E-health, mainly describing the online storage of information and supporting tools for physicians, was represented by ten studies. The majority of studies fell within the m-health domain, a rapidly growing field that leverages mobile devices like smartphones and tablets to deliver healthcare services, monitor patients remotely, and support self-management. Given that interventions could encompass elements from different domains, multiple mentions or references to different domains is possible. As stated, there were instances where multiple reports featured identical telemedicine interventions [ 34 , 35 , 36 , 37 , 38 , 39 ]. In order not to bias the results, identical interventions were counted as one during the analysis of the telemedicine domains and components. The distribution of telemedicine domains is illustrated in Fig.  2 a.

Distribution of the telemedicine ( a ) and telehealth domains ( b ) (Source: own depiction)

The studies within the telehealth domain were mostly epidemiological studies. Six studies described online patient registries for one or more NMDs [ 43 , 44 , 47 , 56 , 58 , 72 ]. The remaining two studies were categorised under health education. One study introduced a blended curriculum focusing on physical examinations for patients with NMDs [ 45 ] while another detailed a virtual neuromuscular ultrasound course [ 61 ]. The distribution of the telehealth domain can be seen in Fig.  2 b.

The second smallest domain was e-health (Fig.  3 ). Within this domain, three studies incorporated electronic health records [ 69 , 89 , 90 ]. Health information was the subject of five studies, with two of these not providing an intervention but instead investigating patients’ computer use and information seeking behaviour [ 42 , 50 ]. Only two interventions described clinical decision support systems, one supporting physicians during the diagnostic phase [ 53 ] and another supporting patients with advanced care planning [ 71 ]. A singular app used a function for physician order entries, specifically for nutrition plan entries [ 86 ].

Distribution of the e-Health ( a ) and m-health domains ( b ) (Source: own depiction)

Most included studies contained m-health components (Fig.  3 ). Among the various m-health interventions analysed, helplines represented the smallest category. Specifically, four interventions provided emergency telephone support, and one included useful helpline numbers in their app [ 35 , 60 , 81 , 89 , 90 ].

The predominant categories within the m-health domain were clinical support and remote data collection. Nine studies reported interventions with synchronous consultations and data collection [ 40 , 51 , 57 , 62 , 65 , 73 , 75 , 81 , 85 ]. To illustrate, Christodoulou et al. conducted telephone-based cognitive-behavioural screening in ALS patients [ 85 ], demonstrating how telemedicine can seamlessly combine remote data collection processes with distance consultations. Another example was the remote application of the ALS Functioning Rating Scale during teleconsultations [ 62 ]. An alternative approach identified involving clinical support and remote data collection occurring asynchronously, utilising specially designed devices or mobile applications for data collection [ 35 , 66 , 68 , 69 , 70 , 88 , 90 ]. In this approach, clinical consultation was offered either on demand or automatically triggered based on the collected data.

Fourteen studies used clinical support without remote data collection, including home exercise programs [ 40 , 82 , 83 , 84 ], psychological interventions [ 39 , 87 ] and pure teleconsultation [ 52 , 54 , 55 , 60 , 77 , 78 , 79 ]. In contrast, 12 studies focussed on pure remote data collection without clinical support. This included, accelerometers [ 74 , 80 ], physical assessments [ 63 , 64 , 67 , 76 ] or the assessment of the patient’s nutritional status [ 86 ] or disease-related health [ 37 , 46 , 89 ]. Additionally, Cesareo et al. as well as Wasilewska et al. examined remote pulmonary monitoring devices [ 41 , 49 ].

Barriers and facilitators for the implementation of telemedicine

CFIR was used to assess factors that may facilitate or hinder the implementation of telemedicine. This framework consists of five domains: the inner setting, the outer setting, the implementation process, the intervention characteristics, and the characteristics of the individuals. Relevant information was found in 22 studies, with a predominant focus on patient and carer perspectives [ 34 , 36 , 37 , 41 , 42 , 48 , 49 , 50 , 51 , 52 , 55 , 62 , 63 , 66 , 69 , 73 , 74 , 75 , 76 , 77 , 78 , 83 ]. As a result, no information regarding the inner/outer setting or the implementation process was gathered. All statements focused on the intervention characteristics or the characteristics of the individuals. Thus, the following section is structured according to the two domains and their constructs.

Intervention characteristics

A summary of mentioned barriers and facilitators can be seen in Table  2 .

General characteristics

This category summarises all barriers and facilitators directly linked to the intervention that could not be categorised elsewhere. The most common barrier encountered during the implementation of telemedicine interventions were malfunctions related to internet connectivity or end devices. Examples included software errors [ 51 ], faulty data transmission [ 34 ] or a poor internet connection [ 83 ]. Additionally, it was reported, that the internet and necessary end devices, such as smartphones, tablets, or computers, were often not available [ 48 , 50 , 63 ].

Relative advantage

A major factor for patients was the reduced time and travel burden [ 34 , 51 , 52 , 62 , 76 , 78 ]. In more advanced stages of the diseases travelling with medical equipment became almost impossible, making telemedicine vital for house-bound patients [ 78 ].

Telemonitoring and the remote data collection provided multiple advantages, with patients and caregivers highlighting the timeliness of actions in case of alerts [ 34 , 73 ]. Continuous monitoring also proved beneficial for in-person visits, as medical staff stated that appointments could be used more efficiently with data being analysed beforehand [ 69 ]. Some disadvantages regarding telemedicine were acknowledged. Caregivers and physicians noted the lack of physical evaluation as problematic [ 51 , 52 ]. Additionally, an emotional distance and a lack of informal encounters between patients and healthcare workers was reported [ 52 , 55 ].

Adaptability

Patients appreciated the flexibility of online exercise programs, which were easier to integrate into their daily routines [ 83 ]. It was seen as important to be able to choose the main form of communication [ 55 , 62 ]. For example, patients with speech difficulties communicating via E-Mail was preferred.

Interventions were easier implemented if participants were thoroughly informed about the telemedicine service and if a computer-literate person was on-site [ 78 ]. The duration and frequency of sessions was another major point. Overall, more frequent, and shorter sessions were perceived as less fatiguing [ 78 ].

Design and quality

Critical considerations included the presentation, design, and quality of telemedicine products, emphasizing features like accessible closing mechanisms for wearable devices and age-appropriate designs [ 49 , 74 ].

From a patient’s perspective telemedicine was cost-saving due to reduced travel [ 34 , 48 ]. Nevertheless, acquisition costs could be a barrier for some. Institutional perspectives indicated potential savings, ranging from 20 to 89%, depending on the approach, making costs a crucial factor [ 50 , 77 ].

Characteristics of individuals

The second domain related to the characteristics of individuals. This includes all stakeholders such as patients, caregivers, and healthcare workers. Table 3 depicts the barriers and facilitators relating to the characteristics of individuals.

Knowledge and Beliefs about the Intervention

The CFIR highlights the importance of an individual’s pre-existing knowledge and beliefs about the intervention [ 30 ]. Trust in the intervention was vital for patients using telemonitoring [ 34 , 36 , 52 , 69 , 78 ]. This includes being confident that the transmitted data was monitored and that providers would act in the case of abnormalities.

Self-efficacy

Easy to use technology was seen as an enabler for telemedicine implementation, as it reassured the user in their abilities. Accordingly, barriers arose if patients could not or did not feel confident in using technological devices [ 50 , 51 , 69 ]. Lack of confidence led patients to use technology on rare occasions and only if deemed necessary [ 36 ].

Other personal attributes

Lastly, this category summarises all personal traits of stakeholders that might impact the implementation of the intervention [ 30 ]. Younger, higher-educated patients embraced technology more readily [ 42 , 75 ]. Another enabler was telemonitoring improving patient empowerment, symptom awareness, and communication [ 34 , 36 , 51 , 69 ]. However, some found constant disease confrontation challenging [ 69 ]. Lastly, a personal connection with medical staff enabled telemedicine use [ 36 ].

This systematic review presents a comprehensive overview of the current status of telemedicine applications for patients with NMDs. The primary objective was to classify the identified interventions according to the dimensions of telemedicine. While some studies within this review explored the epidemiology of NMDs, and two interventions provided education for clinical staff, it's clear that certain aspects of telemedicine in public health remain under-studied.

E-health, encompassing health information, an electronic health record or physician order entries/treatment instructions, was comparatively underutilised, with only a subset of interventions included. Moreover, decision support systems were rarely investigated. The predominant focus of most interventions was on clinical support and remote data collection.

The second phase of the analysis concentrated on the implementation process, with a specific focus on identifying barriers and facilitators associated with both the intervention itself and the individuals involved. In comparison to traditional care, telemedicine often demonstrated a relative advantage. The high motivation demonstrated by NMD patients and their caregivers in integrating telemedicine into their care plan is a testament to the potential of telemedicine as a transformative force in healthcare.

Telemedicine was often perceived as a resource-saving, less fatiguing alternative, particularly offering increased accessibility for homebound patients. The lack of physical touch and reduced personal connections emerged as significant barriers. Additionally, the accessibility of technology played a pivotal role, as inadequate design hindered some patients from using telemedicine services. The acceptance and uptake of telemedicine services often depended on the readiness of patients and their caregivers to embrace and adapt to new digital solutions. Recognising the importance of patient empowerment, fostering the development of essential skills and confidence in utilising technology is crucial for enabling patients to actively engage in their healthcare.

Clinical and policy implications

The COVID-19 pandemic created an unprecedented opportunity for the development and implementation of telehealth. Disruptions in healthcare access, caused by social distancing and hygiene guidelines, led healthcare practitioners to expand telemedicine services to ensure the continuity of care [ 91 , 92 ]. This trend extended to the field of neuromuscular disease care as well [ 48 , 62 , 91 , 93 , 94 ]. The American Academy of Neurology's "Telehealth Position Statement" endorsed telemedicine, citing benefits such as improved access, reduced costs, and enhanced comfort, aligning with findings in this review [ 95 ].

Our findings further highlighted important considerations for the successful implementation of telemedicine. Firstly, it is essential to recognise that not all geographic locations are equally suited for telehealth. Remote areas with insufficient internet or cell phone coverage, as well as low-income households with a lack of digital technologies, may encounter difficulties in participating in telemedicine interventions [ 96 ]. Secondly, careful selection of the target population is vital, as the attitude and willingness of users significantly impact technology uptake [ 34 , 36 , 52 , 69 , 78 ]. The acceptance and efficacy of telemedicine interventions are inherently intertwined with diverse cultural attitudes towards healthcare and technology.

Therefore, understanding cultural factors is critical to discern how these variables may influence the successful integration of telehealth programs across diverse patient populations. A systematic analysis of cultural competence would provide valuable insights to refine and customise approaches, meeting the distinctive needs of diverse communities. Such considerations not only enhance the inclusivity of telemedicine but also contribute to its overall effectiveness and acceptance among a broad spectrum of individuals.

As the results have shown, it is vital to adapt telemedicine to the specific and evolving needs of patients with NMDs. These needs not only vary from patient to patient but also change over time as the disease progresses [ 5 ]. Therefore, when designing telemedicine technology for patients with NMDs, emphasis should be placed on adaptability, flexibility and accessibility [ 49 , 55 , 62 , 74 , 83 ].

Designing telemedicine technology that caters for the unique challenges faced by patients with physical disabilities and cognitive impairments is crucial for fostering inclusive healthcare [ 49 , 74 ]. User interfaces need to incorporate accessibility features, such as voice commands, large fonts, and intuitive navigation, to accommodate individuals with motor challenges or cognitive limitations. Additionally, instructions and information must be presented in various accessible formats, accommodating diverse learning needs [ 97 ].

Prioritising plain language and ensuring readability at lower literacy levels is essential. This approach not only makes instructions universally accessible but also empowers all patients to effectively participate in telemedicine interactions. By incorporating these considerations into the design, telemedicine can better serve the needs of patients with NMDs, promoting inclusivity and enhancing the overall effectiveness of healthcare delivery [ 97 ].

Health policies and regulatory frameworks play a significant role in influencing the development and adoption of telehealth practices. A nuanced understanding of these regulations, encompassing aspects such as licensure, reimbursement, and liability, is essential for gaining comprehensive insights into the complex landscape that shapes and governs telemedicine [ 96 ]. The intricate web of reimbursement policies directly influences the economic viability of telemedicine services, impacting both healthcare providers and patients. By navigating and understanding these policy and regulatory intricacies, stakeholders in the telemedicine ecosystem can strategically address and potentially overcome barriers, facilitating a more widespread and effective implementation of telehealth services [ 96 ].

This review reveals that telemedicine interventions for patients with NMDs exist but have yet to realise their full protentional. Firstly, the heavy focus on ALS care should be expanded to encompass all diagnostic groups within the NMD spectrum. Especially the high availability of mHealth applications, which could be seamlessly integrated into care plans. This integration has the potential to enhance continuity of care, simultaneously easing the burden on the healthcare system and reducing appointment frequency for patients [ 69 ].

The incorporation of long-term patient data through remote monitoring holds numerous advantages [ 98 , 99 ]. Continuous data collection could offer enhanced insights into disease progression, thereby improving disease management. Given the degenerative nature of most NMDs, there is a speculation that long-term data could help in detecting early signs of deterioration, facilitating quicker adaption of treatments. Furthermore, detailed information about disease progression could contribute to health prognosis, empowering both patients and healthcare professionals to better plan and coordinate care [ 98 , 99 ]. It is evident that the full benefits of telemonitoring remain undiscovered, making it an important and interesting area for future research. The exploration of these untapped potentials could significantly advance the effectiveness and scope of telemedicine in the context of NMDs.

Research and evaluation opportunities

The current telemedicine landscape yields promising results, particularly in its role in supporting rare disease research through the establishment of disease registries. These registries systematically collect patient data related to disease progression and treatment, forming the foundation for observational studies [ 100 , 101 ]. These studies offer critical insights into the management and progression of rare disease, contributing to evidence-based clinical decisions and facilitating the recruitment of participants for clinical trial.

National and international patient registries are pivotal for studying prevalence and incidence, enhancing our understanding of rare diseases like neuromuscular disorders [ 100 , 101 ]. The establishment of global patient registries becomes especially important for pooling data on rare diseases. International collaborations can help bridge the gap in research for understudied NMDs. By fostering collaboration and sharing data on a global scale, telemedicine-supported registries contribute significantly to advancing our understanding and management of rare diseases.

The results of our systematic review highlight a gap in the research on telemedicine for NMDs. Except for ALS, most NMDs are underrepresented in the current body of literature. Future research should include a more diverse range of diagnostic groups and undertake a comparative analysis of challenges and solutions. This would lead to a higher external validity and faster adaption of telemedicine solutions.

While teleconsultation and remote monitoring for NMDs are well described, other critical domains within telemedicine have received comparatively limited attention. These research gaps should be addressed in the future. Most importantly, implementation science has a critical role in the successful deployment of telemedicine interventions for NMDs. As seen in this systematic review studies, the focus needs to be on patients, caregivers, and health care practitioners, as well as the intervention itself.

It is noteworthy that there is underreporting of crucial aspects, such as the inner and outer settings, as well as the implementation process, in telemedicine interventions for NMDs. Additionally, there is need for research examining the impact of health policies and clinical guidelines on the adoption and implementation of telemedicine. The lack of implementation research has been described in the systematic review by Helleman et al., who analysed telemedicine for ALS patients [ 25 ]. Implementation science is needed to improve the efficiency and uptake of future telemedicine interventions for NMDs [ 102 ].

While our systematic review focused on highlighting the barriers and facilitators of telemedicine, we fully recognise the importance of addressing the validation challenges associated with digital health data. Future research and healthcare policies should emphasise the need for robust validation processes to ensure the reliability and clinical relevance of digital outcomes in telemedicine interventions.

Limitations

Despite an extensive search string, additional search terms might have yielded more results, especially considering synonyms for neuromuscular diseases. A more specific search for individual diagnostic groups would have been more inclusive, but the sheer number of NMDs made this unfeasible.

The literature databases used represent common sources of clinical evidence, but they may not comprehensively cover health policies, management, and health education related to NMDs, which might be found in other types of databases.

The absence of experimental study designs in the individual studies was notable, with most included studies being cross-sectional or observational. However, as this review aims to provide an overview of interventions, this description suffices.

The majority of included studies are from high-income countries, and the extent of telemedicine utilisation in low- and middle-income countries remains unclear. The variation in target population size and time horizon in NMD research reflects the complexity and rarity of these conditions, suggesting a need for longer follow-up times in future studies to better describe long-term outcomes.

This systematic review offers a comprehensive view of the telemedicine landscape in the context of NMDs. While domains like teleconsultation and telemonitoring have received extensive attention and reporting in the literature, other critical domains, such as decision support tools and informational support, are notably lacking in research and documentation. To further understand, develop and implement telemedicine solutions and to close existing gaps in NMD-specific healthcare provision, policies and guidelines are needed. By actively integrating telemedicine into existing healthcare plans and maintaining a commitment to ongoing updates and improvements, healthcare systems can optimise care delivery, enhance patient outcomes, and ensure that individuals with NMDs receive the high-quality care they deserve. In addition, more high-quality studies are needed to close research gaps concerning the implementation process of telemedicine and prove the respective efficiency and effectiveness in the long run.

Availability of data and materials

Due to the nature of the paper, no primary data was generated. All data analysed during this study are included in this published article and its supplementary information files.

Abbreviations

Amyotrophic lateral sclerosis

Anti-synthetase syndrome

Becker muscular dystrophy

Consolidated framework for implementation research

Congenital myopathy

Congenital muscular dystrophy

Charcot-Marie-tooth disease

Dermatomyositis

Duchenne muscular dystrophy

Emery-Dreifuss muscular dystrophy

Facioscapulohumeral muscular dystrophy

Hereditary spastic paraparesis

International classification of diseases

Juvenile dermatomyositis

Lambert-Eaton-myasthenic-syndrome

Limb-Girdle muscular dystrophy

Myotonic dystrophy

Muscular dystrophy

Muscular dystrophy association

Myofibrillar myopathies

Myasthenia gravis

Muscle glycogenosis

Necrotizing myositis

Neuromuscular disease

Overlap myositis

Pompe disease

Polymyositis

Post-Polio syndrome

Randomized controlled trial

Spinal and bulbar muscular atrophy

Spinal muscular atrophy

Transthyretin familial amyloid polyneuropathy

Mary P, Servais L, Vialle R. Neuromuscular diseases: diagnosis and management. Orthopaed Traumatol: Surg Res. 2018;104(1, Supplement):89-S95. Available from: https://www.sciencedirect.com/science/article/pii/S187705681730333X .

Google Scholar  

Aitken M, Mercer EJ, Mckemey A. Understanding neuromuscular disease care: current state and future prospects. IQVIA Institute; 2018.

Bonne G, Rivier F, Hamroun D. The 2018 version of the gene table of monogenic neuromuscular disorders (nuclear genome). Neuromusc Disord: NMD 2017; 27(12). Available from: https://pubmed.ncbi.nlm.nih.gov/29961566/ .

Deenen JC, Horlings CG, Verschuuren JJ, Verbeek AL, van Engelen BG. The epidemiology of neuromuscular disorders: a comprehensive overview of the literature. J Neuromuscul Dis. 2015;2(1):73–85.

Article   PubMed   Google Scholar  

Feldman EL, Russell JW, Löscher WN, Grisold W, Meng S. Atlas of Neuromuscular diseases: a practical guideline. 3rd ed. Cham: Springer International Publishing; Imprint Springer; 2021. Springer eBook Collection.

Book   Google Scholar  

Craig BM, Hartman JD, Owens MA, Brown DS. Prevalence and losses in quality-adjusted life years of child health conditions: a burden of disease analysis. Matern Child Health J. 2016;20(4):862–9.

Article   PubMed   PubMed Central   Google Scholar  

Birnkrant DJ, Bushby K, Bann CM, Alman BA, Apkon SD, Blackwell A, et al. Diagnosis and management of Duchenne muscular dystrophy, part 2: respiratory, cardiac, bone health, and orthopaedic management. Lancet Neurol. 2018;17(4):347–61. Available from: https://pubmed.ncbi.nlm.nih.gov/29395990/ .

Birnkrant DJ, Bushby K, Bann CM, Apkon SD, Blackwell A, Brumbaugh D, et al. Diagnosis and management of Duchenne muscular dystrophy, part 1: diagnosis, and neuromuscular, rehabilitation, endocrine, and gastrointestinal and nutritional management. Lancet Neurol. 2018;17(3):251–67. Available from: https://pubmed.ncbi.nlm.nih.gov/29395989/ .

Hulisz D. Amyotrophic lateral sclerosis: disease state overview. The American journal of managed care. 2018;24(15 Suppl):S320–6. Available from: https://pubmed.ncbi.nlm.nih.gov/30207670/ .

PubMed   Google Scholar  

Khan F, Ng L, Amatya B, Brand C, Turner-Stokes L. Multidisciplinary care for Guillain-Barré syndrome. Europ J Phys Rehabil Med. 2011;47(4):607–12. Available from: https://pubmed.ncbi.nlm.nih.gov/21912364/ .

CAS   Google Scholar  

Mercuri E, Sumner CJ, Muntoni F, Darras BT, Finkel RS. Spinal muscular atrophy. Nat Rev Dis Prim. 2022;8(1):52. Available from: https://pubmed.ncbi.nlm.nih.gov/35927425/ .

Birnkrant DJ, Bushby K, Bann CM, Apkon SD, Blackwell A, Colvin MK, et al. Diagnosis and management of Duchenne muscular dystrophy, part 3: primary care, emergency management, psychosocial care, and transitions of care across the lifespan. Lancet Neurol. 2018;17(5):445–55.

Johnson NE. Myotonic muscular dystrophies. Continuum (Minneap Minn). 2019;25(6):1682–95.

Brandt M, Johannsen L, Inhestern L, Bergelt C. Parents as informal caregivers of children and adolescents with spinal muscular atrophy: a systematic review of quantitative and qualitative data on the psychosocial situation, caregiver burden, and family needs. Orphanet J Rare Dis. 2022;17(1):274. Available from: https://pubmed.ncbi.nlm.nih.gov/35854387/ .

Landfeldt E, Edström J, Buccella F, Kirschner J, Lochmüller H. Duchenne muscular dystrophy and caregiver burden: a systematic review. Dev Med Child Neurol. 2018;60(10):987–96.

Tramonti F, Bonfiglio L, Bongioanni P, Belviso C, Fanciullacci C, Rossi B, et al. Caregiver burden and family functioning in different neurological diseases. Psychol Health Med. 2019;24(1):27–34.

de Wit J, Bakker LA, van Groenestijn AC, van den Berg LH, SchrÖder CD, Visser-Meily JMA, et al. Caregiver burden in amyotrophic lateral sclerosis: a systematic review. Palliat Med. 2018;32(1):231–45. Available from: https://pubmed.ncbi.nlm.nih.gov/28671483/ .

Hamine S, Gerth-Guyette E, Faulx D, Green BB, Ginsburg AS. Impact of mHealth chronic disease management on treatment adherence and patient outcomes: a systematic review. J Med Internet Res. 2015;17(2):e52. Available from: https://pubmed.ncbi.nlm.nih.gov/25803266/ .

Hanlon P, Daines L, Campbell C, McKinstry B, Weller D, Pinnock H. Telehealth interventions to support self-management of long-term conditions: a systematic metareview of diabetes, heart failure, asthma, chronic obstructive pulmonary disease, and cancer. J Med Internet Res. 2017;19(5):e172.

Whitehead L, Seaton P. The effectiveness of self-management mobile phone and tablet apps in long-term condition management: a systematic review. J Med Internet Res. 2016;18(5):e97.

Bashshur RL, Shannon GW, Smith BR, Alverson DC, Antoniotti N, Barsan WG, et al. The empirical foundations of telemedicine interventions for chronic disease management. Telemed J e-health. 2014;20(9):769–800.

Eze ND, Mateus C, Cravo Oliveira Hashiguchi T. Telemedicine in the OECD: An umbrella review of clinical and cost-effectiveness, patient experience and implementation. PLos One. 2020;15(8):0237585.

Article   Google Scholar  

León-Salas B, González-Hernández Y, Infante-Ventura D, de Armas-Castellano A, García-García J, García-Hernández M, et al. Telemedicine for neurological diseases: a systematic review and meta-analysis. Eur J Neurol. 2023;30(1):241–54.

Wang H, Yuan X, Wang J, Sun C, Wang G. Telemedicine maybe an effective solution for management of chronic disease during the COVID-19 epidemic. Prim Health Care Res Dev. 2021;22:e48.

Helleman J, Kruitwagen ET, van den Berg LH, Visser-Meily JMA, Beelen A. The current use of telehealth in ALS care and the barriers to and facilitators of implementation: a systematic review. Amyotroph Lateral Scler Frontotemporal Degener. 2020;21(3–4):167–82.

Article   CAS   PubMed   Google Scholar  

Bashshur R, Shannon G, Krupinski E, Grigsby J. The taxonomy of telemedicine. Telemed J E-health 2011;17(6). Available from: https://pubmed.ncbi.nlm.nih.gov/21718114/ .

Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ. 2009;339:b2535.

Muscular Dystrophy Association. Find a neuromuscular disease | Muscular Dystrophy Association; 2018 [cited 2022 Mar 28.526Z]. Available from: https://www.mda.org/disease/list .

ICD-11 for Mortality and Morbidity Statistics; 2022 [cited 2022 Mar 31.331Z]. Available from: https://icd.who.int/browse11/l-m/en .

Damschroder LJ, Aron DC, Keith RE, Kirsh, SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009;4. Available from: https://pubmed.ncbi.nlm.nih.gov/19664226/ .

Sterne JAC, Savović J, Page MJ, Elbers RG, Blencowe NS, Boutron I, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ. 2019;366:l4898.

Sterne JA, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355:i4919.

Critical Appraisal Tools | JBI; 2022 [cited 2022 Sep 8]. Available from: https://jbi.global/critical-appraisal-tools .

Ando H, Ashcroft-Kelso H, Halhead R, Chakrabarti B, Young CA, Cousins R, et al. Experience of telehealth in people with motor neurone disease using noninvasive ventilation. Disabil Rehabil Assist Technol. 2021;16(5):490–6.

Ando H, Ashcroft-Kelso H, Halhead R, Young CA, Chakrabarti B, Levene P, et al. Incorporating self-reported questions for telemonitoring to optimize care of patients with MND on noninvasive ventilation (MND OptNIVent). Amyotroph Lateral Scler Frontotemporal Degener. 2019;20(5–6):336–47.

Hobson E, Baird W, Bradburn M, Cooper C, Mawson S, Quinn A, et al. Process evaluation and exploration of telehealth in motor neuron disease in a UK specialist centre. BMJ Open. 2019;9(10):e028526. Available from: https://pubmed.ncbi.nlm.nih.gov/31640994/ .

Hobson EV, Baird WO, Bradburn M, Cooper C, Mawson S, Quinn A, et al. Using telehealth in motor neuron disease to increase access to specialist multidisciplinary care: a UK-based pilot and feasibility study. BMJ Open. 2019;9(10):e028525. Available from: https://bmjopen.bmj.com/content/9/10/e028525 .

Martínez O, Amayra I, López-Paz JF, Lázaro E, Caballero P, García I, et al. Effects of teleassistance on the quality of life of people with rare neuromuscular diseases according to their degree of disability. Front Psychol. 2021;12:637413.

Martínez O, Jometón A, Pérez M, Lázaro E, Amayra I, López-Paz JF et al. Effectiveness of teleassistance at improving quality of life in people with neuromuscular diseases. Spanish J Psychol 2014;17. Available from: https://pubmed.ncbi.nlm.nih.gov/26055393/ .

Sobierajska-Rek A, Mański Ł, Jabłońska-Brudło J, Śledzińska K, Ucińska A, Wierzba J. Establishing a telerehabilitation program for patients with Duchenne muscular dystrophy in the COVID-19 pandemic. Wien Klin Wochenschr. 2020;133(7–8):344–50. Available from: https://link.springer.com/article/10.1007/s00508-020-01786-8 .

PubMed   PubMed Central   Google Scholar  

Wasilewska E, Sobierajska-Rek A, Małgorzewicz S, Soliński M, Szalewska D, Jassem E. Is it possible to have home e-monitoring of pulmonary function in our patients with duchenne muscular dystrophy in the covid-19 pandemic?-A one center pilot study. Int J Environ Res Public Health 2021;18(17).

Abdulla S, Vielhaber S, Machts J, Heinze H-J, Dengler R, Petri S. Information needs and information-seeking preferences of ALS patients and their carers. Amyotroph Lateral Scler Frontotemporal Degener. 2014;15(7–8):505–12.

Ambrosini A, Calabrese D, Avato FM, Catania F, Cavaletti G, Pera MC, et al. The Italian neuromuscular registry: a coordinated platform where patient organizations and clinicians collaborate for data collection and multiple usage. Orphanet J Rare Dis. 2018;13(1):176.

Anil R, Kumar A, Alaparthi S, Sharma A, Nye JL, Roy B, et al. Exploring outcomes and characteristics of myasthenia gravis: Rationale, aims and design of registry - The EXPLORE-MG registry. J Neurol Sci. 2020;414:116830.

Benjamin JC, Groner J, Walton J, Noritz G, Gascon GM, Mahan JD. A blended curriculum to improve resident physical exam skills for patients with neuromuscular disability. MedEdPORTAL. 2019;15:10792.

Berry JD, Paganoni S, Carlson K, Burke K, Weber H, Staples P, et al. Design and results of a smartphone-based digital phenotyping study to quantify ALS progression. Ann Clin Transl Neurol. 2019;6(5):873–81.

Bettio C, Salsi V, Orsini M, Calanchi E, Magnotta L, Gagliardelli L, et al. The Italian National Registry for FSHD: an enhanced data integration and an analytics framework towards smart health care and precision medicine for a rare disease. Orphanet J Rare Dis. 2021;16(1):470.

Capozzo R, Zoccolella S, Musio M, Barone R, Accogli M, Logroscino G. Telemedicine is a useful tool to deliver care to patients with Amyotrophic Lateral Sclerosis during COVID-19 pandemic: results from Southern Italy. Amyotroph Lateral Scler Frontotemporal Degener. 2020;21(7–8):542–8.

Cesareo A, Nido SA, Biffi E, Gandossini S, D’Angelo MG, Aliverti A. A Wearable device for breathing frequency monitoring: a pilot study on patients with muscular dystrophy. Sensors (Basel) 2020;20(18).

Climans SA, Piechowicz C, Koopman WJ, Venance SL. Survey of Canadian myotonic dystrophy patients’ access to computer technology. Can J Neurol Sci. 2017;44(5):567–71.

Geronimo A, Simmons Z. Evaluation of remote pulmonary function testing in motor neuron disease. Amyotroph Lateral Scler Frontotemporal Degener. 2019;20(5–6):348–55.

Geronimo A, Wright C, Morris A, Walsh S, Snyder B, Simmons Z. Incorporation of telehealth into a multidisciplinary ALS Clinic: feasibility and acceptability. Amyotroph Lateral Scler Frontotemporal Degener. 2017;18(7–8):555–61.

Grigull L, Lechner W, Petri S, Kollewe K, Dengler R, Mehmecke S, et al. Diagnostic support for selected neuromuscular diseases using answer-pattern recognition and data mining techniques: a proof of concept multicenter prospective trial. BMC Med Inform Decis Mak. 2016;16:31.

Hooshmand S, Cho J, Singh S, Govindarajan R. Satisfaction of telehealth in patients with established neuromuscular disorders. Front Neurol. 2021;12:667813.

James N, Power E, Hogden A, Vucic S. Patients’ perspectives of multidisciplinary home-based e-Health service delivery for motor neurone disease. Disabil Rehabil Assist Technol. 2019;14(7):737–43.

Malek AM, Stickler DE, Antao VC, Horton DK. The National ALS Registry: a recruitment tool for research. Muscle Nerve. 2014;50(5):830–4.

Pulley MT, Brittain R, Hodges W, Frazier C, Miller L, Matyjasik-Liggett M, et al. Multidisciplinary amyotrophic lateral sclerosis telemedicine care: The store and forward method. Muscle Nerve. 2019;59(1):34–9.

Roy AJ, van den Bergh P, van Damme P, Doggen K, van Casteren V. Early stages of building a rare disease registry, methods and 2010 data from the Belgian Neuromuscular Disease Registry (BNMDR). Acta Neurol Belg. 2015;115(2):97–104.

Sobierajska-Rek A, Mański Ł, Jabłońska-Brudło J, Śledzińska K, Wasilewska E, Szalewska D. Respiratory telerehabilitation of boys and young men with Duchenne muscular dystrophy in the COVID-19 Pandemic. Int J Environ Res Public Health 2021;18(12).

Spiliopoulos KC, Kasdaglis N, Veltsista D, Lykouras D, Lagadinou M, Chroni E. Teleneurology in a center for neuromuscular diseases during the COVID-19 pandemic. Acta Neurol Belg. 2022;122(3):721–4.

Tawfik EA, van Alfen N, Cartwright MS, Inkpen P, Kerasnoudis A, Lieba-Samal D, et al. Virtual neuromuscular ultrasound courses during COVID-19 pandemic: Leveraging technology to enhance learning opportunities. Muscle Nerve. 2021;65(1):29–33.

Vasta R, Moglia C, D’Ovidio F, Di Pede F, de Mattei F, Cabras S, et al. Telemedicine for patients with amyotrophic lateral sclerosis during COVID-19 pandemic: an Italian ALS referral center experience. Amyotroph Lateral Scler Frontotemporal Degener. 2021;22(3–4):308–11.

White MK, Leffler M, Rychlec K, Jones C, McSherry C, Walker L, et al. Adapting traditional content validation methods to fit purpose: an example with a novel video assessment and training materials in Duchenne muscular dystrophy (DMD). Qual Life Res. 2019;28(11):2979–88.

Menon D, Alnajjar S, Barnett C, Vijayan J, Katzberg H, Fathi D, et al. Telephone consultation for myasthenia gravis care during the COVID-19 pandemic: Assessment of a novel virtual myasthenia gravis index. Muscle Nerve. 2021;63(6):831–6.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Portaro S, Calabrò RS, Bramanti P, Silvestri G, Torrisi M, Conti-Nibali V, et al. Telemedicine for Facio-Scapulo-Humeral Muscular Dystrophy: A multidisciplinary approach to improve quality of life and reduce hospitalization rate? Disabil Health J. 2017;11(2):306–9.

Trucco F, Pedemonte M, Racca F, Falsaperla R, Romano C, Wenzel A, et al. Tele-monitoring in paediatric and young home-ventilated neuromuscular patients: A multicentre case-control trial. J Telemed Telecare. 2019;25(7):414–24.

Contesse MG, Sapp ATL, Apkon SD, Lowes LP, Dalle Pazze L, Leffler MG. Reliability and construct validity of the Duchenne Video Assessment. Muscle Nerve. 2021;64(2):180–9.

Garuti G, Bagatti S, Verucchi E, Massobrio M, Spagnolatti L, Vezzani G, et al. Pulmonary rehabilitation at home guided by telemonitoring and access to healthcare facilities for respiratory complications in patients with neuromuscular disease. Eur J Phys Rehabil Med. 2013;49(1):51–7. Available from: https://pubmed.ncbi.nlm.nih.gov/22820817/ .

CAS   PubMed   Google Scholar  

Helleman J, van Eenennaam R, Kruitwagen ET, Kruithof WJ, Slappendel MJ, van den Berg LH, et al. Telehealth as part of specialized ALS care: feasibility and user experiences with “ALS home-monitoring and coaching.” Amyotroph Lateral Scler Frontotemporal Degener. 2020;21(3–4):183–92.

Kamei T, Yamamoto Y, Kanamori T, Nakayama Y, Porter SE. Detection of early-stage changes in people with chronic diseases: A telehome monitoring-based telenursing feasibility study. Nurs Health Sci. 2018;20(3):313–22.

Levi BH, Simmons Z, Hanna C, Brothers A, Lehman E, Farace E, et al. Advance care planning for patients with amyotrophic lateral sclerosis. Amyotroph Lateral Scler Frontotemporal Degener. 2017;18(5–6):388–96.

Longinetti E, Regodón Wallin A, Samuelsson K, Press R, Zachau A, Ronnevi L-O, et al. The Swedish motor neuron disease quality registry. Amyotroph Lateral Scler Frontotemporal Degener. 2018;19(7–8):528–37.

de Marchi F, Sarnelli MF, Serioli M, de Marchi I, Zani E, Bottone N, et al. Telehealth approach for amyotrophic lateral sclerosis patients: the experience during COVID-19 pandemic. Acta Neurol Scand. 2021;143(5):489–96.

McErlane F, Davies EH, Ollivier C, Mayhew A, Anyanwu O, Harbottle V, et al. Wearable technologies for children with chronic illnesses: an exploratory approach. Ther Innov Regul Sci. 2021;55(4):799–806.

Naveen R, Sundaram TG, Agarwal V, Gupta L. Teleconsultation experience with the idiopathic inflammatory myopathies: a prospective observational cohort study during the COVID-19 pandemic. Rheumatol Int. 2020;41(1):67–76.

Newton J, Jayaprakash K, Glasmacher SA, McEleney A, Bethell A, Fraser E, et al. Excellent reliability of the ALSFRS-R administered via videoconferencing: a study of people with motor neuron disease in Scotland. J Neurol Sci. 2020;416:116991.

Paganoni S, van de Rijn M, Drake K, Burke K, Doyle M, Ellrodt AS, et al. Adjusted cost analysis of video televisits for the care of people with amyotrophic lateral sclerosis. Muscle Nerve. 2019;60(2):147–54.

Roman A, Baylor C, Johnson L, Barton M. Expanding Availability of Speech-Generating Device Evaluation and Treatment to People With Amyotrophic Lateral Sclerosis (pALS) Through Telepractice: Perspectives of pALS and Communication Partners. Am J Speech Lang Pathol. 2021;30(5):2098–114.

Selkirk SM, Washington MO, McClellan F, Flynn B, Seton JM, Strozewski R. Delivering tertiary centre specialty care to ALS patients via telemedicine: a retrospective cohort analysis. Amyotroph Lateral Scler Frontotemporal Degener. 2017;18(5–6):324–32.

van Eijk RPA, Bakers JNE, Bunte TM, de Fockert AJ, Eijkemans MJC, van den Berg LH. Accelerometry for remote monitoring of physical activity in amyotrophic lateral sclerosis: a longitudinal cohort study. J Neurol. 2019;266(10):2387–95.

Vitacca M, Comini L, Assoni G, Fiorenza D, Gilè S, Bernocchi P, et al. Tele-assistance in patients with amyotrophic lateral sclerosis: long term activity and costs. Disabil Rehabil Assist Technol. 2012;7(6):494–500.

Alexanderson H, Munters LA, Dastmalchi M, Loell I, Heimbürger M, Opava CH, et al. Resistive home exercise in patients with recent-onset polymyositis and dermatomyositis—a randomized controlled single-blinded study with a 2-year followup. J Rheumatol. 2014;41(6):1124–32.

Astley C, Sieczkowska SM, Marques IG, Ihara BP, Lindoso L, Lavorato SSM, et al. Home-based exercise program for adolescents with juvenile dermatomyositis quarantined during COVID-19 pandemic: a mixed methods study. Pediatr Rheumatol Online J. 2021;19(1):159.

Bankolé L-C, Millet GY, Temesi J, Bachasson D, Ravelojaona M, Wuyam B, et al. Safety and efficacy of a 6-month home-based exercise program in patients with facioscapulohumeral muscular dystrophy: A randomized controlled trial. Medicine (Baltimore). 2016;95(31): e4497.

Christodoulou G, Gennings C, Hupf J, Factor-Litvak P, Murphy J, Goetz RR, et al. Telephone based cognitive-behavioral screening for frontotemporal changes in patients with amyotrophic lateral sclerosis (ALS). Amyotroph Lateral Scler Frontotemporal Degener. 2016;17(7–8):482–8.

Wills AM, Garry J, Hubbard J, Mezoian T, Breen CT, Ortiz-Miller C et al. Nutritional counseling with or without mobile health technology: a randomized open-label standard-of-care-controlled trial in ALS. BMC Neurol 2019; 19.

de Wit J, Beelen A, Drossaert CHC, Kolijn R, van den Berg LH, SchrÖder CD, et al. Blended psychosocial support for partners of patients with ALS and PMA: results of a randomized controlled trial. Amyotroph Lateral Scler Frontotemporal Degener. 2020;21(5–6):344–54.

Palumbo A, Ielpo N, Calabrese B, Corchiola D, Garropoli R, Gramigna V et al. SIMpLE: A mobile cloud-based system for health monitoring of people with ALS. Sensors (Basel) 2021; 21(21).

Ricci G, Baldanzi S, Seidita F, Proietti C, Carlini F, Peviani S, et al. A mobile app for patients with Pompe disease and its possible clinical applications. Neuromusc Disord: NMD. 2018;28(6):471–5.

Scalvini S, Bernocchi P, Zanelli E, Comini L, Vitacca M. Maugeri centre for telehealth and telecare: a real-life integrated experience in chronic patients. J Telemed Telecare. 2018;24(7):500–7.

El-Hassar L, Amara A, Sanson B, Lacatus O, Amir Belhouchet A, Kroneman M, et al. Telemedicine in neuromuscular diseases during covid-19 pandemic: ERN-NMD European survey. J Neuromuscul Dis. 2022;Preprint(Preprint):1–12.

Monaghesh E, Hajizadeh A. The role of telehealth during COVID-19 outbreak: a systematic review based on current evidence. BMC Public Health. 2020;20(1):1193.

Bombaci A, Abbadessa G, Trojsi F, Leocani L, Bonavita S, Lavorgna L. Telemedicine for management of patients with amyotrophic lateral sclerosis through COVID-19 tail. Neurol Sci. 2021;42(1):9–13.

Giannotta M, Petrelli C, Pini A. Telemedicine applied to neuromuscular disorders: focus on the COVID-19 pandemic era. Acta myologica: myopathies and cardiomyopathies. 2022;41(1):30–6. Available from: https://pubmed.ncbi.nlm.nih.gov/35465343/ .

Hatcher-Martin JM, Busis NA, Cohen BH, Wolf RA, Jones EC, Anderson ER, et al. American academy of neurology telehealth position statement. Neurology. 2021;97(7):334–9. Available from: https://n.neurology.org/content/97/7/334 .

Kruse CS, Williams K, Bohls J, Shamsi W. Telemedicine and health policy: a systematic review. Health Policy Technol. 2021;10(1):209–29.

Phuong J, Ordóñez P, Cao J, Moukheiber M, Moukheiber L, Caspi A, et al. Telehealth and digital health innovations: a mixed landscape of access. PLOS Digit Health. 2023;2(12):e0000401.

Beswick E, Fawcett T, Hassan Z, et al. A systematic review of digital technology to evaluate motor function and disease progression in motor neuron disease. J Neurol. 2022;269:6254–68.

Serrano LP, Maita KC, Avila FR, Torres-Guzman RA, Garcia JP, Eldaly AS, Haider CR, Felton CL, Paulson MR, Maniaci MJ, Forte AJ. Benefits and challenges of remote patient monitoring as perceived by health care practitioners: a systematic review. Permanente J. 2023;27(4):100.

Lacaze P, Millis N, Fookes M, Zurynski Y, Jaffe A, Bellgard M, Winship I, McNeil J, Bittles AH. Rare disease registries: a call to action. Intern Med J. 2017;47(9):1075–9.

Kölker S, Gleich F, Mütze U, Opladen T. Rare disease registries are key to evidence-based personalized medicine: highlighting the European experience. Front Endocrinol. 2022;4(13):832063.

Peters DH, Adam T, Alonge O, Akua Agyepong I, Tran N. Implementation research: what it is and how to do it. BMJ 2013;347. Available from: https://www.bmj.com/content/347/bmj.f6753.full

Download references

Acknowledgements

Not applicable.

The authors did not receive any funding for this paper.

Author information

Authors and affiliations.

Chair of Healthcare Management and Health Services Research, University of Bayreuth, Bayreuth, Germany

Deniz Senyel, Katja Senn & Klaus Nagels

Chair of Digital Health and Innovation, Department of Public Health, School of Psychology and Public Health, La Trobe University, Melbourne, Australia

Deniz Senyel & James Boyd

You can also search for this author in PubMed   Google Scholar

Contributions

DS and KS prepared the study protocol as well as performed the literature search and study selection. DS conducted the data extraction and analysis. The report was written by DS with contributions by KS. JB and KN supervised the complete process. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Deniz Senyel .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., supplementary material 2., supplementary material 3., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Senyel, D., Senn, K., Boyd, J. et al. A systematic review of telemedicine for neuromuscular diseases: components and determinants of practice. BMC Digit Health 2 , 17 (2024). https://doi.org/10.1186/s44247-024-00078-9

Download citation

Received : 05 June 2023

Accepted : 22 February 2024

Published : 09 May 2024

DOI : https://doi.org/10.1186/s44247-024-00078-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Telemedicine
• Neuromuscular diseases
• Digitalization

BMC Digital Health

ISSN: 2731-684X

• Submission enquiries: [email protected]
• General enquiries: [email protected]

• Open access
• Published: 13 May 2024

Sexual and reproductive health implementation research in humanitarian contexts: a scoping review

• Alexandra Norton 1 &
• Hannah Tappis 2  

Reproductive Health volume  21 , Article number:  64 ( 2024 ) Cite this article

Metrics details

Meeting the health needs of crisis-affected populations is a growing challenge, with 339 million people globally in need of humanitarian assistance in 2023. Given one in four people living in humanitarian contexts are women and girls of reproductive age, sexual and reproductive health care is considered as essential health service and minimum standard for humanitarian response. Despite growing calls for increased investment in implementation research in humanitarian settings, guidance on appropriate methods and analytical frameworks is limited.

A scoping review was conducted to examine the extent to which implementation research frameworks have been used to evaluate sexual and reproductive health interventions in humanitarian settings. Peer-reviewed papers published from 2013 to 2022 were identified through relevant systematic reviews and a literature search of Pubmed, Embase, PsycInfo, CINAHL and Global Health databases. Papers that presented primary quantitative or qualitative data pertaining to a sexual and reproductive health intervention in a humanitarian setting were included.

Seven thousand thirty-six unique records were screened for inclusion, and 69 papers met inclusion criteria. Of these, six papers explicitly described the use of an implementation research framework, three citing use of the Consolidated Framework for Implementation Research. Three additional papers referenced other types of frameworks used in their evaluation. Factors cited across all included studies as helping the intervention in their presence or hindering in their absence were synthesized into the following Consolidated Framework for Implementation Research domains: Characteristics of Systems, Outer Setting, Inner Setting, Characteristics of Individuals, Intervention Characteristics, and Process.

This review found a wide range of methodologies and only six of 69 studies using an implementation research framework, highlighting an opportunity for standardization to better inform the evidence for and delivery of sexual and reproductive health interventions in humanitarian settings. Increased use of implementation research frameworks such as a modified Consolidated Framework for Implementation Research could work toward both expanding the evidence base and increasing standardization.

Plain English summary

Three hundred thirty-nine million people globally were in need of humanitarian assistance in 2023, and meeting the health needs of crisis-affected populations is a growing challenge. One in four people living in humanitarian contexts are women and girls of reproductive age, and provision of sexual and reproductive health care is considered to be essential within a humanitarian response. Implementation research can help to better understand how real-world contexts affect health improvement efforts. Despite growing calls for increased investment in implementation research in humanitarian settings, guidance on how best to do so is limited. This scoping review was conducted to examine the extent to which implementation research frameworks have been used to evaluate sexual and reproductive health interventions in humanitarian settings. Of 69 papers that met inclusion criteria for the review, six of them explicitly described the use of an implementation research framework. Three used the Consolidated Framework for Implementation Research, a theory-based framework that can guide implementation research. Three additional papers referenced other types of frameworks used in their evaluation. This review summarizes how factors relevant to different aspects of implementation within the included papers could have been organized using the Consolidated Framework for Implementation Research. The findings from this review highlight an opportunity for standardization to better inform the evidence for and delivery of sexual and reproductive health interventions in humanitarian settings. Increased use of implementation research frameworks such as a modified Consolidated Framework for Implementation Research could work toward both expanding the evidence base and increasing standardization.

Peer Review reports

Over the past few decades, the field of public health implementation research (IR) has grown as a means by which the real-world conditions affecting health improvement efforts can be better understood. Peters et al. put forward the following broad definition of IR for health: “IR is the scientific inquiry into questions concerning implementation – the act of carrying an intention into effect, which in health research can be policies, programmes, or individual practices (collectively called interventions)” [ 1 ].

As IR emphasizes real-world circumstances, the context within which a health intervention is delivered is a core consideration. However, much IR implemented to date has focused on higher-resource settings, with many proposed frameworks developed with particular utility for a higher-income setting [ 2 ]. In recognition of IR’s potential to increase evidence across a range of settings, there have been numerous reviews of the use of IR in lower-resource settings as well as calls for broader use [ 3 , 4 ]. There have also been more focused efforts to modify various approaches and frameworks to strengthen the relevance of IR to low- and middle-income country settings (LMICs), such as the work by Means et al. to adapt a specific IR framework for increased utility in LMICs [ 2 ].

Within LMIC settings, the centrality of context to a health intervention’s impact is of particular relevance in humanitarian settings, which present a set of distinct implementation challenges [ 5 ]. Humanitarian responses to crisis situations operate with limited resources, under potential security concerns, and often under pressure to relieve acute suffering and need [ 6 ]. Given these factors, successful implementation of a particular health intervention may require different qualities than those that optimize intervention impact under more stable circumstances [ 7 ]. Despite increasing recognition of the need for expanded evidence of health interventions in humanitarian settings, the evidence base remains limited [ 8 ]. Furthermore, despite its potential utility, there is not standardized guidance on IR in humanitarian settings, nor are there widely endorsed recommendations for the frameworks best suited to analyze implementation in these settings.

Sexual and reproductive health (SRH) is a core aspect of the health sector response in humanitarian settings [ 9 ]. Yet, progress in addressing SRH needs has lagged far behind other services because of challenges related to culture and ideology, financing constraints, lack of data and competing priorities [ 10 ]. The Minimum Initial Service Package (MISP) for SRH in Crisis Situations is the international standard for the minimum set of SRH services that should be implemented in all crisis situations [ 11 ]. However, as in other areas of health, there is need for expanded evidence for planning and implementation of SRH interventions in humanitarian settings. Recent systematic reviews of SRH in humanitarian settings have focused on the effectiveness of interventions and service delivery strategies, as well as factors affecting utilization, but have not detailed whether IR frameworks were used [ 12 , 13 , 14 , 15 ]. There have also been recent reviews examining IR frameworks used in various settings and research areas, but none have explicitly focused on humanitarian settings [ 2 , 16 ].

Given the need for an expanded evidence base for SRH interventions in humanitarian settings and the potential for IR to be used to expand the available evidence, a scoping review was undertaken. This scoping review sought to identify IR approaches that have been used in the last ten years to evaluate SRH interventions in humanitarian settings.

This review also sought to shed light on whether there is a need for a common framework to guide research design, analysis, and reporting for SRH interventions in humanitarian settings and if so, if there are any established frameworks already in use that would be fit-for-purpose or could be tailored to meet this need.

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) extension for scoping reviews was utilized to guide the elements of this review [ 17 ]. The review protocol was retrospectively registered with the Open Science Framework ( https://osf.io/b5qtz ).

Search strategy

A two-fold search strategy was undertaken for this review, which covered the last 10 years (2013–2022). First, recent systematic reviews pertaining to research or evaluation of SRH interventions in humanitarian settings were identified through keyword searches on PubMed and Google Scholar. Four relevant systematic reviews were identified [ 12 , 13 , 14 , 15 ] Table 1 .

Second, a literature search mirroring these reviews was conducted to identify relevant papers published since the completion of searches for the most recent review (April 2017). Additional file 1 includes the search terms that were used in the literature search [see Additional file 1 ].

The literature search was conducted for papers published from April 2017 to December 2022 in the databases that were searched in one or more of the systematic reviews: PubMed, Embase, PsycInfo, CINAHL and Global Health. Searches were completed in January 2023 Table 2 .

Two reviewers screened each identified study for alignment with inclusion criteria. Studies in the four systematic reviews identified were considered potentially eligible if published during the last 10 years. These papers then underwent full-text review to confirm satisfaction of all inclusion criteria, as inclusion criteria were similar but not fully aligned across the four reviews.

Literature search results were exported into a citation manager (Covidence), duplicates were removed, and a step-wise screening process for inclusion was applied. First, all papers underwent title and abstract screening. The remaining papers after abstract screening then underwent full-text review to confirm satisfaction of all inclusion criteria. Title and abstract screening as well as full-text review was conducted independently by both authors; disagreements after full-text review were resolved by consensus.

Data extraction and synthesis

The following content areas were summarized in Microsoft Excel for each paper that met inclusion criteria: publication details including author, year, country, setting [rural, urban, camp, settlement], population [refugees, internally displaced persons, general crisis-affected], crisis type [armed conflict, natural disaster], crisis stage [acute, chronic], study design, research methods, SRH intervention, and intervention target population [specific beneficiaries of the intervention within the broader population]; the use of an IR framework; details regarding the IR framework, how it was used, and any rationale given for the framework used; factors cited as impacting SRH interventions, either positively or negatively; and other key findings deemed relevant to this review.

As the focus of this review was on the approach taken for SRH intervention research and evaluation, the quality of the studies themselves was not assessed.

Twenty papers underwent full-text review due to their inclusion in one or more of the four systematic reviews and meeting publication date inclusion criteria. The literature search identified 7,016 unique papers. After full-text screening, 69 met all inclusion criteria and were included in the review. Figure  1 illustrates the search strategy and screening process.

Flow chart of paper identification

Papers published in each of the 10 years of the review timeframe (2013–2022) were included. 29% of the papers originated from the first five years of the time frame considered for this review, with the remaining 71% papers coming from the second half. Characteristics of included publications, including geographic location, type of humanitarian crisis, and type of SRH intervention, are presented in Table  3 .

A wide range of study designs and methods were used across the papers, with both qualitative and quantitative studies well represented. Twenty-six papers were quantitative evaluations [ 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 ], 17 were qualitative [ 44 , 45 , 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 , 55 , 56 , 57 , 58 , 59 , 60 ], and 26 used mixed methods [ 61 , 62 , 63 , 64 , 65 , 66 , 67 , 68 , 69 , 70 , 71 , 72 , 73 , 74 , 75 , 76 , 77 , 78 , 79 , 80 , 81 , 82 , 83 , 84 , 85 , 86 ]. Within the quantitative evaluations, 15 were observational, while five were quasi-experimental, five were randomized controlled trials, and one was an economic evaluation. Study designs as classified by the authors of this review are summarized in Table  4 .

Six papers (9%) explicitly cited use of an IR framework. Three of these papers utilized the Consolidated Framework for Implementation Research (CFIR) [ 51 , 65 , 70 ]. The CFIR is a commonly used determinant framework that—in its originally proposed form in 2009—is comprised of five domains, each of which has constructs to further categorize factors that impact implementation. The CFIR domains were identified as core content areas influencing the effectiveness of implementation, and the constructs within each domain are intended to provide a range of options for researchers to select from to “guide diagnostic assessments of implementation context, evaluate implementation progress, and help explain findings.” [ 87 ] To allow for consistent terminology throughout this review, the original 2009 CFIR domains and constructs are used.

Guan et al. conducted a mixed methods study to assess the feasibility and effectiveness of a neonatal hepatitis B immunization program in a conflict-affected rural region of Myanmar. Guan et al. report mapping data onto the CFIR as a secondary analysis step. They describe that “CFIR was used as a comprehensive meta-theoretical framework to examine the implementation of the Hepatitis B Virus vaccination program,” and implementation themes from multiple study data sources (interviews, observations, examination of monitoring materials) were mapped onto CFIR constructs. They report their results in two phases – Pre-implementation training and community education, and Implementation – with both anchored in themes that they had mapped onto CFIR domains and constructs. All but six constructs were included in their analysis, with a majority summarized in a table and key themes explored further in the narrative text. They specify that most concerns were identified within the Outer Setting and Process domains, while elements identified within the Inner Setting domain provided strength to the intervention and helped mitigate against barriers [ 70 ].

Sarker et al. conducted a qualitative study to assess provision of maternal, newborn and child health services to Rohingya refugees residing in camps in Cox’s Bazar, Bangladesh. They cite using CFIR as a guide for thematic analysis, applying it after a process of inductive and deductive coding to index these codes into the CFIR domains. They utilized three of the five CFIR domains (Outer Setting, Inner Setting, and Process), stating that the remaining two domains (Intervention Characteristics and Characteristics of Individuals) were not relevant to their analysis. They then proposed two additional CFIR domains, Context and Security, for use in humanitarian contexts. In contrast to Guan et al., CFIR constructs are not used nor mentioned by Sarker et al., with content under each domain instead synthesized as challenges and potential solutions. Regarding the CFIR, Sarker et al. write, “The CFIR guided us for interpretative coding and creating the challenges and possible solutions into groups for further clarification of the issues related to program delivery in a humanitarian crisis setting.” [ 51 ]

Sami et al. conducted a mixed methods case study to assess the implementation of a package of neonatal interventions at health facilities within refugee and internally displaced persons camps in South Sudan. They reference use of the CFIR earlier in the study than Sarker et al., basing their guides for semi-structured focus group discussions on the CFIR framework. They similarly reference a general use of the CFIR framework as they conducted thematic analysis. Constructs are referenced once, but they do not specify whether their application of the CFIR framework included use of domains, constructs, or both. This may be in part because they then applied an additional framework, the World Health Organization (WHO) Health System Framework, to present their findings. They describe a nested approach to their use of these frameworks: “Exploring these [CFIR] constructs within the WHO Health Systems Framework can identify specific entry points to improve the implementation of newborn interventions at critical health system building blocks.” [ 65 ]

Three papers cite use of different IR frameworks. Bolan et al. utilized the Theoretical Domains Framework in their mixed methods feasibility study and pilot cluster randomized trial evaluating pilot use of the Safe Delivery App by maternal and newborn health workers providing basic emergency obstetric and newborn care in facilities in the conflict-affected Maniema province of the Democratic Republic of the Congo (DRC). They used the Theroetical Domains Framework in designing interview questions, and further used it as the coding framework for their analysis. Similar to the CFIR, the Theoretical Domains Framework is a determinant framework that consists of domains, each of which then includes constructs. Bolan et al. utilized the Theoretical Domains Framework at the construct level in interview question development and at the domain level in their analysis, mapping interview responses to eight of the 14 domains [ 83 ]. Berg et al. report using an “exploratory design guided by the principles of an evaluation framework” developed by the Medical Research Council to analyze the implementation process, mechanisms of impact, and outcomes of a three-pillar training intervention to improve maternal and neonatal healthcare in the conflict-affected South Kivu province of the DRC [ 67 , 88 ]. Select components of this evaluation framework were used to guide deductive analysis of focus group discussions and in-depth interviews [ 67 ]. In their study of health workers’ knowledge and attitudes toward newborn health interventions in South Sudan, before and after training and supply provision, Sami et al. report use of the Conceptual Framework of the Role of Attitudes in Evidence-Based Practice Implementation in their analysis process. The framework was used to group codes following initial inductive coding analysis of in-depth interviews [ 72 ].

Three other papers cite use of specific frameworks in their intervention evaluation [ 19 , 44 , 76 ]. As a characteristic of IR is the use of an explicit framework to guide the research, the use of the frameworks in these three papers meets the intention of IR and serves the purpose that an IR framework would have in strengthening the analytical rigor. Castle et al. cite use of their program’s theory of change as a framework for a mixed methods evaluation of the provision of family planning services and more specifically uptake of long-acting reversible contraception use in the DRC. They describe use of the theory of change to “enhance effectiveness of [long-acting reversible contraception] access and uptake.” [ 76 ] Thommesen et al. cite use of the AAAQ (Availability, Accessibility, Acceptability and Quality) framework in their qualitative study assessing midwifery services provided to pregnant women in Afghanistan. This framework is focused on the “underlying elements needed for attainment of optimum standard of health care,” but the authors used it in this paper to evaluate facilitators and barriers to women accessing midwifery services [ 44 ]. Jarrett et al. cite use of the Centers for Disease Control and Prevention’s (CDC) Guidelines for Evaluating Public Health Surveillance Systems to explore the characteristics of a population mobility, mortality and birth surveillance system in South Kivu, DRC. Use of these CDC guidelines is cited as one of four study objectives, and commentary is included in the Results section pertaining to each criteria within these guidelines, although more detail regarding use of these guidelines or the authors’ experience with their use in the study is not provided [ 19 ].

Overall, 22 of the 69 papers either explicitly or implicitly identified IR as relevant to their work. Nineteen papers include a focus on feasibility (seven of which did not otherwise identify the importance of exploring questions concerning implementation), touching on a common outcome of interest in implementation research [ 89 ].

While a majority of papers did not explicitly or implicitly use an IR framework to evaluate their SRH intervention of focus, most identified factors that facilitated implementation when they were present or served as a barrier when absent. Sixty cite factors that served as facilitators and 49 cite factors that served as barriers, with just three not citing either. Fifty-nine distinct factors were identified across the papers.

Three of the six studies that explicitly used an IR framework used the CFIR, and the CFIR is the only IR framework that was used by multiple studies. As previously mentioned, Means et al. put forth an adaptation of the CFIR to increase its relevance in LMIC settings, proposing a sixth domain (Characteristics of Systems) and 11 additional constructs [ 2 ]. Using the expanded domains and constructs as proposed by Means et al., the 59 factors cited by papers in this review were thematically grouped into the six domains: Characteristics of Systems, Outer Setting, Inner Setting, Characteristics of Individuals, Intervention Characteristics, and Process. Within each domain, alignment with CFIR constructs was assessed for, and alignment was found with 29 constructs: eight of Means et al.’s 11 constructs, and 21 of the 39 standard CFIR constructs. Three factors did not align with any construct (all fitting within the Outer Setting domain), and 14 aligned with a construct label but not the associated definition. Table 5 synthesizes the mapping of factors affecting SRH intervention implementation to CFIR domains and constructs, with the construct appearing in italics if it is considered to align with that factor by label but not by definition.

Table 6 lists the CFIR constructs that were not found to have alignment with any factor cited by the papers in this review.

This scoping review sought to assess how IR frameworks have been used to bolster the evidence base for SRH interventions in humanitarian settings, and it revealed that IR frameworks, or an explicit IR approach, are rarely used. All four of the systematic reviews identified with a focus on SRH in humanitarian settings articulate the need for more research examining the effectiveness of SRH interventions in humanitarian settings, with two specifically citing a need for implementation research/science [ 12 , 13 ]. The distribution of papers across the timeframe included in this review does suggest that more research on SRH interventions for crisis-affected populations is taking place, as a majority of relevant papers were published in the second half of the review period. The papers included a wide range of methodologies, which reflect the differing research questions and contexts being evaluated. However, it also invites the question of whether there should be more standardization of outcomes measured or frameworks used to guide analysis and to facilitate increased comparison, synthesis and application across settings.

Three of the six papers that used an IR framework utilized the CFIR. Guan et al. used the CFIR at both a domain and construct level, Sarker et al. used the CFIR at the domain level, and Sami et al. did not specify which CFIR elements were used in informing the focus group discussion guide [ 51 , 65 , 70 ]. It is challenging to draw strong conclusions about the applicability of CFIR in humanitarian settings based on the minimal use of CFIR and IR frameworks within the papers reviewed, although Guan et al. provides a helpful model for how analysis can be structured around CFIR domains and constructs. It is worth considering that the minimal use of IR frameworks, and more specifically CFIR constructs, could be in part because that level of prescriptive categorization does not allow for enough fluidity in humanitarian settings. It also raises questions about the appropriate degree of standardization to pursue for research done in these settings.

The mapping of factors affecting SRH intervention implementation provides an example of how a modified CFIR framework could be used for IR in humanitarian contexts. This mapping exercise found factors that mapped to all five of the original CFIR domains as well as the sixth domain proposed by Means et al. All factors fit well within the definition for the selected domain, indicating an appropriate degree of fit between these existing domains and the factors identified as impacting SRH interventions in humanitarian settings. On a construct level, however, the findings were more variable, with one-quarter of factors not fully aligning with any construct. Furthermore, over 40% of the CFIR constructs (including the additional constructs from Means et al.) were not found to align with any factors cited by the papers in this review, also demonstrating some disconnect between the parameters posed by the CFIR constructs and the factors cited as relevant in a humanitarian context.

It is worth noting that while the CFIR as proposed in 2009 was used in this assessment, as well as in the included papers which used the CFIR, an update was published in 2022. Following a review of CFIR use since its publication, the authors provide updates to construct names and definitions to “make the framework more applicable across a range of innovations and settings.” New constructs and subconstructs were also added, for a total of 48 constructs and 19 subconstructs across the five domains [ 90 ]. A CFIR Outcomes Addendum was also published in 2022, based on recommendations for the CFIR to add outcomes and intended to be used as a complement to the CFIR determinants framework [ 91 ]. These expansions to the CFIR framework may improve applicability of the CFIR in humanitarian settings. Several constructs added to the Outer Setting domain could be of particular utility – critical incidents, local attitudes, and local conditions, each of which could help account for unique challenges faced in contexts of crisis. Sub-constructs added within the Inner Setting domain that seek to clarify structural characteristics and available resources would also be of high utility based on mapping of the factors identified in this review to the original CFIR constructs. As outcomes were not formally included in the CFIR until the 2022 addendum, a separate assessment of implementation outcomes was not undertaken in this review. However, analysis of the factors cited by papers in this review as affecting implementation was derived from the full text of the papers and thus captures content relevant to implementation determinants that is contained within the outcomes.

Given the demonstrated need for additional flexibility within an IR framework for humanitarian contexts, while not a focus of this review, it is worth considering whether a different framework could provide a better fit than the CFIR. Other frameworks have differing points of emphasis that would create different opportunities for flexibility but that do not seem to resolve the challenges experienced in applying the CFIR to a humanitarian context. As one example, the EPIS (Exploration, Preparation, Implementation, Sustainment) Framework considers the impact of inner and outer context on each of four implementation phases; while the constructs within this framework are broader than the CFIR, an emphasis on the intervention characteristics is missing, a domain where stronger alignment within the CFIR is also needed [ 92 ]. Alternatively, the PRISM (Practical, Robust Implementation and Sustainability Model) framework is a determinant and evaluation framework that adds consideration of context factors to the RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) outcomes framework. It has a stronger emphasis on intervention aspects, with sub-domains to account for both organization and patient perspectives within the intervention. While PRISM does include aspects of context, external environment considerations are less robust and intentionally less comprehensive in scope, which would not provide the degree of alignment possible between the Characteristics of Systems and Outer Setting CFIR domains for the considerations unique to humanitarian environments [ 93 ].

Reflecting on their experience with the CFIR, Sarker et al. indicate that it can be a “great asset” in both evaluating current work and developing future interventions. They also encourage future research of humanitarian health interventions to utilize the CFIR [ 51 ]. The other papers that used the CFIR do not specifically reflect on their experience utilizing it, referring more generally to having felt that it was a useful tool [ 65 , 70 ]. On their use of an evaluation framework, Berg et al. reflected that it lent useful structure and helped to identify aspects affecting implementation that otherwise would have gone un-noticed [ 67 ]. The remaining studies that utilized an IR framework did not specifically comment on their experience with its use [ 72 , 83 ]. While a formal IR framework was not engaged by other studies, a number cite a desire for IR to contribute further detail to their findings [ 21 , 37 ].

In their recommendations for strengthening the evidence base for humanitarian health interventions, Ager et al. speak to the need for “methodologic innovation” to develop methodologies with particular applicability in humanitarian settings [ 7 ]. As IR is not yet routinized for SRH interventions, this could be opportune timing for the use of a standardized IR framework to gauge its utility. Using an IR framework to assess factors influencing implementation of the MISP in initial stages of a humanitarian response, and interventions to support more comprehensive SRH service delivery in protracted crises, could lend further rigor and standardization to SRH evaluations, as well as inform strategies to improve MISP implementation over time. Based on categorizing factors identified by these papers as relevant for intervention evaluation, there does seem to be utility to a modified CFIR approach. Given the paucity of formal IR framework use within SRH literature, it would be worth conducting similar scoping exercises to assess for explicit use of IR frameworks within the evidence base for other health service delivery areas in humanitarian settings. In the interim, the recommended approach from this review for future IR on humanitarian health interventions would be a modified CFIR approach with domain-level standardization and flexibility for constructs that may standardize over time with more use. This would enable use of a common analytical framework and vocabulary at the domain level for stakeholders to describe interventions and the factors influencing the effectiveness of implementation, with constructs available to use and customize as most appropriate for specific contexts and interventions.

This review had a number of limitations. As this was a scoping review and a two-part search strategy was used, the papers summarized here may not be comprehensive of those written pertaining to SRH interventions over the past 10 years. Papers from 2013 to 2017 that would have met this scoping review’s inclusion criteria may have been omitted due to being excluded from the systematic reviews. The review was limited to papers available in English. Furthermore, this review did not assess the quality of the papers included or seek to assess the methodology used beyond examination of the use of an IR framework. It does, however, serve as a first step in assessing the extent to which calls for implementation research have been addressed, and identify entry points for strengthening the science and practice of SRH research in humanitarian settings.

With one in 23 people worldwide in need of humanitarian assistance, and financing required for response plans at an all-time high, the need for evidence to guide resource allocation and programming for SRH in humanitarian settings is as important as ever [ 94 ]. Recent research agenda setting initiatives and strategies to advance health in humanitarian settings call for increased investment in implementation research—with priorities ranging from research on effective strategies for expanding access to a full range of contraceptive options to integrating mental health and psychosocial support into SRH programming to capturing accurate and actionable data on maternal and perinatal mortality in a wide range of acute and protracted emergency contexts [ 95 , 96 ]. To truly advance guidance in these areas, implementation research will need to be conducted across diverse humanitarian settings, with clear and consistent documentation of both intervention characteristics and outcomes, as well as contextual and programmatic factors affecting implementation.

Conclusions

Implementation research has potential to increase impact of health interventions particularly in crisis-affected settings where flexibility, adaptability and context-responsive approaches are highlighted as cornerstones of effective programming. There remains significant opportunity for standardization of research in the humanitarian space, with one such opportunity occurring through increased utilization of IR frameworks such as a modified CFIR approach. Investing in more robust sexual and reproductive health research in humanitarian contexts can enrich insights available to guide programming and increase transferability of learning across settings.

Availability of data and materials

The datasets analyzed during the current study are available from the corresponding author on reasonable request.

Abbreviations

Availability, Accessibility, Acceptability and Quality

Centers for Disease Control and Prevention

Consolidated Framework for Implementation Research

Democratic Republic of the Congo

Exploration, Preparation, Implementation, Sustainment

• Implementation research

Low and middle income country

Minimum Initial Service Package

Practical, Robust Implementation and Sustainability Model

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

Reach, Effectiveness, Adoption, Implementation, Maintenance

• Sexual and reproductive health

World Health Organization

Peters DH, et al. Implementation research: what it is and how to do it. RESEARCH METHODS. 2013;347:7.

Means AR, et al. Evaluating and optimizing the consolidated framework for implementation research (CFIR) for use in low- and middle-income countries: a systematic review. Implement Sci. 2020;15(1):17.

Article   PubMed   PubMed Central   Google Scholar  

Alonge O, et al. How is implementation research applied to advance health in low-income and middle-income countries? BMJ Glob Health. 2019;4(2):e001257.

Ridde V, Pérez D, Robert E. Using implementation science theories and frameworks in global health. BMJ Glob Health. 2020;5(4):e002269.

Gaffey MF, et al. Delivering health and nutrition interventions for women and children in different conflict contexts: a framework for decision making on what, when, and how. Lancet (London, England). 2021;397(10273):543–54.

Article   PubMed   Google Scholar  

Singh NS, et al. Delivering health interventions to women, children, and adolescents in conflict settings: what have we learned from ten country case studies? The Lancet. 2021;397(10273):533–42.

Article   Google Scholar  

Ager A, et al. Strengthening the evidence base for health programming in humanitarian crises. Science. 2014;345(6202):1290–2.

Article   CAS   PubMed   Google Scholar  

Blanchet K, et al. Evidence on public health interventions in humanitarian crises. The Lancet. 2017;390(10109):2287–96.

Sphere A. The Sphere Handbook | Standards for quality humanitarian response. 2018.

Google Scholar  

Barot S. In a State of Crisis: Meeting the Sexual and Reproductive Health Needs of Women in Humanitarian Situations. Guttmacher Policy Rev. 2017;20:7.

Crisis, I.-A.W.G.f.R.H.i., Minimum Initial Service Package. 2020: https://www.unfpa.org/resources/minimum-initial-service-package-misp-srh-crisis-situations .

Casey SE. Evaluations of reproductive health programs in humanitarian settings: a systematic review. Confl Heal. 2015;9(1):S1.

Singh NS, et al. A long way to go: a systematic review to assess the utilisation of sexual and reproductive health services during humanitarian crises. BMJ Glob Health. 2018;3(2):e000682.

Singh NS, et al. Evaluating the effectiveness of sexual and reproductive health services during humanitarian crises: A systematic review. PLoS ONE. 2018;13(7):e0199300.

Warren E, et al. Systematic review of the evidence on the effectiveness of sexual and reproductive health interventions in humanitarian crises. BMJ Open. 2015;5(12):e008226.

Dadich A, Piper A, Coates D. Implementation science in maternity care: a scoping review. Implement Sci. 2021;16(1):16.

Tricco AC, et al. PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation. Ann Intern Med. 2018;169(7):467–73.

Devine A, et al. Strategies for the prevention of perinatal hepatitis B transmission in a marginalized population on the Thailand-Myanmar border: a cost-effectiveness analysis. BMC Infect Dis. 2017;17(1):552.

Jarrett P, et al. Evaluation of a population mobility, mortality, and birth surveillance system in South Kivu. Democratic Republic of the Congo Disasters. 2020;44(2):390–407.

PubMed   Google Scholar  

Logie CH, et al. A Psycho-Educational HIV/STI Prevention Intervention for Internally Displaced Women in Leogane, Haiti: Results from a Non-Randomized Cohort Pilot Study. PLoS ONE. 2014;9(2):e89836.

O’Laughlin KN, et al. A cohort study to assess a communication intervention to improve linkage to HIV care in Nakivale Refugee Settlement. Uganda Glob Public Health. 2021;16(12):1848–55.

Adam I. The influence of maternal health education on the place of delivery in conflict settings of Darfur. Sudan Conflict and Health. 2015;9:31.

Adam IF, et al. Relationship between implementing interpersonal communication and mass education campaigns in emergency settings and use of reproductive healthcare services: evidence from Darfur, Sudan. BMJ Open. 2015;5(9):e008285.

Edmond K, et al. Mobile outreach health services for mothers and children in conflict-affected and remote areas: a population-based study from Afghanistan. Arch Dis Child. 2020;105(1):18–25.

Nasir S, et al. Dissemination and implementation of the e-MCHHandbook, UNRWA’s newly released maternal and child health mobile application: a cross-sectional study. BMJ Open. 2020;10(3):e034885.

O’Laughlin KN, et al. Feasibility and acceptability of home-based HIV testing among refugees: a pilot study in Nakivale refugee settlement in southwestern Uganda. BMC Infect Dis. 2018;18(1):332.

Adam I. Evidence from cluster surveys on the association between home-based counseling and use of family planning in conflict-affected Darfur. Int J Gynecol Obstet. 2016;133(2):221–5.

Casey S, et al. Availability of long-acting and permanent family-planning methods leads to increase in use in conflict-affected northern Uganda: Evidence from cross-sectional baseline and endline cluster surveys. Glob Public Health. 2013;8(3):284–97.

Corna F, et al. Supporting maternal mental health of Rohingya refugee women during the perinatal period to promote child health and wellbeing: a field study in Cox’s Bazar. Intervention, the Journal of Mental Health & Psychosocial Support in Conflict Affected Areas. 2019;17(2):160–8.

Glass N, et al. Effectiveness of the Communities Care programme on change in social norms associated with gender-based violence (GBV) with residents in intervention compared with control districts in Mogadishu, Somalia. BMJ Open. 2019;9(3):e023819.

James LE, et al. Development and Testing of a Community-Based Intervention to Address Intimate Partner Violence among Rohingya and Syrian Refugees: A Social Norms-Based Mental Health-Integrated Approach. Int J Environ Res Public Health. 2021;18(21):11674.

Le Roux E, et al. Engaging with faith groups to prevent VAWG in conflict-affected communities: results from two community surveys in the DRC. BMC Int Health Hum Rights. 2020;20(1):27.

Morris CN, et al. When political solutions for acute conflict in Yemen seem distant, demand for reproductive health services is immediate: a programme model for resilient family planning and post-abortion care services. Sex Reprod Health Matters. 2019;27(2):1610279.

Anibueze AU, et al. Impact of counseling visual multimedia on use of family planning methods among displaced Nigerian families. Health Promot Int. 2022;37(3):daac060.

Doocy S, et al. Cash-based assistance and the nutrition status of pregnant and lactating women in the Somalia food crisis: A comparison of two transfer modalities. PLoS ONE. 2020;15(4):e0230989.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Draiko CV, et al. The effect of umbilical cord cleansing with chlorhexidine gel on neonatal mortality among the community births in South Sudan: a quasi-experimental study. Pan Afr Med J. 2021;38:78.

Edmond KM, et al. Can community health worker home visiting improve care-seeking and maternal and newborn care practices in fragile states such as Afghanistan? A population-based intervention study. BMC Med. 2018;16(1):106.

Edmond KM, et al. Conditional cash transfers to improve use of health facilities by mothers and newborns in conflict affected countries, a prospective population based intervention study from Afghanistan. BMC Pregnancy Childbirth. 2019;19(1):193.

Bakesiima R, et al. Effect of peer counselling on acceptance of modern contraceptives among female refugee adolescents in northern Uganda: A randomised controlled trial. PLoS ONE. 2021;16(9):e0256479.

Greene MC, et al. Evaluation of an integrated intervention to reduce psychological distress and intimate partner violence in refugees: Results from the Nguvu cluster randomized feasibility trial. PLoS ONE. 2021;16(6):e0252982.

Gupta J, et al. Gender norms and economic empowerment intervention to reduce intimate partner violence against women in rural Côte d’Ivoire: a randomized controlled pilot study. BMC Int Health Hum Rights. 2013;13(1):46.

Hossain M, et al. Working with men to prevent intimate partner violence in a conflict-affected setting: a pilot cluster randomized controlled trial in rural Côte d’Ivoire. BMC Public Health. 2014;14(1):339.

Vaillant J, et al. Engaging men to transform inequitable gender attitudes and prevent intimate partner violence: a cluster randomised controlled trial in North and South Kivu, Democratic Republic of Congo. BMJ Glob Health. 2020;5(5):e002223.

Thommesen T, et al. “The midwife helped me … otherwise I could have died”: women’s experience of professional midwifery services in rural Afghanistan - a qualitative study in the provinces Kunar and Laghman. BMC Pregnancy Childbirth. 2020;20(1):140.

Awasom-Fru A, et al. Doctors’ experiences providing sexual and reproductive health care at Catholic Hospitals in the conflict-affected North-West region of Cameroon: a qualitative study. Reprod Health. 2022;19(1):126.

Kabakian-Khasholian T, Makhoul J, Ghusayni A. “A person who does not have money does not enter”: a qualitative study on refugee women’s experiences of respectful maternity care. BMC Pregnancy and Childbirth. 2022;22(1):748.

Lilleston P, et al. Evaluation of a mobile approach to gender-based violence service delivery among Syrian refugees in Lebanon. Health Policy Plan. 2018;33(7):767–76.

Mugo NS, et al. Barriers Faced by the Health Workers to Deliver Maternal Care Services and Their Perceptions of the Factors Preventing Their Clients from Receiving the Services: A Qualitative Study in South Sudan. Matern Child Health J. 2018;22(11):1598–606.

Persson M, et al. A qualitative study on health care providers’ experiences of providing comprehensive abortion care in Cox’s Bazar, Bangladesh. Conflict and Health. 2021;15(1):6.

Phanwichatkul T, et al. The perceptions and practices of Thai health professionals providing maternity care for migrant Burmese women: An ethnographic study. Women Birth. 2022;35(4):e356–68.

Sarker M, et al. Effective maternal, newborn and child health programming among Rohingya refugees in Cox’s Bazar, Bangladesh: Implementation challenges and potential solutions. PLoS ONE. 2020;15(3):e0230732.

Tousaw E, et al. “Without this program, women can lose their lives”: migrant women’s experiences with the Safe Abortion Referral Programme in Chiang Mai. Thailand Reprod Health Matters. 2017;25(51):58–68.

Tousaw E, et al. “It is just like having a period with back pain”: exploring women’s experiences with community-based distribution of misoprostol for early abortion on the Thailand-Burma border. Contraception. 2018;97(2):122–9.

West L, et al. Factors in use of family planning services by Syrian women in a refugee camp in Jordan. Journal of Family Planning and Reproductive Health Care. 2017;43(2):96–102.

O’Connell KA, et al. Meeting the Sexual and Reproductive Health Needs of Internally Displaced Persons in Ethiopia’s Somali Region: A Qualitative Process Evaluation. Glob Health Sci Pract. 2022;10(5):e2100818.

Orya E, et al. Strengthening close to community provision of maternal health services in fragile settings: an exploration of the changing roles of TBAs in Sierra Leone and Somaliland. BMC Health Serv Res. 2017;17(1):460.

Perera SM, et al. Barriers to seeking post-abortion care in Paktika Province, Afghanistan: a qualitative study of clients and community members. BMC Womens Health. 2021;21(1):390.

Tanabe M, et al. Piloting community-based medical care for survivors of sexual assault in conflict-affected Karen State of eastern Burma. Confl Heal. 2013;7(1):12.

Tran NT, et al. Clinical outreach refresher trainings in crisis settings (S-CORT): clinical management of sexual violence survivors and manual vacuum aspiration in Burkina Faso, Nepal, and South Sudan. Reprod Health Matters. 2017;25(51):103–13.

Yankah E, et al. Feasibility and acceptability of mobile phone platforms to deliver interventions to address gender-based violence among Syrian adolescent girls and young women in Izmir. Turkey Vulnerable Children and Youth Studies. 2020;15(2):133–43.

Muuo S, et al. Barriers and facilitators to care-seeking among survivors of gender-based violence in the Dadaab refugee complex. Sex Reprod Health Matters. 2020;28(1):1722404.

Amsalu R, et al. Essential newborn care practice at four primary health facilities in conflict affected areas of Bossaso, Somalia: a cross-sectional study. Conflict and Health. 2019;13(13):27.

Myers A, et al. Facilitators and barriers in implementing the Minimum Initial Services Package (MISP) for reproductive health in Nepal post-earthquake. Conflict and Health. 2018;12:35.

Santo L.C.d, et al. Feasibility and acceptability of a video library tool to support community health worker counseling in rural Afghan districts: a cross-sectional assessment. Conflict and Health. 2020;14:56.

Sami S, et al. Understanding health systems to improve community and facility level newborn care among displaced populations in South Sudan: a mixed methods case study. BMC Pregnancy Childbirth. 2018;18(1):325.

Amsalu R, et al. Effectiveness of clinical training on improving essential newborn care practices in Bossaso, Somalia: a pre and postintervention study. BMC Pediatr. 2020;20(1):215.

Berg M, Mwambali SN, Bogren M. Implementation of a three-pillar training intervention to improve maternal and neonatal healthcare in the Democratic Republic Of Congo: a process evaluation study in an urban health zone. Glob Health Action. 2022;15(1):2019391.

Castillo M, et al. Turning Disaster into an Opportunity for Quality Improvement in Essential Intrapartum and Newborn Care Services in the Philippines: Pre- to Posttraining Assessments. Biomed Res Int. 2016;2016:1–9.

Foster AM, Arnott G, Hobstetter M. Community-based distribution of misoprostol for early abortion: evaluation of a program along the Thailand-Burma border. Contraception. 2017;96(4):242–7.

Guan TH, et al. Implementation of a neonatal hepatitis B immunization program in rural Karenni State, Myanmar: A mixed-methods study. PLoS ONE. 2021;16(12):e0261470.

Logie, C.H., et al., Mixed-methods findings from the Ngutulu Kagwero (agents of change) participatory comic pilot study on post-rape clinical care and sexual violence prevention with refugee youth in a humanitarian setting in Uganda. Global Public Health, 2022((Logie C.H., [email protected]) Factor-Inwentash Faculty of Social Work, University of Toronto, Toronto, Canada(Logie C.H., [email protected]) Women’s College Research Institute, Women’s College Hospital, Toronto, Canada(Logie C.H., carmen.l).

Sami S, et al. “You have to take action”: changing knowledge and attitudes towards newborn care practices during crisis in South Sudan. Reprod Health Matters. 2017;25(51):124–39.

Smith JR, et al. Clinical care for sexual assault survivors multimedia training: a mixed-methods study of effect on healthcare providers’ attitudes, knowledge, confidence, and practice in humanitarian settings. Confl Heal. 2013;7(1):14.

Stevens A, et al. Folate supplementation to prevent birth abnormalities: evaluating a community-based participatory action plan for refugees and migrant workers on the Thailand-Myanmar border. Public Health. 2018;161:83–9.

Nguyen Toan T, et al. Strengthening healthcare providers’ capacity for safe abortion and postabortion care services in humanitarian settings: lessons learned from the clinical outreach refresher training model (S-CORT) in Uganda, Nigeria, and the Democratic Republic of Congo. Conflict and Health. 2021;15(1):20.

Castle S, et al. Successful programmatic approaches to facilitating IUD uptake: CARE’s experience in DRC. BMC Womens Health. 2019;19(1):104.

Deitch J, et al. “They Love Their Patients”: Client Perceptions of Quality of Postabortion Care in North and South Kivu, the Democratic Republic of the Congo. Global health, science and practice. 2019;7(Suppl 2):S285–98.

Ferreyra C, et al. Evaluation of a community-based HIV test and start program in a conflict affected rural area of Yambio County, South Sudan. PLoS ONE. 2021;16(7):e0254331.

Ho LS, Wheeler E. Using Program Data to Improve Access to Family Planning and Enhance the Method Mix in Conflict-Affected Areas of the Democratic Republic of the Congo. Glob Health Sci Pract. 2018;6(1):161–77.

Klabbers RE, et al. Health Worker Perspectives on Barriers and Facilitators of Assisted Partner Notification for HIV for Refugees and Ugandan Nationals: A Mixed Methods Study in West Nile Uganda. AIDS Behav. 2021;25(10):3206–22.

Turner C, et al. Neonatal Intensive Care in a Karen Refugee Camp: A 4 Year Descriptive Study. PLoS ONE. 2013;8(8):e72721.

Vries Id, et al. Key lessons from a mixed-method evaluation of a postnatal home visit programme in the humanitarian setting of Gaza. Eastern Mediterr Health J. 2021;27(6):546–52.

Bolan NE, et al. mLearning in the Democratic Republic of the Congo: A Mixed-Methods Feasibility and Pilot Cluster Randomized Trial Using the Safe Delivery App. Global health, science and practice. 2018;6(4):693–710.

Khan MN, et al. Evaluating feasibility and acceptability of a local psycho-educational intervention for pregnant women with common mental problems affected by armed conflict in Swat, Pakistan: A parallel randomized controlled feasibility trial. Int J Soc Psychiatry. 2017;63(8):724–35.

Hynes M, et al. Using a quality improvement approach to improve maternal and neonatal care in North Kivu, Democratic Republic of Congo. Reprod Health Matters. 2017;25(51):140–50.

Gibbs A, et al. The impacts of combined social and economic empowerment training on intimate partner violence, depression, gender norms and livelihoods among women: an individually randomised controlled trial and qualitative study in Afghanistan. BMJ Glob Health. 2020;5(3):e001946.

Damschroder L, et al. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implementation science: IS; 2009.

Moore GF, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015;350:h1258.

Proctor E, et al. Outcomes for Implementation Research: Conceptual Distinctions, Measurement Challenges, and Research Agenda. Adm Policy Ment Health. 2011;38(2):65–76.

Damschroder LJ, et al. The updated Consolidated Framework for Implementation Research based on user feedback. Implement Sci. 2022;17(1):75.

Damschroder LJ, et al. Conceptualizing outcomes for use with the Consolidated Framework for Implementation Research (CFIR): the CFIR Outcomes Addendum. Implement Sci. 2022;17(1):7.

Aarons GA, Hurlburt M, Horwitz SM. Advancing a Conceptual Model of Evidence-Based Practice Implementation in Public Service Sectors. Administration and Policy in Mental Health and Mental Health Services Research. 2011;38(1):4–23.

Feldstein AC, Glasgow RE. A Practical, Robust Implementation and Sustainability Model (PRISM) for Integrating Research Findings into Practice. The Joint Commission Journal on Quality and Patient Safety. 2008;34(4):228–43.

OCHA. Global Humanitarian Overview 2023. 2022 [cited 2023 8/3/2023]; Available from: https://humanitarianaction.info/node/13073/article/glance-0 . Accessed 8 Mar 2023.

Kobeissi L, et al. Setting research priorities for sexual, reproductive, maternal, newborn, child and adolescent health in humanitarian settings. Confl Heal. 2021;15(1):16.

Save the, C., et al. Roadmap to Accelerate Progress for Every Newborn in Humanitarian Settings 2020 – 2024. 2020. p. 52.

Inter-Agency Working Group on Reproductive Health in, C. Inter-Agency Field Manual on Reproductive Health in Humanitarian Settings. 2018.

Download references

Acknowledgements

Not applicable.

The authors received no funding for this study.

Author information

Authors and affiliations.

Duke University School of Medicine, 40 Duke Medicine Circle, Durham, NC, 27710, USA

Alexandra Norton

Johns Hopkins Bloomberg School of Public Health, 615 N. Wolfe St, Baltimore, MD, 21205, USA

Hannah Tappis

You can also search for this author in PubMed   Google Scholar

Contributions

AN and HT designed the scoping review. AN conducted the literature search. AN and HT screened records for inclusion. AN extracted data from included studies. Both authors contributed to synthesis of results. AN drafted the manuscript and both authors contributed to editorial changes.

Corresponding author

Correspondence to Alexandra Norton .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1.

. Literature search terms: Exact search terms used in literature search, with additional detail on the methodology to determine search terms and definitions used for each component of the search

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Norton, A., Tappis, H. Sexual and reproductive health implementation research in humanitarian contexts: a scoping review. Reprod Health 21 , 64 (2024). https://doi.org/10.1186/s12978-024-01793-2

Download citation

Received : 06 November 2023

Accepted : 12 April 2024

Published : 13 May 2024

DOI : https://doi.org/10.1186/s12978-024-01793-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Humanitarian settings

Reproductive Health

ISSN: 1742-4755

• General enquiries: [email protected]

The International Journal of Indian Psychȯlogy

Effect of Counselling Programme on Emotional Functioning and Social Interaction of Children of Divorced Parents: A Psychological Perspective

| Published: May 10, 2024

Divorce is a significant life events that can have profound effects on children wellbeing, including emotional distress and challenges in social relationships. This research paper examines the effectiveness of Counselling interventions for children of divorced parents from a psychological standpoint. Drawing on empirical research, theoretical frameworks, and clinical evidence, the paper explores how Counselling programs can mitigate emotional distress and improve social interaction in children affected by parental divorce. Attachment theory explains here the adverse effect of parental separation on children growth. This systematic review explores the effect of Counselling interventions on the emotional functioning and social interaction of children of divorced parents. Many research Findings support this review that Counselling intervention have a positive impact on the emotional functioning of children, leading to improved coping skills, increased self-esteem and reduced symptoms of anxiety and depression. Additionally, counselling programme facilitate positive social interaction by enhancing communication skills, fostering peer support and promoting healthy relationship dynamics with in the family. However, the effectiveness of counselling may ever be depending on factors such as the child’s age, gender and level of parental conflict. Further research is needed to explore the long-term effect of Counselling intervention and optimal strategies for supporting the wellbeing of children of divorced parents. This review underscores the importance of Counselling as the valuable resource in promoting the emotional resilience and social integration of children facing the challenges of parent’s divorce.

Parental Divorce , Counselling , Emotional Functioning , Social Interaction

This is an Open Access Research distributed under the terms of the Creative Commons Attribution License (www.creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any Medium, provided the original work is properly cited.

© 2024, Chaudhary, N. & Bala, M.

Received: April 29, 2024; Revision Received: May 06, 2024; Accepted: May 10, 2024

Dr. Neetu Chaudhary @ [email protected]

Article Overview

Published in   Volume 12, Issue 2, April-June, 2024