research articles methodology

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base
• Dissertation
• What Is a Research Methodology? | Steps & Tips

What Is a Research Methodology? | Steps & Tips

Published on August 25, 2022 by Shona McCombes and Tegan George. Revised on November 20, 2023.

Your research methodology discusses and explains the data collection and analysis methods you used in your research. A key part of your thesis, dissertation , or research paper , the methodology chapter explains what you did and how you did it, allowing readers to evaluate the reliability and validity of your research and your dissertation topic .

It should include:

• The type of research you conducted
• How you collected and analyzed your data
• Any tools or materials you used in the research
• How you mitigated or avoided research biases
• Why you chose these methods
• Your methodology section should generally be written in the past tense .
• Academic style guides in your field may provide detailed guidelines on what to include for different types of studies.
• Your citation style might provide guidelines for your methodology section (e.g., an APA Style methods section ).

Instantly correct all language mistakes in your text

Upload your document to correct all your mistakes in minutes

Table of contents

How to write a research methodology, why is a methods section important, step 1: explain your methodological approach, step 2: describe your data collection methods, step 3: describe your analysis method, step 4: evaluate and justify the methodological choices you made, tips for writing a strong methodology chapter, other interesting articles, frequently asked questions about methodology.

The only proofreading tool specialized in correcting academic writing - try for free!

The academic proofreading tool has been trained on 1000s of academic texts and by native English editors. Making it the most accurate and reliable proofreading tool for students.

Try for free

Your methods section is your opportunity to share how you conducted your research and why you chose the methods you chose. It’s also the place to show that your research was rigorously conducted and can be replicated .

It gives your research legitimacy and situates it within your field, and also gives your readers a place to refer to if they have any questions or critiques in other sections.

You can start by introducing your overall approach to your research. You have two options here.

Option 1: Start with your “what”

What research problem or question did you investigate?

• Aim to describe the characteristics of something?
• Explore an under-researched topic?
• Establish a causal relationship?

And what type of data did you need to achieve this aim?

• Quantitative data , qualitative data , or a mix of both?
• Primary data collected yourself, or secondary data collected by someone else?
• Experimental data gathered by controlling and manipulating variables, or descriptive data gathered via observations?

Option 2: Start with your “why”

Depending on your discipline, you can also start with a discussion of the rationale and assumptions underpinning your methodology. In other words, why did you choose these methods for your study?

• Why is this the best way to answer your research question?
• Is this a standard methodology in your field, or does it require justification?
• Were there any ethical considerations involved in your choices?
• What are the criteria for validity and reliability in this type of research ? How did you prevent bias from affecting your data?

Once you have introduced your reader to your methodological approach, you should share full details about your data collection methods .

Quantitative methods

In order to be considered generalizable, you should describe quantitative research methods in enough detail for another researcher to replicate your study.

Here, explain how you operationalized your concepts and measured your variables. Discuss your sampling method or inclusion and exclusion criteria , as well as any tools, procedures, and materials you used to gather your data.

Surveys Describe where, when, and how the survey was conducted.

• How did you design the questionnaire?
• What form did your questions take (e.g., multiple choice, Likert scale )?
• Were your surveys conducted in-person or virtually?
• What sampling method did you use to select participants?
• What was your sample size and response rate?

Experiments Share full details of the tools, techniques, and procedures you used to conduct your experiment.

• How did you design the experiment ?
• How did you recruit participants?
• How did you manipulate and measure the variables ?
• What tools did you use?

Existing data Explain how you gathered and selected the material (such as datasets or archival data) that you used in your analysis.

• Where did you source the material?
• How was the data originally produced?
• What criteria did you use to select material (e.g., date range)?

The survey consisted of 5 multiple-choice questions and 10 questions measured on a 7-point Likert scale.

The goal was to collect survey responses from 350 customers visiting the fitness apparel company’s brick-and-mortar location in Boston on July 4–8, 2022, between 11:00 and 15:00.

Here, a customer was defined as a person who had purchased a product from the company on the day they took the survey. Participants were given 5 minutes to fill in the survey anonymously. In total, 408 customers responded, but not all surveys were fully completed. Due to this, 371 survey results were included in the analysis.

• Information bias
• Omitted variable bias
• Regression to the mean
• Survivorship bias
• Undercoverage bias
• Sampling bias

Qualitative methods

In qualitative research , methods are often more flexible and subjective. For this reason, it’s crucial to robustly explain the methodology choices you made.

Be sure to discuss the criteria you used to select your data, the context in which your research was conducted, and the role you played in collecting your data (e.g., were you an active participant, or a passive observer?)

Interviews or focus groups Describe where, when, and how the interviews were conducted.

• How did you find and select participants?
• How many participants took part?
• What form did the interviews take ( structured , semi-structured , or unstructured )?
• How long were the interviews?
• How were they recorded?

Participant observation Describe where, when, and how you conducted the observation or ethnography .

• What group or community did you observe? How long did you spend there?
• How did you gain access to this group? What role did you play in the community?
• How long did you spend conducting the research? Where was it located?
• How did you record your data (e.g., audiovisual recordings, note-taking)?

Existing data Explain how you selected case study materials for your analysis.

• What type of materials did you analyze?
• How did you select them?

In order to gain better insight into possibilities for future improvement of the fitness store’s product range, semi-structured interviews were conducted with 8 returning customers.

Here, a returning customer was defined as someone who usually bought products at least twice a week from the store.

Surveys were used to select participants. Interviews were conducted in a small office next to the cash register and lasted approximately 20 minutes each. Answers were recorded by note-taking, and seven interviews were also filmed with consent. One interviewee preferred not to be filmed.

• The Hawthorne effect
• Observer bias
• The placebo effect
• Response bias and Nonresponse bias
• The Pygmalion effect
• Recall bias
• Social desirability bias
• Self-selection bias

Mixed methods

Mixed methods research combines quantitative and qualitative approaches. If a standalone quantitative or qualitative study is insufficient to answer your research question, mixed methods may be a good fit for you.

Mixed methods are less common than standalone analyses, largely because they require a great deal of effort to pull off successfully. If you choose to pursue mixed methods, it’s especially important to robustly justify your methods.

Receive feedback on language, structure, and formatting

Professional editors proofread and edit your paper by focusing on:

• Academic style
• Vague sentences
• Style consistency

See an example

Next, you should indicate how you processed and analyzed your data. Avoid going into too much detail: you should not start introducing or discussing any of your results at this stage.

In quantitative research , your analysis will be based on numbers. In your methods section, you can include:

• How you prepared the data before analyzing it (e.g., checking for missing data , removing outliers , transforming variables)
• Which software you used (e.g., SPSS, Stata or R)
• Which statistical tests you used (e.g., two-tailed t test , simple linear regression )

In qualitative research, your analysis will be based on language, images, and observations (often involving some form of textual analysis ).

Specific methods might include:

• Content analysis : Categorizing and discussing the meaning of words, phrases and sentences
• Thematic analysis : Coding and closely examining the data to identify broad themes and patterns
• Discourse analysis : Studying communication and meaning in relation to their social context

Mixed methods combine the above two research methods, integrating both qualitative and quantitative approaches into one coherent analytical process.

Above all, your methodology section should clearly make the case for why you chose the methods you did. This is especially true if you did not take the most standard approach to your topic. In this case, discuss why other methods were not suitable for your objectives, and show how this approach contributes new knowledge or understanding.

In any case, it should be overwhelmingly clear to your reader that you set yourself up for success in terms of your methodology’s design. Show how your methods should lead to results that are valid and reliable, while leaving the analysis of the meaning, importance, and relevance of your results for your discussion section .

• Quantitative: Lab-based experiments cannot always accurately simulate real-life situations and behaviors, but they are effective for testing causal relationships between variables .
• Qualitative: Unstructured interviews usually produce results that cannot be generalized beyond the sample group , but they provide a more in-depth understanding of participants’ perceptions, motivations, and emotions.
• Mixed methods: Despite issues systematically comparing differing types of data, a solely quantitative study would not sufficiently incorporate the lived experience of each participant, while a solely qualitative study would be insufficiently generalizable.

Remember that your aim is not just to describe your methods, but to show how and why you applied them. Again, it’s critical to demonstrate that your research was rigorously conducted and can be replicated.

1. Focus on your objectives and research questions

The methodology section should clearly show why your methods suit your objectives and convince the reader that you chose the best possible approach to answering your problem statement and research questions .

2. Cite relevant sources

Your methodology can be strengthened by referencing existing research in your field. This can help you to:

• Show that you followed established practice for your type of research
• Discuss how you decided on your approach by evaluating existing research
• Present a novel methodological approach to address a gap in the literature

3. Write for your audience

Consider how much information you need to give, and avoid getting too lengthy. If you are using methods that are standard for your discipline, you probably don’t need to give a lot of background or justification.

Regardless, your methodology should be a clear, well-structured text that makes an argument for your approach, not just a list of technical details and procedures.

If you want to know more about statistics , methodology , or research bias , make sure to check out some of our other articles with explanations and examples.

• Normal distribution
• Measures of central tendency
• Chi square tests
• Confidence interval
• Quartiles & Quantiles

Methodology

• Cluster sampling
• Stratified sampling
• Thematic analysis
• Cohort study
• Peer review
• Ethnography

Research bias

• Implicit bias
• Cognitive bias
• Conformity bias
• Hawthorne effect
• Availability heuristic
• Attrition bias

Methodology refers to the overarching strategy and rationale of your research project . It involves studying the methods used in your field and the theories or principles behind them, in order to develop an approach that matches your objectives.

Methods are the specific tools and procedures you use to collect and analyze data (for example, experiments, surveys , and statistical tests ).

In shorter scientific papers, where the aim is to report the findings of a specific study, you might simply describe what you did in a methods section .

In a longer or more complex research project, such as a thesis or dissertation , you will probably include a methodology section , where you explain your approach to answering the research questions and cite relevant sources to support your choice of methods.

In a scientific paper, the methodology always comes after the introduction and before the results , discussion and conclusion . The same basic structure also applies to a thesis, dissertation , or research proposal .

Depending on the length and type of document, you might also include a literature review or theoretical framework before the methodology.

Quantitative research deals with numbers and statistics, while qualitative research deals with words and meanings.

Quantitative methods allow you to systematically measure variables and test hypotheses . Qualitative methods allow you to explore concepts and experiences in more detail.

Reliability and validity are both about how well a method measures something:

• Reliability refers to the  consistency of a measure (whether the results can be reproduced under the same conditions).
• Validity   refers to the  accuracy of a measure (whether the results really do represent what they are supposed to measure).

If you are doing experimental research, you also have to consider the internal and external validity of your experiment.

A sample is a subset of individuals from a larger population . Sampling means selecting the group that you will actually collect data from in your research. For example, if you are researching the opinions of students in your university, you could survey a sample of 100 students.

In statistics, sampling allows you to test a hypothesis about the characteristics of a population.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

McCombes, S. & George, T. (2023, November 20). What Is a Research Methodology? | Steps & Tips. Scribbr. Retrieved April 16, 2024, from https://www.scribbr.com/dissertation/methodology/

Is this article helpful?

Shona McCombes

Other students also liked, what is a theoretical framework | guide to organizing, what is a research design | types, guide & examples, qualitative vs. quantitative research | differences, examples & methods, what is your plagiarism score.

• Privacy Policy

Buy Me a Coffee

Home » Research Methodology – Types, Examples and writing Guide

Research Methodology – Types, Examples and writing Guide

Table of Contents

Research Methodology

Definition:

Research Methodology refers to the systematic and scientific approach used to conduct research, investigate problems, and gather data and information for a specific purpose. It involves the techniques and procedures used to identify, collect , analyze , and interpret data to answer research questions or solve research problems . Moreover, They are philosophical and theoretical frameworks that guide the research process.

Structure of Research Methodology

Research methodology formats can vary depending on the specific requirements of the research project, but the following is a basic example of a structure for a research methodology section:

I. Introduction

• Provide an overview of the research problem and the need for a research methodology section
• Outline the main research questions and objectives

II. Research Design

• Explain the research design chosen and why it is appropriate for the research question(s) and objectives
• Discuss any alternative research designs considered and why they were not chosen
• Describe the research setting and participants (if applicable)

III. Data Collection Methods

• Describe the methods used to collect data (e.g., surveys, interviews, observations)
• Explain how the data collection methods were chosen and why they are appropriate for the research question(s) and objectives
• Detail any procedures or instruments used for data collection

IV. Data Analysis Methods

• Describe the methods used to analyze the data (e.g., statistical analysis, content analysis )
• Explain how the data analysis methods were chosen and why they are appropriate for the research question(s) and objectives
• Detail any procedures or software used for data analysis

V. Ethical Considerations

• Discuss any ethical issues that may arise from the research and how they were addressed
• Explain how informed consent was obtained (if applicable)
• Detail any measures taken to ensure confidentiality and anonymity

VI. Limitations

• Identify any potential limitations of the research methodology and how they may impact the results and conclusions

VII. Conclusion

• Summarize the key aspects of the research methodology section
• Explain how the research methodology addresses the research question(s) and objectives

Research Methodology Types

Types of Research Methodology are as follows:

Quantitative Research Methodology

This is a research methodology that involves the collection and analysis of numerical data using statistical methods. This type of research is often used to study cause-and-effect relationships and to make predictions.

Qualitative Research Methodology

This is a research methodology that involves the collection and analysis of non-numerical data such as words, images, and observations. This type of research is often used to explore complex phenomena, to gain an in-depth understanding of a particular topic, and to generate hypotheses.

Mixed-Methods Research Methodology

This is a research methodology that combines elements of both quantitative and qualitative research. This approach can be particularly useful for studies that aim to explore complex phenomena and to provide a more comprehensive understanding of a particular topic.

Case Study Research Methodology

This is a research methodology that involves in-depth examination of a single case or a small number of cases. Case studies are often used in psychology, sociology, and anthropology to gain a detailed understanding of a particular individual or group.

Action Research Methodology

This is a research methodology that involves a collaborative process between researchers and practitioners to identify and solve real-world problems. Action research is often used in education, healthcare, and social work.

Experimental Research Methodology

This is a research methodology that involves the manipulation of one or more independent variables to observe their effects on a dependent variable. Experimental research is often used to study cause-and-effect relationships and to make predictions.

Survey Research Methodology

This is a research methodology that involves the collection of data from a sample of individuals using questionnaires or interviews. Survey research is often used to study attitudes, opinions, and behaviors.

Grounded Theory Research Methodology

This is a research methodology that involves the development of theories based on the data collected during the research process. Grounded theory is often used in sociology and anthropology to generate theories about social phenomena.

Research Methodology Example

An Example of Research Methodology could be the following:

Research Methodology for Investigating the Effectiveness of Cognitive Behavioral Therapy in Reducing Symptoms of Depression in Adults

Introduction:

The aim of this research is to investigate the effectiveness of cognitive-behavioral therapy (CBT) in reducing symptoms of depression in adults. To achieve this objective, a randomized controlled trial (RCT) will be conducted using a mixed-methods approach.

Research Design:

The study will follow a pre-test and post-test design with two groups: an experimental group receiving CBT and a control group receiving no intervention. The study will also include a qualitative component, in which semi-structured interviews will be conducted with a subset of participants to explore their experiences of receiving CBT.

Participants:

Participants will be recruited from community mental health clinics in the local area. The sample will consist of 100 adults aged 18-65 years old who meet the diagnostic criteria for major depressive disorder. Participants will be randomly assigned to either the experimental group or the control group.

Intervention :

The experimental group will receive 12 weekly sessions of CBT, each lasting 60 minutes. The intervention will be delivered by licensed mental health professionals who have been trained in CBT. The control group will receive no intervention during the study period.

Data Collection:

Quantitative data will be collected through the use of standardized measures such as the Beck Depression Inventory-II (BDI-II) and the Generalized Anxiety Disorder-7 (GAD-7). Data will be collected at baseline, immediately after the intervention, and at a 3-month follow-up. Qualitative data will be collected through semi-structured interviews with a subset of participants from the experimental group. The interviews will be conducted at the end of the intervention period, and will explore participants’ experiences of receiving CBT.

Data Analysis:

Quantitative data will be analyzed using descriptive statistics, t-tests, and mixed-model analyses of variance (ANOVA) to assess the effectiveness of the intervention. Qualitative data will be analyzed using thematic analysis to identify common themes and patterns in participants’ experiences of receiving CBT.

Ethical Considerations:

This study will comply with ethical guidelines for research involving human subjects. Participants will provide informed consent before participating in the study, and their privacy and confidentiality will be protected throughout the study. Any adverse events or reactions will be reported and managed appropriately.

Data Management:

All data collected will be kept confidential and stored securely using password-protected databases. Identifying information will be removed from qualitative data transcripts to ensure participants’ anonymity.

Limitations:

One potential limitation of this study is that it only focuses on one type of psychotherapy, CBT, and may not generalize to other types of therapy or interventions. Another limitation is that the study will only include participants from community mental health clinics, which may not be representative of the general population.

Conclusion:

This research aims to investigate the effectiveness of CBT in reducing symptoms of depression in adults. By using a randomized controlled trial and a mixed-methods approach, the study will provide valuable insights into the mechanisms underlying the relationship between CBT and depression. The results of this study will have important implications for the development of effective treatments for depression in clinical settings.

How to Write Research Methodology

Writing a research methodology involves explaining the methods and techniques you used to conduct research, collect data, and analyze results. It’s an essential section of any research paper or thesis, as it helps readers understand the validity and reliability of your findings. Here are the steps to write a research methodology:

• Start by explaining your research question: Begin the methodology section by restating your research question and explaining why it’s important. This helps readers understand the purpose of your research and the rationale behind your methods.
• Describe your research design: Explain the overall approach you used to conduct research. This could be a qualitative or quantitative research design, experimental or non-experimental, case study or survey, etc. Discuss the advantages and limitations of the chosen design.
• Discuss your sample: Describe the participants or subjects you included in your study. Include details such as their demographics, sampling method, sample size, and any exclusion criteria used.
• Describe your data collection methods : Explain how you collected data from your participants. This could include surveys, interviews, observations, questionnaires, or experiments. Include details on how you obtained informed consent, how you administered the tools, and how you minimized the risk of bias.
• Explain your data analysis techniques: Describe the methods you used to analyze the data you collected. This could include statistical analysis, content analysis, thematic analysis, or discourse analysis. Explain how you dealt with missing data, outliers, and any other issues that arose during the analysis.
• Discuss the validity and reliability of your research : Explain how you ensured the validity and reliability of your study. This could include measures such as triangulation, member checking, peer review, or inter-coder reliability.
• Acknowledge any limitations of your research: Discuss any limitations of your study, including any potential threats to validity or generalizability. This helps readers understand the scope of your findings and how they might apply to other contexts.
• Provide a summary: End the methodology section by summarizing the methods and techniques you used to conduct your research. This provides a clear overview of your research methodology and helps readers understand the process you followed to arrive at your findings.

When to Write Research Methodology

Research methodology is typically written after the research proposal has been approved and before the actual research is conducted. It should be written prior to data collection and analysis, as it provides a clear roadmap for the research project.

The research methodology is an important section of any research paper or thesis, as it describes the methods and procedures that will be used to conduct the research. It should include details about the research design, data collection methods, data analysis techniques, and any ethical considerations.

The methodology should be written in a clear and concise manner, and it should be based on established research practices and standards. It is important to provide enough detail so that the reader can understand how the research was conducted and evaluate the validity of the results.

Applications of Research Methodology

Here are some of the applications of research methodology:

• To identify the research problem: Research methodology is used to identify the research problem, which is the first step in conducting any research.
• To design the research: Research methodology helps in designing the research by selecting the appropriate research method, research design, and sampling technique.
• To collect data: Research methodology provides a systematic approach to collect data from primary and secondary sources.
• To analyze data: Research methodology helps in analyzing the collected data using various statistical and non-statistical techniques.
• To test hypotheses: Research methodology provides a framework for testing hypotheses and drawing conclusions based on the analysis of data.
• To generalize findings: Research methodology helps in generalizing the findings of the research to the target population.
• To develop theories : Research methodology is used to develop new theories and modify existing theories based on the findings of the research.
• To evaluate programs and policies : Research methodology is used to evaluate the effectiveness of programs and policies by collecting data and analyzing it.
• To improve decision-making: Research methodology helps in making informed decisions by providing reliable and valid data.

Purpose of Research Methodology

Research methodology serves several important purposes, including:

• To guide the research process: Research methodology provides a systematic framework for conducting research. It helps researchers to plan their research, define their research questions, and select appropriate methods and techniques for collecting and analyzing data.
• To ensure research quality: Research methodology helps researchers to ensure that their research is rigorous, reliable, and valid. It provides guidelines for minimizing bias and error in data collection and analysis, and for ensuring that research findings are accurate and trustworthy.
• To replicate research: Research methodology provides a clear and detailed account of the research process, making it possible for other researchers to replicate the study and verify its findings.
• To advance knowledge: Research methodology enables researchers to generate new knowledge and to contribute to the body of knowledge in their field. It provides a means for testing hypotheses, exploring new ideas, and discovering new insights.
• To inform decision-making: Research methodology provides evidence-based information that can inform policy and decision-making in a variety of fields, including medicine, public health, education, and business.

Advantages of Research Methodology

Research methodology has several advantages that make it a valuable tool for conducting research in various fields. Here are some of the key advantages of research methodology:

• Systematic and structured approach : Research methodology provides a systematic and structured approach to conducting research, which ensures that the research is conducted in a rigorous and comprehensive manner.
• Objectivity : Research methodology aims to ensure objectivity in the research process, which means that the research findings are based on evidence and not influenced by personal bias or subjective opinions.
• Replicability : Research methodology ensures that research can be replicated by other researchers, which is essential for validating research findings and ensuring their accuracy.
• Reliability : Research methodology aims to ensure that the research findings are reliable, which means that they are consistent and can be depended upon.
• Validity : Research methodology ensures that the research findings are valid, which means that they accurately reflect the research question or hypothesis being tested.
• Efficiency : Research methodology provides a structured and efficient way of conducting research, which helps to save time and resources.
• Flexibility : Research methodology allows researchers to choose the most appropriate research methods and techniques based on the research question, data availability, and other relevant factors.
• Scope for innovation: Research methodology provides scope for innovation and creativity in designing research studies and developing new research techniques.

Research Methodology Vs Research Methods

About the author.

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

How to Cite Research Paper – All Formats and...

Data Collection – Methods Types and Examples

Delimitations in Research – Types, Examples and...

Research Paper Format – Types, Examples and...

Research Process – Steps, Examples and Tips

Research Design – Types, Methods and Examples

What Is Research Methodology? A Plain-Language Explanation & Definition (With Examples)

By Derek Jansen (MBA)  and Kerryn Warren (PhD) | June 2020 (Last updated April 2023)

If you’re new to formal academic research, it’s quite likely that you’re feeling a little overwhelmed by all the technical lingo that gets thrown around. And who could blame you – “research methodology”, “research methods”, “sampling strategies”… it all seems never-ending!

In this post, we’ll demystify the landscape with plain-language explanations and loads of examples (including easy-to-follow videos), so that you can approach your dissertation, thesis or research project with confidence. Let’s get started.

Research Methodology 101

• What exactly research methodology means
• What qualitative , quantitative and mixed methods are
• What sampling strategy is
• What data collection methods are
• What data analysis methods are
• How to choose your research methodology
• Example of a research methodology

What is research methodology?

Research methodology simply refers to the practical “how” of a research study. More specifically, it’s about how  a researcher  systematically designs a study  to ensure valid and reliable results that address the research aims, objectives and research questions . Specifically, how the researcher went about deciding:

• What type of data to collect (e.g., qualitative or quantitative data )
• Who  to collect it from (i.e., the sampling strategy )
• How to  collect  it (i.e., the data collection method )
• How to  analyse  it (i.e., the data analysis methods )

Within any formal piece of academic research (be it a dissertation, thesis or journal article), you’ll find a research methodology chapter or section which covers the aspects mentioned above. Importantly, a good methodology chapter explains not just   what methodological choices were made, but also explains  why they were made. In other words, the methodology chapter should justify  the design choices, by showing that the chosen methods and techniques are the best fit for the research aims, objectives and research questions. 

So, it’s the same as research design?

Not quite. As we mentioned, research methodology refers to the collection of practical decisions regarding what data you’ll collect, from who, how you’ll collect it and how you’ll analyse it. Research design, on the other hand, is more about the overall strategy you’ll adopt in your study. For example, whether you’ll use an experimental design in which you manipulate one variable while controlling others. You can learn more about research design and the various design types here .

Need a helping hand?

What are qualitative, quantitative and mixed-methods?

Qualitative, quantitative and mixed-methods are different types of methodological approaches, distinguished by their focus on words , numbers or both . This is a bit of an oversimplification, but its a good starting point for understanding.

Let’s take a closer look.

Qualitative research refers to research which focuses on collecting and analysing words (written or spoken) and textual or visual data, whereas quantitative research focuses on measurement and testing using numerical data . Qualitative analysis can also focus on other “softer” data points, such as body language or visual elements.

It’s quite common for a qualitative methodology to be used when the research aims and research questions are exploratory  in nature. For example, a qualitative methodology might be used to understand peoples’ perceptions about an event that took place, or a political candidate running for president. 

Contrasted to this, a quantitative methodology is typically used when the research aims and research questions are confirmatory  in nature. For example, a quantitative methodology might be used to measure the relationship between two variables (e.g. personality type and likelihood to commit a crime) or to test a set of hypotheses .

As you’ve probably guessed, the mixed-method methodology attempts to combine the best of both qualitative and quantitative methodologies to integrate perspectives and create a rich picture. If you’d like to learn more about these three methodological approaches, be sure to watch our explainer video below.

What is sampling strategy?

Simply put, sampling is about deciding who (or where) you’re going to collect your data from . Why does this matter? Well, generally it’s not possible to collect data from every single person in your group of interest (this is called the “population”), so you’ll need to engage a smaller portion of that group that’s accessible and manageable (this is called the “sample”).

How you go about selecting the sample (i.e., your sampling strategy) will have a major impact on your study.  There are many different sampling methods  you can choose from, but the two overarching categories are probability   sampling and  non-probability   sampling .

Probability sampling  involves using a completely random sample from the group of people you’re interested in. This is comparable to throwing the names all potential participants into a hat, shaking it up, and picking out the “winners”. By using a completely random sample, you’ll minimise the risk of selection bias and the results of your study will be more generalisable  to the entire population. 

Non-probability sampling , on the other hand,  doesn’t use a random sample . For example, it might involve using a convenience sample, which means you’d only interview or survey people that you have access to (perhaps your friends, family or work colleagues), rather than a truly random sample. With non-probability sampling, the results are typically not generalisable .

To learn more about sampling methods, be sure to check out the video below.

What are data collection methods?

As the name suggests, data collection methods simply refers to the way in which you go about collecting the data for your study. Some of the most common data collection methods include:

• Interviews (which can be unstructured, semi-structured or structured)
• Focus groups and group interviews
• Surveys (online or physical surveys)
• Observations (watching and recording activities)
• Biophysical measurements (e.g., blood pressure, heart rate, etc.)
• Documents and records (e.g., financial reports, court records, etc.)

The choice of which data collection method to use depends on your overall research aims and research questions , as well as practicalities and resource constraints. For example, if your research is exploratory in nature, qualitative methods such as interviews and focus groups would likely be a good fit. Conversely, if your research aims to measure specific variables or test hypotheses, large-scale surveys that produce large volumes of numerical data would likely be a better fit.

What are data analysis methods?

Data analysis methods refer to the methods and techniques that you’ll use to make sense of your data. These can be grouped according to whether the research is qualitative  (words-based) or quantitative (numbers-based).

Popular data analysis methods in qualitative research include:

• Qualitative content analysis
• Thematic analysis
• Discourse analysis
• Narrative analysis
• Interpretative phenomenological analysis (IPA)
• Visual analysis (of photographs, videos, art, etc.)

Qualitative data analysis all begins with data coding , after which an analysis method is applied. In some cases, more than one analysis method is used, depending on the research aims and research questions . In the video below, we explore some  common qualitative analysis methods, along with practical examples.  

Moving on to the quantitative side of things, popular data analysis methods in this type of research include:

• Descriptive statistics (e.g. means, medians, modes )
• Inferential statistics (e.g. correlation, regression, structural equation modelling)

Again, the choice of which data collection method to use depends on your overall research aims and objectives , as well as practicalities and resource constraints. In the video below, we explain some core concepts central to quantitative analysis.

How do I choose a research methodology?

As you’ve probably picked up by now, your research aims and objectives have a major influence on the research methodology . So, the starting point for developing your research methodology is to take a step back and look at the big picture of your research, before you make methodology decisions. The first question you need to ask yourself is whether your research is exploratory or confirmatory in nature.

If your research aims and objectives are primarily exploratory in nature, your research will likely be qualitative and therefore you might consider qualitative data collection methods (e.g. interviews) and analysis methods (e.g. qualitative content analysis). 

Conversely, if your research aims and objective are looking to measure or test something (i.e. they’re confirmatory), then your research will quite likely be quantitative in nature, and you might consider quantitative data collection methods (e.g. surveys) and analyses (e.g. statistical analysis).

Designing your research and working out your methodology is a large topic, which we cover extensively on the blog . For now, however, the key takeaway is that you should always start with your research aims, objectives and research questions (the golden thread). Every methodological choice you make needs align with those three components. 

Example of a research methodology chapter

In the video below, we provide a detailed walkthrough of a research methodology from an actual dissertation, as well as an overview of our free methodology template .

Psst… there’s more (for free)

This post is part of our dissertation mini-course, which covers everything you need to get started with your dissertation, thesis or research project. 

You Might Also Like:

199 Comments

Thank you for this simple yet comprehensive and easy to digest presentation. God Bless!

You’re most welcome, Leo. Best of luck with your research!

I found it very useful. many thanks

This is really directional. A make-easy research knowledge.

Thank you for this, I think will help my research proposal

Thanks for good interpretation,well understood.

Good morning sorry I want to the search topic

Thank u more

Thank you, your explanation is simple and very helpful.

Very educative a.nd exciting platform. A bigger thank you and I’ll like to always be with you

That’s the best analysis

So simple yet so insightful. Thank you.

This really easy to read as it is self-explanatory. Very much appreciated…

Thanks for this. It’s so helpful and explicit. For those elements highlighted in orange, they were good sources of referrals for concepts I didn’t understand. A million thanks for this.

Good morning, I have been reading your research lessons through out a period of times. They are important, impressive and clear. Want to subscribe and be and be active with you.

Thankyou So much Sir Derek…

Good morning thanks so much for the on line lectures am a student of university of Makeni.select a research topic and deliberate on it so that we’ll continue to understand more.sorry that’s a suggestion.

Beautiful presentation. I love it.

please provide a research mehodology example for zoology

It’s very educative and well explained

Thanks for the concise and informative data.

This is really good for students to be safe and well understand that research is all about

Thank you so much Derek sir🖤🙏🤗

Very simple and reliable

This is really helpful. Thanks alot. God bless you.

very useful, Thank you very much..

thanks a lot its really useful

in a nutshell..thank you!

Thanks for updating my understanding on this aspect of my Thesis writing.

thank you so much my through this video am competently going to do a good job my thesis

Thanks a lot. Very simple to understand. I appreciate 🙏

Very simple but yet insightful Thank you

This has been an eye opening experience. Thank you grad coach team.

Very useful message for research scholars

Really very helpful thank you

yes you are right and i’m left

Research methodology with a simplest way i have never seen before this article.

wow thank u so much

Good morning thanks so much for the on line lectures am a student of university of Makeni.select a research topic and deliberate on is so that we will continue to understand more.sorry that’s a suggestion.

Very precise and informative.

Thanks for simplifying these terms for us, really appreciate it.

Thanks this has really helped me. It is very easy to understand.

I found the notes and the presentation assisting and opening my understanding on research methodology

Good presentation

Im so glad you clarified my misconceptions. Im now ready to fry my onions. Thank you so much. God bless

Thank you a lot.

thanks for the easy way of learning and desirable presentation.

Thanks a lot. I am inspired

Well written

I am writing a APA Format paper . I using questionnaire with 120 STDs teacher for my participant. Can you write me mthology for this research. Send it through email sent. Just need a sample as an example please. My topic is ” impacts of overcrowding on students learning

Thanks for your comment.

We can’t write your methodology for you. If you’re looking for samples, you should be able to find some sample methodologies on Google. Alternatively, you can download some previous dissertations from a dissertation directory and have a look at the methodology chapters therein.

All the best with your research.

Thank you so much for this!! God Bless

Thank you. Explicit explanation

Thank you, Derek and Kerryn, for making this simple to understand. I’m currently at the inception stage of my research.

Thnks a lot , this was very usefull on my assignment

excellent explanation

I’m currently working on my master’s thesis, thanks for this! I’m certain that I will use Qualitative methodology.

Thanks a lot for this concise piece, it was quite relieving and helpful. God bless you BIG…

I am currently doing my dissertation proposal and I am sure that I will do quantitative research. Thank you very much it was extremely helpful.

Very interesting and informative yet I would like to know about examples of Research Questions as well, if possible.

I’m about to submit a research presentation, I have come to understand from your simplification on understanding research methodology. My research will be mixed methodology, qualitative as well as quantitative. So aim and objective of mixed method would be both exploratory and confirmatory. Thanks you very much for your guidance.

OMG thanks for that, you’re a life saver. You covered all the points I needed. Thank you so much ❤️ ❤️ ❤️

Thank you immensely for this simple, easy to comprehend explanation of data collection methods. I have been stuck here for months 😩. Glad I found your piece. Super insightful.

I’m going to write synopsis which will be quantitative research method and I don’t know how to frame my topic, can I kindly get some ideas..

Thanks for this, I was really struggling.

This was really informative I was struggling but this helped me.

Thanks a lot for this information, simple and straightforward. I’m a last year student from the University of South Africa UNISA South Africa.

its very much informative and understandable. I have enlightened.

An interesting nice exploration of a topic.

Thank you. Accurate and simple🥰

This article was really helpful, it helped me understanding the basic concepts of the topic Research Methodology. The examples were very clear, and easy to understand. I would like to visit this website again. Thank you so much for such a great explanation of the subject.

Thanks dude

Thank you Doctor Derek for this wonderful piece, please help to provide your details for reference purpose. God bless.

Many compliments to you

Great work , thank you very much for the simple explanation

Thank you. I had to give a presentation on this topic. I have looked everywhere on the internet but this is the best and simple explanation.

thank you, its very informative.

Well explained. Now I know my research methodology will be qualitative and exploratory. Thank you so much, keep up the good work

Well explained, thank you very much.

This is good explanation, I have understood the different methods of research. Thanks a lot.

Great work…very well explanation

Thanks Derek. Kerryn was just fantastic!

Great to hear that, Hyacinth. Best of luck with your research!

Its a good templates very attractive and important to PhD students and lectuter

Thanks for the feedback, Matobela. Good luck with your research methodology.

Thank you. This is really helpful.

You’re very welcome, Elie. Good luck with your research methodology.

Well explained thanks

This is a very helpful site especially for young researchers at college. It provides sufficient information to guide students and equip them with the necessary foundation to ask any other questions aimed at deepening their understanding.

Thanks for the kind words, Edward. Good luck with your research!

Thank you. I have learned a lot.

Great to hear that, Ngwisa. Good luck with your research methodology!

Thank you for keeping your presentation simples and short and covering key information for research methodology. My key takeaway: Start with defining your research objective the other will depend on the aims of your research question.

My name is Zanele I would like to be assisted with my research , and the topic is shortage of nursing staff globally want are the causes , effects on health, patients and community and also globally

Thanks for making it simple and clear. It greatly helped in understanding research methodology. Regards.

This is well simplified and straight to the point

Thank you Dr

I was given an assignment to research 2 publications and describe their research methodology? I don’t know how to start this task can someone help me?

Sure. You’re welcome to book an initial consultation with one of our Research Coaches to discuss how we can assist – https://gradcoach.com/book/new/ .

Thanks a lot I am relieved of a heavy burden.keep up with the good work

I’m very much grateful Dr Derek. I’m planning to pursue one of the careers that really needs one to be very much eager to know. There’s a lot of research to do and everything, but since I’ve gotten this information I will use it to the best of my potential.

Thank you so much, words are not enough to explain how helpful this session has been for me!

Thanks this has thought me alot.

Very concise and helpful. Thanks a lot

Thank Derek. This is very helpful. Your step by step explanation has made it easier for me to understand different concepts. Now i can get on with my research.

I wish i had come across this sooner. So simple but yet insightful

really nice explanation thank you so much

I’m so grateful finding this site, it’s really helpful…….every term well explained and provide accurate understanding especially to student going into an in-depth research for the very first time, even though my lecturer already explained this topic to the class, I think I got the clear and efficient explanation here, much thanks to the author.

It is very helpful material

I would like to be assisted with my research topic : Literature Review and research methodologies. My topic is : what is the relationship between unemployment and economic growth?

Its really nice and good for us.

THANKS SO MUCH FOR EXPLANATION, ITS VERY CLEAR TO ME WHAT I WILL BE DOING FROM NOW .GREAT READS.

Short but sweet.Thank you

Informative article. Thanks for your detailed information.

I’m currently working on my Ph.D. thesis. Thanks a lot, Derek and Kerryn, Well-organized sequences, facilitate the readers’ following.

great article for someone who does not have any background can even understand

I am a bit confused about research design and methodology. Are they the same? If not, what are the differences and how are they related?

Thanks in advance.

concise and informative.

Thank you very much

How can we site this article is Harvard style?

Very well written piece that afforded better understanding of the concept. Thank you!

Am a new researcher trying to learn how best to write a research proposal. I find your article spot on and want to download the free template but finding difficulties. Can u kindly send it to my email, the free download entitled, “Free Download: Research Proposal Template (with Examples)”.

Thank too much

Thank you very much for your comprehensive explanation about research methodology so I like to thank you again for giving us such great things.

Good very well explained.Thanks for sharing it.

Thank u sir, it is really a good guideline.

so helpful thank you very much.

Thanks for the video it was very explanatory and detailed, easy to comprehend and follow up. please, keep it up the good work

It was very helpful, a well-written document with precise information.

how do i reference this?

MLA Jansen, Derek, and Kerryn Warren. “What (Exactly) Is Research Methodology?” Grad Coach, June 2021, gradcoach.com/what-is-research-methodology/.

APA Jansen, D., & Warren, K. (2021, June). What (Exactly) Is Research Methodology? Grad Coach. https://gradcoach.com/what-is-research-methodology/

Your explanation is easily understood. Thank you

Very help article. Now I can go my methodology chapter in my thesis with ease

I feel guided ,Thank you

This simplification is very helpful. It is simple but very educative, thanks ever so much

The write up is informative and educative. It is an academic intellectual representation that every good researcher can find useful. Thanks

Wow, this is wonderful long live.

Nice initiative

thank you the video was helpful to me.

Thank you very much for your simple and clear explanations I’m really satisfied by the way you did it By now, I think I can realize a very good article by following your fastidious indications May God bless you

Thanks very much, it was very concise and informational for a beginner like me to gain an insight into what i am about to undertake. I really appreciate.

very informative sir, it is amazing to understand the meaning of question hidden behind that, and simple language is used other than legislature to understand easily. stay happy.

This one is really amazing. All content in your youtube channel is a very helpful guide for doing research. Thanks, GradCoach.

research methodologies

Please send me more information concerning dissertation research.

Nice piece of knowledge shared….. #Thump_UP

This is amazing, it has said it all. Thanks to Gradcoach

This is wonderful,very elaborate and clear.I hope to reach out for your assistance in my research very soon.

This is the answer I am searching about…

realy thanks a lot

Thank you very much for this awesome, to the point and inclusive article.

Thank you very much I need validity and reliability explanation I have exams

Thank you for a well explained piece. This will help me going forward.

Very simple and well detailed Many thanks

This is so very simple yet so very effective and comprehensive. An Excellent piece of work.

I wish I saw this earlier on! Great insights for a beginner(researcher) like me. Thanks a mil!

Thank you very much, for such a simplified, clear and practical step by step both for academic students and general research work. Holistic, effective to use and easy to read step by step. One can easily apply the steps in practical terms and produce a quality document/up-to standard

Thanks for simplifying these terms for us, really appreciated.

Thanks for a great work. well understood .

This was very helpful. It was simple but profound and very easy to understand. Thank you so much!

Great and amazing research guidelines. Best site for learning research

hello sir/ma’am, i didn’t find yet that what type of research methodology i am using. because i am writing my report on CSR and collect all my data from websites and articles so which type of methodology i should write in dissertation report. please help me. i am from India.

how does this really work?

perfect content, thanks a lot

As a researcher, I commend you for the detailed and simplified information on the topic in question. I would like to remain in touch for the sharing of research ideas on other topics. Thank you

Impressive. Thank you, Grad Coach 😍

Thank you Grad Coach for this piece of information. I have at least learned about the different types of research methodologies.

Very useful content with easy way

Thank you very much for the presentation. I am an MPH student with the Adventist University of Africa. I have successfully completed my theory and starting on my research this July. My topic is “Factors associated with Dental Caries in (one District) in Botswana. I need help on how to go about this quantitative research

I am so grateful to run across something that was sooo helpful. I have been on my doctorate journey for quite some time. Your breakdown on methodology helped me to refresh my intent. Thank you.

thanks so much for this good lecture. student from university of science and technology, Wudil. Kano Nigeria.

It’s profound easy to understand I appreciate

Thanks a lot for sharing superb information in a detailed but concise manner. It was really helpful and helped a lot in getting into my own research methodology.

Comment * thanks very much

This was sooo helpful for me thank you so much i didn’t even know what i had to write thank you!

You’re most welcome 🙂

Simple and good. Very much helpful. Thank you so much.

This is very good work. I have benefited.

Thank you so much for sharing

This is powerful thank you so much guys

I am nkasa lizwi doing my research proposal on honors with the university of Walter Sisulu Komani I m on part 3 now can you assist me.my topic is: transitional challenges faced by educators in intermediate phase in the Alfred Nzo District.

Appreciate the presentation. Very useful step-by-step guidelines to follow.

I appreciate sir

wow! This is super insightful for me. Thank you!

Indeed this material is very helpful! Kudos writers/authors.

I want to say thank you very much, I got a lot of info and knowledge. Be blessed.

I want present a seminar paper on Optimisation of Deep learning-based models on vulnerability detection in digital transactions.

Need assistance

Dear Sir, I want to be assisted on my research on Sanitation and Water management in emergencies areas.

I am deeply grateful for the knowledge gained. I will be getting in touch shortly as I want to be assisted in my ongoing research.

The information shared is informative, crisp and clear. Kudos Team! And thanks a lot!

hello i want to study

Hello!! Grad coach teams. I am extremely happy in your tutorial or consultation. i am really benefited all material and briefing. Thank you very much for your generous helps. Please keep it up. If you add in your briefing, references for further reading, it will be very nice.

All I have to say is, thank u gyz.

Good, l thanks

thank you, it is very useful

Trackbacks/Pingbacks

• What Is A Literature Review (In A Dissertation Or Thesis) - Grad Coach - […] the literature review is to inform the choice of methodology for your own research. As we’ve discussed on the Grad Coach blog,…
• Free Download: Research Proposal Template (With Examples) - Grad Coach - […] Research design (methodology) […]
• Dissertation vs Thesis: What's the difference? - Grad Coach - […] and thesis writing on a daily basis – everything from how to find a good research topic to which…

Submit a Comment Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

• Print Friendly

• Interlibrary Loan and Scan & Deliver
• Course Reserves
• Purchase Request
• Collection Development & Maintenance
• Current Negotiations
• Ask a Librarian
• Instructor Support
• Library How-To
• Research Guides
• Research Support
• Study Rooms
• Research Rooms
• Partner Spaces
• Loanable Equipment
• Print, Scan, Copy
• 3D Printers
• Poster Printing
• OSULP Leadership
• Strategic Plan

Scholarly Articles: How can I tell?

• Journal Information
• Literature Review
• Author and affiliation
• Introduction
• Specialized Vocabulary

Methodology

• Research sponsors
• Peer-review

The methodology section or methods section tells you how the author(s) went about doing their research. It should let you know a) what method they used to gather data (survey, interviews, experiments, etc.), why they chose this method, and what the limitations are to this method.

The methodology section should be detailed enough that another researcher could replicate the study described. When you read the methodology or methods section:

• What kind of research method did the authors use? Is it an appropriate method for the type of study they are conducting?
• How did the authors get their tests subjects? What criteria did they use?
• What are the contexts of the study that may have affected the results (e.g. environmental conditions, lab conditions, timing questions, etc.)
• Is the sample size representative of the larger population (i.e., was it big enough?)
• Are the data collection instruments and procedures likely to have measured all the important characteristics with reasonable accuracy?
• Does the data analysis appear to have been done with care, and were appropriate analytical techniques used? 

A good researcher will always let you know about the limitations of his or her research.

• << Previous: Specialized Vocabulary
• Next: Results >>
• Last Updated: Apr 15, 2024 3:26 PM
• URL: https://guides.library.oregonstate.edu/ScholarlyArticle

Contact Info

121 The Valley Library Corvallis OR 97331–4501

Phone: 541-737-3331

Services for Persons with Disabilities

In the Valley Library

• Oregon State University Press
• Special Collections and Archives Research Center
• Undergrad Research & Writing Studio
• Graduate Student Commons
• Tutoring Services
• Northwest Art Collection

Digital Projects

• Oregon Explorer
• Oregon Digital
• ScholarsArchive@OSU
• Digital Publishing Initiatives
• Atlas of the Pacific Northwest
• Marilyn Potts Guin Library  
• Cascades Campus Library
• McDowell Library of Vet Medicine

• Resources Home 🏠
• Try SciSpace Copilot
• Search research papers
• Add Copilot Extension
• Try AI Detector
• Try Paraphraser
• Try Citation Generator
• April Papers
• June Papers
• July Papers

A Comprehensive Guide to Methodology in Research

Table of Contents

Research methodology plays a crucial role in any study or investigation. It provides the framework for collecting, analyzing, and interpreting data, ensuring that the research is reliable, valid, and credible. Understanding the importance of research methodology is essential for conducting rigorous and meaningful research.

In this article, we'll explore the various aspects of research methodology, from its types to best practices, ensuring you have the knowledge needed to conduct impactful research.

What is Research Methodology?

Research methodology refers to the system of procedures, techniques, and tools used to carry out a research study. It encompasses the overall approach, including the research design, data collection methods, data analysis techniques, and the interpretation of findings.

Research methodology plays a crucial role in the field of research, as it sets the foundation for any study. It provides researchers with a structured framework to ensure that their investigations are conducted in a systematic and organized manner. By following a well-defined methodology, researchers can ensure that their findings are reliable, valid, and meaningful.

When defining research methodology, one of the first steps is to identify the research problem. This involves clearly understanding the issue or topic that the study aims to address. By defining the research problem, researchers can narrow down their focus and determine the specific objectives they want to achieve through their study.

How to Define Research Methodology

Once the research problem is identified, researchers move on to defining the research questions. These questions serve as a guide for the study, helping researchers to gather relevant information and analyze it effectively. The research questions should be clear, concise, and aligned with the overall goals of the study.

After defining the research questions, researchers need to determine how data will be collected and analyzed. This involves selecting appropriate data collection methods, such as surveys, interviews, observations, or experiments. The choice of data collection methods depends on various factors, including the nature of the research problem, the target population, and the available resources.

Once the data is collected, researchers need to analyze it using appropriate data analysis techniques. This may involve statistical analysis, qualitative analysis, or a combination of both, depending on the nature of the data and the research questions. The analysis of data helps researchers to draw meaningful conclusions and make informed decisions based on their findings.

Role of Methodology in Research

Methodology plays a crucial role in research, as it ensures that the study is conducted in a systematic and organized manner. It provides a clear roadmap for researchers to follow, ensuring that the research objectives are met effectively. By following a well-defined methodology, researchers can minimize bias, errors, and inconsistencies in their study, thus enhancing the reliability and validity of their findings.

In addition to providing a structured approach, research methodology also helps in establishing the reliability and validity of the study. Reliability refers to the consistency and stability of the research findings, while validity refers to the accuracy and truthfulness of the findings. By using appropriate research methods and techniques, researchers can ensure that their study produces reliable and valid results, which can be used to make informed decisions and contribute to the existing body of knowledge.

Steps in Choosing the Right Research Methodology

Choosing the appropriate research methodology for your study is a critical step in ensuring the success of your research. Let's explore some steps to help you select the right research methodology:

Identifying the Research Problem

The first step in choosing the right research methodology is to clearly identify and define the research problem. Understanding the research problem will help you determine which methodology will best address your research questions and objectives.

Identifying the research problem involves a thorough examination of the existing literature in your field of study. This step allows you to gain a comprehensive understanding of the current state of knowledge and identify any gaps that your research can fill. By identifying the research problem, you can ensure that your study contributes to the existing body of knowledge and addresses a significant research gap.

Once you have identified the research problem, you need to consider the scope of your study. Are you focusing on a specific population, geographic area, or time frame? Understanding the scope of your research will help you determine the appropriate research methodology to use.

Reviewing Previous Research

Before finalizing the research methodology, it is essential to review previous research conducted in the field. This will allow you to identify gaps, determine the most effective methodologies used in similar studies, and build upon existing knowledge.

Reviewing previous research involves conducting a systematic review of relevant literature. This process includes searching for and analyzing published studies, articles, and reports that are related to your research topic. By reviewing previous research, you can gain insights into the strengths and limitations of different methodologies and make informed decisions about which approach to adopt.

During the review process, it is important to critically evaluate the quality and reliability of the existing research. Consider factors such as the sample size, research design, data collection methods, and statistical analysis techniques used in previous studies. This evaluation will help you determine the most appropriate research methodology for your own study.

Formulating Research Questions

Once the research problem is identified, formulate specific and relevant research questions. These questions will guide your methodology selection process by helping you determine what type of data you need to collect and how to analyze it.

Formulating research questions involves breaking down the research problem into smaller, more manageable components. These questions should be clear, concise, and measurable. They should also align with the objectives of your study and provide a framework for data collection and analysis.

When formulating research questions, consider the different types of data that can be collected, such as qualitative or quantitative data. Depending on the nature of your research questions, you may need to employ different data collection methods, such as interviews, surveys, observations, or experiments. By carefully formulating research questions, you can ensure that your chosen methodology will enable you to collect the necessary data to answer your research questions effectively.

Implementing the Research Methodology

After choosing the appropriate research methodology, it is time to implement it. This stage involves collecting data using various techniques and analyzing the gathered information. Let's explore two crucial aspects of implementing the research methodology:

Data Collection Techniques

Data collection techniques depend on the chosen research methodology. They can include surveys, interviews, observations, experiments, or document analysis. Selecting the most suitable data collection techniques will ensure accurate and relevant data for your study.

Data Analysis Methods

Data analysis is a critical part of the research process. It involves interpreting and making sense of the collected data to draw meaningful conclusions. Depending on the research methodology, data analysis methods can include statistical analysis, content analysis, thematic analysis, or grounded theory.

Ensuring the Validity and Reliability of Your Research

In order to ensure the validity and reliability of your research findings, it is important to address these two key aspects:

Understanding Validity in Research

Validity refers to the accuracy and soundness of a research study. It is crucial to ensure that the research methods used effectively measure what they intend to measure. Researchers can enhance validity by using proper sampling techniques, carefully designing research instruments, and ensuring accurate data collection.

Ensuring Reliability in Your Study

Reliability refers to the consistency and stability of the research results. It is important to ensure that the research methods and instruments used yield consistent and reproducible results. Researchers can enhance reliability by using standardized procedures, ensuring inter-rater reliability, and conducting pilot studies.

A comprehensive understanding of research methodology is essential for conducting high-quality research. By selecting the right research methodology, researchers can ensure that their studies are rigorous, reliable, and valid. It is crucial to follow the steps in choosing the appropriate methodology, implement the chosen methodology effectively, and address validity and reliability concerns throughout the research process. By doing so, researchers can contribute valuable insights and advances in their respective fields.

You might also like

AI for Meta-Analysis — A Comprehensive Guide

Cybersecurity in Higher Education: Safeguarding Students and Faculty Data

How To Write An Argumentative Essay

Get science-backed answers as you write with Paperpal's Research feature

What is Research Methodology? Definition, Types, and Examples

Research methodology 1,2 is a structured and scientific approach used to collect, analyze, and interpret quantitative or qualitative data to answer research questions or test hypotheses. A research methodology is like a plan for carrying out research and helps keep researchers on track by limiting the scope of the research. Several aspects must be considered before selecting an appropriate research methodology, such as research limitations and ethical concerns that may affect your research.

The research methodology section in a scientific paper describes the different methodological choices made, such as the data collection and analysis methods, and why these choices were selected. The reasons should explain why the methods chosen are the most appropriate to answer the research question. A good research methodology also helps ensure the reliability and validity of the research findings. There are three types of research methodology—quantitative, qualitative, and mixed-method, which can be chosen based on the research objectives.

What is research methodology ?

A research methodology describes the techniques and procedures used to identify and analyze information regarding a specific research topic. It is a process by which researchers design their study so that they can achieve their objectives using the selected research instruments. It includes all the important aspects of research, including research design, data collection methods, data analysis methods, and the overall framework within which the research is conducted. While these points can help you understand what is research methodology, you also need to know why it is important to pick the right methodology.

Why is research methodology important?

Having a good research methodology in place has the following advantages: 3

• Helps other researchers who may want to replicate your research; the explanations will be of benefit to them.
• You can easily answer any questions about your research if they arise at a later stage.
• A research methodology provides a framework and guidelines for researchers to clearly define research questions, hypotheses, and objectives.
• It helps researchers identify the most appropriate research design, sampling technique, and data collection and analysis methods.
• A sound research methodology helps researchers ensure that their findings are valid and reliable and free from biases and errors.
• It also helps ensure that ethical guidelines are followed while conducting research.
• A good research methodology helps researchers in planning their research efficiently, by ensuring optimum usage of their time and resources.

Writing the methods section of a research paper? Let Paperpal help you achieve perfection

Types of research methodology.

There are three types of research methodology based on the type of research and the data required. 1

• Quantitative research methodology focuses on measuring and testing numerical data. This approach is good for reaching a large number of people in a short amount of time. This type of research helps in testing the causal relationships between variables, making predictions, and generalizing results to wider populations.
• Qualitative research methodology examines the opinions, behaviors, and experiences of people. It collects and analyzes words and textual data. This research methodology requires fewer participants but is still more time consuming because the time spent per participant is quite large. This method is used in exploratory research where the research problem being investigated is not clearly defined.
• Mixed-method research methodology uses the characteristics of both quantitative and qualitative research methodologies in the same study. This method allows researchers to validate their findings, verify if the results observed using both methods are complementary, and explain any unexpected results obtained from one method by using the other method.

What are the types of sampling designs in research methodology?

Sampling 4 is an important part of a research methodology and involves selecting a representative sample of the population to conduct the study, making statistical inferences about them, and estimating the characteristics of the whole population based on these inferences. There are two types of sampling designs in research methodology—probability and nonprobability.

• Probability sampling

In this type of sampling design, a sample is chosen from a larger population using some form of random selection, that is, every member of the population has an equal chance of being selected. The different types of probability sampling are:

• Systematic —sample members are chosen at regular intervals. It requires selecting a starting point for the sample and sample size determination that can be repeated at regular intervals. This type of sampling method has a predefined range; hence, it is the least time consuming.
• Stratified —researchers divide the population into smaller groups that don’t overlap but represent the entire population. While sampling, these groups can be organized, and then a sample can be drawn from each group separately.
• Cluster —the population is divided into clusters based on demographic parameters like age, sex, location, etc.
• Convenience —selects participants who are most easily accessible to researchers due to geographical proximity, availability at a particular time, etc.
• Purposive —participants are selected at the researcher’s discretion. Researchers consider the purpose of the study and the understanding of the target audience.
• Snowball —already selected participants use their social networks to refer the researcher to other potential participants.
• Quota —while designing the study, the researchers decide how many people with which characteristics to include as participants. The characteristics help in choosing people most likely to provide insights into the subject.

What are data collection methods?

During research, data are collected using various methods depending on the research methodology being followed and the research methods being undertaken. Both qualitative and quantitative research have different data collection methods, as listed below.

Qualitative research 5

• One-on-one interviews: Helps the interviewers understand a respondent’s subjective opinion and experience pertaining to a specific topic or event
• Document study/literature review/record keeping: Researchers’ review of already existing written materials such as archives, annual reports, research articles, guidelines, policy documents, etc.
• Focus groups: Constructive discussions that usually include a small sample of about 6-10 people and a moderator, to understand the participants’ opinion on a given topic.
• Qualitative observation : Researchers collect data using their five senses (sight, smell, touch, taste, and hearing).

Quantitative research 6

• Sampling: The most common type is probability sampling.
• Interviews: Commonly telephonic or done in-person.
• Observations: Structured observations are most commonly used in quantitative research. In this method, researchers make observations about specific behaviors of individuals in a structured setting.
• Document review: Reviewing existing research or documents to collect evidence for supporting the research.
• Surveys and questionnaires. Surveys can be administered both online and offline depending on the requirement and sample size.

Let Paperpal help you write the perfect research methods section. Start now!

What are data analysis methods.

The data collected using the various methods for qualitative and quantitative research need to be analyzed to generate meaningful conclusions. These data analysis methods 7 also differ between quantitative and qualitative research.

Quantitative research involves a deductive method for data analysis where hypotheses are developed at the beginning of the research and precise measurement is required. The methods include statistical analysis applications to analyze numerical data and are grouped into two categories—descriptive and inferential.

Descriptive analysis is used to describe the basic features of different types of data to present it in a way that ensures the patterns become meaningful. The different types of descriptive analysis methods are:

• Measures of frequency (count, percent, frequency)
• Measures of central tendency (mean, median, mode)
• Measures of dispersion or variation (range, variance, standard deviation)
• Measure of position (percentile ranks, quartile ranks)

Inferential analysis is used to make predictions about a larger population based on the analysis of the data collected from a smaller population. This analysis is used to study the relationships between different variables. Some commonly used inferential data analysis methods are:

• Correlation: To understand the relationship between two or more variables.
• Cross-tabulation: Analyze the relationship between multiple variables.
• Regression analysis: Study the impact of independent variables on the dependent variable.
• Frequency tables: To understand the frequency of data.
• Analysis of variance: To test the degree to which two or more variables differ in an experiment.

Qualitative research involves an inductive method for data analysis where hypotheses are developed after data collection. The methods include:

• Content analysis: For analyzing documented information from text and images by determining the presence of certain words or concepts in texts.
• Narrative analysis: For analyzing content obtained from sources such as interviews, field observations, and surveys. The stories and opinions shared by people are used to answer research questions.
• Discourse analysis: For analyzing interactions with people considering the social context, that is, the lifestyle and environment, under which the interaction occurs.
• Grounded theory: Involves hypothesis creation by data collection and analysis to explain why a phenomenon occurred.
• Thematic analysis: To identify important themes or patterns in data and use these to address an issue.

How to choose a research methodology?

Here are some important factors to consider when choosing a research methodology: 8

• Research objectives, aims, and questions —these would help structure the research design.
• Review existing literature to identify any gaps in knowledge.
• Check the statistical requirements —if data-driven or statistical results are needed then quantitative research is the best. If the research questions can be answered based on people’s opinions and perceptions, then qualitative research is most suitable.
• Sample size —sample size can often determine the feasibility of a research methodology. For a large sample, less effort- and time-intensive methods are appropriate.
• Constraints —constraints of time, geography, and resources can help define the appropriate methodology.

Got writer’s block? Kickstart your research paper writing with Paperpal now!

How to write a research methodology .

A research methodology should include the following components: 3,9

• Research design —should be selected based on the research question and the data required. Common research designs include experimental, quasi-experimental, correlational, descriptive, and exploratory.
• Research method —this can be quantitative, qualitative, or mixed-method.
• Reason for selecting a specific methodology —explain why this methodology is the most suitable to answer your research problem.
• Research instruments —explain the research instruments you plan to use, mainly referring to the data collection methods such as interviews, surveys, etc. Here as well, a reason should be mentioned for selecting the particular instrument.
• Sampling —this involves selecting a representative subset of the population being studied.
• Data collection —involves gathering data using several data collection methods, such as surveys, interviews, etc.
• Data analysis —describe the data analysis methods you will use once you’ve collected the data.
• Research limitations —mention any limitations you foresee while conducting your research.
• Validity and reliability —validity helps identify the accuracy and truthfulness of the findings; reliability refers to the consistency and stability of the results over time and across different conditions.
• Ethical considerations —research should be conducted ethically. The considerations include obtaining consent from participants, maintaining confidentiality, and addressing conflicts of interest.

Streamline Your Research Paper Writing Process with Paperpal

The methods section is a critical part of the research papers, allowing researchers to use this to understand your findings and replicate your work when pursuing their own research. However, it is usually also the most difficult section to write. This is where Paperpal can help you overcome the writer’s block and create the first draft in minutes with Paperpal Copilot, its secure generative AI feature suite.  

With Paperpal you can get research advice, write and refine your work, rephrase and verify the writing, and ensure submission readiness, all in one place. Here’s how you can use Paperpal to develop the first draft of your methods section.  

• Generate an outline: Input some details about your research to instantly generate an outline for your methods section 
• Develop the section: Use the outline and suggested sentence templates to expand your ideas and develop the first draft.  
• P araph ras e and trim : Get clear, concise academic text with paraphrasing that conveys your work effectively and word reduction to fix redundancies. 
• Choose the right words: Enhance text by choosing contextual synonyms based on how the words have been used in previously published work.  
• Check and verify text : Make sure the generated text showcases your methods correctly, has all the right citations, and is original and authentic. .   

You can repeat this process to develop each section of your research manuscript, including the title, abstract and keywords. Ready to write your research papers faster, better, and without the stress? Sign up for Paperpal and start writing today!

Frequently Asked Questions

Q1. What are the key components of research methodology?

A1. A good research methodology has the following key components:

• Research design
• Data collection procedures
• Data analysis methods
• Ethical considerations

Q2. Why is ethical consideration important in research methodology?

A2. Ethical consideration is important in research methodology to ensure the readers of the reliability and validity of the study. Researchers must clearly mention the ethical norms and standards followed during the conduct of the research and also mention if the research has been cleared by any institutional board. The following 10 points are the important principles related to ethical considerations: 10

• Participants should not be subjected to harm.
• Respect for the dignity of participants should be prioritized.
• Full consent should be obtained from participants before the study.
• Participants’ privacy should be ensured.
• Confidentiality of the research data should be ensured.
• Anonymity of individuals and organizations participating in the research should be maintained.
• The aims and objectives of the research should not be exaggerated.
• Affiliations, sources of funding, and any possible conflicts of interest should be declared.
• Communication in relation to the research should be honest and transparent.
• Misleading information and biased representation of primary data findings should be avoided.

Q3. What is the difference between methodology and method?

A3. Research methodology is different from a research method, although both terms are often confused. Research methods are the tools used to gather data, while the research methodology provides a framework for how research is planned, conducted, and analyzed. The latter guides researchers in making decisions about the most appropriate methods for their research. Research methods refer to the specific techniques, procedures, and tools used by researchers to collect, analyze, and interpret data, for instance surveys, questionnaires, interviews, etc.

Research methodology is, thus, an integral part of a research study. It helps ensure that you stay on track to meet your research objectives and answer your research questions using the most appropriate data collection and analysis tools based on your research design.

Accelerate your research paper writing with Paperpal. Try for free now!

• Research methodologies. Pfeiffer Library website. Accessed August 15, 2023. https://library.tiffin.edu/researchmethodologies/whatareresearchmethodologies
• Types of research methodology. Eduvoice website. Accessed August 16, 2023. https://eduvoice.in/types-research-methodology/
• The basics of research methodology: A key to quality research. Voxco. Accessed August 16, 2023. https://www.voxco.com/blog/what-is-research-methodology/
• Sampling methods: Types with examples. QuestionPro website. Accessed August 16, 2023. https://www.questionpro.com/blog/types-of-sampling-for-social-research/
• What is qualitative research? Methods, types, approaches, examples. Researcher.Life blog. Accessed August 15, 2023. https://researcher.life/blog/article/what-is-qualitative-research-methods-types-examples/
• What is quantitative research? Definition, methods, types, and examples. Researcher.Life blog. Accessed August 15, 2023. https://researcher.life/blog/article/what-is-quantitative-research-types-and-examples/
• Data analysis in research: Types & methods. QuestionPro website. Accessed August 16, 2023. https://www.questionpro.com/blog/data-analysis-in-research/#Data_analysis_in_qualitative_research
• Factors to consider while choosing the right research methodology. PhD Monster website. Accessed August 17, 2023. https://www.phdmonster.com/factors-to-consider-while-choosing-the-right-research-methodology/
• What is research methodology? Research and writing guides. Accessed August 14, 2023. https://paperpile.com/g/what-is-research-methodology/
• Ethical considerations. Business research methodology website. Accessed August 17, 2023. https://research-methodology.net/research-methodology/ethical-considerations/

Paperpal is a comprehensive AI writing toolkit that helps students and researchers achieve 2x the writing in half the time. It leverages 21+ years of STM experience and insights from millions of research articles to provide in-depth academic writing, language editing, and submission readiness support to help you write better, faster.  

Get accurate academic translations, rewriting support, grammar checks, vocabulary suggestions, and generative AI assistance that delivers human precision at machine speed. Try for free or upgrade to Paperpal Prime starting at US$19 a month to access premium features, including consistency, plagiarism, and 30+ submission readiness checks to help you succeed.  

Experience the future of academic writing – Sign up to Paperpal and start writing for free!  

Related Reads:

• Dangling Modifiers and How to Avoid Them in Your Writing 
• Webinar: How to Use Generative AI Tools Ethically in Your Academic Writing
• Research Outlines: How to Write An Introduction Section in Minutes with Paperpal Copilot
• How to Paraphrase Research Papers Effectively

Language and Grammar Rules for Academic Writing

Climatic vs. climactic: difference and examples, you may also like, what is hedging in academic writing  , how to use ai to enhance your college..., how to use paperpal to generate emails &..., ai in education: it’s time to change the..., is it ethical to use ai-generated abstracts without..., do plagiarism checkers detect ai content, word choice problems: how to use the right..., how to avoid plagiarism when using generative ai..., what are journal guidelines on using generative ai..., types of plagiarism and 6 tips to avoid....

• Open access
• Published: 07 September 2020

A tutorial on methodological studies: the what, when, how and why

• Lawrence Mbuagbaw   ORCID: orcid.org/0000-0001-5855-5461 1 , 2 , 3 ,
• Daeria O. Lawson 1 ,
• Livia Puljak 4 ,
• David B. Allison 5 &
• Lehana Thabane 1 , 2 , 6 , 7 , 8  

BMC Medical Research Methodology volume  20 , Article number:  226 ( 2020 ) Cite this article

37k Accesses

52 Citations

58 Altmetric

Metrics details

Methodological studies – studies that evaluate the design, analysis or reporting of other research-related reports – play an important role in health research. They help to highlight issues in the conduct of research with the aim of improving health research methodology, and ultimately reducing research waste.

We provide an overview of some of the key aspects of methodological studies such as what they are, and when, how and why they are done. We adopt a “frequently asked questions” format to facilitate reading this paper and provide multiple examples to help guide researchers interested in conducting methodological studies. Some of the topics addressed include: is it necessary to publish a study protocol? How to select relevant research reports and databases for a methodological study? What approaches to data extraction and statistical analysis should be considered when conducting a methodological study? What are potential threats to validity and is there a way to appraise the quality of methodological studies?

Appropriate reflection and application of basic principles of epidemiology and biostatistics are required in the design and analysis of methodological studies. This paper provides an introduction for further discussion about the conduct of methodological studies.

Peer Review reports

The field of meta-research (or research-on-research) has proliferated in recent years in response to issues with research quality and conduct [ 1 , 2 , 3 ]. As the name suggests, this field targets issues with research design, conduct, analysis and reporting. Various types of research reports are often examined as the unit of analysis in these studies (e.g. abstracts, full manuscripts, trial registry entries). Like many other novel fields of research, meta-research has seen a proliferation of use before the development of reporting guidance. For example, this was the case with randomized trials for which risk of bias tools and reporting guidelines were only developed much later – after many trials had been published and noted to have limitations [ 4 , 5 ]; and for systematic reviews as well [ 6 , 7 , 8 ]. However, in the absence of formal guidance, studies that report on research differ substantially in how they are named, conducted and reported [ 9 , 10 ]. This creates challenges in identifying, summarizing and comparing them. In this tutorial paper, we will use the term methodological study to refer to any study that reports on the design, conduct, analysis or reporting of primary or secondary research-related reports (such as trial registry entries and conference abstracts).

In the past 10 years, there has been an increase in the use of terms related to methodological studies (based on records retrieved with a keyword search [in the title and abstract] for “methodological review” and “meta-epidemiological study” in PubMed up to December 2019), suggesting that these studies may be appearing more frequently in the literature. See Fig.  1 .

Trends in the number studies that mention “methodological review” or “meta-

epidemiological study” in PubMed.

The methods used in many methodological studies have been borrowed from systematic and scoping reviews. This practice has influenced the direction of the field, with many methodological studies including searches of electronic databases, screening of records, duplicate data extraction and assessments of risk of bias in the included studies. However, the research questions posed in methodological studies do not always require the approaches listed above, and guidance is needed on when and how to apply these methods to a methodological study. Even though methodological studies can be conducted on qualitative or mixed methods research, this paper focuses on and draws examples exclusively from quantitative research.

The objectives of this paper are to provide some insights on how to conduct methodological studies so that there is greater consistency between the research questions posed, and the design, analysis and reporting of findings. We provide multiple examples to illustrate concepts and a proposed framework for categorizing methodological studies in quantitative research.

What is a methodological study?

Any study that describes or analyzes methods (design, conduct, analysis or reporting) in published (or unpublished) literature is a methodological study. Consequently, the scope of methodological studies is quite extensive and includes, but is not limited to, topics as diverse as: research question formulation [ 11 ]; adherence to reporting guidelines [ 12 , 13 , 14 ] and consistency in reporting [ 15 ]; approaches to study analysis [ 16 ]; investigating the credibility of analyses [ 17 ]; and studies that synthesize these methodological studies [ 18 ]. While the nomenclature of methodological studies is not uniform, the intents and purposes of these studies remain fairly consistent – to describe or analyze methods in primary or secondary studies. As such, methodological studies may also be classified as a subtype of observational studies.

Parallel to this are experimental studies that compare different methods. Even though they play an important role in informing optimal research methods, experimental methodological studies are beyond the scope of this paper. Examples of such studies include the randomized trials by Buscemi et al., comparing single data extraction to double data extraction [ 19 ], and Carrasco-Labra et al., comparing approaches to presenting findings in Grading of Recommendations, Assessment, Development and Evaluations (GRADE) summary of findings tables [ 20 ]. In these studies, the unit of analysis is the person or groups of individuals applying the methods. We also direct readers to the Studies Within a Trial (SWAT) and Studies Within a Review (SWAR) programme operated through the Hub for Trials Methodology Research, for further reading as a potential useful resource for these types of experimental studies [ 21 ]. Lastly, this paper is not meant to inform the conduct of research using computational simulation and mathematical modeling for which some guidance already exists [ 22 ], or studies on the development of methods using consensus-based approaches.

When should we conduct a methodological study?

Methodological studies occupy a unique niche in health research that allows them to inform methodological advances. Methodological studies should also be conducted as pre-cursors to reporting guideline development, as they provide an opportunity to understand current practices, and help to identify the need for guidance and gaps in methodological or reporting quality. For example, the development of the popular Preferred Reporting Items of Systematic reviews and Meta-Analyses (PRISMA) guidelines were preceded by methodological studies identifying poor reporting practices [ 23 , 24 ]. In these instances, after the reporting guidelines are published, methodological studies can also be used to monitor uptake of the guidelines.

These studies can also be conducted to inform the state of the art for design, analysis and reporting practices across different types of health research fields, with the aim of improving research practices, and preventing or reducing research waste. For example, Samaan et al. conducted a scoping review of adherence to different reporting guidelines in health care literature [ 18 ]. Methodological studies can also be used to determine the factors associated with reporting practices. For example, Abbade et al. investigated journal characteristics associated with the use of the Participants, Intervention, Comparison, Outcome, Timeframe (PICOT) format in framing research questions in trials of venous ulcer disease [ 11 ].

How often are methodological studies conducted?

There is no clear answer to this question. Based on a search of PubMed, the use of related terms (“methodological review” and “meta-epidemiological study”) – and therefore, the number of methodological studies – is on the rise. However, many other terms are used to describe methodological studies. There are also many studies that explore design, conduct, analysis or reporting of research reports, but that do not use any specific terms to describe or label their study design in terms of “methodology”. This diversity in nomenclature makes a census of methodological studies elusive. Appropriate terminology and key words for methodological studies are needed to facilitate improved accessibility for end-users.

Why do we conduct methodological studies?

Methodological studies provide information on the design, conduct, analysis or reporting of primary and secondary research and can be used to appraise quality, quantity, completeness, accuracy and consistency of health research. These issues can be explored in specific fields, journals, databases, geographical regions and time periods. For example, Areia et al. explored the quality of reporting of endoscopic diagnostic studies in gastroenterology [ 25 ]; Knol et al. investigated the reporting of p -values in baseline tables in randomized trial published in high impact journals [ 26 ]; Chen et al. describe adherence to the Consolidated Standards of Reporting Trials (CONSORT) statement in Chinese Journals [ 27 ]; and Hopewell et al. describe the effect of editors’ implementation of CONSORT guidelines on reporting of abstracts over time [ 28 ]. Methodological studies provide useful information to researchers, clinicians, editors, publishers and users of health literature. As a result, these studies have been at the cornerstone of important methodological developments in the past two decades and have informed the development of many health research guidelines including the highly cited CONSORT statement [ 5 ].

Where can we find methodological studies?

Methodological studies can be found in most common biomedical bibliographic databases (e.g. Embase, MEDLINE, PubMed, Web of Science). However, the biggest caveat is that methodological studies are hard to identify in the literature due to the wide variety of names used and the lack of comprehensive databases dedicated to them. A handful can be found in the Cochrane Library as “Cochrane Methodology Reviews”, but these studies only cover methodological issues related to systematic reviews. Previous attempts to catalogue all empirical studies of methods used in reviews were abandoned 10 years ago [ 29 ]. In other databases, a variety of search terms may be applied with different levels of sensitivity and specificity.

Some frequently asked questions about methodological studies

In this section, we have outlined responses to questions that might help inform the conduct of methodological studies.

Q: How should I select research reports for my methodological study?

A: Selection of research reports for a methodological study depends on the research question and eligibility criteria. Once a clear research question is set and the nature of literature one desires to review is known, one can then begin the selection process. Selection may begin with a broad search, especially if the eligibility criteria are not apparent. For example, a methodological study of Cochrane Reviews of HIV would not require a complex search as all eligible studies can easily be retrieved from the Cochrane Library after checking a few boxes [ 30 ]. On the other hand, a methodological study of subgroup analyses in trials of gastrointestinal oncology would require a search to find such trials, and further screening to identify trials that conducted a subgroup analysis [ 31 ].

The strategies used for identifying participants in observational studies can apply here. One may use a systematic search to identify all eligible studies. If the number of eligible studies is unmanageable, a random sample of articles can be expected to provide comparable results if it is sufficiently large [ 32 ]. For example, Wilson et al. used a random sample of trials from the Cochrane Stroke Group’s Trial Register to investigate completeness of reporting [ 33 ]. It is possible that a simple random sample would lead to underrepresentation of units (i.e. research reports) that are smaller in number. This is relevant if the investigators wish to compare multiple groups but have too few units in one group. In this case a stratified sample would help to create equal groups. For example, in a methodological study comparing Cochrane and non-Cochrane reviews, Kahale et al. drew random samples from both groups [ 34 ]. Alternatively, systematic or purposeful sampling strategies can be used and we encourage researchers to justify their selected approaches based on the study objective.

Q: How many databases should I search?

A: The number of databases one should search would depend on the approach to sampling, which can include targeting the entire “population” of interest or a sample of that population. If you are interested in including the entire target population for your research question, or drawing a random or systematic sample from it, then a comprehensive and exhaustive search for relevant articles is required. In this case, we recommend using systematic approaches for searching electronic databases (i.e. at least 2 databases with a replicable and time stamped search strategy). The results of your search will constitute a sampling frame from which eligible studies can be drawn.

Alternatively, if your approach to sampling is purposeful, then we recommend targeting the database(s) or data sources (e.g. journals, registries) that include the information you need. For example, if you are conducting a methodological study of high impact journals in plastic surgery and they are all indexed in PubMed, you likely do not need to search any other databases. You may also have a comprehensive list of all journals of interest and can approach your search using the journal names in your database search (or by accessing the journal archives directly from the journal’s website). Even though one could also search journals’ web pages directly, using a database such as PubMed has multiple advantages, such as the use of filters, so the search can be narrowed down to a certain period, or study types of interest. Furthermore, individual journals’ web sites may have different search functionalities, which do not necessarily yield a consistent output.

Q: Should I publish a protocol for my methodological study?

A: A protocol is a description of intended research methods. Currently, only protocols for clinical trials require registration [ 35 ]. Protocols for systematic reviews are encouraged but no formal recommendation exists. The scientific community welcomes the publication of protocols because they help protect against selective outcome reporting, the use of post hoc methodologies to embellish results, and to help avoid duplication of efforts [ 36 ]. While the latter two risks exist in methodological research, the negative consequences may be substantially less than for clinical outcomes. In a sample of 31 methodological studies, 7 (22.6%) referenced a published protocol [ 9 ]. In the Cochrane Library, there are 15 protocols for methodological reviews (21 July 2020). This suggests that publishing protocols for methodological studies is not uncommon.

Authors can consider publishing their study protocol in a scholarly journal as a manuscript. Advantages of such publication include obtaining peer-review feedback about the planned study, and easy retrieval by searching databases such as PubMed. The disadvantages in trying to publish protocols includes delays associated with manuscript handling and peer review, as well as costs, as few journals publish study protocols, and those journals mostly charge article-processing fees [ 37 ]. Authors who would like to make their protocol publicly available without publishing it in scholarly journals, could deposit their study protocols in publicly available repositories, such as the Open Science Framework ( https://osf.io/ ).

Q: How to appraise the quality of a methodological study?

A: To date, there is no published tool for appraising the risk of bias in a methodological study, but in principle, a methodological study could be considered as a type of observational study. Therefore, during conduct or appraisal, care should be taken to avoid the biases common in observational studies [ 38 ]. These biases include selection bias, comparability of groups, and ascertainment of exposure or outcome. In other words, to generate a representative sample, a comprehensive reproducible search may be necessary to build a sampling frame. Additionally, random sampling may be necessary to ensure that all the included research reports have the same probability of being selected, and the screening and selection processes should be transparent and reproducible. To ensure that the groups compared are similar in all characteristics, matching, random sampling or stratified sampling can be used. Statistical adjustments for between-group differences can also be applied at the analysis stage. Finally, duplicate data extraction can reduce errors in assessment of exposures or outcomes.

Q: Should I justify a sample size?

A: In all instances where one is not using the target population (i.e. the group to which inferences from the research report are directed) [ 39 ], a sample size justification is good practice. The sample size justification may take the form of a description of what is expected to be achieved with the number of articles selected, or a formal sample size estimation that outlines the number of articles required to answer the research question with a certain precision and power. Sample size justifications in methodological studies are reasonable in the following instances:

Comparing two groups

Determining a proportion, mean or another quantifier

Determining factors associated with an outcome using regression-based analyses

For example, El Dib et al. computed a sample size requirement for a methodological study of diagnostic strategies in randomized trials, based on a confidence interval approach [ 40 ].

Q: What should I call my study?

A: Other terms which have been used to describe/label methodological studies include “ methodological review ”, “methodological survey” , “meta-epidemiological study” , “systematic review” , “systematic survey”, “meta-research”, “research-on-research” and many others. We recommend that the study nomenclature be clear, unambiguous, informative and allow for appropriate indexing. Methodological study nomenclature that should be avoided includes “ systematic review” – as this will likely be confused with a systematic review of a clinical question. “ Systematic survey” may also lead to confusion about whether the survey was systematic (i.e. using a preplanned methodology) or a survey using “ systematic” sampling (i.e. a sampling approach using specific intervals to determine who is selected) [ 32 ]. Any of the above meanings of the words “ systematic” may be true for methodological studies and could be potentially misleading. “ Meta-epidemiological study” is ideal for indexing, but not very informative as it describes an entire field. The term “ review ” may point towards an appraisal or “review” of the design, conduct, analysis or reporting (or methodological components) of the targeted research reports, yet it has also been used to describe narrative reviews [ 41 , 42 ]. The term “ survey ” is also in line with the approaches used in many methodological studies [ 9 ], and would be indicative of the sampling procedures of this study design. However, in the absence of guidelines on nomenclature, the term “ methodological study ” is broad enough to capture most of the scenarios of such studies.

Q: Should I account for clustering in my methodological study?

A: Data from methodological studies are often clustered. For example, articles coming from a specific source may have different reporting standards (e.g. the Cochrane Library). Articles within the same journal may be similar due to editorial practices and policies, reporting requirements and endorsement of guidelines. There is emerging evidence that these are real concerns that should be accounted for in analyses [ 43 ]. Some cluster variables are described in the section: “ What variables are relevant to methodological studies?”

A variety of modelling approaches can be used to account for correlated data, including the use of marginal, fixed or mixed effects regression models with appropriate computation of standard errors [ 44 ]. For example, Kosa et al. used generalized estimation equations to account for correlation of articles within journals [ 15 ]. Not accounting for clustering could lead to incorrect p -values, unduly narrow confidence intervals, and biased estimates [ 45 ].

Q: Should I extract data in duplicate?

A: Yes. Duplicate data extraction takes more time but results in less errors [ 19 ]. Data extraction errors in turn affect the effect estimate [ 46 ], and therefore should be mitigated. Duplicate data extraction should be considered in the absence of other approaches to minimize extraction errors. However, much like systematic reviews, this area will likely see rapid new advances with machine learning and natural language processing technologies to support researchers with screening and data extraction [ 47 , 48 ]. However, experience plays an important role in the quality of extracted data and inexperienced extractors should be paired with experienced extractors [ 46 , 49 ].

Q: Should I assess the risk of bias of research reports included in my methodological study?

A : Risk of bias is most useful in determining the certainty that can be placed in the effect measure from a study. In methodological studies, risk of bias may not serve the purpose of determining the trustworthiness of results, as effect measures are often not the primary goal of methodological studies. Determining risk of bias in methodological studies is likely a practice borrowed from systematic review methodology, but whose intrinsic value is not obvious in methodological studies. When it is part of the research question, investigators often focus on one aspect of risk of bias. For example, Speich investigated how blinding was reported in surgical trials [ 50 ], and Abraha et al., investigated the application of intention-to-treat analyses in systematic reviews and trials [ 51 ].

Q: What variables are relevant to methodological studies?

A: There is empirical evidence that certain variables may inform the findings in a methodological study. We outline some of these and provide a brief overview below:

Country: Countries and regions differ in their research cultures, and the resources available to conduct research. Therefore, it is reasonable to believe that there may be differences in methodological features across countries. Methodological studies have reported loco-regional differences in reporting quality [ 52 , 53 ]. This may also be related to challenges non-English speakers face in publishing papers in English.

Authors’ expertise: The inclusion of authors with expertise in research methodology, biostatistics, and scientific writing is likely to influence the end-product. Oltean et al. found that among randomized trials in orthopaedic surgery, the use of analyses that accounted for clustering was more likely when specialists (e.g. statistician, epidemiologist or clinical trials methodologist) were included on the study team [ 54 ]. Fleming et al. found that including methodologists in the review team was associated with appropriate use of reporting guidelines [ 55 ].

Source of funding and conflicts of interest: Some studies have found that funded studies report better [ 56 , 57 ], while others do not [ 53 , 58 ]. The presence of funding would indicate the availability of resources deployed to ensure optimal design, conduct, analysis and reporting. However, the source of funding may introduce conflicts of interest and warrant assessment. For example, Kaiser et al. investigated the effect of industry funding on obesity or nutrition randomized trials and found that reporting quality was similar [ 59 ]. Thomas et al. looked at reporting quality of long-term weight loss trials and found that industry funded studies were better [ 60 ]. Kan et al. examined the association between industry funding and “positive trials” (trials reporting a significant intervention effect) and found that industry funding was highly predictive of a positive trial [ 61 ]. This finding is similar to that of a recent Cochrane Methodology Review by Hansen et al. [ 62 ]

Journal characteristics: Certain journals’ characteristics may influence the study design, analysis or reporting. Characteristics such as journal endorsement of guidelines [ 63 , 64 ], and Journal Impact Factor (JIF) have been shown to be associated with reporting [ 63 , 65 , 66 , 67 ].

Study size (sample size/number of sites): Some studies have shown that reporting is better in larger studies [ 53 , 56 , 58 ].

Year of publication: It is reasonable to assume that design, conduct, analysis and reporting of research will change over time. Many studies have demonstrated improvements in reporting over time or after the publication of reporting guidelines [ 68 , 69 ].

Type of intervention: In a methodological study of reporting quality of weight loss intervention studies, Thabane et al. found that trials of pharmacologic interventions were reported better than trials of non-pharmacologic interventions [ 70 ].

Interactions between variables: Complex interactions between the previously listed variables are possible. High income countries with more resources may be more likely to conduct larger studies and incorporate a variety of experts. Authors in certain countries may prefer certain journals, and journal endorsement of guidelines and editorial policies may change over time.

Q: Should I focus only on high impact journals?

A: Investigators may choose to investigate only high impact journals because they are more likely to influence practice and policy, or because they assume that methodological standards would be higher. However, the JIF may severely limit the scope of articles included and may skew the sample towards articles with positive findings. The generalizability and applicability of findings from a handful of journals must be examined carefully, especially since the JIF varies over time. Even among journals that are all “high impact”, variations exist in methodological standards.

Q: Can I conduct a methodological study of qualitative research?

A: Yes. Even though a lot of methodological research has been conducted in the quantitative research field, methodological studies of qualitative studies are feasible. Certain databases that catalogue qualitative research including the Cumulative Index to Nursing & Allied Health Literature (CINAHL) have defined subject headings that are specific to methodological research (e.g. “research methodology”). Alternatively, one could also conduct a qualitative methodological review; that is, use qualitative approaches to synthesize methodological issues in qualitative studies.

Q: What reporting guidelines should I use for my methodological study?

A: There is no guideline that covers the entire scope of methodological studies. One adaptation of the PRISMA guidelines has been published, which works well for studies that aim to use the entire target population of research reports [ 71 ]. However, it is not widely used (40 citations in 2 years as of 09 December 2019), and methodological studies that are designed as cross-sectional or before-after studies require a more fit-for purpose guideline. A more encompassing reporting guideline for a broad range of methodological studies is currently under development [ 72 ]. However, in the absence of formal guidance, the requirements for scientific reporting should be respected, and authors of methodological studies should focus on transparency and reproducibility.

Q: What are the potential threats to validity and how can I avoid them?

A: Methodological studies may be compromised by a lack of internal or external validity. The main threats to internal validity in methodological studies are selection and confounding bias. Investigators must ensure that the methods used to select articles does not make them differ systematically from the set of articles to which they would like to make inferences. For example, attempting to make extrapolations to all journals after analyzing high-impact journals would be misleading.

Many factors (confounders) may distort the association between the exposure and outcome if the included research reports differ with respect to these factors [ 73 ]. For example, when examining the association between source of funding and completeness of reporting, it may be necessary to account for journals that endorse the guidelines. Confounding bias can be addressed by restriction, matching and statistical adjustment [ 73 ]. Restriction appears to be the method of choice for many investigators who choose to include only high impact journals or articles in a specific field. For example, Knol et al. examined the reporting of p -values in baseline tables of high impact journals [ 26 ]. Matching is also sometimes used. In the methodological study of non-randomized interventional studies of elective ventral hernia repair, Parker et al. matched prospective studies with retrospective studies and compared reporting standards [ 74 ]. Some other methodological studies use statistical adjustments. For example, Zhang et al. used regression techniques to determine the factors associated with missing participant data in trials [ 16 ].

With regard to external validity, researchers interested in conducting methodological studies must consider how generalizable or applicable their findings are. This should tie in closely with the research question and should be explicit. For example. Findings from methodological studies on trials published in high impact cardiology journals cannot be assumed to be applicable to trials in other fields. However, investigators must ensure that their sample truly represents the target sample either by a) conducting a comprehensive and exhaustive search, or b) using an appropriate and justified, randomly selected sample of research reports.

Even applicability to high impact journals may vary based on the investigators’ definition, and over time. For example, for high impact journals in the field of general medicine, Bouwmeester et al. included the Annals of Internal Medicine (AIM), BMJ, the Journal of the American Medical Association (JAMA), Lancet, the New England Journal of Medicine (NEJM), and PLoS Medicine ( n  = 6) [ 75 ]. In contrast, the high impact journals selected in the methodological study by Schiller et al. were BMJ, JAMA, Lancet, and NEJM ( n  = 4) [ 76 ]. Another methodological study by Kosa et al. included AIM, BMJ, JAMA, Lancet and NEJM ( n  = 5). In the methodological study by Thabut et al., journals with a JIF greater than 5 were considered to be high impact. Riado Minguez et al. used first quartile journals in the Journal Citation Reports (JCR) for a specific year to determine “high impact” [ 77 ]. Ultimately, the definition of high impact will be based on the number of journals the investigators are willing to include, the year of impact and the JIF cut-off [ 78 ]. We acknowledge that the term “generalizability” may apply differently for methodological studies, especially when in many instances it is possible to include the entire target population in the sample studied.

Finally, methodological studies are not exempt from information bias which may stem from discrepancies in the included research reports [ 79 ], errors in data extraction, or inappropriate interpretation of the information extracted. Likewise, publication bias may also be a concern in methodological studies, but such concepts have not yet been explored.

A proposed framework

In order to inform discussions about methodological studies, the development of guidance for what should be reported, we have outlined some key features of methodological studies that can be used to classify them. For each of the categories outlined below, we provide an example. In our experience, the choice of approach to completing a methodological study can be informed by asking the following four questions:

What is the aim?

Methodological studies that investigate bias

A methodological study may be focused on exploring sources of bias in primary or secondary studies (meta-bias), or how bias is analyzed. We have taken care to distinguish bias (i.e. systematic deviations from the truth irrespective of the source) from reporting quality or completeness (i.e. not adhering to a specific reporting guideline or norm). An example of where this distinction would be important is in the case of a randomized trial with no blinding. This study (depending on the nature of the intervention) would be at risk of performance bias. However, if the authors report that their study was not blinded, they would have reported adequately. In fact, some methodological studies attempt to capture both “quality of conduct” and “quality of reporting”, such as Richie et al., who reported on the risk of bias in randomized trials of pharmacy practice interventions [ 80 ]. Babic et al. investigated how risk of bias was used to inform sensitivity analyses in Cochrane reviews [ 81 ]. Further, biases related to choice of outcomes can also be explored. For example, Tan et al investigated differences in treatment effect size based on the outcome reported [ 82 ].

Methodological studies that investigate quality (or completeness) of reporting

Methodological studies may report quality of reporting against a reporting checklist (i.e. adherence to guidelines) or against expected norms. For example, Croituro et al. report on the quality of reporting in systematic reviews published in dermatology journals based on their adherence to the PRISMA statement [ 83 ], and Khan et al. described the quality of reporting of harms in randomized controlled trials published in high impact cardiovascular journals based on the CONSORT extension for harms [ 84 ]. Other methodological studies investigate reporting of certain features of interest that may not be part of formally published checklists or guidelines. For example, Mbuagbaw et al. described how often the implications for research are elaborated using the Evidence, Participants, Intervention, Comparison, Outcome, Timeframe (EPICOT) format [ 30 ].

Methodological studies that investigate the consistency of reporting

Sometimes investigators may be interested in how consistent reports of the same research are, as it is expected that there should be consistency between: conference abstracts and published manuscripts; manuscript abstracts and manuscript main text; and trial registration and published manuscript. For example, Rosmarakis et al. investigated consistency between conference abstracts and full text manuscripts [ 85 ].

Methodological studies that investigate factors associated with reporting

In addition to identifying issues with reporting in primary and secondary studies, authors of methodological studies may be interested in determining the factors that are associated with certain reporting practices. Many methodological studies incorporate this, albeit as a secondary outcome. For example, Farrokhyar et al. investigated the factors associated with reporting quality in randomized trials of coronary artery bypass grafting surgery [ 53 ].

Methodological studies that investigate methods

Methodological studies may also be used to describe methods or compare methods, and the factors associated with methods. Muller et al. described the methods used for systematic reviews and meta-analyses of observational studies [ 86 ].

Methodological studies that summarize other methodological studies

Some methodological studies synthesize results from other methodological studies. For example, Li et al. conducted a scoping review of methodological reviews that investigated consistency between full text and abstracts in primary biomedical research [ 87 ].

Methodological studies that investigate nomenclature and terminology

Some methodological studies may investigate the use of names and terms in health research. For example, Martinic et al. investigated the definitions of systematic reviews used in overviews of systematic reviews (OSRs), meta-epidemiological studies and epidemiology textbooks [ 88 ].

Other types of methodological studies

In addition to the previously mentioned experimental methodological studies, there may exist other types of methodological studies not captured here.

What is the design?

Methodological studies that are descriptive

Most methodological studies are purely descriptive and report their findings as counts (percent) and means (standard deviation) or medians (interquartile range). For example, Mbuagbaw et al. described the reporting of research recommendations in Cochrane HIV systematic reviews [ 30 ]. Gohari et al. described the quality of reporting of randomized trials in diabetes in Iran [ 12 ].

Methodological studies that are analytical

Some methodological studies are analytical wherein “analytical studies identify and quantify associations, test hypotheses, identify causes and determine whether an association exists between variables, such as between an exposure and a disease.” [ 89 ] In the case of methodological studies all these investigations are possible. For example, Kosa et al. investigated the association between agreement in primary outcome from trial registry to published manuscript and study covariates. They found that larger and more recent studies were more likely to have agreement [ 15 ]. Tricco et al. compared the conclusion statements from Cochrane and non-Cochrane systematic reviews with a meta-analysis of the primary outcome and found that non-Cochrane reviews were more likely to report positive findings. These results are a test of the null hypothesis that the proportions of Cochrane and non-Cochrane reviews that report positive results are equal [ 90 ].

What is the sampling strategy?

Methodological studies that include the target population

Methodological reviews with narrow research questions may be able to include the entire target population. For example, in the methodological study of Cochrane HIV systematic reviews, Mbuagbaw et al. included all of the available studies ( n  = 103) [ 30 ].

Methodological studies that include a sample of the target population

Many methodological studies use random samples of the target population [ 33 , 91 , 92 ]. Alternatively, purposeful sampling may be used, limiting the sample to a subset of research-related reports published within a certain time period, or in journals with a certain ranking or on a topic. Systematic sampling can also be used when random sampling may be challenging to implement.

What is the unit of analysis?

Methodological studies with a research report as the unit of analysis

Many methodological studies use a research report (e.g. full manuscript of study, abstract portion of the study) as the unit of analysis, and inferences can be made at the study-level. However, both published and unpublished research-related reports can be studied. These may include articles, conference abstracts, registry entries etc.

Methodological studies with a design, analysis or reporting item as the unit of analysis

Some methodological studies report on items which may occur more than once per article. For example, Paquette et al. report on subgroup analyses in Cochrane reviews of atrial fibrillation in which 17 systematic reviews planned 56 subgroup analyses [ 93 ].

This framework is outlined in Fig.  2 .

A proposed framework for methodological studies

Conclusions

Methodological studies have examined different aspects of reporting such as quality, completeness, consistency and adherence to reporting guidelines. As such, many of the methodological study examples cited in this tutorial are related to reporting. However, as an evolving field, the scope of research questions that can be addressed by methodological studies is expected to increase.

In this paper we have outlined the scope and purpose of methodological studies, along with examples of instances in which various approaches have been used. In the absence of formal guidance on the design, conduct, analysis and reporting of methodological studies, we have provided some advice to help make methodological studies consistent. This advice is grounded in good contemporary scientific practice. Generally, the research question should tie in with the sampling approach and planned analysis. We have also highlighted the variables that may inform findings from methodological studies. Lastly, we have provided suggestions for ways in which authors can categorize their methodological studies to inform their design and analysis.

Availability of data and materials

Data sharing is not applicable to this article as no new data were created or analyzed in this study.

Abbreviations

Consolidated Standards of Reporting Trials

Evidence, Participants, Intervention, Comparison, Outcome, Timeframe

Grading of Recommendations, Assessment, Development and Evaluations

Participants, Intervention, Comparison, Outcome, Timeframe

Preferred Reporting Items of Systematic reviews and Meta-Analyses

Studies Within a Review

Studies Within a Trial

Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet. 2009;374(9683):86–9.

PubMed   Google Scholar  

Chan AW, Song F, Vickers A, Jefferson T, Dickersin K, Gotzsche PC, Krumholz HM, Ghersi D, van der Worp HB. Increasing value and reducing waste: addressing inaccessible research. Lancet. 2014;383(9913):257–66.

PubMed   PubMed Central   Google Scholar  

Ioannidis JP, Greenland S, Hlatky MA, Khoury MJ, Macleod MR, Moher D, Schulz KF, Tibshirani R. Increasing value and reducing waste in research design, conduct, and analysis. Lancet. 2014;383(9912):166–75.

Higgins JP, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, Savovic J, Schulz KF, Weeks L, Sterne JA. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ. 2011;343:d5928.

Moher D, Schulz KF, Altman DG. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet. 2001;357.

Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JP, Clarke M, Devereaux PJ, Kleijnen J, Moher D. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 2009;6(7):e1000100.

Shea BJ, Hamel C, Wells GA, Bouter LM, Kristjansson E, Grimshaw J, Henry DA, Boers M. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol. 2009;62(10):1013–20.

Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, Moher D, Tugwell P, Welch V, Kristjansson E, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. Bmj. 2017;358:j4008.

Lawson DO, Leenus A, Mbuagbaw L. Mapping the nomenclature, methodology, and reporting of studies that review methods: a pilot methodological review. Pilot Feasibility Studies. 2020;6(1):13.

Puljak L, Makaric ZL, Buljan I, Pieper D. What is a meta-epidemiological study? Analysis of published literature indicated heterogeneous study designs and definitions. J Comp Eff Res. 2020.

Abbade LPF, Wang M, Sriganesh K, Jin Y, Mbuagbaw L, Thabane L. The framing of research questions using the PICOT format in randomized controlled trials of venous ulcer disease is suboptimal: a systematic survey. Wound Repair Regen. 2017;25(5):892–900.

Gohari F, Baradaran HR, Tabatabaee M, Anijidani S, Mohammadpour Touserkani F, Atlasi R, Razmgir M. Quality of reporting randomized controlled trials (RCTs) in diabetes in Iran; a systematic review. J Diabetes Metab Disord. 2015;15(1):36.

Wang M, Jin Y, Hu ZJ, Thabane A, Dennis B, Gajic-Veljanoski O, Paul J, Thabane L. The reporting quality of abstracts of stepped wedge randomized trials is suboptimal: a systematic survey of the literature. Contemp Clin Trials Commun. 2017;8:1–10.

Shanthanna H, Kaushal A, Mbuagbaw L, Couban R, Busse J, Thabane L: A cross-sectional study of the reporting quality of pilot or feasibility trials in high-impact anesthesia journals Can J Anaesthesia 2018, 65(11):1180–1195.

Kosa SD, Mbuagbaw L, Borg Debono V, Bhandari M, Dennis BB, Ene G, Leenus A, Shi D, Thabane M, Valvasori S, et al. Agreement in reporting between trial publications and current clinical trial registry in high impact journals: a methodological review. Contemporary Clinical Trials. 2018;65:144–50.

Zhang Y, Florez ID, Colunga Lozano LE, Aloweni FAB, Kennedy SA, Li A, Craigie S, Zhang S, Agarwal A, Lopes LC, et al. A systematic survey on reporting and methods for handling missing participant data for continuous outcomes in randomized controlled trials. J Clin Epidemiol. 2017;88:57–66.

CAS   PubMed   Google Scholar  

Hernández AV, Boersma E, Murray GD, Habbema JD, Steyerberg EW. Subgroup analyses in therapeutic cardiovascular clinical trials: are most of them misleading? Am Heart J. 2006;151(2):257–64.

Samaan Z, Mbuagbaw L, Kosa D, Borg Debono V, Dillenburg R, Zhang S, Fruci V, Dennis B, Bawor M, Thabane L. A systematic scoping review of adherence to reporting guidelines in health care literature. J Multidiscip Healthc. 2013;6:169–88.

Buscemi N, Hartling L, Vandermeer B, Tjosvold L, Klassen TP. Single data extraction generated more errors than double data extraction in systematic reviews. J Clin Epidemiol. 2006;59(7):697–703.

Carrasco-Labra A, Brignardello-Petersen R, Santesso N, Neumann I, Mustafa RA, Mbuagbaw L, Etxeandia Ikobaltzeta I, De Stio C, McCullagh LJ, Alonso-Coello P. Improving GRADE evidence tables part 1: a randomized trial shows improved understanding of content in summary-of-findings tables with a new format. J Clin Epidemiol. 2016;74:7–18.

The Northern Ireland Hub for Trials Methodology Research: SWAT/SWAR Information [ https://www.qub.ac.uk/sites/TheNorthernIrelandNetworkforTrialsMethodologyResearch/SWATSWARInformation/ ]. Accessed 31 Aug 2020.

Chick S, Sánchez P, Ferrin D, Morrice D. How to conduct a successful simulation study. In: Proceedings of the 2003 winter simulation conference: 2003; 2003. p. 66–70.

Google Scholar  

Mulrow CD. The medical review article: state of the science. Ann Intern Med. 1987;106(3):485–8.

Sacks HS, Reitman D, Pagano D, Kupelnick B. Meta-analysis: an update. Mount Sinai J Med New York. 1996;63(3–4):216–24.

CAS   Google Scholar  

Areia M, Soares M, Dinis-Ribeiro M. Quality reporting of endoscopic diagnostic studies in gastrointestinal journals: where do we stand on the use of the STARD and CONSORT statements? Endoscopy. 2010;42(2):138–47.

Knol M, Groenwold R, Grobbee D. P-values in baseline tables of randomised controlled trials are inappropriate but still common in high impact journals. Eur J Prev Cardiol. 2012;19(2):231–2.

Chen M, Cui J, Zhang AL, Sze DM, Xue CC, May BH. Adherence to CONSORT items in randomized controlled trials of integrative medicine for colorectal Cancer published in Chinese journals. J Altern Complement Med. 2018;24(2):115–24.

Hopewell S, Ravaud P, Baron G, Boutron I. Effect of editors' implementation of CONSORT guidelines on the reporting of abstracts in high impact medical journals: interrupted time series analysis. BMJ. 2012;344:e4178.

The Cochrane Methodology Register Issue 2 2009 [ https://cmr.cochrane.org/help.htm ]. Accessed 31 Aug 2020.

Mbuagbaw L, Kredo T, Welch V, Mursleen S, Ross S, Zani B, Motaze NV, Quinlan L. Critical EPICOT items were absent in Cochrane human immunodeficiency virus systematic reviews: a bibliometric analysis. J Clin Epidemiol. 2016;74:66–72.

Barton S, Peckitt C, Sclafani F, Cunningham D, Chau I. The influence of industry sponsorship on the reporting of subgroup analyses within phase III randomised controlled trials in gastrointestinal oncology. Eur J Cancer. 2015;51(18):2732–9.

Setia MS. Methodology series module 5: sampling strategies. Indian J Dermatol. 2016;61(5):505–9.

Wilson B, Burnett P, Moher D, Altman DG, Al-Shahi Salman R. Completeness of reporting of randomised controlled trials including people with transient ischaemic attack or stroke: a systematic review. Eur Stroke J. 2018;3(4):337–46.

Kahale LA, Diab B, Brignardello-Petersen R, Agarwal A, Mustafa RA, Kwong J, Neumann I, Li L, Lopes LC, Briel M, et al. Systematic reviews do not adequately report or address missing outcome data in their analyses: a methodological survey. J Clin Epidemiol. 2018;99:14–23.

De Angelis CD, Drazen JM, Frizelle FA, Haug C, Hoey J, Horton R, Kotzin S, Laine C, Marusic A, Overbeke AJPM, et al. Is this clinical trial fully registered?: a statement from the International Committee of Medical Journal Editors*. Ann Intern Med. 2005;143(2):146–8.

Ohtake PJ, Childs JD. Why publish study protocols? Phys Ther. 2014;94(9):1208–9.

Rombey T, Allers K, Mathes T, Hoffmann F, Pieper D. A descriptive analysis of the characteristics and the peer review process of systematic review protocols published in an open peer review journal from 2012 to 2017. BMC Med Res Methodol. 2019;19(1):57.

Grimes DA, Schulz KF. Bias and causal associations in observational research. Lancet. 2002;359(9302):248–52.

Porta M (ed.): A dictionary of epidemiology, 5th edn. Oxford: Oxford University Press, Inc.; 2008.

El Dib R, Tikkinen KAO, Akl EA, Gomaa HA, Mustafa RA, Agarwal A, Carpenter CR, Zhang Y, Jorge EC, Almeida R, et al. Systematic survey of randomized trials evaluating the impact of alternative diagnostic strategies on patient-important outcomes. J Clin Epidemiol. 2017;84:61–9.

Helzer JE, Robins LN, Taibleson M, Woodruff RA Jr, Reich T, Wish ED. Reliability of psychiatric diagnosis. I. a methodological review. Arch Gen Psychiatry. 1977;34(2):129–33.

Chung ST, Chacko SK, Sunehag AL, Haymond MW. Measurements of gluconeogenesis and Glycogenolysis: a methodological review. Diabetes. 2015;64(12):3996–4010.

CAS   PubMed   PubMed Central   Google Scholar  

Sterne JA, Juni P, Schulz KF, Altman DG, Bartlett C, Egger M. Statistical methods for assessing the influence of study characteristics on treatment effects in 'meta-epidemiological' research. Stat Med. 2002;21(11):1513–24.

Moen EL, Fricano-Kugler CJ, Luikart BW, O’Malley AJ. Analyzing clustered data: why and how to account for multiple observations nested within a study participant? PLoS One. 2016;11(1):e0146721.

Zyzanski SJ, Flocke SA, Dickinson LM. On the nature and analysis of clustered data. Ann Fam Med. 2004;2(3):199–200.

Mathes T, Klassen P, Pieper D. Frequency of data extraction errors and methods to increase data extraction quality: a methodological review. BMC Med Res Methodol. 2017;17(1):152.

Bui DDA, Del Fiol G, Hurdle JF, Jonnalagadda S. Extractive text summarization system to aid data extraction from full text in systematic review development. J Biomed Inform. 2016;64:265–72.

Bui DD, Del Fiol G, Jonnalagadda S. PDF text classification to leverage information extraction from publication reports. J Biomed Inform. 2016;61:141–8.

Maticic K, Krnic Martinic M, Puljak L. Assessment of reporting quality of abstracts of systematic reviews with meta-analysis using PRISMA-A and discordance in assessments between raters without prior experience. BMC Med Res Methodol. 2019;19(1):32.

Speich B. Blinding in surgical randomized clinical trials in 2015. Ann Surg. 2017;266(1):21–2.

Abraha I, Cozzolino F, Orso M, Marchesi M, Germani A, Lombardo G, Eusebi P, De Florio R, Luchetta ML, Iorio A, et al. A systematic review found that deviations from intention-to-treat are common in randomized trials and systematic reviews. J Clin Epidemiol. 2017;84:37–46.

Zhong Y, Zhou W, Jiang H, Fan T, Diao X, Yang H, Min J, Wang G, Fu J, Mao B. Quality of reporting of two-group parallel randomized controlled clinical trials of multi-herb formulae: A survey of reports indexed in the Science Citation Index Expanded. Eur J Integrative Med. 2011;3(4):e309–16.

Farrokhyar F, Chu R, Whitlock R, Thabane L. A systematic review of the quality of publications reporting coronary artery bypass grafting trials. Can J Surg. 2007;50(4):266–77.

Oltean H, Gagnier JJ. Use of clustering analysis in randomized controlled trials in orthopaedic surgery. BMC Med Res Methodol. 2015;15:17.

Fleming PS, Koletsi D, Pandis N. Blinded by PRISMA: are systematic reviewers focusing on PRISMA and ignoring other guidelines? PLoS One. 2014;9(5):e96407.

Balasubramanian SP, Wiener M, Alshameeri Z, Tiruvoipati R, Elbourne D, Reed MW. Standards of reporting of randomized controlled trials in general surgery: can we do better? Ann Surg. 2006;244(5):663–7.

de Vries TW, van Roon EN. Low quality of reporting adverse drug reactions in paediatric randomised controlled trials. Arch Dis Child. 2010;95(12):1023–6.

Borg Debono V, Zhang S, Ye C, Paul J, Arya A, Hurlburt L, Murthy Y, Thabane L. The quality of reporting of RCTs used within a postoperative pain management meta-analysis, using the CONSORT statement. BMC Anesthesiol. 2012;12:13.

Kaiser KA, Cofield SS, Fontaine KR, Glasser SP, Thabane L, Chu R, Ambrale S, Dwary AD, Kumar A, Nayyar G, et al. Is funding source related to study reporting quality in obesity or nutrition randomized control trials in top-tier medical journals? Int J Obes. 2012;36(7):977–81.

Thomas O, Thabane L, Douketis J, Chu R, Westfall AO, Allison DB. Industry funding and the reporting quality of large long-term weight loss trials. Int J Obes. 2008;32(10):1531–6.

Khan NR, Saad H, Oravec CS, Rossi N, Nguyen V, Venable GT, Lillard JC, Patel P, Taylor DR, Vaughn BN, et al. A review of industry funding in randomized controlled trials published in the neurosurgical literature-the elephant in the room. Neurosurgery. 2018;83(5):890–7.

Hansen C, Lundh A, Rasmussen K, Hrobjartsson A. Financial conflicts of interest in systematic reviews: associations with results, conclusions, and methodological quality. Cochrane Database Syst Rev. 2019;8:Mr000047.

Kiehna EN, Starke RM, Pouratian N, Dumont AS. Standards for reporting randomized controlled trials in neurosurgery. J Neurosurg. 2011;114(2):280–5.

Liu LQ, Morris PJ, Pengel LH. Compliance to the CONSORT statement of randomized controlled trials in solid organ transplantation: a 3-year overview. Transpl Int. 2013;26(3):300–6.

Bala MM, Akl EA, Sun X, Bassler D, Mertz D, Mejza F, Vandvik PO, Malaga G, Johnston BC, Dahm P, et al. Randomized trials published in higher vs. lower impact journals differ in design, conduct, and analysis. J Clin Epidemiol. 2013;66(3):286–95.

Lee SY, Teoh PJ, Camm CF, Agha RA. Compliance of randomized controlled trials in trauma surgery with the CONSORT statement. J Trauma Acute Care Surg. 2013;75(4):562–72.

Ziogas DC, Zintzaras E. Analysis of the quality of reporting of randomized controlled trials in acute and chronic myeloid leukemia, and myelodysplastic syndromes as governed by the CONSORT statement. Ann Epidemiol. 2009;19(7):494–500.

Alvarez F, Meyer N, Gourraud PA, Paul C. CONSORT adoption and quality of reporting of randomized controlled trials: a systematic analysis in two dermatology journals. Br J Dermatol. 2009;161(5):1159–65.

Mbuagbaw L, Thabane M, Vanniyasingam T, Borg Debono V, Kosa S, Zhang S, Ye C, Parpia S, Dennis BB, Thabane L. Improvement in the quality of abstracts in major clinical journals since CONSORT extension for abstracts: a systematic review. Contemporary Clin trials. 2014;38(2):245–50.

Thabane L, Chu R, Cuddy K, Douketis J. What is the quality of reporting in weight loss intervention studies? A systematic review of randomized controlled trials. Int J Obes. 2007;31(10):1554–9.

Murad MH, Wang Z. Guidelines for reporting meta-epidemiological methodology research. Evidence Based Med. 2017;22(4):139.

METRIC - MEthodological sTudy ReportIng Checklist: guidelines for reporting methodological studies in health research [ http://www.equator-network.org/library/reporting-guidelines-under-development/reporting-guidelines-under-development-for-other-study-designs/#METRIC ]. Accessed 31 Aug 2020.

Jager KJ, Zoccali C, MacLeod A, Dekker FW. Confounding: what it is and how to deal with it. Kidney Int. 2008;73(3):256–60.

Parker SG, Halligan S, Erotocritou M, Wood CPJ, Boulton RW, Plumb AAO, Windsor ACJ, Mallett S. A systematic methodological review of non-randomised interventional studies of elective ventral hernia repair: clear definitions and a standardised minimum dataset are needed. Hernia. 2019.

Bouwmeester W, Zuithoff NPA, Mallett S, Geerlings MI, Vergouwe Y, Steyerberg EW, Altman DG, Moons KGM. Reporting and methods in clinical prediction research: a systematic review. PLoS Med. 2012;9(5):1–12.

Schiller P, Burchardi N, Niestroj M, Kieser M. Quality of reporting of clinical non-inferiority and equivalence randomised trials--update and extension. Trials. 2012;13:214.

Riado Minguez D, Kowalski M, Vallve Odena M, Longin Pontzen D, Jelicic Kadic A, Jeric M, Dosenovic S, Jakus D, Vrdoljak M, Poklepovic Pericic T, et al. Methodological and reporting quality of systematic reviews published in the highest ranking journals in the field of pain. Anesth Analg. 2017;125(4):1348–54.

Thabut G, Estellat C, Boutron I, Samama CM, Ravaud P. Methodological issues in trials assessing primary prophylaxis of venous thrombo-embolism. Eur Heart J. 2005;27(2):227–36.

Puljak L, Riva N, Parmelli E, González-Lorenzo M, Moja L, Pieper D. Data extraction methods: an analysis of internal reporting discrepancies in single manuscripts and practical advice. J Clin Epidemiol. 2020;117:158–64.

Ritchie A, Seubert L, Clifford R, Perry D, Bond C. Do randomised controlled trials relevant to pharmacy meet best practice standards for quality conduct and reporting? A systematic review. Int J Pharm Pract. 2019.

Babic A, Vuka I, Saric F, Proloscic I, Slapnicar E, Cavar J, Pericic TP, Pieper D, Puljak L. Overall bias methods and their use in sensitivity analysis of Cochrane reviews were not consistent. J Clin Epidemiol. 2019.

Tan A, Porcher R, Crequit P, Ravaud P, Dechartres A. Differences in treatment effect size between overall survival and progression-free survival in immunotherapy trials: a Meta-epidemiologic study of trials with results posted at ClinicalTrials.gov. J Clin Oncol. 2017;35(15):1686–94.

Croitoru D, Huang Y, Kurdina A, Chan AW, Drucker AM. Quality of reporting in systematic reviews published in dermatology journals. Br J Dermatol. 2020;182(6):1469–76.

Khan MS, Ochani RK, Shaikh A, Vaduganathan M, Khan SU, Fatima K, Yamani N, Mandrola J, Doukky R, Krasuski RA: Assessing the Quality of Reporting of Harms in Randomized Controlled Trials Published in High Impact Cardiovascular Journals. Eur Heart J Qual Care Clin Outcomes 2019.

Rosmarakis ES, Soteriades ES, Vergidis PI, Kasiakou SK, Falagas ME. From conference abstract to full paper: differences between data presented in conferences and journals. FASEB J. 2005;19(7):673–80.

Mueller M, D’Addario M, Egger M, Cevallos M, Dekkers O, Mugglin C, Scott P. Methods to systematically review and meta-analyse observational studies: a systematic scoping review of recommendations. BMC Med Res Methodol. 2018;18(1):44.

Li G, Abbade LPF, Nwosu I, Jin Y, Leenus A, Maaz M, Wang M, Bhatt M, Zielinski L, Sanger N, et al. A scoping review of comparisons between abstracts and full reports in primary biomedical research. BMC Med Res Methodol. 2017;17(1):181.

Krnic Martinic M, Pieper D, Glatt A, Puljak L. Definition of a systematic review used in overviews of systematic reviews, meta-epidemiological studies and textbooks. BMC Med Res Methodol. 2019;19(1):203.

Analytical study [ https://medical-dictionary.thefreedictionary.com/analytical+study ]. Accessed 31 Aug 2020.

Tricco AC, Tetzlaff J, Pham B, Brehaut J, Moher D. Non-Cochrane vs. Cochrane reviews were twice as likely to have positive conclusion statements: cross-sectional study. J Clin Epidemiol. 2009;62(4):380–6 e381.

Schalken N, Rietbergen C. The reporting quality of systematic reviews and Meta-analyses in industrial and organizational psychology: a systematic review. Front Psychol. 2017;8:1395.

Ranker LR, Petersen JM, Fox MP. Awareness of and potential for dependent error in the observational epidemiologic literature: A review. Ann Epidemiol. 2019;36:15–9 e12.

Paquette M, Alotaibi AM, Nieuwlaat R, Santesso N, Mbuagbaw L. A meta-epidemiological study of subgroup analyses in cochrane systematic reviews of atrial fibrillation. Syst Rev. 2019;8(1):241.

Download references

Acknowledgements

This work did not receive any dedicated funding.

Author information

Authors and affiliations.

Department of Health Research Methods, Evidence and Impact, McMaster University, Hamilton, ON, Canada

Lawrence Mbuagbaw, Daeria O. Lawson & Lehana Thabane

Biostatistics Unit/FSORC, 50 Charlton Avenue East, St Joseph’s Healthcare—Hamilton, 3rd Floor Martha Wing, Room H321, Hamilton, Ontario, L8N 4A6, Canada

Lawrence Mbuagbaw & Lehana Thabane

Centre for the Development of Best Practices in Health, Yaoundé, Cameroon

Lawrence Mbuagbaw

Center for Evidence-Based Medicine and Health Care, Catholic University of Croatia, Ilica 242, 10000, Zagreb, Croatia

Livia Puljak

Department of Epidemiology and Biostatistics, School of Public Health – Bloomington, Indiana University, Bloomington, IN, 47405, USA

David B. Allison

Departments of Paediatrics and Anaesthesia, McMaster University, Hamilton, ON, Canada

Lehana Thabane

Centre for Evaluation of Medicine, St. Joseph’s Healthcare-Hamilton, Hamilton, ON, Canada

Population Health Research Institute, Hamilton Health Sciences, Hamilton, ON, Canada

You can also search for this author in PubMed   Google Scholar

Contributions

LM conceived the idea and drafted the outline and paper. DOL and LT commented on the idea and draft outline. LM, LP and DOL performed literature searches and data extraction. All authors (LM, DOL, LT, LP, DBA) reviewed several draft versions of the manuscript and approved the final manuscript.

Corresponding author

Correspondence to Lawrence Mbuagbaw .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for publication

Competing interests.

DOL, DBA, LM, LP and LT are involved in the development of a reporting guideline for methodological studies.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Mbuagbaw, L., Lawson, D.O., Puljak, L. et al. A tutorial on methodological studies: the what, when, how and why. BMC Med Res Methodol 20 , 226 (2020). https://doi.org/10.1186/s12874-020-01107-7

Download citation

Received : 27 May 2020

Accepted : 27 August 2020

Published : 07 September 2020

DOI : https://doi.org/10.1186/s12874-020-01107-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Methodological study
• Meta-epidemiology
• Research methods
• Research-on-research

BMC Medical Research Methodology

ISSN: 1471-2288

• General enquiries: [email protected]

• Data, AI, & Machine Learning
• Managing Technology
• Social Responsibility
• Workplace, Teams, & Culture
• AI & Machine Learning
• Diversity & Inclusion
• Big ideas Research Projects
• Artificial Intelligence and Business Strategy
• Responsible AI
• Future of the Workforce
• Future of Leadership
• All Research Projects
• AI in Action
• Most Popular
• The Truth Behind the Nursing Crisis
• Work/23: The Big Shift
• Coaching for the Future-Forward Leader
• Measuring Culture

The spring 2024 issue’s special report looks at how to take advantage of market opportunities in the digital space, and provides advice on building culture and friendships at work; maximizing the benefits of LLMs, corporate venture capital initiatives, and innovation contests; and scaling automation and digital health platform.

• Past Issues
• Upcoming Events
• Video Archive
• Me, Myself, and AI
• Three Big Points

AI and Statistics: Perfect Together

Many companies develop AI models without a solid foundation on which to base predictions — leading to mistrust and failures. Here’s how statistics can help improve results.

• Data, AI, & Machine Learning
• AI & Machine Learning
• IT Governance & Leadership
• Technology Implementation

Carolyn Geason-Beissel/MIT SMR | Getty Images

People are often unsure why artificial intelligence and machine learning algorithms work. More importantly, people can’t always anticipate when they won’t work. Ali Rahimi, an AI researcher at Google, received a standing ovation at a 2017 conference when he referred to much of what is done in AI as “ alchemy ,” meaning that developers don’t have solid grounds for predicting which algorithms will work and which won’t, or for choosing one AI architecture over another. To put it succinctly, AI lacks a basis for inference: a solid foundation on which to base predictions and decisions.

This makes AI decisions tough (or impossible) to explain and hurts trust in AI models and technologies — trust that is necessary for AI to reach its potential. As noted by Rahimi, this is an unsolved problem in AI and machine learning that keeps tech and business leaders up at night because it dooms many AI models to fail in deployment.

Get Updates on Leading With AI and Data

Get monthly insights on how artificial intelligence impacts your organization and what it means for your company and customers.

Please enter a valid email address

Thank you for signing up

Privacy Policy

Fortunately, help for AI teams and projects is available from an unlikely source: classical statistics. This article will explore how business leaders can apply statistical methods and statistics experts to address the problem.

Holdout Data: A Tempting but Flawed Approach

Some AI teams view a trained AI model as the basis for inference, especially when that model predicts well on a holdout set of the original data. It’s tempting to make such an argument, but it’s a stretch. Holdout data is nothing more than a sample of the data collected at the same time, and under the same circumstances, as the training data. Thus, a trained AI model, in and of itself, does not provide a trusted basis for inference for predictions on future data observed under different circumstances.

What’s worse, many teams working on AI models fail to clearly define the business problem to be solved . This means that the team members are hard-pressed to tell business leaders whether the training data is the right data . Any one of these three issues (bad foundation, wrong problem, or wrong data) can prove disastrous in deployment — and statistics experts on AI teams can help prevent them.

Many IT leaders and data scientists feel that statistics is an old technology that is no longer needed in a big data and AI era.

About the Authors

Thomas C. Redman is president of Data Quality Solutions and author of People and Data: Uniting to Transform Your Organization (KoganPage, 2023). Roger W. Hoerl is the Brate-Peschel Professor of Statistics at Union College in Schenectady, New York, and coauthor with Ronald D. Snee of Leading Holistic Improvement With Lean Six Sigma 2.0 , 2nd ed. (Pearson FT Press, 2018).

More Like This

Add a comment cancel reply.

You must sign in to post a comment. First time here? Sign up for a free account : Comment on articles and get access to many more articles.

This paper is in the following e-collection/theme issue:

Published on 17.4.2024 in Vol 26 (2024)

This is a member publication of University College London (Jisc)

Twitter Analysis of Health Care Workers’ Sentiment and Discourse Regarding Post–COVID-19 Condition in Children and Young People: Mixed Methods Study

Authors of this article:

Original Paper

• Macarena Chepo 1 * , RN, BSN, MPH, PhD   ; 
• Sam Martin 2, 3 * , MSc, PhD   ; 
• Noémie Déom 2 , MSc   ; 
• Ahmad Firas Khalid 4 , MD, PhD   ; 
• Cecilia Vindrola-Padros 2 , BA, MA, PhD  

1 School of Nursing, Universidad Andrés Bello, Santiago, Chile

2 Department of Targeted Intervention, University College London, London, United Kingdom

3 Oxford Vaccine Group, Churchill Hospital, University of Oxford, Oxford, United Kingdom

4 Canadian Institutes of Health Research Health System Impact Fellowship, Centre for Implementation Research, Ottawa Hospital Research Institute, Otawa, ON, Canada

*these authors contributed equally

Corresponding Author:

Sam Martin, MSc, PhD

Department of Targeted Intervention

University College London

Charles Bell House 43-45

Foley Street

London, W1W 7TY

United Kingdom

Phone: 44 (0)20 3108 3232

Email: [email protected]

Background: The COVID-19 pandemic has had a significant global impact, with millions of cases and deaths. Research highlights the persistence of symptoms over time (post–COVID-19 condition), a situation of particular concern in children and young people with symptoms. Social media such as Twitter (subsequently rebranded as X) could provide valuable information on the impact of the post–COVID-19 condition on this demographic.

Objective: With a social media analysis of the discourse surrounding the prevalence of post–COVID-19 condition in children and young people, we aimed to explore the perceptions of health care workers (HCWs) concerning post–COVID-19 condition in children and young people in the United Kingdom between January 2021 and January 2022. This will allow us to contribute to the emerging knowledge on post–COVID-19 condition and identify critical areas and future directions for researchers and policy makers.

Methods: From a pragmatic paradigm, we used a mixed methods approach. Through discourse, keyword, sentiment, and image analyses, using Pulsar and InfraNodus, we analyzed the discourse about the experience of post–COVID-19 condition in children and young people in the United Kingdom shared on Twitter between January 1, 2021, and January 31, 2022, from a sample of HCWs with Twitter accounts whose biography identifies them as HCWs.

Results: We obtained 300,000 tweets, out of which (after filtering for relevant tweets) we performed an in-depth qualitative sample analysis of 2588 tweets. The HCWs were responsive to announcements issued by the authorities regarding the management of the COVID-19 pandemic in the United Kingdom. The most frequent sentiment expressed was negative. The main themes were uncertainty about the future, policies and regulations, managing and addressing the COVID-19 pandemic and post–COVID-19 condition in children and young people, vaccination, using Twitter to share scientific literature and management strategies, and clinical and personal experiences.

Conclusions: The perceptions described on Twitter by HCWs concerning the presence of the post–COVID-19 condition in children and young people appear to be a relevant and timely issue and responsive to the declarations and guidelines issued by health authorities over time. We recommend further support and training strategies for health workers and school staff regarding the manifestations and treatment of children and young people with post–COVID-19 condition.

Introduction

More than 3 years after the outbreak of the COVID-19 pandemic [ 1 ], the social, political, and economic impact of this phenomenon has been more than significant, considering >700 million worldwide cases and nearly 7 million people’s deaths [ 2 ]. Given the scale of the phenomenon, it is imperative for all countries to thoroughly examine the lessons gleaned from the pandemic, particularly regarding a matter that has raised significant concern among the populace: the long-term effects experienced by individuals who have had COVID-19, spanning weeks, months, or even years after their initial infection [ 3 ]. This phenomenon, referred to as post–COVID-19 condition (or more commonly “long COVID”), warrants careful consideration and analysis [ 4 ].

There is increasing information regarding the clinical manifestation of this condition, particularly in the adult population. The worldwide prevalence has been estimated at approximately 50% to 70% in individuals hospitalized during acute COVID-19 infection and 10% to 12% in vaccinated cases [ 5 ]. While children and young people have a low likelihood of severe COVID-19 infection [ 6 ], the information available to date indicates that the presence of post–COVID-19 condition in this group may be as disabling as in adults, reaching a prevalence rate of 23.4% (range 3.7%-66.5%) [ 7 ].

An agreed definition by the World Health Organization indicates that post–COVID-19 condition in children and young people is a condition that occurs “in individuals with a history of confirmed or probable SARS-CoV-2 infection when experiencing symptoms lasting at least two months which initially occurred within three months of acute COVID-19” [ 8 ]. Post–COVID-19 condition strongly impacts daily functioning and can develop or continue after COVID-19 infection and may fluctuate or relapse over time [ 4 , 8 , 9 ].

Among the symptoms most frequently attributable to post–COVID-19 condition in children and young people are fatigue, altered smell or anosmia, and anxiety [ 8 ]. However, other symptoms have also been reported, such as sleep disturbances, difficulty in concentrating, abdominal pain, myalgia or arthralgia, earache or ringing in ears, mood swings, persistent chest pain, stomach pain, light sensitivity, diarrhea, heart palpitations, and skin lesions [ 8 , 10 ]. One of England’s most significant studies is the Children and Young People With Long COVID study by Stephenson et al [ 11 ]. This national research matched longitudinal and cohort studies in adolescent individuals aged 11 to 17 years and found the presence of symptoms in 35.4% of the adolescent individuals who tested positive at baseline and 8.3% who of the adolescent individuals who tested negative at baseline. A total of 3 months after testing, 66.5% of those who tested positive and 53.3% of those who tested negative had any symptoms [ 11 ]. However, Stephenson et al [ 12 ] recently indicated that in a 6-month follow-up, the prevalence of specific symptoms reported at the time of the polymerase chain reaction testing decreased over time, where, for example, the prevalence of chills, fever, myalgia, cough, and sore throat among those who tested positive decreased from 10% to 25% to <3%.

As research on the symptoms, prevalence, and treatment of post–COVID-19 condition in children and young people continues, it is essential to add to the literature by developing studies that determine the condition’s impact on this group, considering that they are experiencing a range of unwanted symptoms that disrupt their quality of life and that of their families.

Considering that listening to the voices of families and health workers could be helpful to broaden the knowledge achieved in post–COVID-19 condition in children and young people, a powerful tool could be social media, such as Twitter (subsequently rebranded as X). With >3729 million daily active users, Twitter has become one of the most important social platforms in the world [ 13 ]. People used Twitter during the COVID-19 pandemic for different purposes, such as world leaders communicating with citizens [ 14 , 15 ], organizations monitoring movement [ 16 ], scientists studying public discourse around the pandemic [ 17 , 18 ], and researchers performing sentiment analysis [ 19 - 21 ]. In the case of physicians and health care workers (HCWs), Twitter has been used to share and evaluate scientific evidence, guidelines, and technical advice [ 22 - 24 ] and track the course and burden of disease [ 25 ].

Using the social media monitoring platform Pulsar [ 26 ], we aimed to explore HCWs’ perceptions concerning post–COVID condition in children and young people in the United Kingdom between January 2021 and January 2022. We aimed to contribute to the emerging knowledge on post–COVID-19 condition in children and young people and identify critical areas and future directions for researchers and policy makers.

We considered a mixed methods approach to be a pragmatic research paradigm. We analyzed data by conducting a Collaborative and Digital Analysis of Big Qualitative Data in Time Sensitive Contexts (LISTEN) [ 27 ]. This mixed methods analysis consisted of iterative cycles intercalating team discussion and using digital text and discourse analytics tools to analyze related social media data [ 27 ]. We used the LISTEN method to perform quantitative and qualitative analyses of Twitter posts, extracted through the Pulsar platform [ 26 ], related to the experience of post–COVID-19 condition in children and young people in the United Kingdom (eg, phrases, words, hashtags, videos, and images), published between January 1, 2021, and January 31, 2022. We created an advanced Boolean search for keywords mentioning “long COVID” and corelated words, hashtags, and symptoms; furthermore, we filtered for user accounts who identified as HCWs in their Twitter biography description ( Multimedia Appendix 1 ).

Quantitative analysis of all tweets included the following: (1) engagement analysis, where we specifically measured reactions to posts, for example, a retweet, a share, or a comment or quote made toward a tweet; (2) sentiment and emotion analysis, where we measured the positive or negative sentiment in the words and tone of each post within the context of post–COVID-19 condition and HCW’s roles ( Multimedia Appendix 2 ); (3) emotion analysis, where we measured the emotions expressed in the tweets, classified as sadness, anger, disgust, fear, and joy; (4) frequency analysis, where we observed the frequency of keywords and themes in the data set; (5) segmentation analysis, where we measured the key connections or relationships between keywords and their frequent use in the same context; (6) demographic analysis, where we measured the occupation, gender (man or woman or nonbinary or unknown), and city of origin related to the users posting tweets; and (7) analyses, where we evaluated the most influential accounts and the most mentioned websites.

Big qualitative analysis was carried out through thematic discourse analysis of the data sample, using InfraNodus [ 28 ], specifically analyzing the key themes and topics of concern expressed throughout the data set. A codebook was constructed based on the mapping of themes agreed upon by 3 researchers (ND, SM, and MC; Multimedia Appendix 3 ).

The principal investigators (ND, AFK, SM, and MC) interpreted and analyzed the data collected, following the recommendations for rigorous research provided by Creswell and Poth [ 29 ]. Using the LISTEN method [ 27 ], we aimed to show that the integration of qualitative insights through thematic analysis with the quantitative backing of topic modeling can offer a comprehensive view of the discourse. This mixed methods approach allows us to capture the richness of qualitative data while leveraging the objectivity of quantitative measures. Our initial data harvest of the larger corpus data from the Pulsar platform captured 300,000 tweets; this data harvest helped to underpin the software’s sentiment analysis modeling of this specific data set, providing a robust quantitative foundation. The addition of further qualitative data analyses from a smaller qualitative sample allowed for an in-depth understanding of nuanced conversations, particularly when exploring new or complex phenomena such as post–COVID-19 condition in children and young people, with the provision of insights into the context, subtext, and sentiment behind the tweets offering valuable snapshots of public perception and discourse. We used an iterative mixed methods approach, iterating between team discussions and using digital analytics tools to discern relevant themes from the Twitter data corpus. Specifically, we used InfraNodus for thematic analysis, which incorporates a topic modeling script for analyzing and identifying key topics of concern with a data set and provides a structured and objective interpretation of the data. The coding process involved 3 independent researchers (MC, SM, and ND), each with expertise in health care, social network analysis, and digital global health. When initial coding disagreements arose, we meticulously tagged any queries and discussed the posts in question. These instances led to 3 structured meetings wherein the research team deliberated collaboratively to resolve conflicting interpretations. This approach resulted in an 81.99% (2122/2588) initial intercoder agreement rate for the tweets analyzed. For the remaining instances where consensus was not initially reached, the majority rule was applied to finalize theme codings. To quantify the reliability of our coding procedure, with 81.99% (2122/2588) of the tweets coded identically, we used the Cohen κ score, which provides a measure of interrater agreement adjusted for chance. Including the calculation of all variations, this score was calculated to be approximately κ=0.70, indicating good agreement among the coders.

Ethical Considerations

The study only collected data from publicly accessible social networks that have been anonymized by various means, particularly by replacing all usernames and links with anonymous text and summaries of tweets that have been edited, retaining the original message, avoiding direct quotations being identifiable, and ensuring that no information is provided on the identity of the individuals who posted the content studied on the platform.

Internet research requires researchers to carefully consider guidelines to determine whether ethics approval and informed consent are needed [ 30 ]. On the basis of the terms set out by the Research Ethics Committee at the University College London [ 31 ], the study was considered exempt from formal ethics approval for the following reasons: (1) study involving information freely available in the public domain, such as published biographies, newspaper accounts of an individual’s activities, and published minutes of a meeting, that although is considered personal under the Data Protection Act, would not require ethics review; and (2) study involving anonymized records and data sets in the public domain, such as data sets available through the Office for National Statistics or the UK Data Archive where appropriate permissions have already been obtained and it is not possible to identify individuals from the information provided.

Therefore, we anonymized all records and data sets collected during the study to make identification impossible. We removed social media usernames from the data samples. No direct or easily traceable quotes have been included. These measures align with best practices [ 32 - 35 ]. While this study was beyond the scope of the human ethics committee, we adhered to the principles of ethics: beneficence, nonmaleficence, autonomy, and justice [ 36 ]. We collected and analyzed data through secure encrypted servers via the Meltwater and InfraNodus platforms.

Audience Analysis

During the period from January 2021 to January 2022, we obtained 300,000 tweets from 936 accounts. After filtering for relevant posts (refer to inclusion and exclusion criteria in Multimedia Appendix 1 ), we analyzed a sample of 2588 tweets using mixed methods analysis. In terms of gender (man, woman, nonbinary, or unknown), 32.88% (851/2588) were female individuals, 23.49% (608/2588) were male individuals, and 43.59% (1128/2588) were unknown. According to the description given in the user’s biography, the most frequently self-reported terms were “NHS” (582/2588, 22.49%), “health” (230/2588, 8.89%), “medical” (168/2588, 6.49%), “nurse” (166/2588, 6.41%), “clinical” (160/2588, 6.18%), “mum” (158/2588, 6.11%), “doctor” (145/2588, 5.6%), and “GP” (145/2588, 5.6%). In terms of city, tweets came mainly from London (958/2588, 37.02%), Newcastle upon Tyne (326/2588, 12.6%), Redcar (160/2588, 6.18%), Manchester (140/2588, 5.41%), and Bradford (111/2588, 4.29%).

Regarding profession described in the user’s biography, the most frequently mentioned roles were nurses (176/2588, 6.8%); medical roles, for example, paramedic and nursing assistant (173/2588, 6.68%); clinical roles, for example, surgeon, physiotherapist, and anesthesiologist (160/2588, 6.18%); general practitioners (GPs), for example, hospital GP or local surgery GP (142/2588, 5.49%); and physician (140/2588, 5.41%). The most frequent organization affiliated with was the National Health Service (587/2588, 22.68%).

Most Influential Accounts

One of the accounts that generated the highest number of mentions and, therefore, some of the most influence, as they were the ones that talked the most about post–COVID-19 condition in children and young people, was the account for @longcovidkids (593/2588, 22.91% tweets), related to the most shared website longcovidkids.org [ 37 ] , an international UK-based charity for families and children living with post–COVID-19 condition. Although the account was created in October 2020, it was first mentioned in our data collection timeline on January 1, 2021. It offers web support services, funding, and research participation and represents children and young people living with post–COVID-19 condition in expert forums, research panels, health organizations, and parliamentary groups. The other most shared web pages were theguardian.com (the United Kingdom) [ 38 ], bbc.co.uk (the United Kingdom) [ 39 ], peoplewith.com (the United States) [ 40 ], and ncbi.nlm.nih.gov (the United States) [ 41 ]. This shows that in the United Kingdom, there was a mixed influence of UK and US link resources linked to HCW Twitter users in the United Kingdom.

Keyword Analysis

The volume of social media engagement in the discussion about the post–COVID-19 condition experience in children and young people in the United Kingdom reached 1400 posts, 1550 engagements, and 1.9 million impressions. Overall, comments were very responsive to government decisions regarding the vaccination program and school closures ( Multimedia Appendix 4 ). During the first peak of comments in January 2021, the amount of discourse expanded leading up to March 2021, when there were different announcements of school closures, and the guidelines were delivered regarding the priority groups of the vaccination program (frontline HCW and people aged >80 years first). The highest engagement was between June and July 2021, which coincides with the government announcement regarding the availability of vaccines for people aged >18 years. The third peak of comments occurred in September 2021, the same month the authorities announced the extension of the vaccination program to children aged 12 to 15 years.

Top Keywords Analysis

The top words in posts associated with children and young people’s experience of post–COVID-19 condition in the United Kingdom were “Children” (352/2588, 13.6%), “kids” (160/2588, 6.18%), “people” (158/2588, 6.11%), “Young” (148/2588, 5.72%), and “schools” (83/2588, 3.21%). The top hashtags were #longcovid (1387/2588, 53.59%), #longcovidkids (448/2588, 17.31%), #covid19 (370/2588, 14.3%), and #covid (176/2588, 6.8%).

Sentiment and Emotions Analysis

According to sentiment analysis, 99.38% (2572/2588) of the posts reflected negative sentiments and 0.62% (16/2588) reflected positive sentiments. Negative sentiments were mainly associated with comments on hospitalization figures related to the COVID-19 pandemic, criticism of pandemic mitigation policies, and vaccination of children and young people. Furthermore, positive sentiments mainly concerned acknowledgments around decreasing numbers of community support groups.

The primary emotions identified were as follows:

• Sadness (1752/2588, 67.7%), such as in the following tweet:

@[Username] Really upset, after my tough on-call last night. Hospitalisations are still going up, and Gov announcement minismises the effect of long-COVID in adults and children. It’s so hard to keep spirits up today. But we’ll try and continue doing our best in the NHS.

• Joy (367/2588, 14.18%), such as in the following tweet:

@[Username] It’s been an amazing day! [...] I’ve been able to share the experience I’ve gained treating children and adolescents with Long COVID over the last year.

• Fear (233/2588, 9%), as seen in the following tweet:

@[Username] It’s really urgent that young people get the message that they need to get vaccinated. Long COVID is ruining many people’s lives! It’s not a lie or hypochondria, there are real, physiological changes, please understand!

Segmentation Analysis

This analysis revealed the critical clusters of conversation around the main topics of concern within the discourse network around post–COVID-19 condition. Comments were distributed in 4 key conversation segments as follows:

• People, schools, and prevention (1734/2588, 67%): Most of the comments related to measures taken in terms of COVID-19 prevention in schools, concern about the risk of exposure, and sharing experiences of infection in schools.
• Health, adults, and impact (401/2588, 15.49%): Comments mainly reflected concerns and uncertainty about the long-term effect of post–COVID-19 condition on both children and young people and adults.
• Cases, virus, and risk (326/2588, 12.6%): Comments reflected worries about the associated risks and long-term consequences attributable to post–COVID-19 condition (in both adults and children and young people) and the constant mutation of the virus, which will create a permanent risk in the population.
• Months, distress, and symptoms (106/2588, 4.1%): Some HCWs used Twitter to share how children and young people experience post–COVID-19 condition and the extent of these symptoms. Some HCWs exemplified certain typical manifestations, such as fatigue.

Discourse Analysis by Theme

To better understand the topics discussed from the segmentation analysis, we performed a discourse analysis of the key co-occurring themes and topics of concern shared within discussions regarding post–COVID-19 condition in children and young people. The following themes emerged ( Textbox 1 ): concern or uncertainty for the future, school attendance, mask protection from COVID-19, vaccine uptake, infection rates, policy (support or skepticism), understanding and visualizing symptoms, child mental health, access to care, community support, and research ( Figures 1 and 2 ).

• Concern for the future or uncertainty (615/2588, 23.76% tweets): Most comments showed a concern for the future, focusing on shared statistics regarding the rate and spread of infection in children and young people and how this would affect future health outcomes. Furthermore, this group expressed concern regarding political decisions; the presence of illness in loved ones; the eventual overload and response capacity of the health system in the face of an increase in post–COVID-19 condition cases; and the need for training of health care workers (HCWs) to deal with comorbid, potentially long-term symptoms ( Figure 1 A).
• Schools (460/2588, 17.77% tweets): Comments aimed to promote vaccination policies for schoolchildren and flexible measures regarding teachers’ work and attendance, considering cases of people with prolonged symptoms. In addition, several tweets expressed dissatisfaction with school risk mitigation measures, such as the use of face masks and air filters ( Figure 1 B).
• Vaccine (386/2588, 14.9% tweets): Most tweets from this group showed their disapproval of the constant changes in the government’s decisions regarding schools and priority groups for vaccination. Between March and June 2021, the first set of tweets criticized the lack of priority in the vaccination program for schoolchildren and other at-risk groups (such as teachers). Once the authorities announced a vaccination program for schoolchildren aged 12 to 15 years ( Multimedia Appendix 4 ), most comments promoted vaccination for this group. A few comments (78/2588, 3.01%) shared concerns about the vaccine’s efficacy for children, based on the experiences of COVID-19 reinfection in adults despite having received the recommended initial doses. However, to a lesser extent (26/2588, 1%), there was a refusal to vaccinate children, citing fear of possible adverse effects. Nonetheless, it is worth noting that the community frequently refuted such comments ( Figure 1 C).
• Share statistics (334/2588, 12.91% tweets): Frequently, HCWs shared statistical data, such as the number of affected children and young people, the number of post–COVID-19 condition cases, and hospital admissions and deaths. Some of these data were used to validate the existence of the post–COVID-19 phenomenon or to express concern about it ( Figure 1 D).
• Policy (316/2588, 12.21% tweets): The comments were responsive to the policies emanating from the authorities over time ( Multimedia Appendix 4 ). There were 5 main criticisms, including changes in school closure or opening policies; HCWs question why the authorities ignore the evidence of post–COVID-19 cases in children and young people, leading them to question whether decision makers have sufficient training to control the pandemic adequately; the failure to include teachers and school workers in the COVID-19 vaccination program as well as the younger population; the lack of mitigation measures in schools, such as improvements in ventilation systems and mandatory use of masks; and the herd immunity as a plan in the government’s hidden agenda , that is, to promote work and activate the economy ( Figure 1 E).
• “Proof” (280/2588, 10.82% tweets): Most tweets in this group argued regarding the existence of children and young people with post–COVID-19 condition through pictures; statistics; scientific papers; and personal, family, and professional experiences ( Figure 1 F).
• Signs and symptoms (189/2588, 7.3% tweets): Among the symptoms described, chronic fatigue and exhaustion were the most frequent symptoms, which prevent normal activities. Other symptoms were respiratory: dyspnea, chronic cough, and shortness of breath; gastrointestinal: acute or intense abdominal pain, nausea, bloating, gastroparesis, and change in smell or taste; muscular: severe joint pain, “painful foot” and difficulty with physical activity; mental health: anxiety and low mood; topical: rash, skin rashes, and redness and pain in the eyes; and nonspecific symptoms, such as chest pain, heart palpitations, constant high body temperature, precocious puberty, hormonal changes, and erectile dysfunction ( Figure 2 A).
• Face masks (119/2588, 4.6% tweets): Face masks were widely promoted, especially in schools, because HCWs considered them as a practical and straightforward strategy to control the pandemic ( Figure 2 B).
• Skepticism (101/2588, 3.9% tweets): Comments showed reticence toward post–COVID-19 condition in children and young people. Some of the arguments focused on a perceived lack of clarity in the clinical manifestations and stressed the need to better differentiate the post–COVID-19 condition from other related symptomatologies, such as mood disorders (eg, depression and anxiety due to confinement). In contrast, several arguments agreed on the need for more scientific evidence, arguing that post–COVID-19 condition in children and young people are isolated. Other users claimed not to know of such cases instead of calling post–COVID-19 condition in children and young people SMS text message an exaggeration. In addition, several arguments favored releasing restrictions for children and young people, particularly arguments related to the use of masks, because of possible associated risks, for example, hypoxia ( Figure 2 C).
• Mental health (54/2588, 2.09% tweets): Symptoms attributable to mental health problems in children and young people were also a concern. For instance, HCWs mentioned sadness, fear of infecting their family, anxiety regarding sick parents, stress, night terrors, self-harm, and suicidal ideation. Furthermore, users discussed a perceived lack of specific support for children and young people and their families in situations such as hospitalization; prolonged COVID-19 condition; admission to intensive care; and death of a family member, schoolmate, or teacher, all situations that triggered permanent stress in these groups ( Figure 2 D).
• Community support or asking for advice (93/2588, 3.59% tweets): Some HCWs used Twitter to ask for guidance on a specific issue or share experiences of having post–COVID-19 condition or caring for children and young people or family members. Furthermore, they shared informative infographics provided by experts regarding post–COVID-19 condition in children and young people ( Figure 2 E).
• Access to health care or treatment (72/2588, 2.78% tweets): Some HCWs mentioned the lack of specialist (cardiology) support, concerns regarding prolonged National Health Service burnout, and criticisms regarding how follow-up was carried out concerning the relative symptomatology of children and young people with post–COVID-19 condition. At the same time, opening new centers for children and young people with post–COVID-19 condition generated different reactions. On the one hand, some HCWs recognized it as a substantial development, but on the other hand, some HCWs recognized it as proof of the existence of post–COVID-19 condition in children and young people, which raised concerns for the future ( Figure 2 F).
• Research (52/2588, 2% tweets): Under this theme, tweets largely promoted study on post–COVID-19 condition in children and young people or highlighted the need for further study on the subject ( Figure 2 G).
• Images (57/2588, 2.2% tweets): Images shared were primarily from scientific studies, including infographics (from organizations such as National Health Service or @LongCovidKids) and visualization of children and young people’s symptoms, such as rashes, COVID-19 toe, and joint pain. Most infographics shared by organizations (and not individuals), such as the organization LongCovidKids, were related to statistics, such as the number of children and young people with post–COVID-19 condition or the quantification of the type of symptoms experienced. Shared photographs tended to show the more “visually recognizable” symptoms of post–COVID-19 condition, such as skin lesions, rashes, or inflammation. The less visible symptoms, such as chronic fatigue and neurological issues, were represented with photographs of children and young people lying, sleeping under blankets, or duvets or on hospital beds ( Figure 2 H).

Principal Findings

Our primary objective was to explore HCWs’ perceptions concerning post–COVID-19 condition in children and young people in the United Kingdom between January 2021 and January 2022. Our findings indicated that comments made by HCWs on Twitter were responsive to announcements issued by authorities regarding the management of the COVID-19 pandemic in the United Kingdom and associated regulations on the operation of schools. The most frequent feelings and emotions were negative, mainly sadness. In turn, we identified relevant themes for HCWs, such as uncertainty or concern about the future; policies; and regulations for the prevention, management, and addressing both COVID-19 and post–COVID-19 condition in children and young people; vaccination; and the use of Twitter as a strategy to share scientific literature, management strategies, and clinical and personal experiences.

Concern from HCWs regarding the policies for addressing the COVID-19 pandemic in the children and young people in the United Kingdom (including vaccination and schools) was a recurring theme in our findings. Furthermore, concern regarding the side effects of the COVID-19 vaccine and how the vaccine might interact with preexisting physiological symptoms of post–COVID-19 condition in children and young people was a topic of discussion. Similarly, the constant change in policy making in the United Kingdom, as public health bodies and governments have tried to understand and adapt to the emergence of post–COVID-19 condition, have added to the strength of this ongoing debate [ 42 ]. The lack of up-to-date evidence on post–COVID-19 condition in children and young people prompted HCWs to rely on Twitter during the pandemic to communicate relevant information. Twitter has a broad audience reach; is used as a communication tool by politicians, health bodies, and other key influences; and facilitates real-time updates [ 43 ]. During the pandemic, HCWs, primarily those in frontline roles and local response coordination, have often been challenged to become credible spokespersons for pandemic information [ 44 ]. Such credibility directly influences public confidence and decision-making, ultimately determining the success or failure of a public health intervention [ 43 ].

Furthermore, failures in risk communication could explain the presence of uncertainty and negative feelings associated with school regulations. When people are upset, distressed, or fearful, they often do not trust the authority, decrease the perceived validity of the communication received, and find information processing difficult [ 45 ]. In this regard, Fotheringham et al [ 46 ] indicated that during 2020, school leaders in the United Kingdom faced pressures and challenges related to translating and enacting school policies, particularly with the perceived lack of agency shared by the government concerning being able to translate centrally issued guidelines. In turn, Tomson et al [ 47 ] reported that the pandemic has negatively impacted the well-being of leaders in all types of schools and across all demographic groups, affecting their ability to think clearly and solve work-related problems. Given that the protection and care of children and young people health during the COVID-19 pandemic ultimately rests with school leaders, the search for support strategies that focus on the needs of these groups becomes an urgent necessity.

Findings in Relation to Other Studies

Using Twitter’s information, this is one of the first studies to capture health professionals’ perceptions of prolonged COVID-19 in the children and young people in the United Kingdom. However, other studies have addressed post–COVID-19 condition on this social network. Callard and Peregov [ 48 ] reviewed how, through social platforms such as Twitter, patients made the persistence and heterogeneity of COVID-19 symptoms visible, thus catapulting the inclusion of post–COVID-19 condition as a relevant phenomenon in clinical and policy debates. In contrast, other authors in the last 2 years have explored on various platforms (including Twitter) the persistence of symptoms and emotional impact after months of suspected and confirmed diagnosis of COVID-19 [ 49 - 55 ], including the period of vaccination. Furthermore, others have explored web discussions regarding this phenomenon [ 56 ]. Several of these authors agree on a perceived lack of support and specific resources from governmental bodies, a lack of information or clarity in the instructions given, and the absence of formal mechanisms to allow the voices of patients and the community to be heard. The above point is critical as it highlights the gap between the needs of the population and the response provided by policy makers, which not only translates into a gap in access to health services but also limits citizen participation in decision-making on the issues that affect their own health and increases distrust toward regulations and instructions issued by the government.

Implications for Policy and Practice

Several policy recommendations and implications are targeted at various stakeholders to consider while implementing future policy guidelines to address post–COVID-19 health care delivery. First, policy makers should consider investing appropriate resources to collect data regarding post–COVID-19 condition in children and young people, specifically on the impact of COVID-19 on the mental health of children and young people. This implies working closely with researchers to streamline data collection and reporting on post–COVID-19 condition. Second, policy makers should consider providing a basic level of psychosocial support with access to quality mental health and psychosocial support services for HCWs, school staff, parents, and children and young people experiencing post–COVID-19 condition. This implies strengthening health systems, community-based programming, and mobilization. Policies must include documenting the impact of mental health and psychosocial support interventions and innovative approaches to be more widely disseminated and scaled up across different contexts and target population groups. Third, to address the criticism around frequent changes in school closure and opening policies, decision makers should develop clear, easy-to-understand school mitigation plans informed by the best available evidence. The plans should incorporate teachers, school workers, and parents to ensure all voices are included in the policy plan. Fourth, policy makers should adopt a shared decision-making approach incorporating HCWs in the decision-making process for managing the COVID-19 pandemic. Finally, government decision makers should set post–COVID-19 pandemic recovery policies informed from a health equity perspective and how this affects children and young people living with post–COVID-19 condition, factoring in childhood, family income, housing, domestic violence, access to health care, and racism.

In terms of the needed clearer road map for recommendations to support training strategies for HCWs and school staff regarding post–COVID-19 condition in children and young people, we have outlined the following 10 steps.

Step 1: Data Collection and Analysis

Our study underlines the critical need for comprehensive data on post–COVID-19 condition’s impact on the mental health of children and young people. As a first step, it is recommended that policy makers should allocate resources for the systematic collection and analysis of data on post–COVID-19 condition in children and young people, particularly focusing on mental health outcomes. These data should be used to identify the most prevalent symptoms and the most effective treatment strategies. In this context, it is recommended that experts emphasize the importance of early detection and medical consultation for mental health issues in children and young people diagnosed with post–COVID-19 condition, including mood changes, irritability, social withdrawal, memory problems, difficulty in concentrating, anxiety, depression, posttraumatic stress, school absenteeism, and suicidal ideation [ 57 , 58 ]. This entails working closely with researchers to streamline data collection and reporting on post–COVID-19 condition.

Step 2: Psychosocial Support Framework

It has been noted that globally, programs for managing post–COVID-19 condition in children and young people are heterogeneous, ranging from the use of physiotherapy, pediatric occupational therapy, and psychological support to interventions aimed at lifestyle modifications [ 59 ]. This diversity could impact differential outcomes in the treatment, recovery, and timely and effective rehabilitation of children and young people with post–COVID-19 condition. Upon analyzing the wider literature and the social media data in this study, it is recommended that a basic level of psychosocial support should be established. This would involve ensuring access to quality mental health services for HCWs, school staff, parents, and children and young people with post–COVID-19 condition. This framework should be integrated into the health system and community-based programming, emphasizing the mobilization of resources and strengthening of support networks. It is suggested that the psychosocial support framework should facilitate access to quality mental health services and support networks that are robust and responsive. Community engagement gleaned from further Twitter discourse analysis should be a helpful guide in the development of these services to ensure they meet the real and expressed needs of children and young people with post–COVID-19 condition. Practical examples of basic psychosocial support include using web support services; individual or group therapy sessions; school-based emotional support programs; and counseling sessions aimed at parents, family members, or school staff.

Step 3: Educational Mitigation Plans

The frequent policy changes around school closures highlight the necessity for stable and clear educational mitigation plans. It is recommended that these plans should be directly informed by the evidence collected and further analysis of sentiments and emotions surrounding post–COVID-19 condition in schools. Incorporating the viewpoints of teachers, parents, and school staff, as identified in our thematic analysis, will ensure that the mitigation strategies are comprehensive, feasible, and sensitive to the psychosocial impact on children and young people. School staff and policy makers should collaborate to develop clear, evidence-informed educational mitigation plans. These plans should be straightforward and involve teachers, school workers, and parents in their creation, ensuring a unified approach that considers the voices of all stakeholders.

Step 4: Shared Decision-Making in Health Care

In health care settings, the adoption of a shared decision-making model is crucial, enabling HCWs to actively contribute to the formulation of COVID-19 and post–COVID-19 policies. This inclusive approach ensures that frontline workers can provide valuable insights toward policy development. To facilitate this, the establishment of advisory committees composed of representatives from HCWs is recommended. This committee can convene regularly to deliberate on key decisions pertaining to the COVID-19 pandemic management, including prevention measures, resource distribution, and vaccination strategies. Such collaborative groups have demonstrated effectiveness in identifying priority needs within the context of a pandemic [ 60 ].

Step 5: Health Equity in Policy Setting

Post–COVID-19 recovery policies should be set with a health equity lens. This means considering factors such as family income, housing, domestic violence, access to health care, and racism and how these factors affect children and young people living with post–COVID-19 condition. Our findings emphasize the importance of framing post–COVID-19 recovery policies through a lens of health equity. The concerns raised by HCWs regarding the socioeconomic impacts, such as family income and access to health care, underline the need for policies that address not just the medical aspects of post–COVID-19 condition but also the social determinants of health. An equitable approach will ensure that children and young people from diverse backgrounds receive appropriate support.

Step 6: Documenting and Disseminating Interventions

It is vital to document the impact of mental health and psychosocial support interventions. In this context, it is crucial to implement innovative strategies to disseminate unbiased information about post–COVID-19 condition among health care professionals and educators working with children and young people, ensuring it reaches different contexts and populations. These strategies may include creating interactive multimedia resources, such as videos and mobile apps; organizing webinars; actively using social media; and forming web support groups. These groups will provide a space where patients, health care professionals, and educators can share their experiences and knowledge regarding post–COVID-19 condition. These actions will not only help reduce isolation and social stigma but also strengthen support for these groups considered vulnerable [ 61 ].

Step 7: Developing a Clear Communication Strategy

Policy makers must develop a clear communication strategy to address frequent policy changes and mitigate confusion. This strategy should be informed by the data collected and analysis conducted in Step 1. The data reveal a palpable sense of uncertainty and frustration due to frequent policy shifts, underscoring the need for a clear and consistent communication strategy. This strategy should be grounded in the evidence gathered from the health care community’s discourse and aim to minimize confusion by providing timely, transparent, and reliable information regarding post–COVID-19 policies and support services.

Step 8: Training and Support Strategies

On the basis of the findings of the comprehensive data analysis, specific training and support strategies should be developed for HCWs and school staff. These strategies should be focused on the practical aspects of identifying and managing post–COVID-19 condition in children and young people. For instance, training sessions could include practical workshops on recognizing post–COVID-19 symptoms in children and adolescents, conducting diagnostic assessments, and implementing appropriate treatment and support interventions.

Step 9: Continuous Feedback and Policy Adaptation

The continuous evolution of the post–COVID-19 phenomenon demands an iterative approach to policy making. On the basis of our study, we recommend establishing feedback mechanisms with HCWs and school staff to monitor the reception and effectiveness of implemented policies. This feedback, coupled with ongoing research, should inform policy adaptations to ensure they remain aligned with the evolving landscape of post–COVID-19 condition and its impact on children and young people.

Step 10: Evaluation and Research

Finally, there should be a commitment to ongoing evaluation and research. This will involve not only monitoring the implementation of the abovementioned steps but also supporting new research to fill any remaining gaps in understanding the long-term effects of COVID-19 on children and young people.

This sequence of steps is designed to be iterative and responsive, ensuring that the recommendations from the study are translated into concrete actions that adapt to emerging data and research findings.

Strengths and Limitations

A key strength of this study is that our social media analysis of post–COVID-19 condition contributes toward an emerging understanding of reported experiential, emotional, and practical dimensions of post–COVID-19 condition in children and young people specifically and questions of vaccine hesitancy in children and young people with post–COVID-19 condition. This is one of the few studies to collect HCWs’ perceptions regarding post–COVID-19 condition in children and young people in the United Kingdom using information from Twitter. We identify key areas that need considering attention and focus, such as the provision of psychosocial support with access to quality mental health resources to alleviate the impact of post–COVID-19 condition in children and young people and the development of clear post–COVID-19 pandemic recovery guidelines that are informed by health equity perspective, and how this affects children and young people living with post–COVID-19 condition.

One of the limitations this study acknowledges is the definition of post–COVID-19 condition in children and young people. When data were collected, the lack of consensus on the definition of post–COVID-19 condition in children and young people forced us to formulate a definition of post–COVID-19 condition in children and young people based on the available literature. Furthermore, this study is limited to the perceptions of people who used descriptors in their web biography attributable to HCWs; therefore, our results only represent some HCWs in the United Kingdom and those in other countries. In turn, this research collected data from Twitter only; therefore, further inquiry into HCWs’ perceptions of post–COVID-19 condition in children and young people required expanding to other data sources or social networks and including languages other than English. We acknowledge that demographic factors, geographic location, and individual daily activities of social media users can significantly influence language use and word choice, introducing potential biases in tweet-based data. Such biases are inherent in any analysis of social media content and can affect the generalizability of findings. For instance, our study relies on Twitter data, which do not encompass the full spectrum of global or the UK public opinion on post–COVID-19 condition in children and young people. While Twitter serves as a valuable platform for capturing real-time sentiments and experiences, it is not fully representative of all demographics and geographic regions. Our results may reflect the perspectives of more vocal or active social media users, which may not correspond to the silent majority or those without access to social media. In addition, the absence of geotagged information for many users limits our ability to conduct a more nuanced spatial analysis of the sentiments expressed.

Furthermore, our study is built upon the recognition that social media data may overrepresent certain demographic groups while underrepresenting others, such as the older population or those without reliable internet access. This skew can influence the apparent prevalence of certain views or experiences of post–COVID-19 condition. Moreover, individuals’ patterns of daily life, reflected in their social media use and content, contribute additional layers of complexity and potential bias to the discourse analyzed.

Consistent with scholarly precedents on the subject [ 62 , 63 ], our study acknowledges these biases as intrinsic limitations of social media–based research. Although our analysis did not control for these factors, we recognize their potential impact on our results. Future studies would benefit from incorporating a broader array of data sources, including interviews or focus groups, to provide a more representative and comprehensive understanding of post–COVID-19 condition in children and young people. This approach would complement our Twitter-based findings and help mitigate the biases inherent in social media data.

Conclusions

More than a year after the start of the COVID-19 pandemic, the perceptions described on Twitter by HCWs concerning the presence of post–COVID-19 condition in children and young people appear to be a relevant and timely issue as well as very responsive to the declarations and guidelines issued by the health authorities over time. The most prominent group within the discourse studied was the activist or lobbying organization @LongCovidKids, which shared the most tweets and images over the period studied. We recommend that future research focus on how web health activism is organized and carried out for children and young people with post–COVID-19 condition. Such a strategy would allow for a better understanding of the scope and impact of this phenomenon and how it can influence decision-making. Furthermore, we suggest different mitigation strategies, support, and training of HCWs and school staff regarding manifestations and treatment of post–COVID-19 condition in children and young people across all demographic areas.

Acknowledgments

The authors would like to thank the Rapid Research Evaluation and Assessment Lab, Department of Targeted Intervention, University College London, London, United Kingdom, whose support has been essential for developing this project.

Conflicts of Interest

None declared.

Filters used for the search strategy on Twitter.

Sentiment analysis framework: attitudes toward post–COVID-19 condition in children and young people.

Theme codebook: examples of tweets that fit into main themes tagged for mention of children and young people with post–COVID-19 condition.

Timeline of national governmental policies and guidelines regarding children and young people.

• World Health Organization. COVID-19 Public Health Emergency of International Concern (PHEIC) Global research and innovation forum. Global Research Collaboration for Infectious Disease Preparedness. Feb 12, 2020. URL: https:/​/www.​who.int/​publications/​m/​item/​covid-19-public-health-emergency-of-international-concern-(pheic)-global-research-and-inno vation-forum [accessed 2022-06-06]
• Johns Hopkins University of Medicine. COVID-19 Dashboard by the Center for Systems Science and Engineering (CSSE) at Johns Hopkins University (JHU). Coronavirus Resource Center. Jan 22, 2020. URL: https://coronavirus.jhu.edu/map.html [accessed 2023-06-09]
• Nature Editorial. Long COVID and kids: more research is urgently needed. Nature. Feb 08, 2022;602(7896):183. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Long COVID or post-COVID conditions. Centers for Disease Control and Prevention. URL: https://www.cdc.gov/coronavirus/2019-ncov/long-term-effects/index.html [accessed 2023-06-09]
• Davis HE, McCorkell L, Vogel JM, Topol E. Long COVID: major findings, mechanisms and recommendations. Nat Rev Microbiol. Mar 13, 2023;21(3):133-146. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Zimmermann P, Curtis N. Why is COVID-19 less severe in children? A review of the proposed mechanisms underlying the age-related difference in severity of SARS-CoV-2 infections. Arch Dis Child. Dec 01, 2020;106(5):429-439. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Zheng YB, Zeng N, Yuan K, Tian SS, Yang YB, Gao N, et al. Prevalence and risk factor for long COVID in children and adolescents: a meta-analysis and systematic review. J Infect Public Health. Mar 07, 2023;16(5):660-672. [ FREE Full text ] [ CrossRef ] [ Medline ]
• World Health Organization. A clinical case definition for post COVID-19 condition in children and adolescents by expert consensus. World Health Organization. Feb 16, 2023. URL: https:/​/www.​who.int/​publications/​i/​item/​WHO-2019-nCoV-Post -COVID-19-condition-CA-Clinical-case-definition-2023-1 [accessed 2024-03-26]
• Stephenson T, Allin B, Nugawela MD, Rojas N, Dalrymple E, Pinto Pereira S, CLoCk Consortium, et al. Long COVID (post-COVID-19 condition) in children: a modified Delphi process. Arch Dis Child. Jun 17, 2022;107(7):674-680. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Esposito S, Principi N, Azzari C, Cardinale F, Di Mauro G, Galli L, et al. Italian intersociety consensus on management of long Covid in children. Ital J Pediatr. Mar 09, 2022;48(1):1-9. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Stephenson T, Pinto Pereira SM, Shafran R, de Stavola BL, Rojas N, McOwat K, CLoCk Consortium, et al. Physical and mental health 3 months after SARS-CoV-2 infection (long COVID) among adolescents in England (CLoCk): a national matched cohort study. Lancet Child Adolesc Health. Feb 08, 2022;6(4):230-239. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Stephenson T, Pinto Pereira SM, Nugawela MD, McOwat K, Simmons R, Chalder T, et al. CLoCk Consortium. Long COVID-six months of prospective follow-up of changes in symptom profiles of non-hospitalised children and young people after SARS-CoV-2 testing: a national matched cohort study (The CLoCk) study. PLoS One. Mar 6, 2023;18(3):e0277704. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Twitter users, stats, data and trends. Data Reportal. May 11, 2023. URL: https://datareportal.com/essential-twitter-stats#:~ :text=Based [accessed 2023-06-09]
• Rufai SR, Bunce C. World leaders' usage of Twitter in response to the COVID-19 pandemic: a content analysis. J Public Health (Oxf). Aug 18, 2020;42(3):510-516. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Haman M. The use of Twitter by state leaders and its impact on the public during the COVID-19 pandemic. Heliyon. Nov 2020;6(11):e05540. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Huang X, Li Z, Jiang Y, Li X, Porter D. Twitter reveals human mobility dynamics during the COVID-19 pandemic. PLoS One. Nov 10, 2020;15(11):e0241957. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Wicke P, Bolognesi MM. Framing COVID-19: how we conceptualize and discuss the pandemic on Twitter. PLoS One. Sep 30, 2020;15(9):e0240010. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Chang CH, Monselise M, Yang CC. What are people concerned about during the pandemic? detecting evolving topics about COVID-19 from Twitter. J Healthc Inform Res. Jan 17, 2021;5(1):70-97. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Xue J, Chen J, Hu R, Chen C, Zheng C, Su Y, et al. Twitter discussions and emotions about the COVID-19 pandemic: machine learning approach. J Med Internet Res. Nov 25, 2020;22(11):e20550. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Boon-Itt S, Skunkan Y. Public perception of the COVID-19 pandemic on Twitter: sentiment analysis and topic modeling study. JMIR Public Health Surveill. Nov 11, 2020;6(4):e21978. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Valdez D, Ten Thij M, Bathina K, Rutter LA, Bollen J. Social media insights into US mental health during the COVID-19 pandemic: longitudinal analysis of Twitter data. J Med Internet Res. Dec 14, 2020;22(12):e21418. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rashid MA, Yip SW, Gill D, Arfeen Z. Sharing is caring: an analysis of #FOAMed Twitter posts during the COVID-19 pandemic. Postgrad Med J. Mar 14, 2022;98(1157):199-204. [ CrossRef ] [ Medline ]
• Ghosh P, Schwartz G, Narouze S. Twitter as a powerful tool for communication between pain physicians during COVID-19 pandemic. Reg Anesth Pain Med. Feb 21, 2021;46(2):187-188. [ CrossRef ] [ Medline ]
• Burgos LM, Gil Ramirez A, Utengen A, Thamman R. Use of Twitter during COVID-19 pandemic: an opportunity for continuing medical education in cardiology. Medicina (B Aires). 2020;80 Suppl 6:122-123. [ FREE Full text ] [ Medline ]
• Margus C, Brown N, Hertelendy AJ, Safferman MR, Hart A, Ciottone GR. Emergency physician Twitter use in the COVID-19 pandemic as a potential predictor of impending surge: retrospective observational study. J Med Internet Res. Jul 14, 2021;23(7):e28615. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Pulsar TRAC- Audience intelligence platform and social listening tool. Pulsar. URL: https://www.pulsarplatform.com/solutions/pulsar-trac/ [accessed 2022-03-01]
• Vera San Juan N, Aceituno D, Djellouli N, Sumray K, Regenold N, Syversen A, et al. Mental health and well-being of healthcare workers during the COVID-19 pandemic in the UK: contrasting guidelines with experiences in practice. BJPsych Open. Dec 10, 2020;7(1):e15. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Paranyushkin D. InfraNodus: generating insight using text network analysis. In: Proceedings of the World Wide Web Conference. 2019. Presented at: WWW '19; May 13-17, 2019; San Francisco, CA. [ CrossRef ]
• Creswell JW, Poth CN. Qualitative Inquiry and Research Design: Choosing Among Five Approaches 4th Edition. Thousand Oaks, CA. SAGE Publications, Inc; 2017.
• Eysenbach G, Till JE. Ethical issues in qualitative research on internet communities. BMJ. Nov 10, 2001;323(7321):1103-1105. [ FREE Full text ] [ CrossRef ] [ Medline ]
• UCL research ethics. University College London. URL: https://www.ucl.ac.uk/research-ethics/do-i-need-ethical-approval# When [accessed 2023-06-12]
• Martin S, Kilich E, Dada S, Kummervold PE, Denny C, Paterson P, et al. "Vaccines for pregnant women…?! Absurd" - Mapping maternal vaccination discourse and stance on social media over six months. Vaccine. Sep 29, 2020;38(42):6627-6637. [ CrossRef ] [ Medline ]
• Kummervold PE, Martin S, Dada S, Kilich E, Denny C, Paterson P, et al. Categorizing vaccine confidence with a transformer-based machine learning model: analysis of nuances of vaccine sentiment in Twitter discourse. JMIR Med Inform. Oct 08, 2021;9(10):e29584. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ahmed W, Bath PA, Demartini G. Using Twitter as a data source: an overview of ethical, legal, and methodological challenges. In: Woodfield K, editor. The Ethics of Online Research. Bingley, UK. Emerald Publishing Limited; Dec 12, 2017.
• Ayers JW, Caputi TL, Nebeker C, Dredze M. Don't quote me: reverse identification of research participants in social media studies. NPJ Digit Med. Aug 2, 2018;1(1):30. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Varkey B. Principles of clinical ethics and their application to practice. Med Princ Pract. Jun 4, 2021;30(1):17-28. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Long Covid Kids homepage. Long Covid Kids. URL: https://es.longcovidkids.org/ [accessed 2023-06-12]
• The Guardian homepage. The Guardian. URL: https://www.theguardian.com/uk [accessed 2023-06-12]
• Welcome to the BBC. BBC. URL: https://www.bbc.com/ [accessed 2023-06-12]
• People with health. PeopleWith. URL: https://peoplewith.com/ [accessed 2023-06-12]
• Welcome to NCBI. National Institutes of Health National Library of Medicine. URL: https://www.ncbi.nlm.nih.gov/ [accessed 2023-06-12]
• Edobor M. Trust and public discourse during the Covid‐19 pandemic. IPPR Progress Rev. Feb 15, 2021;27(4):354-360. [ CrossRef ]
• Nielsen RK, Fletcher R, Kalogeropoulos A, Simon F. Communications in the coronavirus crisis: lessons for the second wave. Reuters Institute for the Study of Journalism. Oct 27, 2020. URL: https://reutersinstitute.politics.ox.ac.uk/communications-coronavirus-crisis-lessons-second-wave [accessed 2024-03-26]
• Zohar T, Negev M, Sirkin M, Levine H. Trust in COVID-19 policy among public health professionals in Israel during the first wave of the pandemic: a cross-sectional study. Isr J Health Policy Res. Apr 11, 2022;11(1):20. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Enria L, Waterlow N, Rogers NT, Brindle H, Lal S, Eggo RM, et al. Trust and transparency in times of crisis: results from an online survey during the first wave (April 2020) of the COVID-19 epidemic in the UK. PLoS One. Feb 16, 2021;16(2):e0239247. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fotheringham P, Harriott T, Healy G, Arenge G, Wilson E. Pressures and influences on school leaders navigating policy development during the COVID‐19 pandemic. Brit Educ Res J. Jul 30, 2021;48(2):201-227. [ CrossRef ]
• Thomson P, Greany T, Martindale N. The trust deficit in England: emerging research evidence about school leaders and the pandemic. J Educ Admin History. Sep 13, 2021;53(3-4):296-300. [ CrossRef ]
• Callard F, Perego E. How and why patients made long COVID. Soc Sci Med. Jan 2021;268:113426. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Davis HE, Assaf GS, McCorkell L, Wei H, Low RJ, Re'em Y, et al. Characterizing long COVID in an international cohort: 7 months of symptoms and their impact. EClinicalMedicine. Aug 2021;38:101019. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Miyake E, Martin S. Long Covid: online patient narratives, public health communication and vaccine hesitancy. Digit Health. Nov 29, 2021;7:20552076211059649. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Matharaarachchi S, Domaratzki M, Katz A, Muthukumarana S. Discovering long COVID symptom patterns: association rule mining and sentiment analysis in social media tweets. JMIR Form Res. Sep 07, 2022;6(9):e37984. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bhattacharyya A, Seth A, Rai S. The effects of long COVID-19, its severity, and the need for immediate attention: analysis of clinical trials and Twitter data. Front Big Data. Dec 15, 2022;5:1051386. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Samannodi M, Alwafi H, Naser AY, Al Qurashi AA, Qedair JT, Salawati E, et al. Determinants of post-COVID-19 conditions among SARS-CoV-2-infected patients in Saudi Arabia: a web-based cross-sectional study. Diseases. Aug 23, 2022;10(3):55. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Santarossa S, Rapp A, Sardinas S, Hussein J, Ramirez A, Cassidy-Bushrow AE, et al. Understanding the #longCOVID and #longhaulers conversation on Twitter: multimethod study. JMIR Infodemiology. 2022;2(1):e31259. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Déguilhem A, Malaab J, Talmatkadi M, Renner S, Foulquié P, Fagherazzi G, et al. Identifying profiles and symptoms of patients with long COVID in France: data mining infodemiology study based on social media. JMIR Infodemiology. Nov 22, 2022;2(2):e39849. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Awoyemi T, Ebili U, Olusanya A, Ogunniyi KE, Adejumo AV. Twitter sentiment analysis of long COVID syndrome. Cureus. Jun 2022;14(6):e25901. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Malone LA, Morrow A, Chen Y, Curtis D, de Ferranti SD, Desai M, et al. Multi-disciplinary collaborative consensus guidance statement on the assessment and treatment of postacute sequelae of SARS-CoV-2 infection (PASC) in children and adolescents. PM R. Oct 2022;14(10):1241-1269. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Shachar-Lavie I, Shorer M, Segal H, Fennig S, Ashkenazi-Hoffnung L. Mental health among children with long COVID during the COVID-19 pandemic. Eur J Pediatr. Apr 14, 2023;182(4):1793-1801. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Brackel CL, Noij LC, Vijverberg SJ, Legghe CL, Maitland-van der Zee AH, van Goudoever JB, et al. International care programs for pediatric post-COVID condition (long COVID) and the way forward. Pediatr Res. Jan 29, 2024. (forthcoming). [ CrossRef ] [ Medline ]
• Ripp J, Peccoralo L, Charney D. Attending to the emotional well-being of the health care workforce in a New York City health system during the COVID-19 pandemic. Acad Med. Aug 2020;95(8):1136-1139. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Buonsenso D, Camporesi A, Morello R, De Rose C, Fracasso M, Chieffo DP, et al. Social stigma in children with long COVID. Children (Basel). Sep 07, 2023;10(9):1518. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Mitchell L, Frank MR, Harris KD, Dodds PS, Danforth CM. The geography of happiness: connecting twitter sentiment and expression, demographics, and objective characteristics of place. PLoS One. May 29, 2013;8(5):e64417. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Padilla JJ, Kavak H, Lynch CJ, Gore RJ, Diallo SY. Temporal and spatiotemporal investigation of tourist attraction visit sentiment on Twitter. PLoS One. Jun 14, 2018;13(6):e0198857. [ FREE Full text ] [ CrossRef ] [ Medline ]

Abbreviations

Edited by A Mavragani; submitted 20.06.23; peer-reviewed by R Gore, A Wahbeh; comments to author 02.11.23; revised version received 14.02.24; accepted 08.03.24; published 17.04.24.

©Macarena Chepo, Sam Martin, Noémie Déom, Ahmad Firas Khalid, Cecilia Vindrola-Padros. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 17.04.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.

ORIGINAL RESEARCH article

Developing key indicators for sustainable food system: a comprehensive application of stakeholder consultations and delphi method provisionally accepted.

• 1 Institute for Population and Social Research, Mahidol University, Thailand

The final, formatted version of the article will be published soon.

The overall status of the food system in Thailand is currently unknown. Although several national and international reports describe Thailand food system, they are not accurate and relevant to inform policies. This study aims to develop indicators which measure Thailand's sustainable food system. We adopted seven-dimensional metrics proposed by Gustafson to facilitate a comparative analysis of food systems, namely (1) food nutrient adequacy; (2) ecosystem stability; (3) food availability and affordability; (4) sociocultural well-being; (5) food safety; (6) resilience; and (7) waste and loss reduction. Three rounds of the Delphi method were convened to assess the proposed indicators using the Item Objective Congruence (IOC) by 48 Thai stakeholders recruited from the government, NGOs, and academia. IOC is a procedure used in test development for evaluating content validity at the item development stage. In each round, the average IOC for each item was carefully considered, together with stakeholders' comments on whether to retain, remove, or recruit new indicators. The communication through mail and email was sent out so that stakeholders could assess independently. A total of 88 and 73 indicators went to the first and second round Delphi assessment; this resulted in 62 final indicators after the third round. In conclusion, these 62 indicators and 190 sub-indicators are too many for policy uses. As an ongoing indicator development, we plan that these 62 indicators will be further tested in different settings to assess data feasibility. After field tests, the final prioritized indicators will be submitted for policy decisions for regular national monitoring and informing policy towards sustainable food systems in Thailand.

Keywords: Sustainable food system, indicator, Food security, resilience, Agriculture, Delphi method

Received: 08 Jan 2024; Accepted: 17 Apr 2024.

Copyright: © 2024 Rittirong, Chuenglertsiri, Nitnara and Phulkerd. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

* Correspondence: Dr. Jongjit Rittirong, Institute for Population and Social Research, Mahidol University, Salaya, Thailand

People also looked at

• SUGGESTED TOPICS
• The Magazine
• Newsletters
• Managing Yourself
• Managing Teams
• Work-life Balance
• The Big Idea
• Data & Visuals
• Reading Lists
• Case Selections
• HBR Learning
• Topic Feeds
• Account Settings
• Email Preferences

Transformations That Work

• Michael Mankins
• Patrick Litre

More than a third of large organizations have some type of transformation program underway at any given time, and many launch one major change initiative after another. Though they kick off with a lot of fanfare, most of these efforts fail to deliver. Only 12% produce lasting results, and that figure hasn’t budged in the past two decades, despite everything we’ve learned over the years about how to lead change.

Clearly, businesses need a new model for transformation. In this article the authors present one based on research with dozens of leading companies that have defied the odds, such as Ford, Dell, Amgen, T-Mobile, Adobe, and Virgin Australia. The successful programs, the authors found, employed six critical practices: treating transformation as a continuous process; building it into the company’s operating rhythm; explicitly managing organizational energy; using aspirations, not benchmarks, to set goals; driving change from the middle of the organization out; and tapping significant external capital to fund the effort from the start.

Lessons from companies that are defying the odds

Idea in Brief

The problem.

Although companies frequently engage in transformation initiatives, few are actually transformative. Research indicates that only 12% of major change programs produce lasting results.

Why It Happens

Leaders are increasingly content with incremental improvements. As a result, they experience fewer outright failures but equally fewer real transformations.

The Solution

To deliver, change programs must treat transformation as a continuous process, build it into the company’s operating rhythm, explicitly manage organizational energy, state aspirations rather than set targets, drive change from the middle out, and be funded by serious capital investments.

Nearly every major corporation has embarked on some sort of transformation in recent years. By our estimates, at any given time more than a third of large organizations have a transformation program underway. When asked, roughly 50% of CEOs we’ve interviewed report that their company has undertaken two or more major change efforts within the past five years, with nearly 20% reporting three or more.

• Michael Mankins is a leader in Bain’s Organization and Strategy practices and is a partner based in Austin, Texas. He is a coauthor of Time, Talent, Energy: Overcome Organizational Drag and Unleash Your Team’s Productive Power (Harvard Business Review Press, 2017).
• PL Patrick Litre leads Bain’s Global Transformation and Change practice and is a partner based in Atlanta.

Partner Center

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Front Res Metr Anal

The Use of Research Methods in Psychological Research: A Systematised Review

Salomé elizabeth scholtz.

1 Community Psychosocial Research (COMPRES), School of Psychosocial Health, North-West University, Potchefstroom, South Africa

Werner de Klerk

Leon t. de beer.

2 WorkWell Research Institute, North-West University, Potchefstroom, South Africa

Research methods play an imperative role in research quality as well as educating young researchers, however, the application thereof is unclear which can be detrimental to the field of psychology. Therefore, this systematised review aimed to determine what research methods are being used, how these methods are being used and for what topics in the field. Our review of 999 articles from five journals over a period of 5 years indicated that psychology research is conducted in 10 topics via predominantly quantitative research methods. Of these 10 topics, social psychology was the most popular. The remainder of the conducted methodology is described. It was also found that articles lacked rigour and transparency in the used methodology which has implications for replicability. In conclusion this article, provides an overview of all reported methodologies used in a sample of psychology journals. It highlights the popularity and application of methods and designs throughout the article sample as well as an unexpected lack of rigour with regard to most aspects of methodology. Possible sample bias should be considered when interpreting the results of this study. It is recommended that future research should utilise the results of this study to determine the possible impact on the field of psychology as a science and to further investigation into the use of research methods. Results should prompt the following future research into: a lack or rigour and its implication on replication, the use of certain methods above others, publication bias and choice of sampling method.

Introduction

Psychology is an ever-growing and popular field (Gough and Lyons, 2016 ; Clay, 2017 ). Due to this growth and the need for science-based research to base health decisions on (Perestelo-Pérez, 2013 ), the use of research methods in the broad field of psychology is an essential point of investigation (Stangor, 2011 ; Aanstoos, 2014 ). Research methods are therefore viewed as important tools used by researchers to collect data (Nieuwenhuis, 2016 ) and include the following: quantitative, qualitative, mixed method and multi method (Maree, 2016 ). Additionally, researchers also employ various types of literature reviews to address research questions (Grant and Booth, 2009 ). According to literature, what research method is used and why a certain research method is used is complex as it depends on various factors that may include paradigm (O'Neil and Koekemoer, 2016 ), research question (Grix, 2002 ), or the skill and exposure of the researcher (Nind et al., 2015 ). How these research methods are employed is also difficult to discern as research methods are often depicted as having fixed boundaries that are continuously crossed in research (Johnson et al., 2001 ; Sandelowski, 2011 ). Examples of this crossing include adding quantitative aspects to qualitative studies (Sandelowski et al., 2009 ), or stating that a study used a mixed-method design without the study having any characteristics of this design (Truscott et al., 2010 ).

The inappropriate use of research methods affects how students and researchers improve and utilise their research skills (Scott Jones and Goldring, 2015 ), how theories are developed (Ngulube, 2013 ), and the credibility of research results (Levitt et al., 2017 ). This, in turn, can be detrimental to the field (Nind et al., 2015 ), journal publication (Ketchen et al., 2008 ; Ezeh et al., 2010 ), and attempts to address public social issues through psychological research (Dweck, 2017 ). This is especially important given the now well-known replication crisis the field is facing (Earp and Trafimow, 2015 ; Hengartner, 2018 ).

Due to this lack of clarity on method use and the potential impact of inept use of research methods, the aim of this study was to explore the use of research methods in the field of psychology through a review of journal publications. Chaichanasakul et al. ( 2011 ) identify reviewing articles as the opportunity to examine the development, growth and progress of a research area and overall quality of a journal. Studies such as Lee et al. ( 1999 ) as well as Bluhm et al. ( 2011 ) review of qualitative methods has attempted to synthesis the use of research methods and indicated the growth of qualitative research in American and European journals. Research has also focused on the use of research methods in specific sub-disciplines of psychology, for example, in the field of Industrial and Organisational psychology Coetzee and Van Zyl ( 2014 ) found that South African publications tend to consist of cross-sectional quantitative research methods with underrepresented longitudinal studies. Qualitative studies were found to make up 21% of the articles published from 1995 to 2015 in a similar study by O'Neil and Koekemoer ( 2016 ). Other methods in health psychology, such as Mixed methods research have also been reportedly growing in popularity (O'Cathain, 2009 ).

A broad overview of the use of research methods in the field of psychology as a whole is however, not available in the literature. Therefore, our research focused on answering what research methods are being used, how these methods are being used and for what topics in practice (i.e., journal publications) in order to provide a general perspective of method used in psychology publication. We synthesised the collected data into the following format: research topic [areas of scientific discourse in a field or the current needs of a population (Bittermann and Fischer, 2018 )], method [data-gathering tools (Nieuwenhuis, 2016 )], sampling [elements chosen from a population to partake in research (Ritchie et al., 2009 )], data collection [techniques and research strategy (Maree, 2016 )], and data analysis [discovering information by examining bodies of data (Ktepi, 2016 )]. A systematised review of recent articles (2013 to 2017) collected from five different journals in the field of psychological research was conducted.

Grant and Booth ( 2009 ) describe systematised reviews as the review of choice for post-graduate studies, which is employed using some elements of a systematic review and seldom more than one or two databases to catalogue studies after a comprehensive literature search. The aspects used in this systematised review that are similar to that of a systematic review were a full search within the chosen database and data produced in tabular form (Grant and Booth, 2009 ).

Sample sizes and timelines vary in systematised reviews (see Lowe and Moore, 2014 ; Pericall and Taylor, 2014 ; Barr-Walker, 2017 ). With no clear parameters identified in the literature (see Grant and Booth, 2009 ), the sample size of this study was determined by the purpose of the sample (Strydom, 2011 ), and time and cost constraints (Maree and Pietersen, 2016 ). Thus, a non-probability purposive sample (Ritchie et al., 2009 ) of the top five psychology journals from 2013 to 2017 was included in this research study. Per Lee ( 2015 ) American Psychological Association (APA) recommends the use of the most up-to-date sources for data collection with consideration of the context of the research study. As this research study focused on the most recent trends in research methods used in the broad field of psychology, the identified time frame was deemed appropriate.

Psychology journals were only included if they formed part of the top five English journals in the miscellaneous psychology domain of the Scimago Journal and Country Rank (Scimago Journal & Country Rank, 2017 ). The Scimago Journal and Country Rank provides a yearly updated list of publicly accessible journal and country-specific indicators derived from the Scopus® database (Scopus, 2017b ) by means of the Scimago Journal Rank (SJR) indicator developed by Scimago from the algorithm Google PageRank™ (Scimago Journal & Country Rank, 2017 ). Scopus is the largest global database of abstracts and citations from peer-reviewed journals (Scopus, 2017a ). Reasons for the development of the Scimago Journal and Country Rank list was to allow researchers to assess scientific domains, compare country rankings, and compare and analyse journals (Scimago Journal & Country Rank, 2017 ), which supported the aim of this research study. Additionally, the goals of the journals had to focus on topics in psychology in general with no preference to specific research methods and have full-text access to articles.

The following list of top five journals in 2018 fell within the abovementioned inclusion criteria (1) Australian Journal of Psychology, (2) British Journal of Psychology, (3) Europe's Journal of Psychology, (4) International Journal of Psychology and lastly the (5) Journal of Psychology Applied and Interdisciplinary.

Journals were excluded from this systematised review if no full-text versions of their articles were available, if journals explicitly stated a publication preference for certain research methods, or if the journal only published articles in a specific discipline of psychological research (for example, industrial psychology, clinical psychology etc.).

The researchers followed a procedure (see Figure 1 ) adapted from that of Ferreira et al. ( 2016 ) for systematised reviews. Data collection and categorisation commenced on 4 December 2017 and continued until 30 June 2019. All the data was systematically collected and coded manually (Grant and Booth, 2009 ) with an independent person acting as co-coder. Codes of interest included the research topic, method used, the design used, sampling method, and methodology (the method used for data collection and data analysis). These codes were derived from the wording in each article. Themes were created based on the derived codes and checked by the co-coder. Lastly, these themes were catalogued into a table as per the systematised review design.

Systematised review procedure.

According to Johnston et al. ( 2019 ), “literature screening, selection, and data extraction/analyses” (p. 7) are specifically tailored to the aim of a review. Therefore, the steps followed in a systematic review must be reported in a comprehensive and transparent manner. The chosen systematised design adhered to the rigour expected from systematic reviews with regard to full search and data produced in tabular form (Grant and Booth, 2009 ). The rigorous application of the systematic review is, therefore discussed in relation to these two elements.

Firstly, to ensure a comprehensive search, this research study promoted review transparency by following a clear protocol outlined according to each review stage before collecting data (Johnston et al., 2019 ). This protocol was similar to that of Ferreira et al. ( 2016 ) and approved by three research committees/stakeholders and the researchers (Johnston et al., 2019 ). The eligibility criteria for article inclusion was based on the research question and clearly stated, and the process of inclusion was recorded on an electronic spreadsheet to create an evidence trail (Bandara et al., 2015 ; Johnston et al., 2019 ). Microsoft Excel spreadsheets are a popular tool for review studies and can increase the rigour of the review process (Bandara et al., 2015 ). Screening for appropriate articles for inclusion forms an integral part of a systematic review process (Johnston et al., 2019 ). This step was applied to two aspects of this research study: the choice of eligible journals and articles to be included. Suitable journals were selected by the first author and reviewed by the second and third authors. Initially, all articles from the chosen journals were included. Then, by process of elimination, those irrelevant to the research aim, i.e., interview articles or discussions etc., were excluded.

To ensure rigourous data extraction, data was first extracted by one reviewer, and an independent person verified the results for completeness and accuracy (Johnston et al., 2019 ). The research question served as a guide for efficient, organised data extraction (Johnston et al., 2019 ). Data was categorised according to the codes of interest, along with article identifiers for audit trails such as authors, title and aims of articles. The categorised data was based on the aim of the review (Johnston et al., 2019 ) and synthesised in tabular form under methods used, how these methods were used, and for what topics in the field of psychology.

The initial search produced a total of 1,145 articles from the 5 journals identified. Inclusion and exclusion criteria resulted in a final sample of 999 articles ( Figure 2 ). Articles were co-coded into 84 codes, from which 10 themes were derived ( Table 1 ).

Journal article frequency.

Codes used to form themes (research topics).

These 10 themes represent the topic section of our research question ( Figure 3 ). All these topics except, for the final one, psychological practice , were found to concur with the research areas in psychology as identified by Weiten ( 2010 ). These research areas were chosen to represent the derived codes as they provided broad definitions that allowed for clear, concise categorisation of the vast amount of data. Article codes were categorised under particular themes/topics if they adhered to the research area definitions created by Weiten ( 2010 ). It is important to note that these areas of research do not refer to specific disciplines in psychology, such as industrial psychology; but to broader fields that may encompass sub-interests of these disciplines.

Topic frequency (international sample).

In the case of developmental psychology , researchers conduct research into human development from childhood to old age. Social psychology includes research on behaviour governed by social drivers. Researchers in the field of educational psychology study how people learn and the best way to teach them. Health psychology aims to determine the effect of psychological factors on physiological health. Physiological psychology , on the other hand, looks at the influence of physiological aspects on behaviour. Experimental psychology is not the only theme that uses experimental research and focuses on the traditional core topics of psychology (for example, sensation). Cognitive psychology studies the higher mental processes. Psychometrics is concerned with measuring capacity or behaviour. Personality research aims to assess and describe consistency in human behaviour (Weiten, 2010 ). The final theme of psychological practice refers to the experiences, techniques, and interventions employed by practitioners, researchers, and academia in the field of psychology.

Articles under these themes were further subdivided into methodologies: method, sampling, design, data collection, and data analysis. The categorisation was based on information stated in the articles and not inferred by the researchers. Data were compiled into two sets of results presented in this article. The first set addresses the aim of this study from the perspective of the topics identified. The second set of results represents a broad overview of the results from the perspective of the methodology employed. The second set of results are discussed in this article, while the first set is presented in table format. The discussion thus provides a broad overview of methods use in psychology (across all themes), while the table format provides readers with in-depth insight into methods used in the individual themes identified. We believe that presenting the data from both perspectives allow readers a broad understanding of the results. Due a large amount of information that made up our results, we followed Cichocka and Jost ( 2014 ) in simplifying our results. Please note that the numbers indicated in the table in terms of methodology differ from the total number of articles. Some articles employed more than one method/sampling technique/design/data collection method/data analysis in their studies.

What follows is the results for what methods are used, how these methods are used, and which topics in psychology they are applied to . Percentages are reported to the second decimal in order to highlight small differences in the occurrence of methodology.

Firstly, with regard to the research methods used, our results show that researchers are more likely to use quantitative research methods (90.22%) compared to all other research methods. Qualitative research was the second most common research method but only made up about 4.79% of the general method usage. Reviews occurred almost as much as qualitative studies (3.91%), as the third most popular method. Mixed-methods research studies (0.98%) occurred across most themes, whereas multi-method research was indicated in only one study and amounted to 0.10% of the methods identified. The specific use of each method in the topics identified is shown in Table 2 and Figure 4 .

Research methods in psychology.

Research method frequency in topics.

Secondly, in the case of how these research methods are employed , our study indicated the following.

Sampling −78.34% of the studies in the collected articles did not specify a sampling method. From the remainder of the studies, 13 types of sampling methods were identified. These sampling methods included broad categorisation of a sample as, for example, a probability or non-probability sample. General samples of convenience were the methods most likely to be applied (10.34%), followed by random sampling (3.51%), snowball sampling (2.73%), and purposive (1.37%) and cluster sampling (1.27%). The remainder of the sampling methods occurred to a more limited extent (0–1.0%). See Table 3 and Figure 5 for sampling methods employed in each topic.

Sampling use in the field of psychology.

Sampling method frequency in topics.

Designs were categorised based on the articles' statement thereof. Therefore, it is important to note that, in the case of quantitative studies, non-experimental designs (25.55%) were often indicated due to a lack of experiments and any other indication of design, which, according to Laher ( 2016 ), is a reasonable categorisation. Non-experimental designs should thus be compared with experimental designs only in the description of data, as it could include the use of correlational/cross-sectional designs, which were not overtly stated by the authors. For the remainder of the research methods, “not stated” (7.12%) was assigned to articles without design types indicated.

From the 36 identified designs the most popular designs were cross-sectional (23.17%) and experimental (25.64%), which concurred with the high number of quantitative studies. Longitudinal studies (3.80%), the third most popular design, was used in both quantitative and qualitative studies. Qualitative designs consisted of ethnography (0.38%), interpretative phenomenological designs/phenomenology (0.28%), as well as narrative designs (0.28%). Studies that employed the review method were mostly categorised as “not stated,” with the most often stated review designs being systematic reviews (0.57%). The few mixed method studies employed exploratory, explanatory (0.09%), and concurrent designs (0.19%), with some studies referring to separate designs for the qualitative and quantitative methods. The one study that identified itself as a multi-method study used a longitudinal design. Please see how these designs were employed in each specific topic in Table 4 , Figure 6 .

Design use in the field of psychology.

Design frequency in topics.

Data collection and analysis —data collection included 30 methods, with the data collection method most often employed being questionnaires (57.84%). The experimental task (16.56%) was the second most preferred collection method, which included established or unique tasks designed by the researchers. Cognitive ability tests (6.84%) were also regularly used along with various forms of interviewing (7.66%). Table 5 and Figure 7 represent data collection use in the various topics. Data analysis consisted of 3,857 occurrences of data analysis categorised into ±188 various data analysis techniques shown in Table 6 and Figures 1 – 7 . Descriptive statistics were the most commonly used (23.49%) along with correlational analysis (17.19%). When using a qualitative method, researchers generally employed thematic analysis (0.52%) or different forms of analysis that led to coding and the creation of themes. Review studies presented few data analysis methods, with most studies categorising their results. Mixed method and multi-method studies followed the analysis methods identified for the qualitative and quantitative studies included.

Data collection in the field of psychology.

Data collection frequency in topics.

Data analysis in the field of psychology.

Results of the topics researched in psychology can be seen in the tables, as previously stated in this article. It is noteworthy that, of the 10 topics, social psychology accounted for 43.54% of the studies, with cognitive psychology the second most popular research topic at 16.92%. The remainder of the topics only occurred in 4.0–7.0% of the articles considered. A list of the included 999 articles is available under the section “View Articles” on the following website: https://methodgarden.xtrapolate.io/ . This website was created by Scholtz et al. ( 2019 ) to visually present a research framework based on this Article's results.

This systematised review categorised full-length articles from five international journals across the span of 5 years to provide insight into the use of research methods in the field of psychology. Results indicated what methods are used how these methods are being used and for what topics (why) in the included sample of articles. The results should be seen as providing insight into method use and by no means a comprehensive representation of the aforementioned aim due to the limited sample. To our knowledge, this is the first research study to address this topic in this manner. Our discussion attempts to promote a productive way forward in terms of the key results for method use in psychology, especially in the field of academia (Holloway, 2008 ).

With regard to the methods used, our data stayed true to literature, finding only common research methods (Grant and Booth, 2009 ; Maree, 2016 ) that varied in the degree to which they were employed. Quantitative research was found to be the most popular method, as indicated by literature (Breen and Darlaston-Jones, 2010 ; Counsell and Harlow, 2017 ) and previous studies in specific areas of psychology (see Coetzee and Van Zyl, 2014 ). Its long history as the first research method (Leech et al., 2007 ) in the field of psychology as well as researchers' current application of mathematical approaches in their studies (Toomela, 2010 ) might contribute to its popularity today. Whatever the case may be, our results show that, despite the growth in qualitative research (Demuth, 2015 ; Smith and McGannon, 2018 ), quantitative research remains the first choice for article publication in these journals. Despite the included journals indicating openness to articles that apply any research methods. This finding may be due to qualitative research still being seen as a new method (Burman and Whelan, 2011 ) or reviewers' standards being higher for qualitative studies (Bluhm et al., 2011 ). Future research is encouraged into the possible biasness in publication of research methods, additionally further investigation with a different sample into the proclaimed growth of qualitative research may also provide different results.

Review studies were found to surpass that of multi-method and mixed method studies. To this effect Grant and Booth ( 2009 ), state that the increased awareness, journal contribution calls as well as its efficiency in procuring research funds all promote the popularity of reviews. The low frequency of mixed method studies contradicts the view in literature that it's the third most utilised research method (Tashakkori and Teddlie's, 2003 ). Its' low occurrence in this sample could be due to opposing views on mixing methods (Gunasekare, 2015 ) or that authors prefer publishing in mixed method journals, when using this method, or its relative novelty (Ivankova et al., 2016 ). Despite its low occurrence, the application of the mixed methods design in articles was methodologically clear in all cases which were not the case for the remainder of research methods.

Additionally, a substantial number of studies used a combination of methodologies that are not mixed or multi-method studies. Perceived fixed boundaries are according to literature often set aside, as confirmed by this result, in order to investigate the aim of a study, which could create a new and helpful way of understanding the world (Gunasekare, 2015 ). According to Toomela ( 2010 ), this is not unheard of and could be considered a form of “structural systemic science,” as in the case of qualitative methodology (observation) applied in quantitative studies (experimental design) for example. Based on this result, further research into this phenomenon as well as its implications for research methods such as multi and mixed methods is recommended.

Discerning how these research methods were applied, presented some difficulty. In the case of sampling, most studies—regardless of method—did mention some form of inclusion and exclusion criteria, but no definite sampling method. This result, along with the fact that samples often consisted of students from the researchers' own academic institutions, can contribute to literature and debates among academics (Peterson and Merunka, 2014 ; Laher, 2016 ). Samples of convenience and students as participants especially raise questions about the generalisability and applicability of results (Peterson and Merunka, 2014 ). This is because attention to sampling is important as inappropriate sampling can debilitate the legitimacy of interpretations (Onwuegbuzie and Collins, 2017 ). Future investigation into the possible implications of this reported popular use of convenience samples for the field of psychology as well as the reason for this use could provide interesting insight, and is encouraged by this study.

Additionally, and this is indicated in Table 6 , articles seldom report the research designs used, which highlights the pressing aspect of the lack of rigour in the included sample. Rigour with regards to the applied empirical method is imperative in promoting psychology as a science (American Psychological Association, 2020 ). Omitting parts of the research process in publication when it could have been used to inform others' research skills should be questioned, and the influence on the process of replicating results should be considered. Publications are often rejected due to a lack of rigour in the applied method and designs (Fonseca, 2013 ; Laher, 2016 ), calling for increased clarity and knowledge of method application. Replication is a critical part of any field of scientific research and requires the “complete articulation” of the study methods used (Drotar, 2010 , p. 804). The lack of thorough description could be explained by the requirements of certain journals to only report on certain aspects of a research process, especially with regard to the applied design (Laher, 20). However, naming aspects such as sampling and designs, is a requirement according to the APA's Journal Article Reporting Standards (JARS-Quant) (Appelbaum et al., 2018 ). With very little information on how a study was conducted, authors lose a valuable opportunity to enhance research validity, enrich the knowledge of others, and contribute to the growth of psychology and methodology as a whole. In the case of this research study, it also restricted our results to only reported samples and designs, which indicated a preference for certain designs, such as cross-sectional designs for quantitative studies.

Data collection and analysis were for the most part clearly stated. A key result was the versatile use of questionnaires. Researchers would apply a questionnaire in various ways, for example in questionnaire interviews, online surveys, and written questionnaires across most research methods. This may highlight a trend for future research.

With regard to the topics these methods were employed for, our research study found a new field named “psychological practice.” This result may show the growing consciousness of researchers as part of the research process (Denzin and Lincoln, 2003 ), psychological practice, and knowledge generation. The most popular of these topics was social psychology, which is generously covered in journals and by learning societies, as testaments of the institutional support and richness social psychology has in the field of psychology (Chryssochoou, 2015 ). The APA's perspective on 2018 trends in psychology also identifies an increased amount of psychology focus on how social determinants are influencing people's health (Deangelis, 2017 ).

This study was not without limitations and the following should be taken into account. Firstly, this study used a sample of five specific journals to address the aim of the research study, despite general journal aims (as stated on journal websites), this inclusion signified a bias towards the research methods published in these specific journals only and limited generalisability. A broader sample of journals over a different period of time, or a single journal over a longer period of time might provide different results. A second limitation is the use of Excel spreadsheets and an electronic system to log articles, which was a manual process and therefore left room for error (Bandara et al., 2015 ). To address this potential issue, co-coding was performed to reduce error. Lastly, this article categorised data based on the information presented in the article sample; there was no interpretation of what methodology could have been applied or whether the methods stated adhered to the criteria for the methods used. Thus, a large number of articles that did not clearly indicate a research method or design could influence the results of this review. However, this in itself was also a noteworthy result. Future research could review research methods of a broader sample of journals with an interpretive review tool that increases rigour. Additionally, the authors also encourage the future use of systematised review designs as a way to promote a concise procedure in applying this design.

Our research study presented the use of research methods for published articles in the field of psychology as well as recommendations for future research based on these results. Insight into the complex questions identified in literature, regarding what methods are used how these methods are being used and for what topics (why) was gained. This sample preferred quantitative methods, used convenience sampling and presented a lack of rigorous accounts for the remaining methodologies. All methodologies that were clearly indicated in the sample were tabulated to allow researchers insight into the general use of methods and not only the most frequently used methods. The lack of rigorous account of research methods in articles was represented in-depth for each step in the research process and can be of vital importance to address the current replication crisis within the field of psychology. Recommendations for future research aimed to motivate research into the practical implications of the results for psychology, for example, publication bias and the use of convenience samples.

Ethics Statement

This study was cleared by the North-West University Health Research Ethics Committee: NWU-00115-17-S1.

Author Contributions

All authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

• Aanstoos C. M. (2014). Psychology . Available online at: http://eds.a.ebscohost.com.nwulib.nwu.ac.za/eds/detail/detail?sid=18de6c5c-2b03-4eac-94890145eb01bc70%40sessionmgr4006&vid$=$1&hid$=$4113&bdata$=$JnNpdGU9ZWRzL~WxpdmU%3d#AN$=$93871882&db$=$ers
• American Psychological Association (2020). Science of Psychology . Available online at: https://www.apa.org/action/science/
• Appelbaum M., Cooper H., Kline R. B., Mayo-Wilson E., Nezu A. M., Rao S. M. (2018). Journal article reporting standards for quantitative research in psychology: the APA Publications and Communications Board task force report . Am. Psychol. 73 :3. 10.1037/amp0000191 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Bandara W., Furtmueller E., Gorbacheva E., Miskon S., Beekhuyzen J. (2015). Achieving rigor in literature reviews: insights from qualitative data analysis and tool-support . Commun. Ass. Inform. Syst. 37 , 154–204. 10.17705/1CAIS.03708 [ CrossRef ] [ Google Scholar ]
• Barr-Walker J. (2017). Evidence-based information needs of public health workers: a systematized review . J. Med. Libr. Assoc. 105 , 69–79. 10.5195/JMLA.2017.109 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Bittermann A., Fischer A. (2018). How to identify hot topics in psychology using topic modeling . Z. Psychol. 226 , 3–13. 10.1027/2151-2604/a000318 [ CrossRef ] [ Google Scholar ]
• Bluhm D. J., Harman W., Lee T. W., Mitchell T. R. (2011). Qualitative research in management: a decade of progress . J. Manage. Stud. 48 , 1866–1891. 10.1111/j.1467-6486.2010.00972.x [ CrossRef ] [ Google Scholar ]
• Breen L. J., Darlaston-Jones D. (2010). Moving beyond the enduring dominance of positivism in psychological research: implications for psychology in Australia . Aust. Psychol. 45 , 67–76. 10.1080/00050060903127481 [ CrossRef ] [ Google Scholar ]
• Burman E., Whelan P. (2011). Problems in / of Qualitative Research . Maidenhead: Open University Press/McGraw Hill. [ Google Scholar ]
• Chaichanasakul A., He Y., Chen H., Allen G. E. K., Khairallah T. S., Ramos K. (2011). Journal of Career Development: a 36-year content analysis (1972–2007) . J. Career. Dev. 38 , 440–455. 10.1177/0894845310380223 [ CrossRef ] [ Google Scholar ]
• Chryssochoou X. (2015). Social Psychology . Inter. Encycl. Soc. Behav. Sci. 22 , 532–537. 10.1016/B978-0-08-097086-8.24095-6 [ CrossRef ] [ Google Scholar ]
• Cichocka A., Jost J. T. (2014). Stripped of illusions? Exploring system justification processes in capitalist and post-Communist societies . Inter. J. Psychol. 49 , 6–29. 10.1002/ijop.12011 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Clay R. A. (2017). Psychology is More Popular Than Ever. Monitor on Psychology: Trends Report . Available online at: https://www.apa.org/monitor/2017/11/trends-popular
• Coetzee M., Van Zyl L. E. (2014). A review of a decade's scholarly publications (2004–2013) in the South African Journal of Industrial Psychology . SA. J. Psychol . 40 , 1–16. 10.4102/sajip.v40i1.1227 [ CrossRef ] [ Google Scholar ]
• Counsell A., Harlow L. (2017). Reporting practices and use of quantitative methods in Canadian journal articles in psychology . Can. Psychol. 58 , 140–147. 10.1037/cap0000074 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Deangelis T. (2017). Targeting Social Factors That Undermine Health. Monitor on Psychology: Trends Report . Available online at: https://www.apa.org/monitor/2017/11/trend-social-factors
• Demuth C. (2015). New directions in qualitative research in psychology . Integr. Psychol. Behav. Sci. 49 , 125–133. 10.1007/s12124-015-9303-9 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Denzin N. K., Lincoln Y. (2003). The Landscape of Qualitative Research: Theories and Issues , 2nd Edn. London: Sage. [ Google Scholar ]
• Drotar D. (2010). A call for replications of research in pediatric psychology and guidance for authors . J. Pediatr. Psychol. 35 , 801–805. 10.1093/jpepsy/jsq049 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Dweck C. S. (2017). Is psychology headed in the right direction? Yes, no, and maybe . Perspect. Psychol. Sci. 12 , 656–659. 10.1177/1745691616687747 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Earp B. D., Trafimow D. (2015). Replication, falsification, and the crisis of confidence in social psychology . Front. Psychol. 6 :621. 10.3389/fpsyg.2015.00621 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Ezeh A. C., Izugbara C. O., Kabiru C. W., Fonn S., Kahn K., Manderson L., et al.. (2010). Building capacity for public and population health research in Africa: the consortium for advanced research training in Africa (CARTA) model . Glob. Health Action 3 :5693. 10.3402/gha.v3i0.5693 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Ferreira A. L. L., Bessa M. M. M., Drezett J., De Abreu L. C. (2016). Quality of life of the woman carrier of endometriosis: systematized review . Reprod. Clim. 31 , 48–54. 10.1016/j.recli.2015.12.002 [ CrossRef ] [ Google Scholar ]
• Fonseca M. (2013). Most Common Reasons for Journal Rejections . Available online at: http://www.editage.com/insights/most-common-reasons-for-journal-rejections
• Gough B., Lyons A. (2016). The future of qualitative research in psychology: accentuating the positive . Integr. Psychol. Behav. Sci. 50 , 234–243. 10.1007/s12124-015-9320-8 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Grant M. J., Booth A. (2009). A typology of reviews: an analysis of 14 review types and associated methodologies . Health Info. Libr. J. 26 , 91–108. 10.1111/j.1471-1842.2009.00848.x [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Grix J. (2002). Introducing students to the generic terminology of social research . Politics 22 , 175–186. 10.1111/1467-9256.00173 [ CrossRef ] [ Google Scholar ]
• Gunasekare U. L. T. P. (2015). Mixed research method as the third research paradigm: a literature review . Int. J. Sci. Res. 4 , 361–368. Available online at: https://ssrn.com/abstract=2735996 [ Google Scholar ]
• Hengartner M. P. (2018). Raising awareness for the replication crisis in clinical psychology by focusing on inconsistencies in psychotherapy Research: how much can we rely on published findings from efficacy trials? Front. Psychol. 9 :256. 10.3389/fpsyg.2018.00256 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Holloway W. (2008). Doing intellectual disagreement differently . Psychoanal. Cult. Soc. 13 , 385–396. 10.1057/pcs.2008.29 [ CrossRef ] [ Google Scholar ]
• Ivankova N. V., Creswell J. W., Plano Clark V. L. (2016). Foundations and Approaches to mixed methods research , in First Steps in Research , 2nd Edn. K. Maree (Pretoria: Van Schaick Publishers; ), 306–335. [ Google Scholar ]
• Johnson M., Long T., White A. (2001). Arguments for British pluralism in qualitative health research . J. Adv. Nurs. 33 , 243–249. 10.1046/j.1365-2648.2001.01659.x [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Johnston A., Kelly S. E., Hsieh S. C., Skidmore B., Wells G. A. (2019). Systematic reviews of clinical practice guidelines: a methodological guide . J. Clin. Epidemiol. 108 , 64–72. 10.1016/j.jclinepi.2018.11.030 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Ketchen D. J., Jr., Boyd B. K., Bergh D. D. (2008). Research methodology in strategic management: past accomplishments and future challenges . Organ. Res. Methods 11 , 643–658. 10.1177/1094428108319843 [ CrossRef ] [ Google Scholar ]
• Ktepi B. (2016). Data Analytics (DA) . Available online at: https://eds-b-ebscohost-com.nwulib.nwu.ac.za/eds/detail/detail?vid=2&sid=24c978f0-6685-4ed8-ad85-fa5bb04669b9%40sessionmgr101&bdata=JnNpdGU9ZWRzLWxpdmU%3d#AN=113931286&db=ers
• Laher S. (2016). Ostinato rigore: establishing methodological rigour in quantitative research . S. Afr. J. Psychol. 46 , 316–327. 10.1177/0081246316649121 [ CrossRef ] [ Google Scholar ]
• Lee C. (2015). The Myth of the Off-Limits Source . Available online at: http://blog.apastyle.org/apastyle/research/
• Lee T. W., Mitchell T. R., Sablynski C. J. (1999). Qualitative research in organizational and vocational psychology, 1979–1999 . J. Vocat. Behav. 55 , 161–187. 10.1006/jvbe.1999.1707 [ CrossRef ] [ Google Scholar ]
• Leech N. L., Anthony J., Onwuegbuzie A. J. (2007). A typology of mixed methods research designs . Sci. Bus. Media B. V Qual. Quant 43 , 265–275. 10.1007/s11135-007-9105-3 [ CrossRef ] [ Google Scholar ]
• Levitt H. M., Motulsky S. L., Wertz F. J., Morrow S. L., Ponterotto J. G. (2017). Recommendations for designing and reviewing qualitative research in psychology: promoting methodological integrity . Qual. Psychol. 4 , 2–22. 10.1037/qup0000082 [ CrossRef ] [ Google Scholar ]
• Lowe S. M., Moore S. (2014). Social networks and female reproductive choices in the developing world: a systematized review . Rep. Health 11 :85. 10.1186/1742-4755-11-85 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Maree K. (2016). Planning a research proposal , in First Steps in Research , 2nd Edn, ed Maree K. (Pretoria: Van Schaik Publishers; ), 49–70. [ Google Scholar ]
• Maree K., Pietersen J. (2016). Sampling , in First Steps in Research, 2nd Edn , ed Maree K. (Pretoria: Van Schaik Publishers; ), 191–202. [ Google Scholar ]
• Ngulube P. (2013). Blending qualitative and quantitative research methods in library and information science in sub-Saharan Africa . ESARBICA J. 32 , 10–23. Available online at: http://hdl.handle.net/10500/22397 . [ Google Scholar ]
• Nieuwenhuis J. (2016). Qualitative research designs and data-gathering techniques , in First Steps in Research , 2nd Edn, ed Maree K. (Pretoria: Van Schaik Publishers; ), 71–102. [ Google Scholar ]
• Nind M., Kilburn D., Wiles R. (2015). Using video and dialogue to generate pedagogic knowledge: teachers, learners and researchers reflecting together on the pedagogy of social research methods . Int. J. Soc. Res. Methodol. 18 , 561–576. 10.1080/13645579.2015.1062628 [ CrossRef ] [ Google Scholar ]
• O'Cathain A. (2009). Editorial: mixed methods research in the health sciences—a quiet revolution . J. Mix. Methods 3 , 1–6. 10.1177/1558689808326272 [ CrossRef ] [ Google Scholar ]
• O'Neil S., Koekemoer E. (2016). Two decades of qualitative research in psychology, industrial and organisational psychology and human resource management within South Africa: a critical review . SA J. Indust. Psychol. 42 , 1–16. 10.4102/sajip.v42i1.1350 [ CrossRef ] [ Google Scholar ]
• Onwuegbuzie A. J., Collins K. M. (2017). The role of sampling in mixed methods research enhancing inference quality . Köln Z Soziol. 2 , 133–156. 10.1007/s11577-017-0455-0 [ CrossRef ] [ Google Scholar ]
• Perestelo-Pérez L. (2013). Standards on how to develop and report systematic reviews in psychology and health . Int. J. Clin. Health Psychol. 13 , 49–57. 10.1016/S1697-2600(13)70007-3 [ CrossRef ] [ Google Scholar ]
• Pericall L. M. T., Taylor E. (2014). Family function and its relationship to injury severity and psychiatric outcome in children with acquired brain injury: a systematized review . Dev. Med. Child Neurol. 56 , 19–30. 10.1111/dmcn.12237 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Peterson R. A., Merunka D. R. (2014). Convenience samples of college students and research reproducibility . J. Bus. Res. 67 , 1035–1041. 10.1016/j.jbusres.2013.08.010 [ CrossRef ] [ Google Scholar ]
• Ritchie J., Lewis J., Elam G. (2009). Designing and selecting samples , in Qualitative Research Practice: A Guide for Social Science Students and Researchers , 2nd Edn, ed Ritchie J., Lewis J. (London: Sage; ), 1–23. [ Google Scholar ]
• Sandelowski M. (2011). When a cigar is not just a cigar: alternative perspectives on data and data analysis . Res. Nurs. Health 34 , 342–352. 10.1002/nur.20437 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Sandelowski M., Voils C. I., Knafl G. (2009). On quantitizing . J. Mix. Methods Res. 3 , 208–222. 10.1177/1558689809334210 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Scholtz S. E., De Klerk W., De Beer L. T. (2019). A data generated research framework for conducting research methods in psychological research .
• Scimago Journal & Country Rank (2017). Available online at: http://www.scimagojr.com/journalrank.php?category=3201&year=2015
• Scopus (2017a). About Scopus . Available online at: https://www.scopus.com/home.uri (accessed February 01, 2017).
• Scopus (2017b). Document Search . Available online at: https://www.scopus.com/home.uri (accessed February 01, 2017).
• Scott Jones J., Goldring J. E. (2015). ‘I' m not a quants person'; key strategies in building competence and confidence in staff who teach quantitative research methods . Int. J. Soc. Res. Methodol. 18 , 479–494. 10.1080/13645579.2015.1062623 [ CrossRef ] [ Google Scholar ]
• Smith B., McGannon K. R. (2018). Developing rigor in quantitative research: problems and opportunities within sport and exercise psychology . Int. Rev. Sport Exerc. Psychol. 11 , 101–121. 10.1080/1750984X.2017.1317357 [ CrossRef ] [ Google Scholar ]
• Stangor C. (2011). Introduction to Psychology . Available online at: http://www.saylor.org/books/
• Strydom H. (2011). Sampling in the quantitative paradigm , in Research at Grass Roots; For the Social Sciences and Human Service Professions , 4th Edn, eds de Vos A. S., Strydom H., Fouché C. B., Delport C. S. L. (Pretoria: Van Schaik Publishers; ), 221–234. [ Google Scholar ]
• Tashakkori A., Teddlie C. (2003). Handbook of Mixed Methods in Social & Behavioural Research . Thousand Oaks, CA: SAGE publications. [ Google Scholar ]
• Toomela A. (2010). Quantitative methods in psychology: inevitable and useless . Front. Psychol. 1 :29. 10.3389/fpsyg.2010.00029 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Truscott D. M., Swars S., Smith S., Thornton-Reid F., Zhao Y., Dooley C., et al.. (2010). A cross-disciplinary examination of the prevalence of mixed methods in educational research: 1995–2005 . Int. J. Soc. Res. Methodol. 13 , 317–328. 10.1080/13645570903097950 [ CrossRef ] [ Google Scholar ]
• Weiten W. (2010). Psychology Themes and Variations . Belmont, CA: Wadsworth. [ Google Scholar ]

Help | Advanced Search

Computer Science > Computation and Language

Title: leave no context behind: efficient infinite context transformers with infini-attention.

Abstract: This work introduces an efficient method to scale Transformer-based Large Language Models (LLMs) to infinitely long inputs with bounded memory and computation. A key component in our proposed approach is a new attention technique dubbed Infini-attention. The Infini-attention incorporates a compressive memory into the vanilla attention mechanism and builds in both masked local attention and long-term linear attention mechanisms in a single Transformer block. We demonstrate the effectiveness of our approach on long-context language modeling benchmarks, 1M sequence length passkey context block retrieval and 500K length book summarization tasks with 1B and 8B LLMs. Our approach introduces minimal bounded memory parameters and enables fast streaming inference for LLMs.

Submission history

Access paper:.

• HTML (experimental)
• Other Formats

References & Citations

• Google Scholar
• Semantic Scholar

BibTeX formatted citation

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

Neolithic women in Europe were tied up and buried alive in ritual sacrifices, study suggests

The research found evidence of the "incaprettamento" method of murder at 14 Neolithic sites in Europe.

The murder of sacrificial victims by "incaprettamento" — tying their neck to their legs bent behind their back, so that they effectively strangled themselves — seems to have been a tradition across much of Neolithic Europe, with a new study identifying more than a dozen such murders over more than 2,000 years.

The study comes after a reassessment of an ancient tomb that was discovered more than 20 years ago at Saint-Paul-Trois-Châteaux near Avignon, in southern France. The tomb mimics a silo, or pit where grain was stored, and it held the remains of three women who were buried there about 5,500 years ago.

The new study, published Wednesday (April 10) in the journal Science Advances , reinterprets the positions of two of the skeletons and suggests the individuals were deliberately killed — first by tying them up in the manner called "incaprettamento" and then by burying them while they were still alive, perhaps for an agricultural ritual.

Study senior author Eric Crubézy , a biological anthropologist at Paul Sabatier University in Toulouse, France, told Live Science that there was a lot of agricultural symbolism to the tomb. He noted that a wooden structure built over it was aligned with the sun at the solstices and that several broken stones for grinding grain were found nearby. "You have the alignment, you have the silo, you have the broken stones — so it seems that it was a rite related to agriculture."

Related: Skull of Neolithic 'bog body' from Denmark was smashed by 8 heavy blows in violent murder

To investigate the idea of human sacrifice at Saint-Paul-Trois-Châteaux, Crubézy, who worked on the initial discovery of the tomb, and colleagues examined earlier archaeological studies of tomb sites throughout Europe. The team included forensic pathologist Bertrand Ludes , of Paris Cité University and the study's lead author.

They found evidence of 20 probable cases of sacrificial murders using incaprettamento at 14 Neolithic (New Stone Age) sites dating to between 5400 and 3500 B.C. They also found papers describing Mesolithic (Middle Stone Age) rock art in the Addaura Cave in Sicily, made between 14000 and 11000 B.C., that seems to depict two human figures bound in the incaprettamento manner.

Sign up for the Live Science daily newsletter now

Get the world’s most fascinating discoveries delivered straight to your inbox.

Crubézy said it appears incaprettamento originated as a sacrificial custom in the Mesolithic period, before agriculture, and later came to be used for human sacrifices associated with agriculture in the Neolithic period.

As a method of human sacrifice, incaprettamento seems to have been widespread across much of Neolithic Europe, with evidence of the practice at sites ranging from the Czech Republic to Spain. The earliest is a tomb near Brno-Bohunice in the Czech Republic that is dated to about 5400 B.C., and the latest is the tomb at Saint-Paul-Trois-Châteaux, suggesting that the practice persisted for more than 2,000 years, Crubézy said.

Gruesome murders

The bindings used to tie the two individuals at Saint-Paul-Trois-Châteaux have long since decayed, but a few features of their skeletons — such as the unusual positions of their legs — suggest how they died, Crubézy said.

The third woman in the tomb seems to have been older and likely died from natural causes, the researchers found. She was also interred normally for the time, on her side in the center of the tomb. This suggests that she had been ceremonially buried after her natural death and that the two younger women had been sacrificed to be buried with her, he said.

— 15 people were brutally murdered 5,000 years ago, but the bodies were buried with care

— 2 waves of mass murder struck prehistoric Denmark, genetic study reveals

— Why were dozens of people butchered 6,200 years ago and buried in a Neolithic death pit?

The two sacrificial victims seem to have been pinned down with heavy fragments of stones used for grinding grain, indicating that, despite their bindings, they were still alive when they were buried, he said.

Today, the gruesome incaprettamento murder method is associated with the Italian Mafia , who have sometimes used it as a form of warning or reprimand.

Crubézy said it wasn't known why incaprettamento was used for Stone Age human sacrifices, but it might have been because a person bound in this way could be seen as strangling themselves, rather than being killed by someone else.

Tom Metcalfe is a freelance journalist and regular Live Science contributor who is based in London in the United Kingdom. Tom writes mainly about science, space, archaeology, the Earth and the oceans. He has also written for the BBC, NBC News, National Geographic, Scientific American, Air & Space, and many others.

32 astonishing ancient burials, from 'vampire' decapitations to riches for the afterlife

Drowned land off Australia was an Aboriginal hotspot in last ice age, 4,000 stone artifacts reveal

AI pinpoints where psychosis originates in the brain

• DAR "Crubézy said it wasn't known why incaprettamento was used for Stone Age human sacrifices, but it might have been because a person bound in this way could be seen as strangling themselves, rather than being killed by someone else." This would make no sense because a person obviously could not bind THEMSELVES in this manner! Reply
• Lemmy Caution The article rather confusingly presents us with two widely separate years in which the burial of the two sacrificial victims is thought to have taken place. First we are told these burials are estimated to have occurred around the year 5,400 BCE. Further along in the narrative, on the other hand, we are informed these sacrificial burials took place approximately 5,500 years ago. The obvious problem with this divergent chronology is that there is a substantial difference, which is to say all of 1900 years, between 5,500 years ago and the earlier noted figure of 5,400 BCE, which amounts to fully 7400 years ago. Reply
• View All 2 Comments

Most Popular

• 2 Infamous boat-sinking orcas spotted hundreds of miles from where they should be, baffling scientist
• 3 Pluto's huge white 'heart' has a surprisingly violent origin, new study suggests
• 4 Most massive stellar black hole in the Milky Way discovered 'extremely close' to Earth
• 5 2,000-foot-wide 'potentially hazardous' asteroid has just made its closest approach to Earth — and you can see it with a telescope
• 2 Tired of your laptop battery degrading? New 'pulse current' charging process could double its lifespan.
• 3 Infamous boat-sinking orcas spotted hundreds of miles from where they should be, baffling scientist