delphi technique case study

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Front Public Health

Agile Delphi methodology: A case study on how technology impacts burnout syndrome in the post-pandemic era

Fuensanta medina-dominguez.

1 Computer Science and Engineering Department, Universidad Carlos III de Madrid, Leganés, Madrid, Spain

Maria-Isabel Sanchez-Segura

Antonio de amescua-seco, germán-lenin dugarte-peña.

2 Universidad Francisco de Vitoria, Madrid, Spain

Santiago Villalba Arranz

3 Unidad Técnica de Diseño, Innovación y Desarrollo, Instituto Regional de Seguridad y Salud en el Trabajo de la Comunidad de Madrid, Madrid, Spain

Associated Data

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Introduction

In the post-pandemic era, many habits in different areas of our lives have changed. The exponential growth in the use of technology to perform work activities is one of them. At the same time, there has been a marked increase in burnout syndrome. Is this a coincidence? Could they be two interconnected developments? What if they were? Can we use technology to mitigate this syndrome? This article presents the agile Delphi methodology (MAD), an evolved version of the Delphi method, adapted to the needs of modern-day society.

To drive Occupational Health and Safety (OHS) experts to reach a consensus on what technological and non-technological factors could be causing the burnout syndrome experienced by workers in the post-pandemic era, MAD has been used in a specific case study. This study formally presents MAD and describes the stages enacted to run Delphi experiments agilely.

MAD is more efficient than the traditional Delphi methodology, reducing the time taken to reach a consensus and increasing the quality of the resulting products.

OHS experts identified factors that affect and cause an increase in burnout syndrome as well as mechanisms to mitigate their effects. The next step is to evaluate whether, as the experts predict, burnout syndrome decreases with the mechanisms identified in this case study.

1. Introduction

It is difficult to ignore the extreme social and economic shake-up that we have been experiencing since early 2020. The global health crisis caused by COVID-19, which has enveloped the entire planet, has changed our work habits, among many other things. According to Moss ( 1 ), in April 2020, 81% of workplaces were closed, leaving 2.6 billion people (knowledge workers) locked up and working from home. However, not only did they lock themselves up to work but they also spent their leisure time online, that is, first-world citizens were suddenly living their whole life through information and communication technologies. An example was the growth in popularity of the Zoom platform, which increased from 10 million to 200 million daily users over a few months.

In 2018, the Spanish National Institute for Occupational Health and Safety developed an interesting study ( 2 ) to analyze the psychosocial impact of the use of information and communication technologies (ICT) in the workplace. This study concluded that they had both positive and negative effects but, of course, did not account for the situation experienced over the last few years as a result of the COVID-19 pandemic. Although the situation is no longer as extreme as it was in 2020, COVID-19 certainly triggered and left important changes in its wake that have made us consider our hyperconnected society and its potential connection to the exponential increase in one of the most important psychosocial risks that affect workers, which is the well-known burnout syndrome.

This is not a new syndrome. Much has been written about burnout syndrome since 1974, but it was not until 2019 that it was finally included by the World Health Organization (WHO) in its 10th revision of the International Classification of Diseases (ICD-10) and described as a syndrome conceptualized as a result of chronic work-related stress that has not been successfully managed . It is now classified in the 11th revision of the International Classification of Diseases (ICD-11) as an occupational phenomenon ( 3 ).

Although different methods for measuring burnout syndrome have been used since 1981 ( 4 ), the landscape has changed a great deal since then. We want to put it on the table and reanalyze this psychosocial risk considering the new landscape left behind by the precipitous change suffered as a consequence of the above health situation. We aimed to explore a potential relationship between burnout syndrome and the use of digital technologies, given the exponential growth that both are experiencing. This interest raises the following questions: Could the prominent role of technology in the lives of working people by increasing the number of people suffering from this syndrome? If this is the case and we cannot avoid the use of technology, what could we do to make technology less of a problem? What could we do to convert technology itself into an instrument to mitigate burnout syndrome?

Based on the reliability of its results, one of the best-known and most used techniques to determine whether or not there is consensus among a group of experts on certain criteria is the Delphi method ( 5 ). This method requires some adaptations to the agility of 21st-century demands for more efficient and effective adoption. Therefore, we present a series of adjustments made to this method in the “Research method” section. In the “Case study” section, we report the results of its application with the specific objective of determining whether or not there is consensus regarding the use of technology affecting workers in such a way as to cause burnout syndrome.

2. Literature review

This section first describes burnout syndrome and related works that pinpoint the factors and causes that may potentially exacerbate this syndrome, followed by the traditional Delphi methodology used as the baseline for this case study.

2.1. Burnout syndrome

The World Health Organization (WHO) announced burnout syndrome as the disease of the century. It is not a new syndrome. In 1974, psychiatrist Herbert J. Freudenberger started to investigate burnout. His studies focused mainly on the medical sector. He observed that his colleagues tended to lose empathy with their patients and suffer from exhaustion ( 6 ). The syndrome was, in its early days, related to the health and emergency sectors, such as police and firefighters. However, the syndrome is no longer confined to specific sectors ( 7 , 8 ). There were early warning signs of burnout before the COVID-19 pandemic. However, during and after the pandemic, the incidence of burnout has increased significantly, and, consequently, the syndrome has come to be known as a “second” or “silent” pandemic ( 8 ).

In 2019, the WHO classified it as an occupational risk, and it was included in the International Statistical Classification of Diseases and Related Health Problems (ICD-11), which came into force on 1 January 2022 ( 8 ). The WHO defined burnout as a syndrome conceptualized as resulting from chronic workplace stress that has not been successfully managed ( 3 ). However, according to the foremost expert on burnout, Christina Maslach, the new WHO classification in the ICD-11 is concerning because categorizing burnout as a disease was an attempt by the WHO to provide definitions for what is wrong with people instead of what is wrong with companies ( 9 ). This should lead us to consider the organization that employs the worker, and not the worker, as responsible for burnout.

Additionally, when we analyzed the impact from not only the human (health) but also the economic point of view, we came across interesting statistics. In the United States, burnout costs $500 billion, and up to 550 million days are lost due to work stress. Every year in Europe, mental health costs between 3 and 5% of the GDP of the region. Burnout has a very big impact on the world economy, which is set to increase in proportion to the burnout trend.

To better understand the impact of burnout syndrome, several studies analyzed the factors that are affecting and causing an increase in employee burnout syndrome. Some of the identified factors are high-stress levels ( 10 ), workplace conflicts ( 11 ), social support among colleagues ( 12 ), job satisfaction ( 13 ), work–family reconciliation ( 14 ), the sense of control and autonomy ( 15 ), personal skills, training in communication skills ( 16 ), earnings from employment ( 17 ), unfair treatment at work, unmanageable workload, role ambiguity, deficiencies in communication and support from managers, unreasonable time pressure, and so on. However, none of these studies reflect technology as a cause of burnout. In contrast, the advantages of technology use at companies (productivity, efficiency, job satisfaction, etc.) are widely acknowledged ( 18 ). However, it was recently found that technology may also have a negative connotation and be synonymous with techno-stress, leading to burned-out employees whose performance drops ( 19 – 21 ).

The identification of the factors that increase burnout is just as important as analyzing the mechanisms that mitigate this syndrome. Therefore, we present a study carried out with experts in occupational risk prevention. We analyzed and explored the organizational factors that affect employees, increasing their burnout, as well as the mechanisms that they consider companies should implement to mitigate this syndrome.

Although there are other bodies of research identifying factors and methods, the difference and originality of this investigation in comparison to other studies with the same or similar objectives are that we used a technique called the Delphi method, where the participants, who are recognized experts with a lot of knowledge and experience on the subject, attempted to reach a consensus based on anonymous reflection and sharing of opinions.

2.2. Delphi method

The Delphi method was created in the United States in the 1950s. It has its origin in the philosophical field, where group knowledge is valued over individual knowledge, that is, considering that the relevant information accumulated by a group of experts is always equal to or greater than that of the individual in particular ( 5 , 22 ). The Delphi method is based on the recognition of the superiority of group judgment over individual judgment. Therefore, the objective of the method is to obtain the most reliable consensus opinion from a group of experts ( 5 ).

The special characteristics of this method are as follows:

• - Highly efficient expert selection process: A mechanism has been created to easily identify potential experts. Everyday jobs are performed by workers in physically distributed spaces. Therefore, it is very important to consider the opinions of a broad spectrum of experts from different countries and organizational cultures to assure the maximum possible diversity.
• - Iterative process: The process is divided into several rounds. Participants express their opinions in each round. In between rounds, they have the opportunity to reflect on both their own opinions and those issued by the other experts.
• - Regular feedback: The opinion of the experts on the problem being analyzed is communicated before each round.
• - Anonymity: The experts do not know the source of each response. Anonymity has the advantage of preventing dominant members of the group from influencing or inhibiting other participants. Additionally, none of the experts communicate directly.
• - Availability of group statistical results, if required. Statistics have been associated with response types to maximize efficiency and ensure that the results are analyzable.

Although there are different approaches to organizing Delphi activities ( 23 – 25 ), they all follow the same philosophy, consisting of the steps shown in Figure 1 .

Traditional Delphi method.

It is a methodology that is currently used in many scientific fields, like health sciences ( 26 , 27 ), tourism ( 28 ), and emerging technologies ( 29 ) because of its advantages over other techniques. These advantages, related to methodology and method application, are that the methodology is verifiable, understandable, and holistic ( 30 ); bridges the gap between qualitative and quantitative methods ( 31 ); has a controlled feedback process; and accommodates various statistical analysis techniques to interpret the data ( 32 ), among many others ( 33 ). However, it also has limitations. Li et al. ( 34 ) identified several limitations of the method, one of which is that a Delphi study is time-consuming to complete because the process includes multiple iterations or rounds. Additionally, the Delphi technique is very sensitive to design characteristics and the clarity of the question formulation. Furthermore, as the procedure depends on the quality of the feedback provided, the result must be carefully and responsibly analyzed ( 35 ).

To overcome the identified limitations, we modified the traditional Delphi process and propose the agile Delphi methodology, which will be explained in the “Research method” section.

3. Research method

This section explains the MAD, which adopts some original contributions designed to overcome the deficiencies identified in the traditional Delphi method. We modified the traditional Delphi method by adding specific techniques based on the agile philosophy to some steps of the traditional method. These modifications aimed to improve method efficiency and efficacy and overcome the limitations identified in the “Literature review” section.

The transition from the traditional Delphi method toward an agile methodology requires a cultural change. We propose the use of the Plan-Do-Check-Act (PDCA) methodology that underpins the agile philosophy ( 36 , 37 ). Figure 2 illustrates the agile Delphi methodology, where the agile methodology (PDCA) is combined with the steps of the traditional Delphi methodology. Each step of the agile methodology includes the unmodified steps of the traditional Delphi method (black type), the modified steps of the traditional Delphi method, including the techniques and mechanisms that we contributed (blue type), and new steps not existing in the traditional Delphi process but needed in MAD (red type).

Agile Delphi methodology.

This methodology, MAD, is agile (it follows an agile philosophy), iterative (it repeats the loop or cycle, called rounds in traditional Delphi, where three rounds are recommended), and incremental (it gradually increases the level of detail of the results about whether consensus is reached or not).

Agile Delphi methodology (MAD) defines two work teams:

• Delphi process owner team. This team is composed of three experts on OHS, one woman and two men who are experienced in OHS, specifically burnout syndrome. This team is involved in the following MAD methodology steps: identify the topic and objectives, establish the action plan, select and form the panel of experts, design the questionnaire, collect questionnaire feedback, review the modified questionnaire, interpret results, and make decisions.
• Delphi expert team. This team is composed of four Delphi process experts, two women and two men, who all have experience with and knowledge of methodologies and processes. This team is involved in the following MAD methodology steps: identify the topic and objectives; establish an action plan; design a questionnaire, review a questionnaire, collect feedback on the questionnaire, review the modified questionnaire, start communication, collect questionnaire responses, analyze response data, interpret results and make decisions, and prepare and communicate the results.

In addition, there are two well-established roles:

• Experts: A group whose responsibility is to issue judgments and opinions, which is the core of the method.
• Coordinator: A group that coordinates the process with a variable number of members ranging from two to five people. They coordinate each work team.

Each step of the agile Delphi methodology is described as follows: First, each step of the methodology is framed within the respective PDCA phase ( 36 , 37 ) and then described according to the following descriptors:

• Type: either “traditional” if it was already part of the Delphi method, “modified” if an agile technique has been added to the Delphi method to carry out the step in MAD, and “new step” if this step was not previously part of the Delphi method.
• Description: description of the step.
• Activities: activities to be performed in the step.
• Teams and roles involved in the step.
• Deliverable: deliverable of the step.

3.1. Phase 1: Plan

The purpose of this phase is to describe the problem on which the experts are to reflect. For this purpose, the activities performed in this phase are to identify the topic and study objectives, establish the action plan, and select and form the panel of experts. The following lists the steps of the Plan phase.

3.1.1. Step 1-MAD: Identify the topic and objectives

Type: Modified step.

Description: In this step, the topic, goal, and objectives are defined. The recommended technique for use in this step is SMART (ER) ( 38 ). The Delphi process owner team identifies the topic, the goal, and the objectives to be studied, making sure that the objectives are specific, measurable, achievable, relevant, time-bound, engaging, and rewarding.

Activities to be carried out:

• Activity 1: Briefly describe the topic.
• Activity 2: Identify the general goal.
• Activity 3: Break down the general goal into specific objectives.
• Activity 4: Check if each goal is SMART (ER).

Team and roles involved:

• Team: Delphi process owner team and Delphi expert team
• Role: Coordinator

Deliverable: Table 1 stores the information from this step.

Description of the topic and objectives.

3.1.2. Step 2-MAD: Establish an action plan

Type: New step.

Description: In this step, the action plan is described, defining the activities to be developed as part of MAD, the timeline (start, finish), the person/s responsible for each activity, and deliverables. The recommended technique for use in this step is the GANTT chart ( 39 ).

• Activity 1: Define activities, sequence activities, estimate resources, estimate durations, and develop a schedule.
• Activity 2: Prepare the GANTT chart with the above information.
• Activity 3: Review the GANTT chart (both teams).

List of experts.

Deliverable: A GANTT chart stores the information from this step.

3.1.3. Step 3-MAD: Select and form the expert panel

Description: The panel of experts who are to share their knowledge and experience for the study is selected and formed.

The Delphi process owner team identifies the potential experts that are to participate in the Delphi process. The topic to be studied must be taken into account to identify and select the most suitable experts. The expert selection criteria document is available at: https://www.promiseinnovatech.com/images/files/MAD-Methodology/ExpertNetwork.pdf .

• Activity 1: Define the areas of expertise.
• Activity 2: Define the typology of experts: type of expertise (occupational hazards, etc.), academic qualifications, and years of experience.
• Activity 3: Identify and brief experts. Select 15 to 20 experts on the subject under study from different disciplines. For topic evaluation, representativeness is based on the quality rather than the quantity of the experts.
• Activity 4: Contact the experts to ask them to participate in the MAD. To be able to confirm or refuse participation, they must be sent the following information: topic, description, and objectives of the process to be carried out, as well as the number of rounds to be implemented and the estimated time of the process.
• Activity 5: Complete Table 2 .

Team and actors involved:

• Team: Delphi process owner team
• Actor: Coordinator

Deliverable: Table 2 stores the information from this step.

3.2. Phase 2: Do

The purpose of this phase is to conduct a questionnaire round. Therefore, the questionnaire for the round is designed, and the expert responses to the questionnaire are reviewed. The steps of the Do phase are detailed as follows.

3.2.1. Step 4-MAD: Design and review the questionnaire

This step is composed of five sub-steps:

• Step 4.1 Design questionnaire.
• Step 4.2. Review questionnaire.
• Step 4.3. Collect feedback on the questionnaire.
• Step 4.4. Review the modified questionnaire.
• Step 4.5. Start communication.

3.2.1.1. Step 4.1-MAD: Design questionnaire

Description: The Delphi process owner team designs the questionnaire for each of the rounds. The questionnaire must be aligned with the objectives specified in Step 1 and will include three types of questions:

• Questions requiring numerical data as a response. For example, “How many hours do you spend answering email?”
• [ ] Physical office location
• [ ] Working hours
• [ ] Telecommuting
• [ ] Social benefits (tickets to theme parks and health insurance)
• [ ] Salary”
• Open-ended questions, that is, respondents can write whatever they regard to be relevant and give their own judgments.

Recommended number of questions: 15

• Activity 1: Design questions aligned with the specific objectives of the Delphi process. The questionnaire must include (i) sociodemographic questions to provide an overall description of the group of people answering the questionnaire, and (ii) questions that cover the specific objectives defined in Step 1. The questions must be closed-ended, precise, unambiguous, and understandable, phrased in a language that is appropriate and understandable for experts. Each question must refer to a single objective or concept; that is, there should not be two questions in one. Each question should be aligned with a specific objective. Questions should be placed in a logical order, from easiest to hardest.
• Activity 2: Design response. Depending on the questionnaire type, descriptive or analytic, the responses will be numerical values, categories, yes/no, an ordinal scale, or text (open-ended responses).
• Activity 3: Test or pilot the designed questionnaire with a small number of people to analyze the adequacy of each question and evaluate the clarity of the approach, number of questions, and intelligibility of the content.
• Activity 4: Make the appropriate modifications based on the conclusions of the test or pilot survey.

Draft questionnaire.

Deliverable: Table 3 stores the information from this step.

3.2.1.2. Step 4.2-MAD: Review questionnaire

Description: The Delphi expert team reviews the questionnaire designed by the Delphi process owner team.

• Activity 1: Analyze the adequacy of each questionnaire item, taking into account the objectives established in Step 1.
• Activity 2: Analyze the clarity and approach of questions and answers, the number of questions, and the instructions for experts.
• Activity 3: Analyze which statistics to apply with each question. We are aware that not just any statistic can be used with any type of response. If the statistic that can be used for each questionnaire item has not been previously analyzed, it may not be possible to statistically analyze the data at the end of the MAD process. For this reason, we related the type of response to the statistic to be used to ensure that the results are analyzable.
• Activity 4: Recommend actions to solve the problems identified in the questionnaire.

Review of the draft questionnaire.

• Team: Delphi expert team

Deliverable: Table 4 stores the information from this step.

3.2.1.3. Step 4.3-MAD: Apply feedback on the questionnaire

Description: The Delphi process owner team will review the Delphi process actions recommended by the expert team and make the necessary modifications to the questionnaire.

• Activity 1: Review Table 4 for each question.
• Activity 2: Carry out the recommended actions on each of the questionnaire items, modifying the questions or answers in Table 4 .
• Activity 3: After applying feedback, fill in the “ Has it been done? ” section of Table 4 with Y (yes) or N (no). If N is entered, specify why the recommended actions have not been carried out.

Deliverable: Fully completed Table 4 stores the information.

3.2.1.4. Step 4.4-MAD: Review the modified questionnaire

Description: The Delphi expert team reviews the modifications made to the questionnaire and approves the final questionnaire.

• Activity 1: Review the modified questionnaire.
• Activity 2: Check if all the actions have been carried out.
• Activity 3: Discuss the recommended actions that were not carried out and decide on the final questionnaire.

Final questionnaire.

Deliverable: Table 5 stores the information.

3.2.1.5. Step 4.5-MAD: Start communication

The Delphi expert team informs the panel that round x is starting and provides access to the questionnaire with the instructions for completion.

Record of the round.

• - Team: Delphi process expert team
• - Actor: Coordinator

Deliverable: Table 6 stores the information.

3.2.2. Step 5-MAD: Answer the questionnaire

The experts answer the questionnaire before the finishing date of the round. The Delphi process expert team checks if all the experts have answered the questionnaire and sends a reminder of the date on which the round ends 3 days before the end of the round.

• Activity 1: Answer the questionnaire (experts).

Record of communication with experts.

• Team: Delphi experts team
• Actor: Coordinator and experts

Deliverable: Table 7 is completed by the coordinator of the agile Delphi process expert team.

3.3. Phase 3: Check

The purpose of this phase is to check the expert responses. To do this, the data are analyzed. The steps of the Check phase are explained as follows.

3.3.1. Step 6-MAD: Analyze response data

The Delphi expert team conducts a statistical analysis of the expert responses to each of the questionnaire items.

Activities to be completed:

• Mean, median, mode, maximum, minimum, standard deviation, and quartiles
• The interquartile range: RQ = Q3 – Q1
• Relative interquartile range RIR = (Q3 – Q1)/Q2 (median)
• Percentage of responses in the range of the median ± 1
• Kendall rank correlation coefficient
• Chronbach's alpha
• Activity 2: Provide summary statistics and plot graphs.

3.4. Phase 4: ACT

The purpose of this phase is to interpret the results. In addition, the Delphi expert team decides, based on the results, whether another round is needed, and the results of this round are reported to the experts. The steps of the Act phase are detailed as follows.

3.4.1. Step 7-MAD: Interpret results and make decisions

The Delphi expert and process owner teams analyze the results. They make decisions, interpret responses, and evaluate the actions to be taken in the next round.

• Activity 1: Interpret statistical results.
• Activity 2: Evaluate decision-making.

Conclusions and decision making.

3.4.2. Step 8-MAD: Prepare and report results

The Delphi process expert team reports the results and notifies the experts ahead of the next round.

The activities to be carried out:

• Activity 1: Give feedback to the experts and design the questionnaire for the next round.
• Activity 2: Communicate the feedback obtained from the round to the experts. Feedback should include information on the responses from the previous round, considering the type of questions asked in the respective round.

At the end of a PDCA cycle, an assessment of whether or not it is necessary to carry out a new round is conducted. If so, another PDCA cycle starts, with the exception that Phase 1 is not carried out from scratch: the objectives, experts, and action plan are reviewed, taking into account the actions of the previous round.

To apply the MAD methodology, companies do not need to invest in a specific tool. The methodology is designed for use with a spreadsheet and a form for the questionnaires. These are both tools that are fully accessible to companies as part of a Google or Microsoft package.

4. Case study

Now that the MAD methodology has been defined, we have to measure its efficiency against the traditional Delphi method. For this purpose, we developed a case study to test MAD. This section reports several findings related to:

• The comparison of the agile Delphi methodology (MAD) with the traditional Delphi methodology.
• The results of applying MAD to analyze whether SMEs are considering burnout syndrome as a psychosocial risk, what factors affect or increase employee burnout, and what mechanisms could be implemented to mitigate and reduce burnout syndrome at the workplace.

4.1. Research design

In this case study, three MAD rounds were carried out:

Round 1: The questionnaire was composed of nine questions. There were seven questions based on the sociodemographic and professional characteristics of the participants and two multiple-choice questions to check what the OHS risks and benefits of digital transformation are and discover if burnout is one. The estimated questionnaire response time was 6 min. Experts had 8 days to respond online and received one reminder by email.

Round 2: The questionnaire was composed of eight questions. The questions were devised to discover whether there was consensus about the following:

• Whether enterprises consider burnout syndrome to be a psychosocial risk?
• What ICT and non-ICT mechanisms mitigate burnout syndrome?
• What factors affect this syndrome?

The estimated questionnaire response time was 8 min.

Round 3: The questionnaire was composed of nine questions. The estimated questionnaire response time was 15 min. The questions asked in this round were designed to prompt deeper reflections from experts on the factors and mechanisms that affect and mitigate burnout syndrome.

Participants: A total of 16 experts, experienced and knowledgeable in occupational risk prevention, participated; 56% were women and 44% were men. More than 90% of the experts were in the over 45 year age group, which denotes the extent of expert knowledge and experience. Participation was voluntary and kept confidential throughout the study. The participants do not know each other or know who is participating in the process.

Research questions: The research questions are shown in Table 9 .

Research questions.

ICT, information and communication technology.

Abbreviation: ICT for information and communication technology.

Results: The results are reported in Sections A and B below.

4.1.1. (A) Results of applying the agile Delphi methodology vs. the traditional Delphi method

We carried out two independent studies: Case Study A (CS-A) following the traditional Delphi method and Case Study B (CS-B) following the agile Delphi methodology:

• The traditional Delphi method followed the iterative method that repeats the steps to try to reach a consensus among the experts on a topic, in this case, burnout syndrome.
• The agile Delphi methodology took an agile approach, following the agile, iterative, and incremental methods explained in the “Research method” section.

Although the participants in the studies were different people, both studies were conducted according to the same research design using the same number of participants with the same profiles.

In CS-A, the Delphi expert team was composed of researchers from the Universidad Carlos III de Madrid belonging to the IRSST-UC3M Chair: R&D for Intelligent Digital Transformation of Occupational Health and Safety. The Delphi process owner team was composed of two researchers from Universidad Carlos III de Madrid who did not belong to the IRRST-UC3M Chair.

In CS-B, the Delphi expert team was composed of researchers from Universidad Carlos III de Madrid belonging to the IRSST-UC3M Chair: R&D for Intelligent Digital Transformation of Occupational Health and Safety. The Delphi process owner team was composed of two experts in burnout syndrome from the Madrid Regional Health and Safety at Work Institute (IRSST) in Spain.

The number of experts in both cases was 14 for the CS-A group and 16 for the CS-B group. In both cases, the participant profile included experts in occupational risk prevention. The research design was the same (number of rounds and expert profile). The tools used to analyze the data and run the survey were Microsoft Excel and Microsoft Forms, respectively.

Several questions were formulated in order to compare both methods and check if the results of the agile Delphi methodology were better than those of the traditional Delphi method.

Question 1.A: Is the agile Delphi methodology really agile? Is MAD faster than the traditional methodology? How long does it take to build a case with the traditional Delphi method compared with the agile Delphi methodology? To check the time taken, we analyzed the planning of both case studies. Table 10 shows a comparison of both case studies. The panel of experts was different for each case study to avoid bias, but both had similar experience and profiles. Both case studies began on the same date, that is, 1 March (they were developed simultaneously), but CS-B (using MAD) ended almost a month earlier. Therefore, we can state that it takes less time to run the agile Delphi methodology than the traditional methodology. The activities that took less time were the steps for which we had recommended techniques or created mechanisms to improve their efficiency.

CS-B: agile Delphi methodology.

Question 2.A: What is the quality of the questionnaire? How many times do you have to redo the questionnaire or rework questions? Not only is it important to be more efficient from the point of view of the time spent performing the Delphi process, but it is also necessary to check the quality of the questionnaires. The question that we formulated was, “What rework effort was required in order to align the questionnaire items with the identified objectives?” To answer this question, we checked how much rework was required on the questionnaire for each round. Rework means how many times, on average, the questionnaire items had to be redone or revised before they were considered to be complete. As Figure 3 shows, the number of reworks was lower using the techniques and mechanisms defined in MAD, leading to higher-quality questionnaires in fewer iterations.

4.1.2. (B) Results of the burnout syndrome study using agile Delphi

The dossier containing the information gathered at each step of the MAD methodology during case study development is available at https://www.promiseinnovatech.com/products/other-research-outputs .

Psychosocial risk: Although burnout syndrome occupies a prominent place on the list of psychosocial occupational risks, this research was conducted to find out if there is consensus among OHS experts on the following questions.

Question 1.B: Do enterprises consider burnout syndrome to be a psychosocial risk? As an OHS expert, do you consider burnout to be a psychosocial risk? The results in terms of consensus among OHS experts are clear and compelling: 75% of the enterprises employing the experts do consider burnout syndrome to be a psychosocial risk. In addition, there is a unanimous agreement among the experts that burnout syndrome is a psychosocial risk and that OHS must consider it as such.

Technology affects burnout syndrome: In view of the exponential growth of both digital technologies and burnout syndrome, we raised the following questions.

Question 2.B: Do companies have data to corroborate the fact that ICTs have a negative impact on burnout syndrome, that is, that technological resources and tools have a negative effect and help to increase burnout syndrome among employees? A total of 25% of the companies have data that support that ICTs negatively affect worker burnout syndrome.

Question 3.B: Do companies have mechanisms to mitigate the negative impact of ICTs on burnout syndrome, that is, technological resources and tools to help reduce burnout syndrome among employees? A total of 13% of companies do have mechanisms. Although this is a rather low percentage, it is encouraging in the sense that there is hope that ICTs can be used as a lever to reduce burnout syndrome symptoms.

Factors that increase burnout syndrome: There are a lot of studies exploring the factors that affect burnout syndrome. However, these factors have not been studied from an occupational risk prevention perspective. In this study, we formulated the following question.

Question 4.B: Which factors do experts currently consider to have the greatest impact on the increase in burnout syndrome among employees at the workplace?

Figure 4 shows the degree of consensus for the effect of each factor on burnout syndrome (Option 1—the most and Option 4—least) as prioritized by the experts. The y-axis shows the priority. For example, the percentage of experts that consider burnout syndrome to be exacerbated by an inappropriate organizational culture is 68.80%, 12.50% for poorly structured organizational policies, 12.50% for poorly defined processes, and only 6.30% for poor implementation of digital transformation. This result was elicited in round 2. Once the data had been analyzed, this information was delivered to experts in the next round (round 3). The objective of this feedback was for experts to reflect on their responses, considering the opinions of the other experts. In round 3, they were asked the same question to check if they had changed their opinion. They were also asked to think more specifically about each of the factors and provide specific actions to mitigate their effect on burnout. The actions identified for each factor are shown in Table 11 .

Consensus on factors.

Actions for each factor.

Mechanism to mitigate burnout syndrome: Having identified the business factors that affect employee burnout syndrome, the next step was to ask the experts different questions related to reducing the impact of the identified factors. Do the companies have mechanisms in place to prevent and mitigate burnout? Taking into account that enterprises need both ICT and non-ICT mechanisms to mitigate burnout syndrome, which mechanisms, based on your knowledge and experience as an OHS specialist, are more effective for companies to prevent burnout syndrome?

Question 5.B: Does the company you work for have mechanisms to prevent and reduce burnout syndrome? A total of 63% of experts agreed that their enterprise did not have mechanisms in place, 13% stated that their enterprises did, and 25% said N/A (do not know or not applicable).

Question 6.B: Which are the most effective ICT mechanisms to prevent burnout syndrome at companies?

• ICT mechanisms that favor adequate and effective communication channels for relationships between workers, middle managers, and managers of the organization.
• Technological resources are adapted to jobs to facilitate the routine work of its employees. If this is not the case, technological media cease to be facilitators and become inhibitors, which is one of the causes that provoke burnout syndrome.
• Continuous training. ICTs are constantly evolving, and employees need to be prepared for change. The only way to do this is through continuous training and online help systems.
• ICT mechanisms to detect symptoms of mental fatigue and evaluate the state of the worker.
• Digital disconnection tools and protocols. These frameworks will guarantee the right of workers to effectively enjoy their rest time as well as the right to preserve their personal and family privacy.

Question 6.B: Which are the most effective non-ICT mechanisms to prevent burnout syndrome at companies?

• Work autonomy indicates that employees have time flexibility and can make decisions about how to organize their work.
• Implementation of psychosocial and physical health strategies, like mentoring techniques and training to develop skills for emotional management on top of change management techniques from a psychosocial perspective.
• Teamwork promotion, through the improvement of communication and the creation of safe spaces where communication is encouraged.
• Worker reporting and action channels and protocols.

5. Conclusion

Although burnout is not a new syndrome, having been around for more than 40 years now, it is true that the number of cases has increased alarmingly in recent years, to the point that it is now considered to be the “second” or “silent” pandemic. Many studies have been conducted from the point of view of burnout sufferers. However, as psychiatrist Christina Maslach pointed out, we should not overlook the fact that, in the case of burnout syndrome, the organization rather than the individual is at fault; that is, burnout is the responsibility of the employer. To help companies reduce this syndrome, we carried out a study using the Delphi method, in which experts who have extensive knowledge and experience of burnout syndrome participated. The goal of the study was to reach a consensus on the factors that cause burnout and the mechanisms that should be used at companies to mitigate the syndrome among their employees. While there are other works with the same goal, this research is original in that we have used a method where the participants, recognized, knowledgeable, and experienced experts in burnout syndrome, have, across several iterations or rounds, reached a consensus on their responses. They have had the opportunity to reflect on both their own opinions and the viewpoints of the other experts and, if appropriate, modify their own responses in the next round. The traditional Delphi method has several limitations. For instance, it is time-consuming and has shortcomings with respect to the definition of questionnaires and statistics. To overcome these shortcomings, we defined the agile Delphi methodology (MAD), which combines an agile methodology with the traditional Delphi method. To do this, we added some steps and modified others. Additionally, we defined mechanisms or recommended techniques to make the method steps more agile and efficient.

The limitation of this study is that all the participating experts were Spanish. Although these experts are well-acquainted with burnout syndrome in the Spanish context, we cannot be certain that, being a psychosocial risk, it will affect workers equally regardless of the country in which they perform their work activity. To be sure that the results are independent of the country in which the study is carried out, it would have to be replicated in different countries or with experts of different nationalities. In this case, the results would be more generalizable globally. The findings of this study were validated in two ways: one validation compared the agile Delphi methodology (MAD) and the traditional Delphi methodology to check if MAD is more agile and efficient. As the results reported in the “Case study” section show, we can state that the agile Delphi methodology (MAD) is more agile and efficient than the traditional Delphi methodology. The time taken is reduced by increasing the quality of the products, that is, the questionnaires, in fewer iterations. For the other validation, OHS experts identified and discussed the factors causing an increase in burnout syndrome and the mechanisms to mitigate the effects of the identified factors. In this case, experts reached a consensus on the factors that most affect burnout, which are inappropriate organizational culture, poorly structured organizational policies, poorly defined organizational processes, and poorly implemented digital transformation. To help mitigate these factors, experts identified ICT mechanisms, such as creating communication channels, adapting technology to jobs, providing continuous training, online help systems, and digital disconnection tools, and non-ICT mechanisms, such as adopting work autonomy, as well as flexibility and conciliation protocols, and developing psychosocial and physical health strategies.

The next step in this research would be to implement the identified mechanisms at organizations and evaluate whether, as the experts claim, the burnout syndrome rate decreases.

Data availability statement

Ethics statement.

Ethical review and approval was not required for the study on human participants in accordance with the local legislation and institutional requirements. Written informed consent from the participants was not required to participate in this study in accordance with the national legislation and the institutional requirements.

Author contributions

FM-D has designed the work presented in this paper and guided its development. FM-D, M-IS-S, G-LD-P, AA-S, and SV have participated in the reviewing process, analysis, synthesis of the case study summarized in this paper, and writing process. All authors contributed to the article and approved the submitted version.

Funding Statement

This research was funded by the Instituto Regional de Seguridad y Salud en el Trabajo de la Comunidad de Madrid (IRSST), through the IRSST-UC3M Chair: R&D for Intelligent Digital Transformation of Occupational Health and Safety, with the participation of the Knowledge and Culture of Prevention Area at IRSST.

Abbreviations

BS, burnout syndrome; MAD, agile Delphi methodology; PDCA, Plan-Do-Check-Act methodology; SMART (ER), specific, measurable, achievable, relevant, time-bound, engaging, and rewarding technique; ICD-11, International Statistical Classification of Diseases and Related Health Problems; ICT, information and communication technologies; IRSST, Instituto Regional de Seguridad y Salud en el Trabajo (Madrid Regional Health and Safety at Work Institute); OHS, Occupational Health and Safety; NA, I don't know; R&D, research and development; WHO, World Health Organization.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

• Research article
• Open access
• Published: 18 August 2021

Applying real-time Delphi methods: development of a pain management survey in emergency nursing

• Wayne Varndell 1 , 2 , 3 ,
• Margaret Fry 4 , 5 &
• Doug Elliott 4  

BMC Nursing volume  20 , Article number:  149 ( 2021 ) Cite this article

7124 Accesses

13 Citations

8 Altmetric

Metrics details

The modified Delphi technique is widely used to develop consensus on group opinion within health services research. However, digital platforms are offering researchers the capacity to undertake a real-time Delphi, which provides novel opportunities to enhance the process. The aim of this case study is to discuss and reflect on the use of a real-time Delphi method for researchers in emergency nursing and cognate areas of practice. A real-time Delphi method was used to develop a national survey examining knowledge, perceptions and factors influencing pain assessment and management practices among Australian emergency nurses. While designing and completing this real-time Delphi study, a number of areas, emerged that demanded careful consideration and provide guidance to future researchers.

Peer Review reports

The Delphi technique is an established and effective research method with multifaceted applications for health services research. The Delphi technique is uniquely designed to explore health issues and topics where minimal information or agreement currently exists, a relatively common situation within nursing practice. Secondly, the Delphi technique allows for the introduction and integration of viewpoints, opinions, and insights from a wide array of expert stakeholders. With increasing accessibility to the Internet and proliferation of smart device technology, changes from paper-based surveys to the development of online software systems, such as the real-time Delphi method, has significantly extended the potential research for the research population and sample, and efficiency of data collection and analysis. However, a recent systematic review highlighted a gap between available methodological guidance and publishing primary research in conducting real-time Delphi studies [ 1 , 2 ].

In this paper, we seek to examine the methodological gap in applying real-time Delphi methods, by providing a specific case example from a real-time Delphi study conducted to develop a self-reporting survey tool to explore pain management practices of Australian emergency nursing in critically ill adult patients [ 3 ]. Insight into the procedural challenges and enablers encountered in conducting a real-time Delphi study are provided. Importantly, key characteristics of the method are presented, followed by the case-based exemplars to illustrate important methodological considerations. Reflections from the case are then presented, along with recommendations for future researchers considering the use of a real-time Delphi technique approach.

Overview of the Delphi Technique

The Delphi technique was developed in the late 1950s’ by the Research and Development (RAND) Corporation [ 4 ] as a method for enabling a group of individuals to collectively address a complex problem, through a structured group communication process without bringing participants together physically [ 5 ]. Delphi has value in the healthcare sector, as it is characterised by multi-disciplinary teams and hierarchical structures [ 6 ]. The Delphi technique has since become popular with nursing researchers exploring a wide range of topics including role delineation [ 7 , 8 , 9 ], priorities for nursing research [ 10 , 11 , 12 ], standards of practice [ 13 , 14 ] and instrument development [ 15 , 16 ].

The four main characteristics of the classic Delphi method are anonymity, iteration, controlled feedback and statistical aggregation of group responses [ 17 ]. Data collection within the classic Delphi typically includes at least two [ 18 ] or three [ 19 ] rounds of questionnaires facilitated by a moderator. Round one represents what Ziglio [ 20 ] termed the ‘exploration phase’, in which the topic is fully explored using broad open-ended questions. Each following round then becomes part of an ‘evaluation phase’, where results of the previous round, interspersed with controlled feedback from a moderator, are used to frame another set of questions. Each round provides an opportunity for expert panel members to respond to and revise their answer in view of the previous responses from other panel members [ 21 ]. Since its introduction, over 20 variations of the classic Delphi method have evolved, with researchers modifying the approach to suit their needs. Most common Delphi versions include modified, decision, policy, internet, and more recently real-time Delphi, and have empaneled varying numbers of experts ranging from 6 to 1,142 [ 22 , 23 ] (Table  1 ).

The ubiquitous and interactive capacity of the Internet and smart device technology offers benefits that are intimately linked with contemporary research innovations in healthcare [ 24 ]. Two clear limitations of the classic Delphi technique were prolonged study durations and high panel member attrition [ 25 ]. Aiming to overcome these issues, Gordon and Pease [ 26 ] developed the concept of an information technology-enabled contemporaneous extension called real-time Delphi, to improve speed of the data collection process and syntheses of opinions. Conducting a real-time Delphi relies on specially designed software to administer the survey; the functionality or capabilities of which can negatively impact on the success of a study. Initial thoughts of using technology to facilitate the Delphi process emerged as early as 1975 [ 27 ]. The first specifically designed real-time Delphi software was developed in 1998 called Professional Delphi Scan [ 28 ], with the first real-time Delphi surveys performed and published in the early 2000 s [ 29 ]. Since then, several real-time software-based tools have been developed, often by researchers for the purposes of their study [ 30 , 31 , 32 ]. However, these have not been evaluated in detail in the literature.

In a real-time Delphi process, participants are provided with access to an online questionnaire portal for a specific amount of time. On accessing the portal, expert panel members see all their responses to items and the ongoing, hence real-time, anonymised responses from other panel members. The core innovation of real-time Delphi studies is the simultaneous calculation and feedback. Unlike the classic method, in a real-time Delphi participants do not judge at discrete intervals (i.e. rounds), but can change their opinion as often as they like within the set timeframe [ 33 ] (Fig.  1 ).

Real-time Delphi processes

A real-time Delphi case exemplar

A real-time Delphi study was conducted to develop a context specific instrument (i.e. survey) to investigate emergency nurses’ practices in managing acute pain in critically ill adult patients. The following steps were followed in designing and conducting our real-time Delphi study: study design, pilot testing, recruiting experts, retention, data analysis and reporting. Findings from this study are reported elsewhere [ 34 ]. The real-time Delphi method was selected to: maximise participation from expert panel members geographically separated, minimise the amount of time demanded of experts, enable equal flow of information to and from all members, real-time presentation of results to enable experts to reassess and adjust their opinion, and allow panel members a greater degree of expression [ 35 , 36 ]. Prior to commencing the study, a comprehensive literature review was conducted by the research team to generate initial survey items, and used the following questions:

What indicators would signify that acute pain in the critically ill adult patient has or has not been adequately detected?

What indicators would imply that acute pain in the critically ill adult patient has or has not been adequately managed?

What indicators would suggest that acute pain in the critically ill adult patient has or has not been communicated adequately?

A total of 74 items were initially generated from the literature, and organised into six domains: clinical environment, clinical governance, practice, knowledge, beliefs and values, and perception. Next, commercially available real-time Delphi survey systems were evaluated for their suitability. This process was guided by reviewing the literature [ 33 ], trialing available platforms and examining fee structures. Following this review, Surveylet (Calibrum Inc., Utah) was selected [ 37 ]. Survey items were then uploaded into Surveylet software system. Pilot testing was then conducted by the research team to evaluate software settings, automation, flow and ease of navigation. Average time to complete the survey was 38 min (SD 8 min).

An expert panel size of 12 to 15 was selected. Identification and selection of experts occurred in three stages: defining the relevant expertise, identifying individuals with desired knowledge and experience, and retaining panel members. First, a pro forma listing the type of skills, experience, qualifications, relevant professional memberships and academic outputs (e.g. peer-reviewed publications) as traits of a desired expert panel member. Second, the research team added potential experts to the list: names of academics were identified via a review of the pertinent literature, with emergency nursing clinicians identified from contacting the College of Emergency Nursing Australasia. Third, initial contacts were approached and provided with a brief overview of the study; pertinent biographical information was then obtained. In addition, they were invited to nominate other experts to be approached for inclusion. Contacts were then independently ranked, with the top 15 experts invited to participate. Twelve accepted the invitation to participate: eight emergency nurses, most nurse consultants (n = 6), two pain management nurse consultants, and two emergency nursing academics from across Australia with an average of 18 years clinical experience. All experts held postgraduate qualifications and half had published in emergency nursing practice and/or pain management.

In Delphi studies, an a priori level of consensus and stability sought for the items the experts will rate is set by the research team. In this study, consensus was achieved if ≥ 83 % (10 out of 12 panel members) of experts ranked the item ≥ 7 on the 9-point Likert scale. Secondary measures of consensus among experts included stability of response, evaluated using coefficient of quartile variation (< 5 %) and interclass correlations (≥ 0.75) [ 38 , 39 ]. Items were retained if primary and secondary measures were met. Data were analysed using median, range and interquartile range. Descriptive statistics were then developed in tabular form and scatterplots.

The Delphi panel members were introduced to each survey domain as they navigated through the real-time Delphi software system, including descriptions of the domain and response format to rate each item. For ease of navigation, one item was presented per page, which included a real-time statistical summary and anonymised remarks from other experts (Fig.  2 ) [ 37 ].

Example of review screen

Experts were asked to rate the importance of each question using a 9-point Likert scale (1, extremely unimportant to 9, extremely important), and whether the question could be modified to improve its relevance (Yes/No). If modifications were suggested, respondents were able to provide an example of how the proposed question could be revised, which could then be subsequently voted on by the expert panel.

Participation was asynchronous with experts able to independently re-visit the real-time Delphi survey portal and modify their responses at any point in time between 1st February and March 14th 2019 (a total of 35 days). On accessing the survey portal, panel members can engage in the consensus process from the outset by viewing other panel members’ have responded. Panel members could view not only their own quantitative responses but also the median, range and interquartile range of all given quantitative responses. In the same way, panel members could also view all qualitative arguments submitted by panel members including their own. Panel members could then review or change any or all their responses, or add new arguments, up until the survey closed. Prior to launching the Delphi study, the research team piloted accessing the survey portal, data collection and analysis methods.

All panel members participated in the survey, providing on average four responses per survey item. Further, of the 74 items initially proposed, 58 (78.4 %) reached consensus in the first week of the study commencing. Following feedback from the expert panel, of the initial 74 items proposed, 12 (16.2 %) were modified to improve clarity, and a further 17 items were added by the expert panel to improve survey depth. At the conclusion of the real-time Delphi, the final survey contained 91 items.

While completing the real-time Delphi, several areas were identified as needing consideration when using this technique: software selection, rating scale, piloting, recruiting experts, consensus and stability, retention and reporting. Key issues are discussed in the following section.

Software and survey design

This case study has highlighted that to conduct a real-time Delphi requires specialised software. A recent review [ 33 ] independently evaluated the characteristics of four commercially available real-time Delphi software solutions (Risk Assessment and Horizon Scanning, eDelfoi, Global Futures Intelligence System and Surveylet) for their range of features and available question formats; data analytics; user friendliness; and, intuitive system operation (Table  2 ). Surveylet (Calibrum Inc., Utah) [ 37 ] was rated the highest for its flexibility, breadth of inbuilt data management options, anonymity of participants and security. While the Surveylet system can amply conduct a real-time Delphi, the system is sophisticated and requires additional time and guidance to configure correctly. Training is provided by way of video tutorials, and support options are available to assist in survey setup at an additional cost. We recommend that prior to conducting a real-time Delphi, researchers comprehensively review, and where possible, trial available software solutions.

Advantages of conducting an online Delphi study include: reduced data entry errors due to automated entry, fewer instances of panelists missing questions resulting in incomplete data, length of time decreases for data collection, and automated aggregation of results and feedback to panelists [ 40 ]. The principle difference between a conventional online Delphi and real-time Delphi software systems is the immediate calculation and provision of group responses, which can assist in generating time-sensitive guidance. While there are advantages (Table  3 ), there are also challenges, which are principally associated with software complexity [ 35 ] and cost [ 41 ].

Internet accessibility, system navigation difficulties and the inconvenience of entering data into a computer-based data screen are recognised as challenges [ 44 ]. While the internet is a tool for extending the potential research population and sample, navigating an unfamiliar virtual landscape may frustrate panel members and therefore limit the number of completed surveys [ 45 ]. To minimise potential software complexity issues in our study, panel members were sent detailed written instructions on how to access and navigate the real-time Delphi software system, and could attend a one-to-one videoconference with a member of the research team to assist in using the platform [ 17 , 46 ]. Cost-efficiency is often stated as a key benefit of using online survey tools to conduct an electronic survey [ 41 ]. However, in our review of commercially available real-time Delphi software systems, we found that it can become expensive with system providers potentially charging per survey, the number of system administrators or participants enrolled, the duration of the survey, and/or system support to aid survey customisation. While further evidence is needed to substantiate the claim concerning the efficiency of the real-time Delphi method compared to multi-round Delphi designs, current multi-round Delphi studies investigating topics relating to emergency nursing practice, have taken 60 [ 43 ] to 273 [ 13 ] days to complete.

Rating scale

Currently, there is no agreement about what rating scale size should be used in Delphi studies; despite being a common reason cited for study failure [ 47 , 48 , 49 ]. Rating scales used in previous Delphi studies exploring aspects of emergency nursing practice have ranged from 4 to 11 [ 1 ]. While 5 and 7-point scales are the most common forms of Likert scales used in surveys [ 50 , 51 ], 9-point Likert scales are frequently used in Delphi studies, particularly during the consensus process [ 47 , 49 , 52 ]. A wide range of Likert rating scale sizes can be set within Surveylet. In addition, to underline their rating, experts can also detail their reasoning behind their selection, which can be viewed by other panel members. According to Best [ 53 ], accuracy can be improved if experts are provided with both quantitative and qualitative arguments. In a real-time Delphi experts are able to immediately react on each other’s responses, increasing the degree of information experts can interact with, which may aid in recapturing their own point of view [ 26 ].

Despite the administrative complexity of conducting any Delphi method, there is limited discussion on pilot testing in the literature. Pilot testing can be conducted to test and adjust the Delphi survey to improve comprehension [ 54 ]. When using online software, such as when conducting a real-time Delphi, to conduct and collect multiple responses, the potential impact on cost, time, participant motivation and data integrity should an error occur, could jeopardise the overall study. Pilot testing is therefore vital to identify potential technical or system configuration errors, data collection irregularities (i.e. logic settings) and strengthen participant orientation, prior to commencing the study [ 55 ]. Prior to initiating the real-time Delphi, we first verified system configuration and all settings (e.g. timeframe, communication templates), panel member contact details, and that survey items were uploaded correctly. Second, members of the research team independently piloted the survey as mock participants and system administrators, to evaluate the survey flow and ease of navigation. Average time to complete the survey was 38 min (SD 8 min).

Recruiting experts

The formulation of an expert panel and its makeup is of critical importance for all Delphi studies, yet raises methodological concerns that can negatively impact on the quality of the results [ 36 , 56 , 57 ]. Despite criticism in the literature about Delphi as a methodological approach [ 2 , 17 , 35 , 36 , 40 , 58 ], there remains little agreement as to what defines an expert [ 36 ]. Keeney et al. [ 59 ] in their review identified several definitions of ‘expert’ ranging from someone who has knowledge about a specific topic, recognised as a specialist in the field, to an informed individual. A recent systematic review of the Delphi method in emergency nursing [ 1 ] found similar emphasis in the criteria commonly used to identify experts: length of clinical experience, professional role (e.g. educator, clinical nurse consultant), professional college membership, peer-reviewed publications and postgraduate qualifications. From the current literature, it suggests that defining who is an expert may not be about the role they occupy, but what attributes they possess: knowledge and experience [ 36 , 59 , 60 , 61 ].

Recruiting experts in our study required: defining the relevant expertise, identifying individuals with desired knowledge and experience, and retaining panel members. Melynk et al. [ 62 ] suggests that a minimum threshold for participation as an expert on a Delphi panel should include those measurable characteristics that each participant group would acknowledge as those defining expertise, appropriate to the context, scope and aims of the particular study. While selection of panel experts in Delphi studies typically involves non-probability sampling techniques, which potentially reduces representativeness [ 17 , 57 ], the aim of our study was to recruit academic and emergency nurses with knowledge and clinical experience in the phenomena being explored – pain management practices for adult critically ill patients [ 55 ]. To achieve this, the procedure detailed by Delbecq et al. [ 63 ] was followed.

Expert panel size

Presently there is no agreement in the literature concerning expert panel size [ 2 ]. A recent review of 22 Delphi studies within emergency nursing reported a wide range of panel sizes - from fewer than 12 up to 315. Duffield [ 64 ] suggests that when a Delphi panel is homogenous 10 to 15 people are adequate. In a similar Delphi study seeking to develop a self-completed survey to examine triage practice, 12 experts were recruited [ 16 ]. As noted earlier, the target panel size in our study was 12 to 15, however, as Hartman and Baldwin [ 65 ] highlight, due the higher degree of automation of real-time Delphi software systems, typically web-based, the number of experts over a large geographic area participating in a real-time study can be increased.

Keeping participants fully engaged once recruited is challenging [ 40 , 57 ]. High attrition rates can negatively impact on the clarity and validity of results (i.e. item consensus and selection) [ 56 ]. Conducting a classic multi-round Delphi study can be a slower process with respect to receiving and analysing feedback, generating the next survey round and determining consensus, and potentially increases the risk of attrition [ 66 ]. A potential benefit of the real-time Delphi is its expediency [ 67 ]. The much shorter timeframe between panel members submitting their response and getting insights into others’ responses, encourages stronger cognitive examination with the respective issue in question; maximising the validity of results [ 65 ]. Presently there is no formal guidance within the literature as to what constitutes an appropriate timeframe with regards to the real-time Delphi method. However, consideration should be given to the overall consensus process timeframe, to ensure panel members have sufficient time to explore opinions to minimise the potential risk of acquiescence bias. To detect potential acquiescence bias, dispersion measures such as range and coefficient of quartile variation were used.

As noted by Zipfinger [ 42 ], asynchronous participation can also aid in retaining panel members. Panel members are able to access the Delphi portal at any time, 24-hours a day within the set timeframe, making it more convenient to participate and review feedback. Further, panelists can contribute to whatever aspects in the survey they want, especially when having gone through each question at least once [ 68 ].

To maintain panel member engagement, we employed a variety of methods, beginning with participant information sheets. Information sheets were designed based on recommendations from the literature [ 58 , 69 ], to ensure straightforward messaging on the importance and appeal of the study, aims, processes, timeframe, and benefits, all in clearly marked subsections. To further encourage potential experts who may have had little experience in participating in a real-time Delphi study, we detailed how participants would be introduced to the study, the Delphi methodology, availability of one-on-one training sessions in the use of real-time Delphi software system, and access to technical support. Once the study commenced, the real-time Delphi software sent personalised reminder emails at weekly intervals to encourage participants to (re)assess items in a timely fashion, and provided a summary of responses received to date. These emails emphasised that their views mattered and that for the results to be meaningful, it was important to complete the Delphi process. Sending reminder emails once the Delphi has commenced, can potentially increase retention and response activity of experts [ 58 , 70 ]. However, while a recent study examining the experiences of Delphi participants concluded that receiving reminders to participate where not viewed negatively, it did not explore frequency [ 71 ]. At the completion of the real-time Delphi, panel members were sent a certificate thanking them for their commitment to the study [ 58 ], and to provide evidence for their professional development records [ 72 ]. Within our study, the level of response activity appears to suggest retention and engagement strategies were effective.

Consensus and stability

Quantifying the degree of consensus among experts is an important element of Delphi data analysis and interpretation, however reaching a pre-calculated threshold value (e.g. greater than 80 %) of consensus is not the general aim, and rarely is it 100 % [ 73 , 74 ]. Consensus can either be used to determine if agreement exists or as a stopping guideline, and is measured at the conclusion of a preset number of rounds [ 75 ]. Further, as previous studies have demonstrated [ 49 , 76 ], results can be greatly impacted by the level of consensus set and rating scale used [ 48 , 77 ]. Within the emergency nursing literature, consensus thresholds have ranged from 50 % [ 78 ] to 90 % [ 79 ]. Our study used the most common consensus level from previous Delphi studies that had identified survey items as being essential when rated by at least 80 % of the experts [ 1 ].

Stability of consensus is also important, which is best evaluated using measures of dispersion [ 54 , 67 ]. Assessing stability can occur between consecutive rounds, such as in the classic Delphi, or at the conclusion end of the consensus process. While the use of mean, standard deviation and parametric statistics to describe ordinal data is not strictly incorrect when the data is not irregular [ 80 , 81 ], the use of median, range and interquartile range based on Likert-type scales is favoured as they are more robust to being sensitive to outliers [ 57 ]. In classic Delphi, stability is judged between rounds. In real-time Delphi, stability of response is evaluated at the end of the study. In our study, a coefficient of quartile variation (CQV) value less than 5 % was set a priori [ 82 ], and configured in Surveylet as a measure of relative dispersion based on interquartile range. It is also a measure of homogeneity (i.e. internal consistency) appropriate for small sample (i.e. panel) sizes of 15 or less [ 83 ], expressed as:

In addition, interclass correlations were calculated to inferentially determine stability (≥ 0.75) of responses [ 38 , 39 ]. Descriptive statistics were then developed in tabular form and scatterplots. Survey items that met the above consensus and stability criteria were incorporated into the final survey (Table  4 ).

Regardless of what Delphi study design and approach is adopted, attention to rigour of reporting throughout the process is a vital aspect of research. Trustworthiness of the Delphi technique has been debated in the general and nursing literature. Keeney et al. [ 36 ] and Powell [ 84 ] suggest that the Delphi technique should not be judged by psychometric criteria used for more positivist approaches, with several criteria proposed to evaluate trustworthiness of qualitative studies [ 55 , 85 , 86 , 87 , 88 ]. A common purpose among criteria is to support trustworthiness by reporting the process of study design and data analysis accurately.

In our study, we elected to apply the criteria proposed by Lincoln and Guba [ 86 ], based on four concepts; credibility, transferability, dependability and confirmability. Our real time Delphi was based on consensus amongst experienced individuals familiar with the phenomena being explored, across emergency nursing, pain management and academia (credibility and confirmability). Decisions on development of survey questions was arrived at through a documented and auditable; a processes supported by the Surveylet software system (credibility and dependability). The anonymous and continuous process of real-time Delphi research fostered honesty and verification of panelist responses, as panelists could provide feedback and ‘member-checking’ without fear of reprisal from their colleagues (credibility) [ 36 ]. Prior to initiating the real-time Delphi process, we piloted the survey for its structure, flow, ease of navigation and robustness (transferability).

Conclusions

Many papers describe the use of the classic Delphi approach in health services research, yet few provide practical advice on the type and process for undertaken such a research design using the real-time Delphi method. This article presented a case exemplar of a real-time Delphi study and the development of a survey to explore emergency nursing practice. The real-time Delphi method can be of great use in a wide range of time-sensitive health research issues where divergent opinion or little agreement exists. Our experiences have highlighted important strengths and challenges in its deployment, including several methodological issues which may provide guidance to other researchers.

Availability of data and materials

Not applicable.

Abbreviations

Coefficient of Quartile Variation.

Research and Development Corporation.

Standard Deviation.

Varndell W, et al. Use of the Delphi method to generate guidance in emergency nursing practice: A systematic review.International Emergency Nursing, Volume 56, 2021: p. 100867.

McPherson S, Reese C, Wendler MC. Methodology Update: Delphi Studies. Nurs Res. 2018;67(5):404–10.

Article   PubMed   Google Scholar  

Varndell, W, Fry, M, Elliott, D. Pain assessment and interventions by nurses in the emergency department: A national survey.J Clin Nurs. 2020;29:2352–62. https://doi.org/10.1111/jocn.15247 .

Dalkey N, Helmer O. An experimental application of the Delphi method to the use of experts. Manage Sci. 1963;9(3):458–67.

Article   Google Scholar  

Gordon T, Helmer O. Report on a long-range forecasting study. 1964 May 20th 2020]; Available from: https://www.rand.org/content/dam/rand/pubs/papers/2005/P2982.pdf .

Beech R. Go the extra mile — use the Delphi Technique. J Nurs Manag. 1999;7(5):281–8.

Article   CAS   PubMed   Google Scholar  

Roberts-Davis M, Read S. Clinical role clarification: using the Delphi method to establish similarities and differences between nurse practitioners and clinical nurse specialists. J Clin Nurs. 2001;10(1):33–43.

Duffield C. The Delphi technique. Aust J Adv Nurs. 1988;6(2):41–5.

CAS   PubMed   Google Scholar  

White K, Wilkes L. Describing the role of the breast nurse in Australia. European Journal of Oncology Nursing. 1998;2(2):89–98.

Annells M, et al. A Delphi study of district nursing research priorities in Australia. Appl Nurs Res. 2005;18(1):36–43.

Bayley EW, et al. ENA’s Delphi study on national research priorities for emergency nurses in the United States. J Emerg Nurs. 2004;30(1):12–21.

Considine J, et al. Consensus-based clinical research priorities for emergency nursing in Australia. Australasian Emergency Care. 2018;21(2):43–50.

ENA NP Validation Work Team. et al., Nurse Practitioner Delphi Study: competencies for practice in emergency care. J Emerg Nurs. 2010;36(5):439–49.

Okuwa M, et al. Measuring the pressure applied to the skin surrounding pressure ulcers while patients are nursed in the 30 degree position. J Tissue Viability. 2005;15(1):3–8.

Wilkes L, et al. Development of a violence tool in the emergency hospital setting. Nurse Researcher. 2010;17(4):70–82.

Fry M, Burr G. Using the Delphi technique to design a self-reporting triage survey tool. Accid Emerg Nurs. 2001;9(4):235–41.

Rowe G, Wright G. The Delphi technique: Past, present, and future prospects — Introduction to the special issue. Technol Forecast Soc Chang. 2011;78(9):1487–90.

Miranda FBG, Mazzo A. and G. Alves Pereira-Junior, Construction and validation of competency frameworks for thetraining of nurses in emergencies. 26: Revista Latino-Americana de Enfermagem; 2018.

Murphy JP, et al. Emergency department registered nurses’ disaster medicine competencies. An exploratory study utilizing a modified Delphi technique. Int Emerg Nurs. 2019;43:84–91.

Ziglio E. The Delphi Method and its contribution to decision-making, in Gazing into the Oracle: The Delphi Method and itsApplication to Social Policy and Public Health., M. Adler and E. Ziglio, Editors. 1996, Jessica Kingsley Publisher: Bristol, PA. p. 3–33.

Mullen PM. Delphi: myths and reality. J Health Organ Manag. 2003;17(1):37–52.

Strasser S, London L, Kortenbout E. Developing a competence framework and evaluation tool for primary care nursing in South Africa. Educ Health (Abingdon). 2005;18(2):133–44.

Barnette JJ. Delphi methodology: An empirical investigation. Educational Research Quarterly; 1978.

Cole ZD, Donohoe HM, Stellefson ML. Internet-based Delphi research: case based discussion. Environmental management. 2013;51(3):511–23.

Article   PubMed   PubMed Central   Google Scholar  

Bardecki MJ. Participants’ response to the Delphi method: An attitudinal perspective. Technol Forecast Soc Chang. 1984;25(3):281–92.

Gordon T, Pease A. RT Delphi: An efficient, “round-less” almost real time Delphi method. Technol Forecast Soc Chang. 2006;73(4):321–33.

Linstone HA, Turoff M, editors. The Delphi method: techniques and applications / edited by Harold A. Linstone and MurrayTuroff ; with a foreword by Olaf Helmer. Advanced Book Program: Reading, Mass: Addison-Wesley Pub. Co.; 1975.

Kuusi O, Hiltunen E. Signification Process of Future Sign. Editor: Finland Futures Research Centre - Turku School ofEconomics; 2007. Turku.

Glenn J, Gordon T, editors. Futures Research Methodology. Version 3.0 ed. The Millennium Project; 2009.

Gary JE, von der Gracht HA. The future of foresight professionals: Results from a global Delphi study. Futures. 2015;71:132–45.

Keller J, von der Gracht HA. The influence of information and communication technology (ICT) on future foresight processes — Results from a Delphi survey. Technol Forecast Soc Chang. 2014;85:81–92.

Markmann C, Darkow I-L, von der Gracht H. A Delphi-based risk analysis — Identifying and assessing future challenges for supply chain security in a multi-stakeholder environment. Technol Forecast Soc Chang. 2013;80(9):1815–33.

Aengenheyster S, et al. Real-Time Delphi in practice — A comparative analysis of existing software-based tools. Technol Forecast Soc Chang. 2017;118:15–27.

Varndell W, Fry M, Elliott D. Pain Assessment and Interventions by Nurses in the Emergency Department: A National Survey. J Clin Nurs. 2020;29(13–14):2352–62.

Avaella J. Delphi panels: research design, procedures, advantages and challenges. International Journal of Doctoral Studies. 2016;11:305–21.

Keeney S, Hasson F, McKenna H. The Delphi technique in nursing and health research. Ames: IO: Wiley-Blackwell; 2011.

Calibrum. 2020: St George, UT.

Trevelyan E, Robinson N. Delphi methodology in health research: how to do it? European Journal of Integrative Medicine. 2015;7(4):423–8.

Koo T, Li M. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. Journal ofChiropractic Medicine, 2016. 15.

Khodyakov D, et al., Practical Considerations in Using Online Modified-Delphi Approaches to Engage Patients and OtherStakeholders in Clinical Practice Guideline Development. The Patient - Patient-Centered Outcomes Research. 2020;13(1):11–21.

Wright KB. Researching Internet-Based Populations: Advantages and Disadvantages of Online Survey Research, Online Questionnaire Authoring Software Packages, and Web Survey Services. Journal of Computer-Mediated Communication. 2005;10(3):00–0.

Article   CAS   Google Scholar  

Zipfinger S. Computer-Aided Delphi: An Experimental Study of Comparing Round-Based with Real-Time Implementation of the Method. Linz: Trauner Verlag; 2007.

Google Scholar  

de Lemos J, Tweeddale M, Chittock D. Measuring quality of sedation in adult mechanical ventilated critically ill patients: the Vancouver Interaction and Calmness Scale. J Clin Epidemiol. 2000;53:908–19.

Donohoe HM, Needham RD. Moving best practice forward: Delphi characteristics, advantages, potential problems, and solutions. International Journal of Tourism Research. 2009;11(5):415–37.

Hall DA, et al. Recruiting and retaining participants in e-Delphi surveys for core outcome set development: Evaluating the COMiT’ID study. PloS one. 2018;13(7):e0201378–8.

Article   PubMed   PubMed Central   CAS   Google Scholar  

Donohoe H, Stellefson M, Tennant B. Advantages and limitations of the e-Delphi technique: implications for health education researchers. American Journal of Health Education. 2012;43:38–46.

Diamond IR, et al. Defining consensus: A systematic review recommends methodologic criteria for reporting of Delphi studies. J Clin Epidemiol. 2014;67(4):401–9.

Lange T, et al. Comparison of different rating scales for the use in Delphi studies: different scales lead to different consensus and show different test-retest reliability. BMC Med Res Methodol. 2020;20(1):28.

De Meyer D, et al. Delphi procedure in core outcome set development: rating scale and consensus criteria determined outcome selection. J Clin Epidemiol. 2019;111:23–31.

Weijters B, Cabooter E, Schillewaert N. The effect of rating scale format on response styles: The number of response categories and response category labels. Int J Res Mark. 2010;27(3):236–47.

Revilla MA, Saris WE, Krosnick JA. Choosing the Number of Categories in Agree–Disagree Scales. Sociological Methods &Research. 2014;43(1):73-97. https://doi.org/10.1177/0049124113509605 .

Williamson PR, et al. The COMET Handbook: version 1.0. Trials. 2017;18(Suppl 3):280.

Roger JB. An Experiment in Delphi Estimation in Marketing Decision Making. J Mark Res. 1974;11(4):448–52.

Clibbens N, Walters S, Baird W. Delphi research: Issues raised by a pilot study. Nurse Res. 2012;19:37–44.

Polit D, Beck C. Essentials of nursing research: appraising evidence for nursing practice. 9th ed. Philadelphia: Lippincott WIlliams & Wilkins; 2017.

Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs. 2000;32(4):1008–15.

Hsu C-C, Sandford B. The Delphi Technique: Making Sense of Consensus. Practical Assessment Research Evaluation. 2007;12(10):1–8.

Hall DA, et al. Recruiting and retaining participants in e-Delphi surveys for core outcome set development: Evaluating the COMiT’ID study. PLOS ONE. 2018;13(7):e0201378.

Keeney S, Hasson F, McKenna HP. A critical review of the Delphi technique as a research methodology for nursing. Int J Nurs Stud. 2001;38(2):195–200.

Cantrill J, Sibbald B, Buetow S. The Delphi and nominal group techniques in health services research. Int J Pharm Pract. 1996;4(2):67–74.

Kennedy HP. Enhancing Delphi research: methods and results. J Adv Nurs. 2004;45(5):504–11.

Melnyk SA, et al. Mapping the future of supply chain management: a Delphi study. Int J Prod Res. 2009;47(16):4629–53.

Delbecq AL, Ven AHVd, Glenview DHG. Group Technique for Program Planning: A Guide to Nominal Group and DelphiProcesses. Group & Organization Studies. 1976;1(2):256–6.

Duffield C. The Delphi Technique. Australian Journal of Advanced Nursing. 1989;6(2):41–5.

Hartman FT, Baldwin A. Using Technology to Improve Delphi Method. Journal of Computing in Civil Engineering. 1995;9(4):244–9.

Schmalz U, Spinler S, Ringbeck J. Lessons Learned from a Two-Round Delphi-based Scenario Study. MethodsX. 2021;8:101179.

Gnatzy T, et al. Validating an innovative real-time Delphi approach - A methodological comparison between real-time and conventional Delphi studies. Technol Forecast Soc Chang. 2011;78(9):1681–94.

Turoff M, Hiltz S. Computer based Delphi processes, Gazing into the Oracle: The Delphi Method and its Application toSocial Policy and Public Health, M. Adler and E. Ziglio, Editors. 1995, Jessica Kingsley Publishers: London. pp. 56–88.

Ennis L, Wykes T. Sense and readability: participant information sheets for research studies. The British journal of psychiatry: the journal of mental science. 2016;208(2):189–94.

Edwards PJ, Roberts I, Clarke MJ, DiGuiseppi C, Wentz R, Kwan I, Cooper R, Felix LM, Pratap S. Methods to increaseresponse to postal and electronic questionnaires. Cochrane Database of Systematic Reviews. 2009;Issue 3:MR000008.

Turnbull AE, et al. A survey of Delphi panelists after core outcome set development revealed positive feedback and methods to facilitate panel member participation. J Clin Epidemiol. 2018;102:99–106.

Nursing and Midwifery Board of Australia. Guidelines for continuing professional development. NMBA: Melbourne,Victoria; 2016.

Linstone HA, Turoff M. Delphi: A brief look backward and forward. Technol Forecast Soc Chang. 2011;78(9):1712–9.

Warth J, von der Gracht HA, Darkow I-L. A dissent-based approach for multi-stakeholder scenario development — The future of electric drive vehicles. Technol Forecast Soc Chang. 2013;80(4):566–83.

Keeney S, Hasson F, McKenna H. Consulting the oracle: ten lessons from using the Delphi technique in nursing research. J Adv Nurs. 2006;53(2):205–12.

Naylor CD, et al. Placing patients in the queue for coronary revascularization: evidence for practice variations from an expert panel process. Am J Public Health. 1990;80(10):1246–52.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Linstone H, Turoff M. The Delphi Method: Techniques and Applications. Reading: MA: Addison-Wesley; 1975.

Lee FH, et al. Clinical competencies of emergency nurses toward violence against women: a delphi study. J Contin Educ Nurs. 2015;46(6):272–8.

Holanda FL, Marra CC, Cunha I. Assessment of professional competence of nurses in emergencies: created and validated instrument. Rev Bras Enferm. 2018;71(4):1865–74.

Argyrous G. Statistics for research with a guide to SPSS. 2nd ed. London: SAGE; 2005.

Winter JD, Dodou D. Five-Point Likert Items: t test versus Mann-Whitney-Wilcoxon. Practical Assessment,Research and Evaluation. 2010;15:1-16.

Machin D, Campbell MJ, Walters S. Medical statistics: a textbook for health sciences. 4th ed. West Sussex: Wiley; 2007.

Altunkaynak B, Gamgam H. Bootstrap confidence intervals for the coefficient of quartile variation. Communications in Statistics - Simulation Computation. 2019;48(7):2138–46.

Powell C. The Delphi technique: myths and realities. J Adv Nurs. 2003;41(4):376–82.

Emden C, Sandelowski M. The good, the bad and the relative, Part Two: Goodness and the criterion problem in qualitative research. Int J Nurs Pract. 1999;5(1):2–7.

Lincoln Y, Guba E. Naturalistic inquiry. London: Sage; 1985.

Book   Google Scholar  

Neuendorf K. The Content Analysis Guidebook. 2nd ed. London: SAGE Publications; 2016.

Schreier M. Qualitative content analysis in practice. Thousand Oaks: SAGE Publications Inc; 2012.

Download references

Acknowledgements

This publication was not funded or commissioned.

Author information

Authors and affiliations.

Prince of Wales Hospital Emergency Department, NSW, 2031, Randwick, Australia

Wayne Varndell

Faculty of Health, University of Technology Sydney, NSW, 2007, Ultimo, Australia

College of Emergency Nursing Australasia, PO Box 7345, Victoria, 3193, Beaumaris, Australia

School of Nursing & Midwifery, Faculty of Health, University of Technology Sydney, 15 Broadway, NSW, 2007, Ultimo, Australia

Margaret Fry & Doug Elliott

Director Research and Practice Development Nursing and Midwifery Directorate NSLHD, Royal North Shore Hospital, Level 7 Kolling Building, NSW, 2065, St Leonards, Australia

Margaret Fry

You can also search for this author in PubMed   Google Scholar

Contributions

WV: Conceptualisation, Writing – original draft, Writing – review and editing. MF: Conceptualisation, Writing – review and editing. DE: Conceptualisation, Writing – review and editing. All authors read and approved the manuscript.

Corresponding author

Correspondence to Wayne Varndell .

Ethics declarations

Ethics approval and consent to participate.

This study was approved by the South Eastern Sydney Human Research and Ethics Committee (17/162). Before commencing the study, informed written consent forms were obtained from all participants.

Competing interests

All authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Varndell, W., Fry, M. & Elliott, D. Applying real-time Delphi methods: development of a pain management survey in emergency nursing. BMC Nurs 20 , 149 (2021). https://doi.org/10.1186/s12912-021-00661-9

Download citation

Received : 03 March 2021

Accepted : 07 July 2021

Published : 18 August 2021

DOI : https://doi.org/10.1186/s12912-021-00661-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Methodology
• Emergency Nursing
• Group Opinion

BMC Nursing

ISSN: 1472-6955

• General enquiries: [email protected]

• Research article
• Open access
• Published: 15 June 2020

Improving design choices in Delphi studies in medicine: the case of an exemplary physician multi-round panel study with 100% response

• Rebekka Veugelers   ORCID: orcid.org/0000-0002-3118-5613 1 ,
• Menno I. Gaakeer 1 ,
• Peter Patka 2 &
• Robbert Huijsman 3  

BMC Medical Research Methodology volume  20 , Article number:  156 ( 2020 ) Cite this article

9525 Accesses

60 Citations

3 Altmetric

Metrics details

A proper application of the Delphi technique is essential for obtaining valid research results. Medical researchers regularly use Delphi studies, but reports often lack detailed information on methodology and controlled feedback: in the medical literature, papers focusing on Delphi methodology issues are rare. Since the introduction of electronic surveys, details on response times remain scarce. We aim to bridge a number of gaps by providing a real world example covering methodological choices and response times in detail.

The objective of our e(lectronic)-Delphi study was to determine minimum standards for emergency departments (EDs) in the Netherlands. We opted for a two-part design with explicit decision rules. Part 1 focused on gathering and defining items; Part 2 addressed the main research question using an online survey tool. A two-person consensus rule was applied throughout: even after consensus on specific items was reached, panellists could reopen the discussion as long as at least two panellists argued similarly. Per round, the number of reminders sent and individual response times were noted. We also recorded the methodological considerations and evaluations made by the research team prior to as well as during the study.

The study was performed in eight rounds and an additional confirmation round. Response rates were 100% in all rounds, resulting in 100% consensus in Part 1 and 96% consensus in Part 2. Our decision rules proved to be stable and easily applicable. Items with negative advice required more rounds before consensus was reached. Response delays were mostly due to late starts, but once panellists started, they nearly always finished the questionnaire on the same day. Reminders often yielded rapid responses. Intra-individual differences in response time were large, but quick responders remained quick.

Conclusions

We advise those considering Delphi study to follow the CREDES guideline, consider a two-part design, invest in personal commitment of the panellists, set clear decision rules, use a consistent lay-out and send out your reminders early. Adopting this overall approach may assist researchers in future Delphi studies and may help to improve the quality of Delphi designs in terms of improved rigor and higher response rates.

Peer Review reports

Medical researchers commonly use the Delphi technique for consensus studies. A proper execution of this technique is essential for a study’s validity, but the present medical literature on the topic has so far remained rather vague. In this paper, we discuss several methodological issues and panel response characteristics based on our study amongst an expert panel of Emergency Physicians [ 1 ].

The Delphi technique is designed to obtain the most reliable consensus of opinion in a group of experts. It attempts to achieve this by means of a series of questionnaires interspersed with controlled feedback including group statistical responses [ 2 , 3 ]. Anonymity amongst panellists prevents the occurrence of individual dominance that may result from strong verbalization, track records or professional dominance. It also allows panel members to change their opinion on the basis of arguments presented by the other panel members without publicly admitting that they have done so. These advantages are assumed to increase reliability of consensus [ 4 ]. When an online survey tool is applied, the term e-Delphi (electronic) is used.

Since the early nineteen-fifties, the Delphi technique has been widely used in a large number of diverse domains such as the military, business, education, social science and health care [ 4 ]. It can be used for a wide range of complex research aims, including forecasting, issue identification, issue prioritization, ranking, policy formation and concept-framework development [ 5 , 6 ]. However, the method’s versatility is both a strength and a weakness. Practitioners are often willing, and sometimes even eager, to modify Delphi to meet their own decision-making and forecasting needs. In some cases, these modifications are meaningful and contribute to a better understanding of the technique; in other cases, they are random and arbitrary – thus undermining quality and credibility [ 4 ].

Reports of medical studies using Delphi often lack detailed information on how Delphi studies are conducted [ 7 , 8 ]. This may be partly due to medical journal word count limits, but it results in low repeatability and limited insight into external validity. To address this matter, a guideline for reporting (CREDES) was published [ 9 ]. Still, most of the existing methodological literature on Delphi will not be found by using common search strategies for medical research in databases such as PubMed and Embase, and medical researchers may therefore easily overlook these. For novel users, however, such information is crucial, because the method’s apparent simplicity is in contrast with the work and the difficulty involved in its proper execution [ 3 ]. Time it takes to complete a Delphi study is usually underestimated [ 10 ] and respond rates are often disappointingly low [ 10 , 11 ] Deficient application of the technique will lead to poor validity of the results [ 3 , 5 , 12 ]. In addition, little has so far been published on the way in which panellists respond to the presented rounds, what circumstances contribute to high response rates in a panel or how these can be optimized.

With this paper, we provide insight into the methodological challenges encountered in our Delphi study on ED standards and present the solutions we formulated based on the available data and how these turned out. With our work, we aim to aid others who plan a Delphi Design and to help them improve the quality of their study. In addition, we provide details on response times from the current e-Delphi study.

The objective of our e(lectronic)-Delphi study was to determine minimum standards for emergency departments (EDs) in the Netherlands. We based our methodological choices on available literature; we discussed and decided on these within our full research group, and we kept a record of the methodological considerations and evaluations made by the research team before as well as during the study. Per round, the number of reminders sent and individual responses times were noted.

Table  1 presents an overview of items requiring design choices that influence the rigor of a Delphi, based on the CREDES guideline for conducting and reporting [ 9 ].

Purpose and rationale of our Delphi

The study that is used as an example in this article achieved a consensus among Emergency Physicians on minimum operational standards for emergency departments (EDs) in the Netherlands [ 1 ]. It focused on three domains: ED facilities, diagnostics and the availability of medical specialists. Its background is the rapidly changing emergency health care environment, where crowding, an aging patient population and staff shortages put pressure on the quality and availability of ED care. A clear view on the minimum standards of a hospital-based 24/7 ED is important for policy makers set course toward optimal conditions for delivering adequate care for patients with undifferentiated, urgent or emergent complaints. A second driver for the study can be found in the ED context with highly trained physicians, which may be a challenging field to explore the various methodological aspects to design and perform a rigorous Delphi. Here, we set out to formulate clear and repeatable options for addressing known Delphi methodology issues, and we describe how these played out in our study. We do so in order to contribute to the work of medical Delphi users and assist them with the application of the Delphi technique in practice.

Prior information for establishing a panel’s knowledge base

Panellists were provided with information on the study’s purpose and rationale, including the definition of ED that was used. Experts remained anonymous and unknown to each other throughout the entire process. One selection criterion for our panellists was a prior possession of the required knowledge base [ 1 ], and therefore no additional information on ED operational standards was provided. In this way, the risk of influencing the experts’ judgements was eliminated.

Unstructured (classical) or structured (modified) first round

In a ‘classical’ Delphi, the first round is unstructured, allowing free scope for experts to elaborate on issues they deem important. Commonly, however, this round is structured to save time and effort for both the monitor team and the panellists [ 6 ].

We assumed that interpretation differences amongst panellists could exist with respect to the items involved, and that these could thus affect the discussion of the main research question in a negative manner. To avoid such miscommunication on items, we added Part 1 prior to focusing on the main research goal in Part 2, with both parts consisting of several rounds. In Part 1 we combined the collection of all relevant items from the literature and the panel with reaching consensus among panellists regarding the definition of these items. Additionally, adding a consensus path before introducing the main research question allowed the panel and the monitor team to familiarize themselves with the method and the online survey tool. Such clear definitions will also aid interpretability of the results for others.

We started our study in a semi-structured fashion (Part 1, Round 1): panellists were provided with a list of items and a proposed definition for each of these. Panellists were then asked to do two things. First, they were invited to add all items that could possibly be regarded as an ED facility or a diagnostic option for an ED. To stimulate panel members to submit plain as well as more uncommon items in the first round, we made sure that the provided list included items that panellists would most likely consider to be necessary for every ED in Part 2 (for example, every ED needs a toilet) as well as unnecessary (for example, every ED should have an MRI scanner on the shop floor). Next, the panellists rated and commented on the provided definitions. Once all items had been gathered and consensus on the definitions of all items had been reached, we proceeded with Part 2. Part 2 focused on the main research topic: minimum ED standards in three domains (ED facilities, diagnostics and availability of medical specialists), for which the defined items resulting from Part 1 and a list of all medical specialties were used.

Required question type (qualitative or quantitative)

In Part 1, panellists were asked dichotomously if they could agree with a proposed definition. Open text fields were used for panellists to explain their choices and to add their opinions on remarks made by their colleagues. Open fields were also used to include any additional items.

In Part 2, the domains ‘ED facilities’ and ‘diagnostic availability’ started dichotomously, but they were converted into multiple-choice options when two or more panel members suggested a similar adjustment or condition. This was added in the next round, for example “agree, only when condition x is satisfied”.

In the third domain (availability of medical specialists), we first aimed at a yes-no level of necessity. Additionally, we wanted more information on the degree to which the panel felt this was necessary. We used a multiple choice approach to limit the time and effort asked from panellists, for example “agree, 24/7 availability” and “agree, 24/7 availability by phone as well as physically available within 30 minutes”. Again, when two panellists added a similar remark, this was added as an answer option. Items that were selected by fewer than two panellists were omitted in the next round, but panellists were offered the opportunity to disagree with the remaining options when they were not convinced by the motivations given by other panellists.

Define consensus and non-consensus

Consensus was assumed to have been reached when at least 70% agreement was achieved. Our decision rules are presented in Table  2 . When consensus was reached, members could ask for an item to be re-discussed, but a motivation was required. Again, a threshold of two similar motivations was needed for an item to be discussed in the next round. We opted for this approach to enhance the validation of the consensus reached. It also provided panellists with the opportunity to individually avoid process loss due to early closure (through reaching agreement on the first solution that nobody strongly objects to with the aim to achieve consensus rather than aiming for the best possible judgment that is agreed upon wholeheartedly by most). The obvious disadvantage in this case is that it required more time and effort on the part of the panellists.

Clear and transparent guidelines on how to proceed from round to round

We decided not to set a fixed number of rounds. Instead, we set endpoints and decision rules indicating when to add changes and when to accept. Classically, the number of iterations seldom exceeds one or two rounds, when most changes in panellists’ responses are expected [ 6 , 13 ]. However, the possibility to respond to the provided feedback and consensus are essential for improving validity [ 6 ]. The downside of multiple rounds could be that panellists (even ‘holdouts’) can become tired of discussing the same item again and tend to lean towards closure and changing their opinion towards the mean if the number of rounds keep increasing [ 13 , 14 ] or they drop out [ 11 ]. We tried to minimize such weariness amongst panellists by making sure the surveys were easy to fill out, with a clear and consistent presentation, in order to minimize the amount of time needed. In the introduction of each round the purpose of each part/round was explained to the panel and decision rules relevant to the specific round were shared. The survey then started with the pages with items on which no consensus had been reached previously (Fig.  1 ), followed by items on which consensus had been reached in the previous rounds together with all remarks. Only when the panellists did not wish to see an item that had been consented upon earlier return for rediscussing, this item was deleted from the next round.

Components of online survey. The design was identical for each page of the survey and consisted of the following components

When a similar suggestion for adjustment of an item was made by at least two panellists, it was changed and the adjusted item was put forward for discussion again in the next round. This rule was also applied when consensus was reached in the same round as suggestions for change were made; we called this reached consensus ‘preliminary consensus’. The item was adjusted and put forward again in the next round (an example is included in the Results section below). When in Part 2, Round 4 no consensus was reached and no suggestions for change were made, nor major shifts in opinion were seen, we accepted non-consensus with respect to these items.

Strategy for processing results between survey rounds

The online survey tool generated the pooled results per round. Quantitative data were expressed in percentages as statistical group response. Newly generated items were included in the next round. Qualitative data were presented per item; arguments from the last round were presented separately from the arguments from the previous rounds.

Our rule for modification required 2 similar suggestions for adjustment. Two members of the research team (RV and MG) individually judged all comments on similarity. In case of disagreement, the data were discussed in the full research team.

Development of materials/instruments (platform / lay-out / questions)

An online survey tool was used ( SurveyMonkey.com ®) with a clear lay-out that was identical in all rounds (Fig. 1 ). We presented one item per page, including a statistical summary of the former round, and anonymized remarks from prior rounds. Each member of the panel received an individual invitation for each survey. A survey could be closed and continued whenever the panellists wanted this.

Pilot materials / instruments

Surveys were built per round by one researcher (RV). No formal pilot was performed. The functionality of each round was tested using different mail servers and operating systems and checked by a second researcher. When agreed upon, the survey was sent to the panel and a dummy version was sent to the research team for control purposes.

Selection of experts

One of the key aspects of a rigorous Delphi study is the selection of experts for the panel, as this is crucial for the validity of the resulting consensus. Therefore, much thought should be given to assembling a representative panel. Steps in this selection process include identifying the classes of experts, completing these with names, contacting individuals and letting them nominate others for inclusion, ranking individuals on the basis of qualifications, inviting individuals and finally asking for their commitment [ 5 ]. Gaining commitment from high-profile and busy panellists depends on the way in which this process evolves. Incentives to participate could include being invited to join a selective group and the opportunity to learn from the consensus process [ 5 ]. We did not pay our experts, nor did we give them presents of any kind. The ideal number of panellists for a Delphi is not set in stone, but a recommended number is 10 to 18 for ensuring the development of productive group dynamics and for arriving at consensus among experts [ 9 , 15 ].

We recruited our panel among Emergency Physicians (EPs) to guarantee a broad but direct clinical view on the topic. We added the following requirement: 15 to 25 members had to be included from all over the country (i.e. one to two members from each of the country’s 11 predefined regions of acute care), and there had to be a balance between working in academic, semi-academic and rural hospitals. We aimed for EPs with demonstrated competence in management or education (Table  3 ), added names to our list and called the highest-ranking EP in each region. Calls were made by MG, a Dutch EP well known by most EPs as the former Chairman of the Dutch Society of EPs. EPs were informed about the aim of the study, the methodology and the importance of anonymity, and we emphasized the amount of the time that they needed to invest. When other EPs were nominated, we ranked them on our list. Verbal and written informed consent was obtained from all panellists.

Role of research team

The research team designed the study and set the decision rules, in line with the available literature. They applied these rules and closely monitored and evaluated the methodological aspects of the study and managed the overall study process. The summary statistics and panellists’ input were interpreted and discussed by the full research team. Minutes of these meetings were kept in a log. Members were careful not to influence panellists’ opinions, and none of the members had a conflict of interest.

Strategy to improve response rate

Activities to improve response rates started as early as the inclusion process for the panel’s members, when each panellist had personal contact with the main researcher (MG). Response rates can be enhanced by a ‘personal touch’ together with a clear explanation the study process and awareness of the importance of commitment to the study for the validity of the results [ 11 ].

After inclusion, panellists received an e-mail with study details and additional message in the week leading up to each round. The link to the survey was sent by e-mail and a text message was sent by phone. When the research team felt it was necessary, panellists received a reminder. This was done on the basis of prior response times of the individual panellists, expected duration of completion and national holidays. Panellists were prompted by one researcher (MG).

Final draft

According to CREDES, results from a Delphi study should be reviewed by an external board before implementation. However, since emergency care is a multidisciplinary field and support needs to be found amongst the various emergency care stakeholders, our study consensus was not set up for direct implementation [ 1 ], since emergency care is a multidisciplinary field and support amongst the various emergency care stakeholders; it was the starting point for a further discussion about the development of standards for EDs.

Panel participation

The expert panel consisted of 20 EPs. Their experience and backgrounds are shown in Table 3 . Only one EP nominated a more experienced colleague, who agreed to participate instead. Per round, we sent a maximum of two e-mails with the link to the survey, up to four reminders, and we made a maximum of two calls. This led to a 100% response rate without any dropouts.

Item throughput

Part 1 started with 55 items. After four rounds, no items were requested back for re-discussion, and this left us with 63 items and consensus on definitions for all of these (Fig.  2 ). In Part 2, we started with these 63 items and added 29 medical specialties. After four rounds, this resulted in consensus on 62 (98%) items and 27 (93%) specialties (Fig.  3 ), and no items were requested back.

Flowchart for Part 1: gathering and defining items. Preliminary consensus = consensus (agreement 70% or above) but retained based on other decision rule

Flowchart for Part 2. the main research topic: minimum ED standards in three domains (ED facilities, diagnostics and medical specialist availability)

Percentages agreement

The first round in Part 1 resulted in an average agreement of 85% per item, varying from 55 to 100% for different items. For items in which consensus was reached in the first round, average agreement was 89%. The average agreement percentage in the non-consensus group was 63%. If an item needed several rounds to reach consensus, the ultimate average agreement percentage for this item was higher: 98% vs 89%.

We clearly experienced the added value of the decision rule of continuing when two panellists submit similar suggestions for adjustment, irrespective of the agreement percentage: definition agreement for CBRN room (room for treating patient with Chemical, Biological, Radiological and Nuclear contamination) was 79% in Round 2, which was enough agreement to call this a consensus. However, because more than one panellist had made a similar remark for adjustment, we continued the discussion. The results that followed led us to split the definition into two separate items (CBRN room minimum option and CBRN room maximum option), and in the next round 90% consensus was reached for both items. The process was then repeated again, based on the same decision rule, which resulted in 100 and 95% consensus, respectively.

In the first round of Part 2, we found an average agreement of 79% (50–100%) with respect to facilities and 77% (50–100%) with respect to diagnostics. For those items (facilities and diagnostics) in which consensus was reached in the first round, average agreement was 87%. Average agreement on items that did not reach consensus in Round 1 was 59%.

In Part 2, a third domain (availability of medical specialists) was added. On the basis of a yes/no scale, this resulted in agreement for 90% of the 30 medical specialties in the first round; this remained 90%. On the scale used for the degree of this availability, there was little agreement in the first round (only for 10% of specialties), but in the final round consensus was reached for 90% of specialties. A striking change was seen for plastic surgeon necessity: in Round 1, 15% of our panellists supported the idea that a plastic surgeon was not necessary in every ED (= consensus on ‘necessary’), but this changed to 95% (= consensus on ‘not necessary’) when, based on panellists’ input, an option was added for the condition that regional availability should be guaranteed instead.

Agreement with regard to regarded necessity

For items (facilities and diagnostics) that were deemed nessesary, consensus was reached in the first round for 97% of the items concerned (3% in Round 2). When items were considered unnecessary, consensus was reached in the first round for 18% of the items (27% in Round 2, 45% in Round 3, 5% in Round 4, 5% in Round 5). Items that resulted in consensus on necessity mostly took one round, whereas items that resulted in consensus on non-necessity needed an average 2.45 rounds (Fig.  4 ). Of the 22 items that were considered unnecessary, there were 13 items for which the panel set the condition that (part of) their functionalities should be available in another way before consensus was reached.

Deemed necessity and influence on number of rounds to reach consensus

Panel participation and response

We found large differences in response time between panellists and between rounds (Fig.  5 ). Response time consists of the time needed to fill in a questionnaire as well as waiting time until the panellist opens the survey. In our study, once panellists started filling in the questionnaire, they nearly always finished the questionnaire on the same day (on average 80%). We did not keep data on how often and for how long the survey was accessed.

Median time to complete survey per round. The vertical axis displays the response time in days; the horizontal axis lists the individual rounds. The figure represents the median, the interquartile range and two outliers in Part 1 Round 4

In all rounds, panellists were prompted to respond. The timing and method of sending reminders (see Fig.  6 ) was not standardized. Timing depended upon the expected time needed to respond perceived by the research group. This was based on i.e. the panellist’s previous responses, the expected time investment and national holidays. Three panel members did not need any prompts, and one member was prompted 15 times over the eight content rounds (Fig.  7 ). In total, these Delphi rounds took 17 months to complete.

Response times and reminders. The horizontal axis presents the date; the vertical axis presents the cumulative returned surveys. The dotted lines represent one or more reminders that were sent

Number of reminders sent to individual panellists in total and per round

The quality of a Delphi study strongly depends on its design and the quality of its execution. In 2017, guidelines were published on the execution and reporting of such studies (CREDES). Since then, thousands Delphi studies have been published, but the use of CREDES has not been generally adopted. The Equator network used a 4 item quality score for reporting [ 16 ] all of which are also included in CREDES. Nevertheless, is a remaining clear call remains for a Delphi standardization [ 17 ] and researchers continue to study the method itself [ 16 , 18 , 19 , 20 ] with the aim to improve Delphi standardization and quality. However, little has so far been published on practical design choices and how these ultimately play out. This paper tries to fill some of these gaps.

In the first part of our two-part design, we not only collected possible items but also established a common language. We could not find previous studies that did this in a similar way. Defining items took several rounds confirming our assumption that individual items can be interpreted differently by different panellists. After Part 1 had been successfully completed, there appeared to be little miscommunication in the panel responses in Part 2. Another advantage of our two-part structure was that both the panel and the research group became acquainted with the method before focusing on the main research question. A disadvantage may be the time investment needed for extra rounds, but in view of our 100% response rate this does not have to be a problem. Another downside was that panellists were so eager to start answering the main research question that they found it hard to limit themselves to the definitions considered in Part 1. Keeping a steady focus also remained difficult in Part 2, as panellists tended to dwell on the items in other situations. For example, panellists could present arguments specifically related to EDs in cardiac intervention centers, although our study explicitly focused on the minimum requirements for 24/7 EDs only, regardless of size or type.

There are no strict guidelines on the correct number of rounds. The number of rounds is either set in advance or rounds continue until consensus is reached; no further changes take place or return rates diminish [ 10 ]. Having a total number of four rounds in Part 1 and four rounds in Part 2 did not influence the response rate negatively. A low number of rounds is generally thought to increase completion rates, but clear evidence is lacking. We have shown, as was described previously, that response rates can stay high (even as high as 100%) with a high number of rounds. This was previously attributed to a highly motivated panel and multiple reminders [ 17 , 20 ]. To avoid shifting opinions due to study fatigue, it must be made clear to panellists that they do not need to strive for conformity, and that the study will end when no further changes in opinion are presented. We decided to apply clear and strict decision rules to reduce the risk of bias due to subjectivity or inappropriate influence from the research team. Our decision rules worked out well and proved to be applicable without disagreement on their interpretation, and no adjustments or violations were needed. Our approach structured the interpretation of the panellists’ input and the response of the research team. We added a ninth and final confirmative round to offer panellists the option to ask for items to be returned in order to be discussed again. Although no items were asked back, this made sense for two reasons. Firstly, it is in line with our decision rules, which therefore makes our work methodologically sound. Secondly, this could resolve the issue recently raised that true consensus is not merely a majority of vote, considering that some could find an outcome unacceptable [ 21 ].

Questions about the first two domains were presented with yes/no answers and free text to elaborate on the rationale behind panellists’ choices and to add remarks on results from previous rounds. This proved to be effective. For the third domain (availability of medical specialists), we used a structured approach. We wanted yes/no answers as well as opinions on the way in which options should be put into place. We selected multiple options based on common practice (yes, 24/7; yes, on call < 30 min, etc.). In hindsight, we would approach this differently next time. A better option would be to ask yes/no questions and to add a compulsory text box for panellists to indicate what they believe the minimum availability option would be. Such an approach could stimulate out-of-the-box suggestions and possibly create support for such an option in the next round.

Building an online Delphi study requires suitable software and effective lay-out choices. A specific Delphi software programme that provides structure and supports feedback reporting could improve such a study [ 21 ]. In the absence of major players in this specific field, we selected a well-known and commonly applied survey tool. We used clear and identical formats in each round. Each round started with a short introduction detailing the necessary background information and the study’s progress, and each page stated the objective of the study. This is in line with the given advice to be clear about the objective [ 19 ], and repeatedly specifying the objective keeps panellists focused on the goal [ 17 , 20 ]. After the second round in Part 1 of our study, we made all essential questions compulsory and kept filling in the text boxes optional. In hindsight, as mentioned above, this should have been done from the start. Finally, we retained the option to stop and restart at any time, and we also retained the option to go back to the survey at a later moment to make changes or additions for as long as the round was open. This remains a logical choice.

Considerations based on social influencing principles, for example in view of the need for blinding, are of significant importance in the Delphi technique. Generally reported reasons for the blinding of panellists are the following: avoiding group pressure, avoiding situations in which people gear their opinions towards those expressed by the most dominant or most highly respected panellist and ensuring that panellists feel free to change their views. In addition, similar choices based on social principles can help to improve response rates. This was previously suggested by McKenna [ 11 , 22 ] as the ‘personal touch’. In our study the main researcher recruited participants personally by phone. This added not only the effects of liking [ 23 ] (knowing each other and having similar goals), but also authority [ 23 ] (being a well-known and highly respected colleague) as well as reciprocity [ 23 ] (service as provided in the past by being chairman). Setting a similar goal during the recruitment process (a consensus standard as a means towards improving the ED landscape) strengthened the principle of liking. This type of information and personal contact during the recruitment process most likely set the basis for the project’s success.

No single best strategy is known for sending out reminders, but persistence is felt to be important [ 10 ]. We decided to individualize reminders. Considering our 100% response rate, we may conclude that this was an effective strategy. It has been recently shown that most panellists do not disapprove receiving reminders [ 20 ]. However, there is still room for improvement; for example, sending reminders at earlier moments might have shortened response delays in the first round of Part 2. That said, response delays in our study were very rarely due to the time that was needed to complete a survey, but almost always due to delays in opening and starting the survey. Once a panellist had started the survey, the questionnaire was almost always finished in 1 day. Individualizing reminders seemed justified since interpersonal differences proved to be large, while intrapersonal differences were limited. For example, one panellist needed more than one reminder for all but the first survey round, while some needed none, and others needed one reminder in some rounds. If this is representative for other panels, it would seem that panellists who need several reminders in the earlier rounds may also need more at later stages. We would suggest making an early start when late responders need to be reminded. An advantage of sending e-mail reminders was that panellists did not need to search for their link in previous messages. Text messages and telephone reminders, on the other hand, are likely to be perceived as more personal and may therefore have more emotional impact.

We also found differences concerning the time of day when panellists submitted their responses: this was spread over 24 h of the day, most likely corresponding to their shift work as a doctor. Efficacy of the type and the timing of reminders might be influenced by this as well.

Items that resulted in consensus on non-necessity took more rounds (Fig.  4 ). This might be due to the fact that people generally find it harder to say ‘no’ than to say ‘yes’. In our study, panellists wanted to set a condition before saying ‘no’ to an ED item because they wanted to ensure access to proper healthcare for all patients. Taking this type of panellist behaviour into consideration prior to running a study will likely lead to a more accurate estimation of the number of rounds (resources) needed.

There is no rule that specifies which cut-off value for consensus should be used. Commonly applied levels vary between 51 and 80% [ 24 ]. It has been shown that using different consensus definitions on one data set can lead to dramatically different results [ 17 , 25 ]. Interested in the effect on the results we estimate what effect increasing the cut-off value from 70 to 80% would have had on our results. We found that, in general, a majority of items showed an increase in consensus with an increasing number of rounds, although this increase became less and less steep [ 13 ]. Therefore, it seems reasonable to assume that accepted items with consensus in early rounds which varied between 70 and 80% might have resulted in higher consensus if they had been retained in another round. We can confirm this for items that were repeated in extra rounds (following our two-similar decision rule) all nine items passed the 80% mark. In Part 1, we accepted 8% (3/36) of the items within the 70–80% range, and in Part 2 this was 38% (15/63 items) and 31% (9/29) for specialist availability. Eleven of these items were accepted in a first round, four in a second round and three in a third round. In view of these findings, we conclude that changing the cut-off value would most likely not have had major effects on our results.

Response times are shown in Fig. 5 and Fig. 6 . Since response times tend to be influenced by many factors, study researchers should discuss these and offer explanations or interpretations. In our first survey, it took a reminder to motivate half of the panellists to respond. We saw clear effects of sending out reminders: in the last rounds, we sent individualized reminders spread over a short period of time. This was done for logistical reasons so as to be able to contact panellists individually.

The duration of the first round in Part 2 proved to be longer than the duration of other rounds. The time needed to complete this survey was by far the longest, and therefore the research team felt that panellists should be allowed more time to submit their responses. The team did not, however, realize at that moment that response times were mostly influenced by delays in opening the survey rather than by the time that was needed to complete it. The third round was short, most likely due to the fact that by then consensus had been reached for most items. Furthermore, panellists might have had more time because the period concerned was a holiday period and reminders were sent out at an early stage. The longer duration of Round 4 was no surprise: with continuing rounds, panellists generally experience response exhaustions occurs in panellists [ 11 ], especially with busy experts and hard-pressed clinicians [ 10 ].

In conclusion, this article described the methodological considerations and relevant practical aspects of our Delphi study that resulted in a 100% response rate. This exemplifies the value of a systematic approach to design choices. Based on our experience, we advise those considering a Delphi study with the aim to reach consensus on a certain topic to do the following. Adopt the CREDES guideline. Consider a two-part design, including a first part to establish a common language and to familiarize both the panellists and the research team with the online tool that is used. Invest in ensuring personal commitment from the panellists during the recruitment phase. Set clear decision rules to enhance consistency during the process and to keep the process comprehensible for the panellists. Exclude items that have reached consensus from the next rounds, but use a confirmation round in which panellist are given the option to reintroduce such items into the discussion. Design the e-Delhi in a clear and consistent lay-out throughout the full study. Expect items that result in a negative advice to require more rounds before consensus is reached. Finally, send out reminders at an early stage. Our data suggest that delays in survey responses are usually due to participants not opening a survey rather than participants taking a long time to complete it. A rigorous plan for reminders will enhance both high response rates as well as a timely completion of surveys. Adopting this overall approach may assist researchers in the future execution of Delphi studies and may help them to enhance the quality of Delphi designs in terms of improved rigor and higher response rates.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Abbreviations

Chemical, Biological, Radiological and Nuclear

Guidance on Conducting and REporting DElphi Studies

Emergency Department

electronic Delphi

Emergency Physician

Magnetic Resonance Imaging

Gaakeer MI, Veugelers R, Patka P, Huijsman R, on behalf of the Dutch ED Study Group. Minimum operational standards for 24/7 available emergency departments in the Netherlands: a first step taken by emergency physicians using an e-Delphi approach. Eur J Emerg Med. 2019;26:86–93 https://doi.org/10.1097/MEJ.0000000000000494 .

Article   Google Scholar  

Dalkey N, Helmer O. An experimental application of the Delphi method to the use of experts. Manag Sci. 1963;9:3 https://doi.org/10.1287/mnsc.9.3.458 .

Landeta J. Current validity of the Delphi method in social sciences. Technol Forecast Soc Change. 2006;73:5 https://doi.org/10.1016/j.techfore.2005.09.002 .

Gupta AG, Clarke RE. Theory and applications of the Delphi technique: bibliography (1975-1994). Technol Forecast Soc Change. 1996;53:2 https://doi.org/10.1016/S0040-1625(96)00094-7 .

Okli C, Pawlowski SD. The Delphi method as a research tool: an example, design considerations and applications. Inf Manag. 2004;42(1):15–29 https://doi.org/10.1016/j.im.2003.11.002 .

Rowe G, Wright G, Bolger F. Delphi - a reevaluation of research and theory. Technol Forecast Soc Change. 1991;39:235–51.

Diamond IR, Grant RC, Feldman BM, Pencharz PB, Ling SC, Moore AM, et al. Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies. J Clin Epidemiol. 2014;67:401–9 https://doi.org/10.1016/j.jclinepi.2013.12.002 .

de Loë RC, Melnychuk N, Murray D, Plummer R. Advancing the state of policy Delphi practice: a systematic review evaluating methodological evolution, innovation, and opportunities. Technol Forecast Soc Change. 2016;14:78–88 https://doi.org/10.1016/j.techfore.2015.12.009 .

Junger S, Payne SA, Radbruch L, Brearly SG. Guidance on conducting and REporting DElphi studies (CREDES) in palliative care: recommendations based on a methodological systematic review. Palliat Med. 2017;31:8 https://doi.org/10.1177/0269216317690685 .

Keeney S, Hasson F, McKenna H. Consulting the oracle: ten lessons from using the Delphi technique in nursing research. J Adv Nurs. 2006;53:2 https://doi.org/10.1111/j.1365-2648.2006.03716.x .

McKenna HP. The Delphi technique: a worthwhile research approach for nursing? J Adv Nurs. 1994;19:6 https://doi.org/10.1111/j.1365-2648.1994.tb01207.x .

Rieger WG. Directions in Delphi developments: dissertations and their quality. Technol Forecast Soc Change. 1986;29:2 https://doi.org/10.1016/0040-1625 (86)90063-6.

Brockhof K. The performance of forecasting groups in computer dialogue and face to face discussions. In: Linstone H, Turoff M, editors. The Delphi method: techniques and applications. London: Addison-Wesley; 1975. p. 285–311. https://web.njit.edu/~turoff/pubs/delphibook/delphibook.pdf . Accessed 16 Oct 2018.

Google Scholar  

Helmer O. Social technology. New York: Basic Books; 1966.

Paliwoda SJ. Predicting the future using Delphi. Manag Decis. 1983;21:1 https://doi.org/10.1108/eb001309 .

Banno M, Tsujimoto Y, Kataoka Y. Reporting quality of the Delphi technique in reporting guidelines: a protocol for a systematic analysis of the EQUATOR network library. BMJ Open. 2019;9:e024942 https://doi.org/10.1136/bmjopen-2018-024942 .

Humphrey-Murto S, de Wit M. The Delphi Method-more research please. J Clin Epidemiol. 2019;106:136–9.

Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review. PLoS One. 2011;6(6):e20476 Epub 2011 Jun 9. https://doi.org/10.1371/journal.pone.0020476 .

Article   CAS   Google Scholar  

Sinha IP, Smyth RL, Williamson PR. Using the Delphi technique to determine which outcomes to measure in clinical trials: recommendations for the future based on a systematic review of existing studies. PLoS Med. 2011;8(1):e1000393 https://doi.org/10.1371/journal.pmed.1000393 .

Turnbull AE, Dinglas VD, Friedman LA, Chessare CM, Sepúlveda KA, Bingham CO 3rd, et al. A survey of Delphi panellists after core outcome set development revealed positive feedback and methods to facilitate panel member participation. J Clin Epidemiol. 2018;102:99–106 Epub 2018 Jun 30. https://doi.org/10.1016/j.jclinepi.2018.06.007 .

Humphrey-Murto S, Crew R, Shea B, Bartlett SJ, March L, Tugwell P, et al. Consensus building in OMERACT: recommendations for use of the Delphi for Core outcome set development. J Rheumatol. 2019;46(8):1041–6 Epub 2019 Feb 15. https://doi.org/10.3899/jrheum.181094 .

McKenna H. The selection by ward managers of an appropriate nursing model for long-stay psychiatric patient care. J Adv Nurs. 1989;14:9 https://doi.org/10.1111/j.1365-2648.1989.tb01641.x .

Cialdini RB, Goldstein NJ. Social influence: compliancy and conformity. Annu Rev Psychol. 2004;55:1 https://doi.org/10.1146/annurev.psych.55.090902.142015 .

Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs. 2000;32:4 https://doi.org/10.1046/j.1365-2648.2000.t01-1-01567.x .

Grant S, Booth M, Khodyakov D. Lack of preregistered analysis plans allows unacceptable data mining for and selective reportingof consensus in Delphi studies. J Clin Epidemiol. 2018;99:96–105 Epub 2018 Mar 17. https://doi.org/10.1016/j.jclinepi.2018.03.007 .

Download references

Acknowledgements

We are grateful to all members of the Delphi panel for their participation. We thank our language editor Laetis Kuipers for her work and Barry Hage for his work on the graphical presentation of Fig. 6 .

This study was supported by an unrestricted grant from Achmea Healthcare Foundation (Ref. Z634). The funding body assigned the grant based on our study protocol without restrictions on the protocol, data collection or analysis. Neither did they have insight into the collected data or manuscripts prior to publication.

Author information

Authors and affiliations.

Emergency Department, Adrz hospital, P.O. Box 15 4460, AA, Goes, the Netherlands

Rebekka Veugelers & Menno I. Gaakeer

Emergency Department, Erasmus MC, P.O. Box 2040, 3000, CA, Rotterdam, the Netherlands

Peter Patka

Erasmus School of Health Policy & Management, P.O. Box 1738, 3000, DR, Rotterdam, the Netherlands

Robbert Huijsman

You can also search for this author in PubMed   Google Scholar

Contributions

MG initiated and together with RV, PP and RH designed the study. RV designed and built the online surveys, critically revised by MG, PP and RH. RV and MG performed data collection and analysis. RV, MG, PP, RH participated in research team meetings on interpreting data between rounds as well as the concluding data. RV drafted this document; MG, PP and RH revised the manuscript critically for important intellectual content. All authors have read and approved the manuscript. All authors have participated sufficiently in the work to take public responsibility for appropriate portions of the content; and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Corresponding author

Correspondence to Rebekka Veugelers .

Ethics declarations

Ethics approval and consent to participate.

Ethics approval was obtained from the Medical Ethics Committee Erasmus MC of Rotterdam, The Netherlands (Ref. MEC-2014-322). Written informed consent was obtained from all participating panellists.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Veugelers, R., Gaakeer, M.I., Patka, P. et al. Improving design choices in Delphi studies in medicine: the case of an exemplary physician multi-round panel study with 100% response. BMC Med Res Methodol 20 , 156 (2020). https://doi.org/10.1186/s12874-020-01029-4

Download citation

Received : 12 December 2018

Accepted : 24 May 2020

Published : 15 June 2020

DOI : https://doi.org/10.1186/s12874-020-01029-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Delphi technique
• Emergency medicine
• Emergency service
• Response rate
• Decision rule
• Evidence-based emergency medicine
• Epidemiologic research design
• Operational standards
• Tips and tricks

BMC Medical Research Methodology

ISSN: 1471-2288

• General enquiries: [email protected]

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here .

Loading metrics

Open Access

Peer-reviewed

Research Article

Implementing a combined Delphi and Focus Group qualitative methodology in Nexus research designs—The case of the WEFE Nexus in Apokoronas, Crete

Roles Conceptualization, Formal analysis, Investigation, Methodology, Writing – original draft

* E-mail: [email protected]

Affiliation Department of Agricultural Production and Resource Economics, Technical University of Munich, Freising, Germany

Affiliation Department of Psychology, University of Crete, Rethymnon, Crete, Greece

Roles Conceptualization, Investigation, Supervision, Writing – review & editing

Roles Data curation, Methodology

Affiliation Department of Economics, University of Crete, Rethymnon, Crete, Greece

Roles Funding acquisition, Investigation, Supervision, Writing – review & editing

Roles Supervision

• Carolin Canessa, 
• Andreas Vavvos, 
• Sofia Triliva, 
• Iosif Kafkalas, 
• Maria Vrachioli, 
• Johannes Sauer

• Published: July 14, 2022
• https://doi.org/10.1371/journal.pone.0271443
• Reader Comments

In recent years, researchers and policymakers have emphasised the importance of understanding the complex relationships between Water, Energy, Food and Ecosystems (WEFE). The primary reason for capturing these complexities is to understand how decisions made in the water, food and energy sectors can affect one another. Crucially, biodiversity and ecosystem services (E) play a mediating role in these relationships by making material and non-material contributions to all other sectors (W, E, F). The Nexus approach has been widely used for capturing these interdependencies and identifying opportunities for increasing efficiency, reducing trade-offs and building synergies for sustainable resource use across the WEFE nodes. One challenge in using this framework is the need to harmonise the technical and managerial dimensions of the WEFE interlinkages with the perceptions and priorities of local populations directly involved in the use and management of resources. This paper presents a methodological framework that seeks to integrate the perspectives of experts, practitioners and local stakeholders on the WEFE Nexus through the combined application of the Delphi and Focus Group methods. In this paper, the municipality of Apokoronas in Crete, Greece has served as the case in point. The combined framework allowed us to explore the Nexus understanding at the local level and was instrumental in the identification of initiatives for more integrated resource management. The triangulation of results captured the differences in priorities between practitioners and the local community at large, but also, more specifically, it pointed to discrepancies within groups and across WEFE sectors. The outcomes of this paper demonstrate that awareness and learning play a central role in Nexus actions to overcome conflicts and perceived inequalities, and to internalise solutions. The inclusion of the ecosystems node in the traditional WEF Nexus encouraged participants to contemplate the pivotal role of ecosystems in supporting the rest of the WEF sectors.

Citation: Canessa C, Vavvos A, Triliva S, Kafkalas I, Vrachioli M, Sauer J (2022) Implementing a combined Delphi and Focus Group qualitative methodology in Nexus research designs—The case of the WEFE Nexus in Apokoronas, Crete. PLoS ONE 17(7): e0271443. https://doi.org/10.1371/journal.pone.0271443

Editor: Vassilis G. Aschonitis, Soil and Water Resources Institute ELGO-DIMITRA, GREECE

Received: March 24, 2022; Accepted: June 30, 2022; Published: July 14, 2022

Copyright: © 2022 Canessa et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: Data cannot be shared publicly because they involve human research participants’ opinions and other sensitive data. Participants’ consent forms clearly stated that data would not be shared with any parties outside the research project. Data are available from the Ethics Committee (contact via [email protected] .) for researchers who meet the criteria for access to confidential data.

Funding: This study was funded by the EU H2020 PRIMA project SIGMA Nexus - Sustainable Innovation and Governance in the Mediterranean Area for the WEF Nexus, Grant #1943 [SIGMANEXUS] [Call 2019 Section 1 Nexus RIA]. The funding agency had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. The findings and conclusions in this publication are those of the authors and should not be construed to represent any official EU or PRIMA determination or policy.

Competing interests: The authors have declared that no competing interests exist.

1 Introduction

In recent years, researchers and policymakers have emphasised the importance of understanding the complex relationships between water, energy, food and ecosystems [ 1 – 4 ]. The interdependencies between these sectors are commonly referred to as the Water-Energy-Food-Ecosystems (WEFE) Nexus. The implication behind this concept is that in a world of rising global population and increasing pressure on resources, any strategy that is not nodal—focuses on one sector at a time without taking into account its impact on others—can have serious unintended consequences and result in oversimplified and narrow theoretical approaches that ignore the notion of interconnectivity [ 5 , 6 ]. Against this frame of reference, accounting for the interlinkages of WEFE nodes through a Nexus approach has become a priority in the global development agenda [ 7 ]. The Nexus approach is a means to identify opportunities for increasing efficiency, reducing trade-offs and building synergies for sustainable resource use across the nodes [ 8 ].

Despite the fact that the scope of the approach is clear [ 9 ], the implementation of the framework can still be challenging in the face of the complex (spatial, temporal, institutional and jurisdictional) interlinkages across the various sectors [ 10 , 11 ]. One of the thorniest problems to plague the operationalisation of the WEFE concept pertains to grasping fully all the biophysical and socio-economic links within and across nodes. This is possibly the main reason why the ecosystems node is frequently omitted by the majority of Nexus studies [ 12 ]. To date, most published research on the WEFE Nexus only partially includes a biophysical and economic assessment of ecosystem services [ 2 ]. Moreover, the Nexus framework has been criticised as a construct that obscures the power inequalities inherent in the use and management of natural resources, by focusing solely on technical, technological and managerial solutions. Additionally, the concept is perceived as poorly connected with local visions of sustainable and just futures, as well as issues of environmental justice and poverty [ 13 , 14 ].

Since the introduction of the concept in 2011 [ 8 ], researchers and policymakers have made significant efforts to explore the Nexus from a variety of perspectives [ 15 ]. Over the past decade, nearly three-quarters of all studies on the Nexus have been quantitative in nature, favouring its use as a tool to assess the biophysical resource flows between nodes [ 16 ]. At the same time, however, a relatively small body of literature has studied the concept under a qualitative lens [ 15 ], using a wide range of methods and combinations thereof, such as fuzzy cognitive mapping [ 17 ], stakeholder analysis [ 18 , 19 ], expert interviews [ 18 , 20 ] and workshops [ 9 , 17 , 19 ]. Due to the high level of technical expertise required, these studies have often targeted communities of experts and policymakers, and seldom addressed wider groups of stakeholders or citizens. This is also the case in the studies by Smajgl et al. [ 21 ] and Foran et al. [ 22 ], who used the Delphi method to investigate the implications of large-scale development investments on the Water-Energy-Food (WEF) Nexus in the Mekong basin. As a result, stakeholder engagement within the Nexus research agenda is still limited, despite widespread recognition of the potential value of interdisciplinary research [ 23 ]. Several authors agree that participatory processes, convergence thinking, and interdisciplinary approaches can bring new perspectives into the discussion, by supporting knowledge-sharing and informed decision-making [ 9 , 24 ].

A critical reconsideration of the Nexus concept is of particular relevance for the Mediterranean region. The adoption of an integrated perspective in resource management has become a priority, due to the growing demand for resources, increased water scarcity, climate change, the degradation of ecosystem services, and water pollution in the region [ 25 ]. In the face of these developments, scientists and policymakers have emphasised the need to promote food and water security through integrated resource management and inclusive development, which recognise the role of different stakeholders in society [ 26 ]. Several studies have investigated the management of two or three WEFE resources in the Mediterranean, using qualitative methods [ 18 , 27 ]. Karabulut et al. [ 28 ] and Laspidou et al. [ 29 ] have proposed Nexus approaches to quantify the interlinkages among different sectors, including climate, land and environment. Karabulut et al. [ 30 ] solicited expert opinions to assess the cross-sectoral impacts of different European policies on the WEFE Nexus, while Malagó et al. [ 31 ] proposed an analytical framework to identify and balance the strengths and weaknesses of WEFE Nexus practices in the Mediterranean context. To the best of our knowledge, however, no study to date has used either the Delphi or the Focus Group techniques to investigate stakeholder understanding of the WEFE Nexus in the Mediterranean, in a context where equal importance is placed to all four nodes of the Nexus.

A compelling case can be made for adopting a more comprehensive qualitative approach to the study of the WEFE Nexus, as there is documented need to understand the diverse interactions between WEFE nodes and people in the Mediterranean region [ 29 ]. Within this context, this paper explores local stakeholders’ understanding of the WEFE Nexus by applying the Focus Group and Delphi methods to the area of Apokoronas in Crete, Greece. The Delphi (DS) and Focus Group (FG) techniques are used in the same study to investigate how local experts and stakeholders, respectively, view the WEFE Nexus and its concomitant resource management integration strategies. While Delphi attempts to capture the overall existing pressures over the Nexus and is predicated on expert opinions, the FGs investigate the "context-embedded" perspectives of various groups of local stakeholders regarding the interplay between resources, policies and sustainable development. The results of the study were compared by triangulating outcomes from both methodologies [ 32 ]. The following research questions guided the analysis: i) How do diverse stakeholders understand and frame the interconnections and interdependencies across the WEFE sectors? ii) What types of solutions would the stakeholders put in place to improve WEFE integrated resource management? iii) What are the prospects of applying qualitative methodologies in Nexus research designs?

The analysis is structured as follows. Section 2 introduces the case study area and the methodology. It presents the data collection process and the demographic characteristics of the participants, and discusses the appropriateness of the methods used. Section 3 contains an independent presentation of the results of the Delphi surveys and the Focus Groups interviews in light of the research questions. In Section 4, the results from both methodologies are triangulated and policy implications are discussed. Finally, future areas of investigation are proposed.

2 Materials and methods

2.1 study area.

Apokoronas is a district of 15,289 inhabitants located in the north-eastern corner of the Chania prefecture of Crete, at the foot of the White Mountains ( Fig 1 ) [ 33 ]. According to Köppen-Geiger climate classification system, the climate in the study area is warm and temperate. The average maximum temperature is 23.6 °C, the average minimum temperature is 11.3°C and the annual precipitation is around 900 mm. The economy in the study area relies on agriculture, including many small-scale dairy and olive-oil facilities, and tourism. The study area aims to reflect the impact of both physical and anthropogenic drivers affecting the WEFE Nexus. The physical drivers are associated mainly with the impacts of climate change, while the anthropogenic drivers reflect demographic changes, the transition from traditional crops to new water-intensive ones, and the continuous touristic development along with higher hosting standards.

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

https://doi.org/10.1371/journal.pone.0271443.g001

2.1.1 Water in Apokoronas.

The island of Crete is self-sufficient in terms of water supply, but still commonly suffers from water scarcity events. These are mainly due to local climatic conditions and disparities between water availability and demand [ 34 ]. The municipality of Apokoronas registers the highest precipitation and hosts the only natural freshwater lake on the island; yet, it experiences increased conflict over resource allocation. Lake Kournas and its freshwater system belong to the Natura 2000 Network owing to its rich ecosystem and soil [ 35 , 36 ]. The lake of Kournas is the only natural freshwater lake in Crete. Its hydrological system supplies nearby villages and supports a vast peripheral area of arable land, which, especially around the Almyros estuary, is particularly fertile. In the past 30 years, there has been an expansion in service industries on the island (mainly tourism-related) at the expense of farming areas. The model of rapid tourism development, and, in some cases over-tourism, in Crete is subject to increased criticism for its negative impact on natural resources and the environment [ 37 ].

2.1.2 Energy in Apokoronas.

From the energy perspective, Crete’s electrical grid was recently connected to mainland Greece’s power supply. Whilst electricity production on the island is partly based on thermal power units that operate on fossil fuels, the connection to the mainland electrical grid created the conditions for increasing renewable energy production for the years to come. Many inhabitants use small energy generating turbines for pumping water and a biofuel produced from olive kernels for heating purposes. A small hydroelectric unit operates in Almyros river in Georgioupoli, where water from the natural lake and the Lefka Ori mountain springs is used as a source of electricity.

2.1.3 Food in Apokoronas.

Agriculture remains a very important and dynamic sector in the area of Apokoronas with farmers cultivating traditional crops, such as olives and citrus fruits, or newly introduced, water-intensive crops, such as avocados. Other key agricultural sub-sectors include livestock, agro-forestry, and fishing. Farmers engage in a range of interdependent gathering, production and post-harvesting processes and their livelihoods encompass various processing-packaging sector activities. Similarly, people who are employed or own family-run businesses also farm their own land.

2.1.4 Ecosystems in Apokoronas.

The entire freshwater system around Kournas lake, including the marshes, the Almyros stream and estuary in Georgioupolis, are one of the most important in the east Mediterranean region [ 34 ]. The area is a relevant site for breeding, migrant and wintering waterbirds. Species of concern include Egrettagarzetta and Aythya Nyroca. The local ecosystem supports the essential provision, regulation and cultural services, which are pressured by complex, transactional and anthropogenic drivers of change such as tourism, agriculture, inappropriate water and environmental management [ 38 ].

2.2 Delphi study methodology

The Delphi technique refers to a structured, anonymous and iterative round of surveys aimed at eliciting opinions from a panel of experts, who are asked to reach a consensus on critical issues [ 39 ]. The method is used to structure a group communication process, whilst avoiding confrontation among the participants [ 40 ]. Different variations of Delphi studies exist, depending on their objectives [ 41 – 43 ]. In this study, a modified argument-policy Delphi was adopted, which was simultaneously investigating expert understanding of the WEFE Nexus in Apokoronas and opportunities for its integrated management [ 44 ]. The technique was used for exploratory rather than confirmatory purposes. As a result, the consensus measurement, typical of the Delphi technique, was not its main goal but rather the instrument for data analysis.

Following the Delphi criteria of iteration, anonymity and controlled feedback, two rounds of quantitative surveys, preceded by key informant interviews, were developed [ 45 ]. The use of a qualitative round is widely accepted and strongly encouraged [ 46 ]. In terms of structuring the questionnaire, a general-to-specific approach was used to allow experts to evaluate the questions in increasing depth. The questionnaires included a combination of closed- and open-ended questions designed based on the outcomes of the key informant interviews. As outlined in Fig 2 , the survey consists of four sections that cover the expertise background of the interviewees, the sustainable development priorities for the area, the drivers and pressures on the WEFE Nexus in the area, and the possible responses to increase WEFE Nexus integrated management in Apokoronas. A simplified Drivers, Pressures, State, Impact and Response (DPSIR) logic was used to guide the survey design and the interpretation of the results [ 47 ]. The survey concentrated mainly on the Drivers-Pressures-Impact and Responses on the WEFE Nexus, and avoided the details on the “state of the environment” to reduce the complexity of the task for the experts. This approach was previously used by Benitez-Capistros et al. [ 48 ] in combination with the Delphi to investigate proper conservation management strategies in the Galapagos Islands. In the context of this research, the simplified DPRI was found to be the most appropriate approach to explore the relations between socio-ecological systems and the impacts on the four Nexus nodes.

https://doi.org/10.1371/journal.pone.0271443.g002

In the second round of the survey, the experts were introduced to the results of the first round and were asked to revise their answers. Only those questions where no consensus was achieved in the first round were included in the second. For the data analysis of each round, conventional central tendency measures were calculated [ 49 ]. Moreover, different measures of consensus to measure polarisation among individuals were also used: i) the Kendall’s W coefficient of concordance (KW) to rank questions (KW: 0–0.49 for non-consensus, KW: 0.5–1 for consensus) [ 50 ]; ii) the 70% cut-off for agreement and disagreement Likert-scales [ 51 ]; iii) the interquartile range (IQR) and Kendall’s W for the relevance and prioritisation of responses (IQR≤1 for consensus) [ 49 ].

The experts were identified using a snowball technique, based on: i) their professional expertise in one or more of the WEFE Nexus nodes; ii) their familiarity with the area of Apokoronas; iii) their involvement in the integrated management of natural resources [ 52 ]. In Delphi studies, samples are assessed based on the representativeness and expertise of the participants, rather than their numbers [ 43 ]. Participants were selected so that the group composition could reflect the diversity of knowledge stemming from different WEFE sectors and societal expressions. The total number of invited participants was 20, with five participants for each WEFE Nexus sector. Approximately one-third of those invited took part in the first round (n = 7). The absence of attrition rate between rounds suggests that the questionnaire was relevant to the experts and successful in terms of addressing the complexity of the WEFE Nexus thinking [ 53 ]. Regarding the optimal number of Delphi participants, there is enormous variability across studies [ 54 ]. Previous Delphi studies on the Nexus [ 21 , 22 ] and natural resource management [ 46 , 48 ] have used a similar number of participants. The chosen panel reflected the required balanced composition among WEFE sectors and was highly knowledgeable of the case study area. For a more detailed presentation of participants’ characteristics, please refer to S1 Table .

2.3 Focus groups methodology

Six focus groups were organised in the municipality of Apokoronas in Crete, which included thirty-nine participants from the tourism, food services, agricultural and processing sectors, as well as representatives of environmental groups and women’s collectives. The categories were defined based on the literature review and informal discussions. Each focus group comprised four to seven participants. Thirty-seven participants were of Greek ancestry, one was originally from a Balkan state and another from a country in Western Europe. All participants were born or had been living in Crete for over 30 years. The majority had completed their secondary education, attained their High-School diploma, or had higher education (university or technical training) degrees. Many participants held two jobs. For example, they might have owned a small business or worked seasonally in the tourist industry, and farmed their land during the winter months.

The focus group guide followed the structure described in Fig 2 and included questions that concern the interlinkages among the Nexus nodes in the region, and the conflicts between diverse local stakeholders and policy issues. The duration of the focus group discussions ranged from one hour to one hour and a half. The research team was composed of three researchers with extensive knowledge of the context in question. In most cases, the participants knew each other, and there was already trust and intimacy among them. The audiotaped focus group (FG) discussions were transcribed verbatim, cross-checked for accuracy and analysed using Thematic Analysis (TA) procedures, as outlined by Braun and Clarke [ 55 ]. The analysis proceeded through the following steps: familiarisation with the data, recognition of patterns across the data and the identification of themes. Excerpts from the focus groups are used in order to strengthen the line of argumentation.

2.4 Ethical approval and informed consent

The research was performed as part of the SIGMA-Nexus project, a research project funded under the European Commission H2020-PRIMA-2019 call (ID1943). The research project as a whole, including the FG and the DS, was approved by the Research Ethics Committee of the University of Crete (REC-UOC, A.P. 220/15.12.2020). The data for both FG and DS were collected during the summer months of 2021. For the Delphi survey, the online software Questback was used. The focus groups were held in person either in local restaurants or in a venue at the town hall of Apokoronas. In both cases, the participants were asked to read and sign an informed consent and respond to demographic questions.

3.1.1 Sustainable development priorities.

After responding to demographic and expertise-related questions in the first section of the survey, the experts were asked to rank in order of importance a list of socio-economic and environmental objectives (SDG targets) for the area of Apokoronas ( Fig 2 ). The targets, strongly related to the WEFE Nexus domain, were adapted by Malagó et al. [ 31 ]. The idea behind this question is that interconnections among Nexus nodes cannot be identified without taking into consideration local development objectives and policy priorities [ 19 ]. The latter provides a benchmark for assessing the fit and relevance of specific Nexus practices. The SDG targets, in particular, were chosen because of their dual global and local relevance, and their connection with the WEFE Nexus [ 7 ]. As discussed in Section 2.2, the panel’s agreed ranking of priorities was assessed using Kendall’s W, a non-parametric statistic for assessing agreement among panellists [ 49 ]. While an overall consensus among experts was reached for the socio-economic objectives in round one, the consensus threshold for the environmental objectives was reached in the second round. As shown in Fig 3 , in terms of socio-economic objectives, expert opinion gravitated towards the prioritisation of those targets related to sustainable agriculture and food production.

KW after the first round was 0.53 (p < 0.001).

https://doi.org/10.1371/journal.pone.0271443.g003

These two were ranked higher compared to sustainable tourism development, which was quite low on the list. Increasing the employment opportunities and income of small-scale producers were also ranked quite high among the proposed objectives for Apokoronas. This was reinforced by a panellist, who stressed the need to ‘enhance employment in the primary sector as a socio-economic solution for improving the management of resources in the area’ . This seems to be related to the fact that, in Crete, agriculture has become a part-time activity coexisting with seasonal work in the tourism industry. It has been observed that farmers in the area do not prefer farming methods that require full-time commitment or special management practices and, as such, they are subject to income diversification [ 56 ].

In terms of environmental objectives, all targets related to water security were considered a priority. Reduction of water pollution and integrated water resource management were prioritised, followed by the need to increase water use efficiency in agriculture. The role that water and land ecosystems play in the achievement of such goals was also recognised. Fig 4 shows how, between the first and second rounds, the objectives related to protecting the ecosystems and increasing climate resilience moved higher in the ranking, at the expense of those related to the sustainable management of wastewater and clean energy.

KW after the second round was 0.54 (p < 0.001).

https://doi.org/10.1371/journal.pone.0271443.g004

3.1.2 Drivers and impacts.

During the key informant interviews, the experts identified agriculture, tourism, land-use changes, urbanisation and transition to more water-intensive crops as the main drivers of change in Apokoronas. In the Delphi survey, the panel was asked to rate the impact of these five factors on the WEFE sectors. The experts agreed after the first round on the role that the different drivers play in the WEFE. As shown in Fig 5 , according to the experts, tourism development and the transition to more water-intensive (and profitable) crops have the most impact on the WEFE Nexus in Apokoronas.

Mean of the rating on a scale from 1 to 5, where 1 is "very low impact" and 5 is "very high impact". The coefficient of variation was below 0.3 for all ratings.

https://doi.org/10.1371/journal.pone.0271443.g005

3.1.3 Pressures.

Thereafter, the experts were presented with a list of statements to identify the main pressures on the WEFE Nexus and their underlying driving forces. The experts were asked to indicate their level of agreement or disagreement with the proposed statements. The final list of pressures, following the second round of elicitation, is reported in Table 1 . Surprisingly, only 29% (2 out of 7 experts) agreed on tourism development as a driver of pressure on WEFE resources, while opinions about the role of land management practices on the Nexus were mixed.

https://doi.org/10.1371/journal.pone.0271443.t001

The experts have confirmed the presence of increased tension among different water uses (e.g., domestic, tourism and agriculture). They have also confirmed that water stress mainly stems from the inadequate management of water resources, rather than being the result of water scarcity. This corroborates a previous observation by Chartzoulakis et al. [ 57 ] and Tzanakakis et al. [ 34 ].

Further, the role of tourism on the WEFE in Apokoronas was viewed with a high degree of uncertainty. The plurality of opinions expressed can be related to the fact that none of the experts specialised in tourism, but also to the widespread recognition of tourism as an instrument to boost ecosystem preservation, especially in the case of Lake Kournas. This ambiguity was articulated by one expert in the following terms: ’the tourist development of the lake will contribute to its preservation , but not to its upgrading’ . In addition, opinions about climate change, already seen to affect the WEFE resource base and, in particular, agricultural production, were also mixed.

Regarding energy, the key informant interviews pointed out the existence of a link between water and energy production. The panel confirmed the synergy between the two sectors, as well as the potential for this relationship to be exploited in the near future. Nevertheless, some experts also highlighted how rising energy prices limit the development of agricultural holdings, arguing for the promotion of desalination units and, hence, alternative water sources. The relationship between the two sectors is, thus, not seen in terms of trade-offs, but rather in terms of synergies. Despite the fact that the energy sector as a whole does not seem to be highly impacted by the rest of the sectors and is certainly not viewed as an issue of concern for the area by the experts, a plurality of opinions exists about the level of energy use efficiency in agriculture. Another point of disagreement can be found in relation to the adoption of sustainable farming practices, and in particular organic farming. One expert commented on the ’negative impact of agriculture on the ecosystems’ with more sustainable farming practices such as ’organic agriculture’ failing to scale up because of the absence of ’the expected economic results’ and ’environmental consciousness for the majority of farmers’ .

Table 1 reports the underlying causes of pressure on the WEFE Nexus, as identified by the experts. Among 14 proposed statements, seven reached relevant statistical consensus (APMO above 70%). The threshold was not reached in those questions describing the role of farming activities in the WEFE Nexus, most probably because a higher level of knowledge in agriculture was required. Nevertheless, after the second round, the experts converged to the most popular opinions in most cases. The panel did not reach a definitive consensus on whether current water, agricultural, energy and environmental policies are sufficiently integrated taking into account impacts to other sectors. By contrast, they agreed that the existing instruments do not adequately address the impact of climate change on WEFE systems. The panel also argued that there is a lack of coordination among WEFE authorities affecting resource management in the area. In addition, they agreed on the following underlying causes of pressure: local tourism development strategies do not take into account the impact of tourism operations on WEFE systems to any satisfactory degree; the water network is near-obsolete, water losses and reckless use of water are among the main reasons for water use inefficiency; lack of accurate and reliable data, and cumbersome water legislation and pricing mechanisms.

3.1.4 Responses.

In the final section of the survey, a list of 24 suggestions for increasing WEFE Nexus integration and addressing the challenges previously identified in the survey was proposed to the panellists. The experts were asked to rate the relevance of two groups of actions: (a) governance and policy-related responses, and (b) socio-economic and technical responses. In terms of consensus, low heterogeneity was found among expert opinions. In the first round, only six out of twenty-four responses showed an IQR value greater than or equal to 1. After the second round, the experts converged towards the majority opinion and the six innovations registered a reduction in their IQR range. This means that all responses were considered to have certain relevance by the experts.

In the second round, the panel was asked to rank on a scale of 1 to 12, in order of importance, the responses in the two groups. Considering the low heterogeneity of opinions observed in the first round, this ranking aimed to test, using Kendall’s W, the experts’ agreement on their prioritisation. The final ranking is reported in Figs 6 and 7 . For the governance and policy-related responses, a Kendall’s W of 0.43 was obtained. This indicates that despite all suggestions being relevant, a low-moderate level of agreement exists on their prioritisation for WEFE integrated resource management. This disagreement is statistically significant. A higher agreement level was found in the ranking of the socio-economic and technical innovations. The ratings showed a statistically significant Kendall’s W of 0.62.

Experts’ ranking on a scale from 1 to 12 (KW: 0.43).

https://doi.org/10.1371/journal.pone.0271443.g006

Experts’ ranking on a scale from 1 to 12 (KW: 0.62).

https://doi.org/10.1371/journal.pone.0271443.g007

Overall, a number of responses aimed at improving resource management was rated as extremely relevant, such as: i) upgrading water infrastructure; ii) enhancing the management of protected areas (e.g. Natura 2000 sites); iii) developing a WEFE Nexus resource management plan; iv) collecting, in a consolidated database, detailed records of natural resource use by sector; and v) increasing the use of renewable energy sources in the water, food and tourism sectors. Raising awareness by disclosing information about resource use and status was also considered of extreme relevance, followed by the need to create economic incentives for farmers to switch to more sustainable agricultural practices, and the need to implement programmes focusing on environmental education. This is in line with a comment by an expert arguing that ’ the beliefs of the local population on the state of natural resources do not align with the effective conditions of the Nexus resource base ’. Equally interesting is the importance given to the efficient management of protected areas, referring in particular to the Natura 2000 site of Lake Kournas. This suggests that, according to the panel, conservation efforts are relevant for the area. Similarly, the experts highlighted the importance of developing sustainability standards on the part of the farming and tourism sectors.

3.2 Focus groups

The analyses from the FG discussions in the Apokoronas area revealed the following thematic units: i) WEFE Nexus understandings, nature’s endowments and "the water paradox"; ii) Sustainability, existing policies, proposed amendments and enforcement obstacles. Fig 8 visually depicts the first thematic entity and the interconnections among the WEFE Nexus nodes resulting from the focus group analysis, while Table 2 presents the second thematic entity, namely the policy recommendations and solutions to the natural resource sustainability challenges identified by the participants.

https://doi.org/10.1371/journal.pone.0271443.g008

https://doi.org/10.1371/journal.pone.0271443.t002

3.2.1 WEFE Nexus understandings, nature’s endowments and "the water paradox".

Participants provided multiple examples regarding the connections between different nodes of the WEFE Nexus. For example, some of the farmers from the Armenoi village are forced to use brackish water from the Zourbos springs in order to irrigate their avocados, which is considered to affect production. In another example, as an olive-producing municipality, Apokoronas’ many oil producers described how the biofuel produced by processing olive kernels is used to generate energy for heating their homes or running their businesses. Olive kernel biofuel is also used to produce energy to pump water from wells. In terms of economic profitability, the transition to more profitable but also more water-intensive crops (mainly avocado) is increasing concerns regarding the impact of these practices on the Water-Ecosystems-Food sectors of the Nexus. Moreover, despite being a predominantly olive- and dairy-producing municipality, proper management of olive mill wastewater (OMWW) and cheese whey wastewater (CWW) is inconsistent and not regularly enforced in Apokoronas. Focus group discussions veered to the impact of the olive mill by-product, Katsigaros . Participants described how its runoff into aquifers and soils leads to local ecosystem degradation. Along the same lines, participants mentioned how dumping cheese whey wastewater (CWW) is another common practice in the region that adversely affects the local ecosystems.

From the multitude of examples presented in Fig 8 , this analysis will elaborate on three that dominated the focus group discussions: the overuse of plastic bottles, wastewater treatment and renewable energy. The first example concerns the impact of the overuse of bottled water on WEFE nodes. In all the FGs, participants described Apokoronas as an area that is blessed, endowed and privileged by nature, with ample water sources. They expressed their opinion on all the nodes of the Nexus, but discussions were prominently water-centred since water issues are vital for the area. Nevertheless, participants described the fact that their land is ’rich in water’ and endowed by nature, yet faces a multitude of problems related to water management, as ’a paradox’.

Overuse of plastic bottles is the result of the poor quality of tap water, forcing participants to resort to bottled water. There was a sense of injustice that permeated these discussions in the FG, particularly since an award-winning brand of water is bottled and distributed to the rest of Greece and abroad from springs in the area (Lefka Ori, the village of Stylos). Their biggest concern, however, was the deterioration in water quality and water resources, rendering tap water unusable and leading participants to buy expensive bottled water in bulk. As proper recycling policies and the corresponding mentality are lacking, participants described how dependency on bottled water leads to plastic inundation. Plastic gets dumped into the water sources and affects marine, lake and river ecosystems, while microplastics are entering the food chain, accumulating in the bodies of birds and other animals.

Regarding wastewater, participants stated that a wastewater treatment plant (WWTP) had been in the making for years. As many local businesses dump their sewage directly in the water sources and in the natural environment, many nodes of the WEFE Nexus are adversely impacted. This practice has immense negative effects on marine habitats, food quality and food production, as well as environmental sustainability. Concerning wastewater treatment, many participants believed that big businesses had significant advantages over smaller family-owned enterprises. This is a serious point of contention because it affects households and family-run businesses in the entire area in question.

According to the FG participants from the tourism sector, large hotels are partially exempt because many of them have the capacity to process sewage for at least some of their units. Smaller hotels and businesses do not have sewage processing capabilities and shoulder hefty costs for transporting water sewage to another municipality’s sewage processing plant.

In general, there was homogeneity of opinions in the discussions, but there was disagreement over whether small businesses could adopt such sustainable practices. Vassilis, Stelios and Giorgos, whose real names are not revealed for confidentiality purposes, from the tourism focus group vehemently disagreed:

Vassilis : You are obliged to do something about sustainable practices . Stelios : It’s not your obligation , it’s the obligation of the municipality . The municipality is the starting point […] it is the base . For me , the problem starts from your base . If your base does not have a wastewater treatment plant , they can’t come to me and say , ’why do you dump your sewage ? ’ . Giorgos : Since you are obliged to have your own wastewater treatment plant , I can’t understand you . Stelios : Since we have reached this impasse … Vassilis : And what can one do with your wastewater ? Giorgos : And how are you going to function ? Don’t start your business ? If you don’t you confront a more complex predicament .

Vassilis and Giorgos are representatives of large resort hotels, which have facilities such as solar photovoltaic panels and sewage/wastewater treatment plants. They also reuse water to grow some crops that are then used in their kitchens. They explained that not all wastewater is processed due to the limited capacity of the sewage treatment systems they have. In this dialogue, they both claimed that wastewater treatments plants are primarily privately owned facilities and should be ’obligatory for all’ businesses. They both pressed Stelios by persistently interrupting him and posing questions about his responsibilities. Stelios, the owner of a smaller business, strongly disagreed and argued that establishing such practices is not financially possible for his business, and that the management of wastewater should be the responsibility of the municipality. This verbal encounter reflects wider societal conflicts regarding power inequalities, state responsibilities and the capacity of smaller tourist establishments to adopt sustainable practices.

The differences of opinion also centred on renewable energy. Wind energy was a point of contention because many participants believe that wind turbines will lead to land fragmentation (construction or roads in mountainous and forest landscapes), change of land use at the expense of the ecosystems, and are noisy and aesthetically unappealing on mountain peaks. Other participants focused on the low operating costs of wind farms, Crete’s “Meltemi winds” and the White Mountain range in the case study area that forms a natural barrier for winds. Solar energy via solar-PV installations also drew differences in opinion with some participants expressing beliefs that such installations compete with farmland, “take up lots of land” and the panels’ efficiency levels are relatively low and are hampered by storage difficulties.

In the participants’ examples of sustainable (e.g., wastewater recycling) and unsustainable (e.g., the preponderance of bottled water use) practices, the interconnections between the WEFE Nexus nodes were highlighted. The majority of the examples that participants offered focused on the interconnections between the water and food nodes and their crucial role in farming. Despite the fact that many focus group questions pertained to the significance of energy and ecosystems, participants were less reflexive about these two nodes and their impact on their livelihoods. Moreover, participants were hesitant in delving into the complicated relationship between tourism’s contributions to economic growth and its negative environmental and sustainable use of natural resources.

3.2.2 Sustainability, existing policies, proposed amendments and enforcement obstacles.

This theme delineates participants’ policy proposals regarding the sustainable use and management of natural resources. Although participants valued Apokoronas’ natural water capital and its diverse ecosystems, they elaborated on their concerns regarding relevant policies and their enforcement. In this theme, the focus is on three vital policy issues: i) the effects of over-tourism; ii) the pricing of water and the future of local agriculture and farming; and iii) the transparency of natural resources management.

The expansion of the ’tourism industry’, according to the residents who live and work in the area, has created more jobs, economic growth, and possibilities for the advancement of people’s well-being. Nevertheless, they believe that this industry contributes disproportionately to natural resource depletion in all the nodes of the WEFE Nexus and, according to some, these deleterious effects are currently evident in the area. Participants discussed the problems involved in resource sustainability, and described the current state of affairs as a voracious consumption of resources. The participants from the water collective focus group noted that most water bodies, including estuaries, the coastal zone and even the sea, have been degraded mainly to assuage short-term economic gains for a few people. As a result, not much attention is paid to the long-term economic and environmental sustainability of the study area. In two discussions regarding "over-tourism", participants proposed "ecotourism", "agro-tourism", and other more environmentally friendly forms of tourism as solutions for the reduction of trade-offs among sectors.

Local farmers noted that they struggle to compete with large multi-national commodity markets and are under pressure from tourism expansion, facing serious challenges to secure public support. They consider their future to be unviable and unsustainable. As shown in Table 2 , one proposed solution is the development and implementation of policies empowering agricultural directorates, agronomists and farming cooperatives. This includes creating a link between local agricultural and livestock production and the local tourism market. Participants noted that the tourist industry has eclipsed their contribution to Apokoronas’ economy. They discussed the fraying of bonds in their rural communities, and disclosed pressures to reinvent farming in a way that balances ecological, social and economic concerns. Farmers outlined their need for programmes that provide support, guidance and know-how about crops’ water needs, and adaptation to climate change, including biological pest control.

Regarding water management and distribution conflicts, participants proposed transparency and alignment with governmental authorities and an equitable water pricing system. They advocate for a water pricing system that is fair for farmers, household consumers, food and processing businesses and the tourist industry. They enumerated many problems, including over-exploitation of water sources during the summer months, low water efficiency (in farming, businesses and homes), absence of a quality monitoring system, lack of information regarding water quality (publication of water analyses data), an outdated and faulty distribution system with frequent leaks and asbestos pipes, salt-water intrusion, inadequate cooperation amongst the different agencies that are responsible for water, and lack of a modern recycling and reuse system.

One of the six focus groups comprised members of a local water collective. In the following dialogue between two members of the water collective, this sense of vagueness regarding the management of natural resources is brought to the foreground:

Alexia : Have we discussed the asbestos pipes ? Giannis : Yes , there are multiple problems , the asbestos pipes , which […] people do not readily acknowledge in our region [the asbestos pipes] are some kilometres , we do not exactly have data , we do not exactly have data . Alexia : It’s all very unclear , this is the problem . Giannis : Yes , that there is misinformation , data concealment , I do not know exactly what’s happening . The truth is that I can’t say something with absolute certainty .

Members of this group, which operates through open assemblies and participatory processes, expressed their ambivalence, bewilderment and uncertainty towards the existence of old water lines made of asbestos, a material that has been phased out. Giannis replied very hesitantly to Alexia’s request to discuss the topic and explained that, although they are informally conducting their own research, they cannot provide clear evidence. This was a dominant interaction pattern in this focus group, where one participant would disclose a piece of information, but would immediately be challenged by other participants on the reliability of their assertion. The members of the group stated that they are considering ways to demand and gain access to water quality data, arguing that water testing results from certified laboratories are not posted publicly. In this respect, they were vocal about their belief that being informed about the quality of water is a democratic right of all citizens.

Policies in the energy sector and land use were also deemed urgent. The participants stated that policies in the energy sector should include the regulation of energy prices and the provision of viable programming by amending the “Save-Spare” initiative from “energy inefficient buildings” to nearly zero-energy buildings. Energy needs should be met from renewable sources, including those produced on-site or locally. As the participants explained, that different “renewable energies that are storable” and that “replace Crete’s reliance on the fossil fuel mazut” are necessary. The focus group discussion on land use brought up the necessity to change zoning policies that were not fully applied and planning laws that are incomplete. The existing zoning policies are “ sporadically and unfairly enforced leading to the construction of tourist facilities on the coastline ”.

4 Discussion

The use of multiple research methodologies is common practice in qualitative research. The triangulation of results from different methods and data sources allows for a more comprehensive understanding of the study object, provides more robust confirmation of findings and includes diverse perspectives [ 32 ]. The use of DS and FGs in this study aimed specifically to explore whether the combination of these methods could capture more holistically the WEFE Nexus interrelations and the perspectives of different stakeholders. As the result of quantification, biophysical flows among Nexus nodes cannot be easily contested; the management of resources, however, is determined by socio-economic needs, and cultural and historical forces that influence subjective perceptions of interconnectedness, and present challenges and opportunities for their sustainable use.

This study applied two different methodologies bringing together two distinct dimensions of the Nexus approach. On the one hand, the study accounted for the technical and managerial dimensions of the Nexus thinking by asking WEFE experts about the main challenges to sustainable resource use, and relevant solutions in light of local development priorities. On the other hand, the study attempted to capture the perceptions and experiences of local populations regarding the use and management of natural resources, including their views on justice and power inequalities. The integration of these two perspectives may lead to more informed WEFE Nexus policy initiatives, which take into account the voices and perspectives of diverse local stakeholders.

The relevance of both methods was considerable to all three research questions. Regarding the first question about how diverse stakeholders understand and frame the interconnections and interdependencies across the WEFE sectors, the comparison revealed multiple perspectives with different points of convergence. Both groups (Delphi panellists and Focus Group participants) confirmed that Nexus thinking in Apokoronas has a strong focus on water. Bridging the gap between water availability and demand and improving water quality were central concerns. The focus groups participants underlined the power inequalities and injustices and highlighted implications for the rest of the Nexus nodes (e.g. overuse of plastic bottles affecting the ecosystems and the negative impact of the use of brackish water for crop production). The Delphi captured the underlying causes of the " water paradox " pervading the WEFE Nexus, which include, among others, the low-level coordination between water and WEFE authorities, the fast-becoming obsolete water network, and inadequate water legislation and pricing mechanisms. Nonetheless, while the need for improving water pricing policies was firmly pointed out by the focus groups participants, who advocated for a more equitable water pricing system, local experts did not consider this a priority. This is probably due to the experts’ concerns about the different levels of jurisdictions that need to come together in order to effect an overhaul of the pricing system.

Concerning the second research question, on how participants suggested improving the WEFE Nexus, several opportunities for a more integrated WEFE resource management in Apokoronas were found. The groups converged in the main ways to address the major Nexus challenges and resource use conflicts. Both groups acknowledged the importance of environmental education and the proper disclosure of information. The latter was raised by the experts, who called for the public availability of all relevant data for decision-making. Focus group participants attributed a more political meaning to information dissemination, referring to the lack of transparency in the use of natural resources. While the Delphi experts identified areas of macrointervention at the management and policy levels (such as improving the management of protected areas or creating a WEFE Nexus management plan), FG participants reflected on specific steps towards change, such as the strengthening of agronomic services, zoning policies and local value chains. A tendency to prefer top-down interventions to bottom-up grassroots movements can be observed in both groups, suggesting that the reduction of complexity and the integrated management of resources are considered the responsibility of policymakers.

Finally, for the third research question on the potential of qualitative methodologies in Nexus designs, elicitation and reflexive thinking inherent in both methods contributed to identifying node interlinkages and resource management challenges for Apokoronas. FG participants could break down complex interactions even if not conceptualising them as a Nexus. The discussions also allowed the detection of already applied Nexus practices in the area. These include, for instance, the production of biofuel as a by-product of olive farming or the reuse of water from hotel wastewater treatment systems. Despite the clarity with which focus group participants showcased the interlinkages of the four nodes in their area, they concentrated mainly on those resource management problems that had a direct impact on their livelihood (e.g., municipal wastewater sewage treatment), while disregarding other ecosystemic challenges (e.g., biodiversity loss) that are not directly relevant to human well-being. The Delphi approach filled this gap, with experts outlining the role of different factors (drivers, pressures, causes) in determining Nexus sustainability as a whole. Delphi experts focused on challenges at the municipality level, but interpreted them in the regional context. They provided an overall interpretation of the variables that need to be considered when conducting WEFE Nexus assessments and, in that way, a sense of the relative importance of each issue was obtained.

On these grounds, it is possible to contend that the valuable insights of both FG participants and Delphi experts were complementary to one another. On a micro-level, FG discussions centred on the economic concerns of Apokoronas’ residents, regarding how the intricate connections between the decline of agricultural production, the outdated zoning policies and over-tourism pervaded the use of natural resources in the area. On a macro-level, the Delphi experts recognised the primary importance of the protection and restoration of ecosystems in the area, as a means to manage sustainably the rest of the nodes in the nexus. Further, by triangulating the results from the two methods, contrasting viewpoints about the impact of certain activities were identified. An example is the case of the tourism sector, which, although not explicitly considered in the WEFE Nexus, dominated the discussions. Delphi experts considered tourism as the sector with the most impact on the four WEFE nodes. Nonetheless, they did not prioritise sustainable tourism over the strengthening of sustainable agricultural production. Such an ambivalent position regarding tourism expansion was also observed among FG participants, who acknowledged that tourism was an important economic sector and, at the same time, the driver behind the depletion of natural resources in the area. To the contrary, agriculture, a prominent marker of local identity, was seen as retreating in the face of tourism as an economic activity. Lack of adequate knowledge and incentives to adopt improved management practices seems to be at the heart of the WEFE unsustainable farming systems and agro-industrial activities. The absence of expected economic benefits and low environmental consciousness reduces the proclivity among farmers to adopt environmentally friendly innovations. While Delphi captured the environmental and social dimensions of farming sustainability, the indignation of farmers who struggled to make ends meet was evident in the FGs.

This study added useful insights regarding the socio-economic and environmental trade-offs between WEFE sectors. Echoing a point made by several scholars [ 13 , 14 , 58 ], it is argued that neglecting the role of social, political and economic forces and local identities in the interlinkages between natural resources and decision-making can be detrimental. A remarkable study in this respect, which accounted for the context-specific, historical perspectives of people in Crete, is the study of Siamanta and Dunlap [ 58 ]. The authors investigated wind energy development in Crete and argued that local opposition to wind parks is deeply associated with the population’s past struggles against foreign powers. Following Siamanta and Dunlap [ 58 ] and ’ in contrast to the rather depoliticised and ahistorical treatment of social order and context in the dominant energy-water-food Nexus literature ’ [ 22 ], this study explored qualitative approaches to situate the participants’ experiences within their proper social, political and economic contexts. Therefore, the argument proposed in this study is that the WEFE Nexus is not a static, abstract, fixed concept, but rather a flexible, dynamic and empirically grounded construct that accounts for local particularities in the use of natural resources.

5 Conclusion

One of the challenges of the Nexus framework is how to bridge the technical and managerial dimensions of the WEFE interconnections, and the priorities and perceptions of local resource users and stakeholders involved in resource management. Since the introduction of the concept, approximately one-quarter of Nexus studies have been based on qualitative methods [ 15 ]. This study moved in the same direction by applying novel combinations of qualitative methodologies in Nexus research designs. Two methods, Delphi and Focus Groups, have been combined to investigate the WEFE Nexus in Apokoronas (Crete) and explored the added potential of this approach.

Each with its strengths and weaknesses, the two methods have proven complementary to each other. The Delphi was designed to overcome silo approaches in Nexus thinking by eschewing confrontation among WEFE sectoral experts. FGs seek participant interaction to stimulate conversation that would lead to the identification of common concerns and the sharing of opinions. Given the complexity of the Nexus discourse, the risk of unbalanced assessments among nodes, and the technical and societal dimensions pervading the different sectors, the combination of these two techniques provided grounds for a more sober analysis. For instance, FG discussions capture motives and perspectives that cannot be observed in an expert-led Delphi survey. The integration of the two methods is particularly suited for local WEFE Nexus assessments seeking to collect information for policy strategies aimed at improving Nexus resource management. The study identified reciprocal causal relationships among Nexus systems, which offer valuable insights for policy interventions and, from a research perspective, can be used to inform the development of further empirical studies using a Nexus approach. It also revealed nuances and different perspectives even in the face of low heterogeneity between and within groups. Based on the above, it is possible to conclude that such a combination of methods can provide useful insights on sectoral and societal conditions even in heavily diversified contexts.

Supporting information

S1 table. delphi study: socio-demographic characteristics of participants..

https://doi.org/10.1371/journal.pone.0271443.s001

Acknowledgments

The authors are grateful for the contributions of all participants from the area of Apokoronas, who participated in the Focus Groups and the Delphi survey. We want to thank the Organization for the Development of Crete S.A. for facilitating the contacts and providing the maps of the area.

• 1. ICIMOD. Contribution of Himalayan ecosystems to water, energy, and food security in South Asia: A nexus approach. Kathmandu: International Centre for Integrated Mountain Development; 2012.
• View Article
• Google Scholar
• 3. Barchiesi S, Carmona-Moreno C, Dondeynaz C, Biedler M. Proceedings of the workshop on Water-Energy-Food-Ecosystems (WEFE) Nexus and Sustainable Development Goals (SDGs). Luxembourg: Publications Office of the European Union; 2018. https://medrc.org/wp-content/uploads/2019/05/19_jrc_procworkshopwaterenergyfoodecosystemsNexusandsdgs.pdf
• 4. Carmona-Moreno C, Dondeynaz C, Biedler M. Position paper on water, energy, food and ecosystems (WEFE) Nexus and sustainable development goals (SDGS). Luxembourg: Publications Office of the European Union; 2019. https://publications.jrc.ec.europa.eu/repository/handle/JRC114177
• 5. World Economic Forum. Water security: The water-food-energy-climate nexus. Washington: The World Economic Forum Water Initiative; 2011.
• 6. UN General Assembly. Resolution adopted by the General Assembly on 25 September 2015. Transforming our World: The 2030 Agenda for Sustainable Development (A/RES/70/1); 2015.
• 8. Hoff H. Understanding the nexus. The Water, Energy and Food Security Nexus Conference. Stockholm: Stockholm Environment Institute; 2011.
• 11. Sood A, Nicol A, Arulingam I. Unpacking the water-energy-environment-food nexus: Working across systems. Colombo, Sri Lanka: International Water Management Institute (IWMI Working Paper 186); 2019.
• PubMed/NCBI
• 26. United Nations Environment Programme / Mediterranean Action Plan and Plan Bleu. State of the Environment and Development in the Mediterranean. Nairobi; 2020. https://planbleu.org/en/soed-2020-state-of-environment-and-development-in-mediterranean/
• 33. ELSTAT (Hellenic Statistical Authority). Population-Housing Census; 2011. https://www.statistics.gr/2011-census-pop-hous .
• 36. EC (European Commission). NATURA 2000—Standard Data Form, Site GR4340010; 2020. URL: https://natura2000.eea.europa.eu/Natura2000/SDF.aspx?site=GR4340010
• 38. Canessa C, Halytsia O, Abdelradi F, Bazdanis G, Nikolaou T, Sauer J, Vardoulaki J, Vrachioli M. Identification of economic activities and social considerations in WEF Nexus. Deliverable 1.1. SIGMA Nexus; 2021. https://sigma-nexus.eu3
• 39. Dalkey NC. The Delphi method: An experimental study of group opinion. Santa Monica: Rand Corporation; 1969.
• 42. Turoff M, Linstone HA. The Delphi method: Techniques and applications. Reading, Mass.: Addison-Wesley; 2002.
• 47. Kristensen P. The DPSIR framework. In: Workshop on a comprehensive assessment of the vulnerability of water resources to environmental change in Africa using river basin approach. Nairobi: UNEP; 2004. https://wwz.ifremer.fr/dce/content/download/69291/913220/…/DPSIR.pdf
• 51. Kapoor P. A systems approach to documentary maritime fraud. Plymouth: Plymouth Polytechnic; 1987.
• 52. Bogner A, Littig B, Menz W. Interviewing experts. London: Palgrave Macmillan; 2009.

To read this content please select one of the options below:

Please note you do not have access to teaching notes, case study: application of enhanced delphi method for software development and evaluation in medical institutes.

ISSN : 0368-492X

Article publication date: 4 April 2016

In recent years, many development projects of the medical systems encounter difficulties and eventually fail. Failure is often due to very complicated and changeable medical procedures and the inconsistent understanding between system stakeholders, especially the healthcare providers, and information technology staff. Many research results also indicate that poor communication easily results in negative consequences during the implementation of the medical information system. To effectively overcome this obstacle, the purpose of this paper is to propose an enhanced Delphi method to assist in reaching consensus during the software development with some additional steps.

Design/methodology/approach

As an alternative to the traditional way to elicit pertinent feedback from respondents, the enhanced Delphi method stresses the systematic, flexible, and cyclic stages to construct a questionnaire with viewpoints from different types of panelists and a self-assessment procedure as a validating step to measure the improvements in the system implementation.

The better communication between the members of project team does increase the comprehensive assessment of a project.

Originality/value

Based on a practical case, the enhanced Delphi method really demonstrates good performance and effectiveness.

• Communication
• Enhanced Delphi method
• Software evaluation

Yang, T.-H. , Ku, C.-Y. and Liu, M.-N. (2016), "Case study: Application of enhanced Delphi method for software development and evaluation in medical institutes", Kybernetes , Vol. 45 No. 4, pp. 637-649. https://doi.org/10.1108/K-03-2015-0084

Emerald Group Publishing Limited

Copyright © 2016, Emerald Group Publishing Limited

Related articles

We’re listening — tell us what you think, something didn’t work….

Report bugs here

All feedback is valuable

Please share your general feedback

Join us on our journey

Platform update page.

Visit emeraldpublishing.com/platformupdate to discover the latest news and updates

Questions & More Information

Answers to the most commonly asked questions here

• Topic 6: Delphi method – Case studies

by | Apr 30, 2019 | Uncategorized

Find out the results of the Delphi method in the completed research project

• Keywords: Delphi method, Energy Future 2040, Methodology, Future scenario
• Est. time of completion: 30 min

If the presentation is not displaying properly, please visit this  link

← Previous Topic Next Topic →

Course Navigation

• Topic 1: Tackling global challenges
• Topic 2: Introduction to business foresight
• Topic 3: Introduction to explore foresight in research
• Topic 4: Foresight for research (a case study)
• S1 Lesson Summary and References
• Topic 1: Exploring futures with Framework Foresight
• Topic 2: Patent Analysis for technology forecasting
• Topic 3: Time serie analysis
• Topic 4: Text mining in foresight studies
• Topic 5: Roadmapping – Case studies
• Topic 7: Intuitive logics school of scenario construction – Case studies
• S2 Lesson Summary and References
• S2 Lesson Quiz
• Topic 1: From Futures to Strategy – Fail fast
• Topic 2: From Futures to Strategy – Fail cheap
• S3 Lesson Summary and References

Foresight Course

Return to the Foresight Course page.