research project guidelines

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• HHS Author Manuscripts

How to Conduct Responsible Research: A Guide for Graduate Students

Alison l. antes.

1 Department of Medicine, Division of General Medical Sciences, Washington University School of Medicine, St. Louis, Missouri, 314-362-6006

Leonard B. Maggi, Jr.

2 Department of Medicine, Division of Molecular Oncology, Siteman Cancer Center, Washington University School of Medicine, St. Louis, Missouri, 314-362-4102

Researchers must conduct research responsibly for it to have an impact and to safeguard trust in science. Essential responsibilities of researchers include using rigorous, reproducible research methods, reporting findings in a trustworthy manner, and giving the researchers who contributed appropriate authorship credit. This “how-to” guide covers strategies and practices for doing reproducible research and being a responsible author. The article also covers how to utilize decision-making strategies when uncertain about the best way to proceed in a challenging situation. The advice focuses especially on graduate students but is appropriate for undergraduates and experienced researchers. The article begins with an overview of the responsible conduct of research, research misconduct, and ethical behavior in the scientific workplace. The takeaway message is that responsible conduct of research requires a thoughtful approach to doing research to ensure trustworthy results and conclusions and that researchers receive fair credit.

INTRODUCTION

Doing research is stimulating and fulfilling work. Scientists make discoveries to build knowledge and solve problems, and they work with other dedicated researchers. Research is a highly complex activity, so it takes years for beginning researchers to learn everything they need to know to do science well. Part of this large body of knowledge is learning how to do research responsibly. Our purpose in this article is to provide graduate students a guide for how to perform responsible research. Our advice is also relevant to undergraduate researchers and for principal investigators (PIs), postdocs, or other researchers who mentor beginning researchers and wish to share our advice.

We begin by introducing some fundamentals about the responsible conduct of research (RCR), research misconduct, and ethical behavior. We focus on how to do reproducible science and be a responsible author. We provide practical advice for these topics and present scenarios to practice thinking through challenges in research. Our article concludes with decision-making strategies for addressing complex problems.

What is the responsible conduct of research?

To be committed to RCR means upholding the highest standards of honesty, accuracy, efficiency, and objectivity ( Steneck, 2007 ). Each day, RCR requires engaging in research in a conscientious, intentional fashion that yields the best science possible ( “Research Integrity is Much More Than Misconduct,” 2019 ). We adopt a practical, “how-to” approach, discussing the behaviors and habits that yield responsible research. However, some background knowledge about RCR is helpful to frame our discussion.

The scientific community uses many terms to refer to ethical and responsible behavior in research: responsible conduct of research, research integrity, scientific integrity, and research ethics ( National Academies of Science, 2009 ; National Academies of Sciences Engineering and Medicine, 2017 ; Steneck, 2007 ). A helpful way to think about these concepts is “doing good science in a good manner” ( DuBois & Antes, 2018 ). This means that the way researchers do their work, from experimental procedures to data analysis and interpretation, research reporting, and so on, leads to trustworthy research findings and conclusions. It also includes respectful interactions among researchers both within research teams (e.g., between peers, mentors and trainees, and collaborators) and with researchers external to the team (e.g., peer reviewers). We expand on trainee-mentor relationships and interpersonal dynamics with labmates in a companion article ( Antes & Maggi, 2021 ). When research involves human or animal research subjects, RCR includes protecting the well-being of research subjects.

We do not cover all potential RCR topics but focus on what we consider fundamentals for graduate students. Common topics covered in texts and courses on RCR include the following: authorship and publication; collaboration; conflicts of interest; data management, sharing, and ownership; intellectual property; mentor and trainee responsibilities; peer review; protecting human subjects; protecting animal subjects; research misconduct; the role of researchers in society; and laboratory safety. A number of topics prominently discussed among the scientific community in recent years are also relevant to RCR. These include the reproducibility of research ( Baker, 2016 ; Barba, 2016 ; Winchester, 2018 ), diversity and inclusion in science ( Asplund & Welle, 2018 ; Hofstra et al., 2020 ; Meyers, Brown, Moneta-Koehler, & Chalkley, 2018 ; National Academies of Sciences Engineering and Medicine, 2018a ; Roper, 2019 ), harassment and bullying ( Else, 2018 ; National Academies of Sciences Engineering and Medicine, 2018b ; “ No Place for Bullies in Science,” 2018 ), healthy research work environments ( Norris, Dirnagl, Zigmond, Thompson-Peer, & Chow, 2018 ; “ Research Institutions Must Put the Health of Labs First,” 2018 ), and the mental health of graduate students ( Evans, Bira, Gastelum, Weiss, & Vanderford, 2018 ).

The National Institutes of Health (NIH) ( National Institutes of Health, 2009 ) and the National Science Foundation ( National Science Foundation, 2017 ) have formal policies indicating research trainees must receive education in RCR. Researchers are accountable to these funding agencies and the public which supports research through billions in tax dollars annually. The public stands to benefit from, or be harmed by, research. For example, the public may be harmed if medical treatments or social policies are based on untrustworthy research findings. Funding for research, participation in research, and utilization of the fruits of research all rely on public trust ( Resnik, 2011 ). Trustworthy findings are also essential for good stewardship of scarce resources ( Emanuel, Wendler, & Grady, 2000 ). Researchers are further accountable to their peers, colleagues, and scientists more broadly. Trust in the work of other researchers is essential for science to advance. Finally, researchers are accountable for complying with the rules and policies of their universities or research institutions, such as rules about laboratory safety, bullying and harassment, and the treatment of animal research subjects.

What is research misconduct?

When researchers intentionally misrepresent or manipulate their results, these cases of scientific fraud often make the news headlines ( Chappell, 2019 ; O’Connor, 2018 ; Park, 2012 ), and they can seriously undermine public trust in research. These cases also harm trust within the scientific community.

The U.S. defines research misconduct as fabrication, falsification, and plagiarism (FFP) ( Department of Health and Human Services, 2005 ). FFP violate the fundamental ethical principle of honesty. Fabrication is making up data, and falsification is manipulating or changing data or results so they are no longer truthful. Plagiarism is a form of dishonesty because it includes using someone’s words or ideas and portraying them as your own. When brought to light, misconduct involves lengthy investigations and serious consequences, such as ineligibility to receive federal research funding, loss of employment, paper retractions, and, for students, withdrawal of graduate degrees.

One aspect of responsible behavior includes addressing misconduct if you observe it. We suggest a guide titled “Responding to Research Wrongdoing: A User-Friendly Guide” that provides advice for thinking about your options if you think you have observed misconduct ( Keith-Spiegel, Sieber, & Koocher, 2010 ). Your university will have written policies and procedures for investigating allegations of misconduct. Making an allegation is very serious. As Keith-Spiegel et al.’s guide indicates, it is important to know the evidence that supports your claim, and what to expect in the process. We encourage, if possible, talking to the persons involved first. For example, one of us knew of a graduate student who reported to a journal editor their suspicion of falsified data in a manuscript. It turned out that the student was incorrect. Going above the PI directly to the editor ultimately led to the PI leaving the university, and the student had a difficult time finding a new lab to complete their degree. If the student had first spoken to the PI and lab members, they could have learned that their assumptions about the data in the paper were wrong. In turn, they could have avoided accusing the PI of a serious form of scientific misconduct—making up data—and harming everyone’s scientific career.

What shapes ethical behavior in the scientific workplace?

Responsible conduct of research and research misconduct are two sides of a continuum of behavior—RCR upholds the ideals of research and research misconduct violates them. Problematic practices that fall in the middle but are not defined formally as research misconduct have been labeled as detrimental research practices ( National Academies of Sciences Engineering and Medicine, 2017 ). Researchers conducting misleading statistical analyses or PIs providing inadequate supervision are examples of the latter. Research suggests that characteristics of individual researchers and research environments explain (un)ethical behavior in the scientific workplace ( Antes et al., 2007 ; Antes, English, Baldwin, & DuBois, 2018 ; Davis, Riske-Morris, & Diaz, 2007 ; DuBois et al., 2013 ).

These two influences on ethical behavior are helpful to keep in mind when thinking about your behavior. When people think about their ethical behavior, they think about their personal values and integrity and tend to overlook the influence of their environment. While “being a good person” and having the right intentions are essential to ethical behavior, the environment also has an influence. In addition, knowledge of standards for ethical research is important for ethical behavior, and graduate students new to research do not yet know everything they need to. They also have not fully refined their ethical decision-making skills for solving professional problems. We discuss strategies for ethical decision-making in the final section of this article ( McIntosh, Antes, & DuBois, 2020 ).

The research environment influences ethical behavior in a number of ways. For example, if a research group explicitly discusses high standards for research, people will be more likely to prioritize these ideals in their behavior ( Plemmons et al., 2020 ). A mentor who sets a good example is another important factor ( Anderson et al., 2007 ). Research labs must also provide individuals with adequate training, supervision and feedback, opportunities to discuss data, and the psychological safety to feel comfortable communicating about problems, including mistakes ( Antes, Kuykendall, & DuBois, 2019a , 2019b ). On the other hand, unfair research environments, inadequate supervision, poor communication, and severe stress and anxiety may undermine ethical decision-making and behavior; particularly when many of these factors exist together. Thus, (un)ethical behavior is a complex interplay of individual factors (e.g., personality, stress, decision-making skills) and the environment.

For graduate students, it is important to attend to what you are learning and how the environment around you might influence your behavior. You do not know what you do not know, and you necessarily rely on others to teach you responsible practices. So, it is important to be aware. Ultimately, you are accountable for your behavior. You cannot just say “I didn’t know.” Rather, just like you are curious about your scientific questions, maintain a curiosity about responsible behavior as a researcher. If you feel uncomfortable with something, pay attention to that feeling, speak to someone you trust, and seek out information about how to handle the situation. In what follows, we cover key tips for responsible behavior in the areas of reproducibility and authorship that we hope will help you as you begin.

HOW TO DO REPRODUCIBLE SCIENCE

The foremost responsibility of scientists is to ensure they conduct research in such a manner that the findings are trustworthy. Reproducibility is the ability to duplicate results ( Goodman, Fanelli, & Ioannidis, 2016 ). The scientific community has called for greater openness, transparency, and rigor as key remedies for lack of reproducibility ( Munafò et al., 2017 ). As a graduate student, essential to fostering reproducibility is the rigor of your approach to doing experiments and handling data. We discuss how to utilize research protocols, document experiments in a lab notebook, and handle data responsibly.

Utilize research protocols

1. learn and utilize the lab’s protocols.

Research protocols describe the step-by-step procedures for doing an experiment. They are critical for the quality and reproducibility of experiments. Lab members must learn and follow the lab’s protocols with the understanding that they may need to make adjustments based on the requirements of a specific experiment.

Also, it is important to distinguish between the experiment you are performing and analyzing the data from that experiment. For example, the experiment you want to perform might be to determine if loss of a gene blocks cell growth. Several protocols, each with pros and cons, will allow you to examine “cell growth.” Using the wrong experimental protocol can produce data that leads to muddled conclusions. In this example, the gene does block cell growth, but the experiment used to produce the data that you analyze to understand cell growth is wrong, thus giving a result that is a false negative.

When first joining a lab, it is essential to commit to learning the protocols necessary for your assigned research project. Researchers must ensure they are proficient in executing a protocol and can perform their experiments reliably. If you do not feel confident with a protocol, you should do practice runs if possible. Repetition is the best way to work through difficulties with protocols. Often it takes several attempts to work through the steps of a protocol before you will be comfortable performing it. Asking to watch another lab member perform the protocol is also helpful. Be sure to watch closely how steps are performed, as often there are minor steps taken that are not written down. Also, experienced lab members may do things as second nature and not think to explicitly mention them when working through the protocol. Ask questions of other lab members so that you can improve your knowledge and gain confidence with a protocol. It is better to ask a question than potentially ruin a valuable or hard-to-get sample.

Be cautious of differences in the standing protocols in the lab and how you actually perform the experiment. Even the most minor deviations can seriously impact the results and reproducibility of an experiment. As mentioned above, often there are minor things that are done that might not be listed in the protocol. Paying attention and asking questions are the best ways to learn, in addition to adding notes to the protocol if you find minor details are missing.

2. Develop your own protocols

Often you will find that a project requires a protocol that has not been performed in the lab. If performing a new experiment in the lab and no protocol exists, find a protocol and try it. Protocols can be obtained from many different sources. A great source is other labs on campus, as you can speak directly to the person who performs the experiment. There are many journal sources as well, such as Current Protocols, Nature Protocols, Nature Methods, and Cell STAR Methods . These methods journals provide the most detailed protocols for experiments often with troubleshooting tips. Scientific papers are the most common source of protocols. However, keep in mind that due to the common brevity of methods sections, they often omit crucial details or reference other papers that may not contain a complete description of the protocol.

3. Handle mistakes or problems promptly

At some point, everyone encounters problems with a protocol, or realizes they made a mistake. You should be prepared to handle this situation by being able to detail exactly how you performed the experiment. Did you skip a step? Shorten or lengthen a time point? Did you have to make a new buffer or borrow a labmate’s buffer? There are too many ways an experiment can go wrong to list here but being able to recount all the steps you performed in detail will help you work through the problem. Keep in mind that often the best way to understand how to perform an experiment is learning from when something goes wrong. This situation requires you to critically think through what was done and understand the steps taken. When everything works perfectly, it is easy to pay less attention to the details, which can lead to problems down the line.

It is up to you to be attentive and meticulous in the lab. Paying attention to the details may feel like a pain at first, or even seem overwhelming. Practice and repetition will help this focus on details become a natural part of your lab work. Ultimately, this skill will be essential to being a responsible scientist.

Document experiments in a lab notebook

1. recognize the importance of a lab notebook.

Maintaining detailed documentation in a lab notebook allows researchers to keep track of their experiments and generation of data. This detailed documentation helps you communicate about your research with others in the lab, and serves as a basis for preparing publications. It also provides a lasting record for the lab that exists beyond your time in the lab. After graduate students leave the lab, sometimes it is necessary to go back to the results of older experiments. A complete and detailed notebook is essential, or all of the time, effort, and resources are lost.

2. Learn the note-keeping practices in your lab

When you enter a new lab, it is important to understand how the lab keeps notebooks and the expectations for documentation. Being conscientious about documentation will make you a better scientist. In some labs, the PI might routinely examine your notebook, while in other labs you may be expected to maintain a notebook, but it may not be regularly viewed by others. It is tempting to become relaxed in documentation if you think your notebook may not be reviewed. Avoid this temptation; documentation of your ideas and process will improve your ability to think critically about research. Further, even if the PI or lab members do not physically view your notebook, you will need to communicate with them about your experiments. This documentation is necessary to communicate effectively about your work.

3. Organize your lab notebook

Different labs use different formats; some use electronic notebooks while others handwritten notebooks. The contents of a good notebook include the purpose of the experiment, the details of the experimental procedure, the data, and thoughts about the results. To effectively document your experiment, there are 5 critical questions that the information you record should be able to answer.

• Why I am doing this experiment? (purpose)
• What did I do to perform the experiment? (protocol)
• What are the results of what I did? (data, graphs)
• What do I think about the results?
• What do I think are the next steps?

We also recommend a table of contents. It will make the information more useful to you and the lab in the future. The table of contents should list the title of the experiment, the date(s) it was performed, and the page numbers on which it is recorded. Also, make sure that you write clearly and provide a legend or explanation of any shorthand or non-standard abbreviation you use. Often labs will have a combination of written lab notebooks and electronic data. It is important to reference where electronic data are located that go with each experiment. The idea is to make it as easy as possible to understand what you did and where to find all the data (electronic and hard copy) that accompanies your experiment.

Keeping a lab notebook becomes easier with practice. It can be thought of almost like journaling about your experiment. Sometimes people think of it as just a place to paste their protocol and a graph or data. We strongly encourage you to include your thoughts about why you made the decisions you made when conducting the experiment and to document your thoughts about next steps.

4. Commit to doing it the right way

A common reason to become lax in documentation is feeling rushed for time. Although documentation takes time, it saves time in the long-run and fosters good science. Without good notes, you will waste time trying to recall precisely what you did, reproduce your findings, and remember what you thought would be important next steps. The lab notebook helps you think about your research critically and keep your thoughts together. It can also save you time later when writing up results for publication. Further, well-documented data will help you draft a cogent and rigorous dissertation.

Handle data responsibly

1. keep all data.

Data are the product of research. Data include raw data, processed data, analyzed data, figures, and tables. Many data today are electronic, but not all. Generating data requires a lot of time and resources and researchers must treat data with care. The first essential tip is to keep all data. Do not discard data just because the experiment did not turn out as expected. A lot of experiments do not turn out to yield publishable data, but the results are still important for informing next steps.

Always keep the original, raw data. That is, as you process and analyze data, always maintain an unprocessed version of the original data.

Universities and funding agencies have data retention policies. These policies specify the number of years beyond a grant that data must be kept. Some policies also indicate researchers need to retain original data that served as the basis for a publication for a certain number of years. Therefore, your data will be important well beyond your time in graduate school. Most labs require you to keep samples for reanalysis until a paper is published, then the analyzed data are enough. If you leave a lab before a paper is accepted for publication, you are responsible for ensuring your data and original samples are well documented for others to find and use.

2. Document all data

In addition to keeping all data, data must be well-organized and documented. This means that no matter the way you keep your data (e.g., electronic or in written lab notebooks), there is a clear guide—in your lab notebook, a binder, or on a lab hard drive—to finding the data for a particular experiment. For example, it must be clear which data produced a particular graph. Version control of data is also critical. Your documentation should include “metadata” (data about your data) that tracks versions of the data. For example, as you edit data for a table, you should save separate versions of the tables, name the files sequentially, and note the changes that were made to each version.

3. Backup your data

You should backup electronic data regularly. Ideally, your lab has a shared server or cloud storage to backup data. If you are supposed to put your data there, make sure you do it! When you leave the lab, it must be possible to find your data.

4. Perform data analysis honestly and competently

Inappropriate use of statistics is a major concern in the scientific community, as the results and conclusions will be misleading if done incorrectly ( DeMets, 1999 ). Some practices are clearly an abuse of statistics, while other inappropriate practices stem from lack of knowledge. For example, a practice called “p-hacking” describes when researchers “collect or select data or statistical analyses until nonsignificant results become significant” ( Head, Holman, Lanfear, Kahn, & Jennions, 2015 ). In addition to avoiding such misbehavior, it is essential to be proficient with statistics to ensure you do statistical procedures appropriately. Learning statistical procedures and analyzing data takes many years of practice, and your statistics courses may only cover the basics. You will need to know when to consult others for help. In addition to consulting members in your lab or your PI, your university may have statistical experts who can provide consultations.

5. Master pressure to obtain favored results

When you conduct an experiment, the results are the results. As a beginning researcher, it is important to be prepared to manage the frustration of experiments not turning out as expected. It is also important to manage the real or perceived pressure to produce favored results. Investigators can become wedded to a hypothesis, and they can have a difficult time accepting the results. Sometimes you may feel this pressure coming from yourself; for example, if you want to please your PI, or if you want to get results for a certain publication. It is important to always follow the data no matter where it leads.

If you do feel pressure, this situation can be uncomfortable and stressful. If you have been meticulous and followed the above recommendations, this can be one great safeguard. You will be better able to confidently communicate your results to the PI because of your detailed documentation, and you will be more confident in your procedures if the possibility of error is suggested. Typically, with enough evidence that the unexpected results are real, the PI will concede. We recommend seeking the support of friends or colleagues to vent and cope with stress. In the rare case that the PI does not relent, you could turn to an advisor outside the lab if you need advice about how to proceed. They can help you look at the data objectively and also help you think about the interpersonal aspects of navigating this situation.

6. Communicate about your data in the lab

A critical element of reproducible research is communication in the lab. Ideally, there are weekly or bi-weekly meetings to discuss data. You need to develop your communication skills for writing and speaking about data. Often you and your labmates will discuss experimental issues and results informally during the course of daily work. This is an excellent way to hone critical thinking and communication skills about data.

Scenario 1 – The Protocol is Not Working

At the beginning of a rotation during their first year, a graduate student is handed a lab notebook and a pen and is told to keep track of their work. There does not appear to be a specific format to follow. There are standard lab protocols that everyone follows, but minor tweaks to the protocols do not seem to be tracked from experiment to experiment in the standard lab protocol nor in other lab notebooks. After two weeks of trying to follow one of the standard lab protocols, the student still cannot get the experiment to work. The student has included the appropriate positive and negative controls which are failing, making the experiment uninterpretable. After asking others in the lab for help, the graduate student learns that no one currently in the lab has performed this particular experiment. The former lab member who had performed the experiment only lists the standard protocol in their lab notebook.

How should the graduate student start to solve the problem?

Speaking to the PI would be the next logical step. As a first-year student in a lab rotation, the PI should expect this type of situation and provide additional troubleshooting guidance. It is possible that the PI may want to see how the new graduate student thinks critically and handles adversity in the lab. Rather than giving an answer, the PI might ask the student to work through the problem. The PI should give guidance, but it may not be an immediate fix for the problem. If the PI’s suggestions fail to correct the problem, asking a labmate or the PI for the contact information of the former lab member who most recently performed the experiment would be a reasonable next step. The graduate student’s conversations with the PI and labmates in this situation will help them learn a lot about how the people in the lab interact.

Most of the answers for these types of problems will require you as a graduate student to take the initiative to answer. They will require your effort and ingenuity to talk to other lab members, other labs at the university, and even scour the literature for alternatives. While labs have standard protocols, there are multiple ways to do many experiments, and working out an alternative will teach you more than when everything works. Having to troubleshoot problems will result in better standard protocols in the lab and better science.

HOW TO BE A RESPONSIBLE AUTHOR

Researchers communicate their findings via peer-reviewed publications, and publications are important for advancing in a research career. Many graduate students will first author or co-author publications in graduate school. For good advice on how to write a research manuscript, consult the Current Protocols article “How to write a research manuscript” ( Frank, 2018 ). We focus on the issues of assigning authors and reporting your findings responsibly. First, we describe some important basics: journal impact factors, predatory journals, and peer review.

What are journal impact factors?

It is helpful to understand journal impact factors. There is criticism about an overemphasis on impact factors for evaluating the quality or importance of researchers’ work ( DePellegrin & Johnston, 2015 ), but they remain common for this purpose. Journal impact factors reflect the average number of times articles in a journal were cited in the last two years. Higher impact factors place journals at a higher rank. Approximately 2% of journals have an impact factor of 10 or higher. For example, Cell, Science, and Nature have impact factors of approximately 39, 42, and 43, respectively. Journals can be great journals but have lower impact factors; often this is because they focus on a smaller specialty field. For example, Journal of Immunology and Oncogene are respected journals, but their impact factors are about 4 and 7, respectively.

Research trainees often want to publish in journals with the highest possible impact factor because they expect this to be viewed favorably when applying to future positions. We encourage you to bear in mind that many different journals publish excellent science and focus on publishing where your work will reach the desired audience. Also, keep in mind that while a high impact factor can direct you to respectable, high-impact science, it does not guarantee that the science in the paper is good or even correct. You must critically evaluate all papers you read no matter the impact factor.

What are predatory journals?

Predatory journals have flourished over the past few years as publishing science has moved online. An international panel defined predatory journals as follows ( Grudniewicz et al., 2019 ):

Predatory journals and publishers are entities that prioritize self-interest at the expense of scholarship and are characterized by false or misleading information, deviation from best editorial and publication practices, a lack of transparency, and/or the use of aggressive and indiscriminate solicitation practices. (p. 211)

Often young researchers receive emails soliciting them to submit their work to a journal. There are typically small fees (around $99 US) requested but these fees will be much lower than open access fees of reputable journals (often around $2000 US). A warning sign of a predatory journal is outlandish promises, such as 24-hour peer review or immediate publication. You can find a list of predatory journals created by a postdoc in Europe at BeallsList.net ( “Beall’s List of Potential Predatory Journals and Publishers,” 2020 ).

What is peer review?

Peer reviewers are other scientists who have the expertise to evaluate a manuscript. Typically 2 or 3 reviewers evaluate a manuscript. First, an editor performs an initial screen of the manuscript to ensure its appropriateness for the journal and that it meets basic quality standards. At this stage, an editor can decide to reject the manuscript and not send it to review. Not sending a paper for peer review is common in the highest impact journals that receive more submissions per year than can be reviewed and published. For average-impact journals and specialty journals, typically your paper will be sent for peer review.

In general, peer review focuses on three aspects of a manuscript: research design and methods, validity of the data and conclusions, and significance. Peer reviewers assess the merit and rigor of the research design and methodology, and they evaluate the overall validity of the results, interpretations, and conclusions. Essentially, reviewers want to ensure that the data support the claims. Additionally, reviewers evaluate the overall significance, or contribution, of the findings, which involves the novelty of the research and the likelihood that the findings will advance the field. Significance standards vary between journals. Some journals are open to publishing findings that are incremental advancements in a field, while others want to publish only what they deem as major advancements. This feature can distinguish the highest impact journals which seek the most significant advancements and other journals that tend to consider a broader range of work as long as it is scientifically sound. It is important to keep in mind that determining at the stage of review and publication whether a paper is “high impact” is quite subjective. In reality, this can only really be determined in retrospect.

The key ethical issues in peer review are fairness, objectivity, and confidentiality ( Shamoo & Resnik, 2015 ). Peer reviewers are to evaluate the manuscript on its merits and not based on biases related to the authors or the science itself. If reviewers have a conflict of interest, this should be disclosed to the editor. Confidentiality of peer review means that the reviewers should keep private the information; they should not share the information with others or use it to their benefit. Reviewers can ultimately recommend that the manuscript is rejected, revised, and resubmitted (major or minor revisions), or accepted. The editor evaluates the reviewers’ feedback and makes a judgment about rejecting, accepting, or requesting a revision. Sometimes PIs will ask experienced graduate students to assist with peer reviewing a manuscript. This is a good learning opportunity. The PI should disclose to the editor that they included a trainee in preparing the review.

Assign authorship fairly

Authorship gives credit to the people who contributed to the research. This includes thinking of the ideas, designing and performing experiments, interpreting the results, and writing the paper. Two key questions regarding authorship include: 1 - Who will be an author? 2 - What will be the order in which authors are listed? These seem simple on the surface but can get quite complex.

1. Know authorship guidelines

Authorship guidelines published by journals, professional societies, and universities communicate key principles of authorship and standards for earning authorship. The core ethical principle of assigning authorship is fairness in who receives credit for the work. The people who contributed to the work should get credit for it. This seems simply enough, but determining authorship can (and often does) create conflict.

Many universities have authorship guidelines, and you should know the policies at your university. The International Committee of Medical Journal Editors (ICMJE) provides four criteria for determining who should be an author ( International Committee of Medical Journal Editors, 2020 ). These criteria indicate that an author should do all of the following: 1) make “substantial contributions” to the development of the idea or research design, or to acquiring, analyzing, or interpreting the data, 2) write the manuscript or revise it a substantive way, 3) give approval of the final manuscript (i.e., before it is submitted for review, and after it is revised, if necessary), and 4) agree to be responsible for any questions about the accuracy or integrity of the research.

Several types of authorship violate these guidelines and should be avoided. Guest authorship is when respected researchers are added out of appreciation, or to have the manuscript be perceived more favorably to get it published or increase its impact. Gift authorship is giving authorship to reward an individual, or as a favor. Ghost authorship is when someone made significant contributions to the paper but is not listed as an author. To increase transparency, some journals require authors to indicate how each individual contributed to the research and manuscript.

2. Apply the guidelines

Conflicts often arise from disagreements about how much people contributed to the research and whether those contributions merit authorship. The best approach is an open, honest, and ongoing discussion about authorship, which we discuss in #3 below. To have effective, informed conversations about authorship, you must understand how to apply the guidelines to your specific situation. The following is a simple rule of thumb that indicates there are three components of authorship. We do not list giving final approval of the manuscript and agreeing to be accountable, but we do consider these essentials of authorship.

• Thinking – this means contributing to the ideas leading to the hypothesis of the work, designing experiments to address the hypothesis, and/or analyzing the results in the larger context of the literature in the field.
• Doing – this means performing and analyzing the experiments.
• Writing – this means editing a draft, or writing the entire paper. The first author often writes the entire first draft.

In our experience, a first author would typically do all three. They also usually coordinate the writing and editing process. Co-authors are typically very involved in at least two of the three, and are somewhat involved in the other. The PI, who oversees and contributes to all three, is often the last, or “senior author.” The “senior author” is typically the “corresponding author”—the person listed as the individual to contact about the paper. The other co-authors are listed between the first and senior author either alphabetically, or more commonly, in order from the largest to smallest contribution.

Problems in assigning authorship typically arise due to people’s interpretations of #1 (thinking) and #2 (doing)—what and how much each individual contributed to a project’s design, execution, and analysis. Different fields or PIs may have their own slight variations on these guidelines. The potential conflicts associated with assigning authorship lead to the most common recommendation for responsibly assigning authorship: discuss authorship expectations early and revisit them during the project.

3. Discuss authorship with your collaborators

Publications are important for career advancement, so you can see why people might be worried about fairness in assigning authorship. If the problem arises from a lack of a shared understanding about contributions to the research, the only way to clarify this is an open discussion. This discussion should ideally take place very early at the beginning of a project, and should be ongoing. Hopefully you work in a laboratory that makes these discussions a natural part of the research process; this makes it much easier to understand the expectations upfront.

We encourage you to speak up about your interest in making a contribution that would merit authorship, especially if you want to earn first authorship. Sometimes norms about authoring papers in a lab make it clear you are expected to first and co-author publications, but it is best to communicate your interest in earning authorship. If the project is not yours, but you wish to collaborate, you can inquire what you may be able to contribute that would merit authorship.

If it is not a norm in your lab to discuss authorship throughout the life of projects, then as a graduate student you may feel reluctant to speak up. You could initiate a conversation with a more senior graduate student, a postdoc, or your PI, depending on the dynamics in the group. You could ask generally about how the lab approaches assignment of authorship, but discussing a specific project and paper may be best. It may feel awkward to ask, but asking early is less uncomfortable than waiting until the end of the project. If the group is already drafting a manuscript and you are told that your contribution is insufficient for authorship, this situation is much more discouraging than if you had asked earlier about what is expected to earn authorship.

How to report findings responsibly

The most significant responsibility of authors is to present their research accurately and honestly. Deliberately presenting misleading information is clearly unethical, but there are significant judgment calls about how to present your research findings. For example, an author can mislead by overstating the conclusions given what the data support.

1. Commit to presenting your findings honestly

Any good scientific manuscript writer will tell you that you need to “tell a good story.” This means that your paper is organized and framed to draw the reader into the research and convince them of the importance of the findings. But, this story must be sound and justified by the data. Other authors are presenting their findings in the best, most “publishable” light, so it is a balancing act to be persuasive but also responsible in presenting your findings in a trustworthy manner. To present your findings honestly, you must be conscious of how you interpret your data and present your conclusions so that they are accurate and not overstated.

One misbehavior known as “HARKing,” Hypothesis After the Results are Known, occurs when hypotheses are created after seeing the results of an experiment, but they are presented as if they were defined prior to collecting the data ( Munafò et al., 2017 ). This practice should be avoided. HARKing may be driven, in part, by a concern in scientific publishing known as publication bias. This bias is a preference that reviewers, editors, and researchers have for papers describing positive findings instead of negative findings ( Carroll, Toumpakari, Johnson, & Betts, 2017 ). This preference can lead to manipulating one’s practices, such as by HARKing, so that positive findings can be reported.

It is important to note that in addition to avoiding misbehaviors such as HARKing, all researchers are susceptible to a number of more subtle traps in judgment. Even the most well-intentioned researcher may jump to conclusions, discount alternative explanations, or accept results that seem correct without further scrutiny ( Nuzzo, 2015 ). Therefore, researchers must not only commit to presenting their findings honestly but consider how they can counteract such traps by slowing down and increasing their skepticism towards their findings.

2. Provide an appropriate amount of detail

Providing enough detail in a manuscript can be a challenge with the word limits imposed by most journals. Therefore, you will need to determine what details to include and which to exclude, or potentially include in the supplemental materials. Methods sections can be long and are often the first to be shortened, but complete methods are important for others to evaluate the research and to repeat the methods in other studies. Even more significant is making decisions about what experimental data to include and potentially exclude from the manuscript. Researchers must determine what data is required to create a complete scientific story that supports the central hypothesis of the paper. On the other hand, it is not necessary or helpful to include so much data in the manuscript, or in supplemental material, that the central point of the paper is difficult to discern. It is a tricky balance.

3. Follow proper citation practices

Of course, responsible authorship requires avoiding plagiarism. Many researchers think that plagiarism is not a concern for them because they assume it is always done intentionally by “copying and pasting” someone else’s words and claiming them as your own. Sometimes poor writing practices, such as taking notes from references without distinguishing between direct quotes and paraphrased material, can lead to including material that is not quoted properly. More broadly, proper citation practices include accurately and completely referencing prior studies to provide appropriate context for your manuscript.

4. Attend to the other important details

The journal will require several pieces of additional information, such as disclosure of sources of funding and potential conflicts of interest. Typically, graduate students do not have relationships that constitute conflicts of interest, but a PI who is a co-author may. In submitting a manuscript, also make sure to acknowledge individuals not listed as authors but who contributed to the work.

5. Share data and promote transparency

Data sharing is a key facet of promoting transparency in science ( Nosek et al., 2015 ). It will be important to know the expectations of the journals in which you wish to publish. Many top journals now require data sharing; for example, sharing your data files in an online repository so others have access to the data for secondary use. Funding agencies like NIH also increasingly require data sharing. To further foster transparency and public trust in research, researchers must deposit their final peer-reviewed manuscripts that report on research funded by NIH to PubMed Central. PubMed makes biomedical and life science research publicly accessible in a free, online database.

Scenario 2 – Authors In Conflict

To prepare a manuscript for publication, a postdoc’s data is added to a graduate student’s thesis project. After working together to combine the data and write the paper, the postdoc requests co-first authorship on the paper. The graduate student balks at this request on the basis that it is their thesis project. In a weekly meeting with the lab’s PI to discuss the status of the paper, the graduate student states that they should divide the data between the authors as a way to prove that the graduate student should be the sole first author. The PI agrees to this attempt to quantify how much data each person contributed to the manuscript. All parties agree the writing and thinking were equally shared between them. After this assessment, the graduate student sees that the postdoc actually contributed more than half of the data presented in the paper. The graduate student and a second graduate student contributed the remaining data; this means the graduate student contributed much less than half of the data in the paper. However, the graduate student is still adamant that they must be the sole first author of the paper because it is their thesis project.

Is the graduate student correct in insisting that it is their project, so they are entitled to be the sole first author?

Co-first authorship became popular about 10 years ago as a way to acknowledge shared contributions to a paper in which authors worked together and contributed equally. If the postdoc contributed half of the data and worked with the graduate student to combine their interpretations and write the first draft of the paper, then the postdoc did make a substantial contribution. If the graduate student wrote much of the first draft of the paper, contributed significantly to the second half of data, and played a major role in the thesis concept and design, this is also a major contribution. We summarized authorship requirements as contributing to thinking, doing, and writing, and we noted that a first author usually contributes to all of these. The graduate student has met all 3 elements to claim first authorship. However, it appears that the postdoc has also met these 3 requirements. Thus, it is at least reasonable for the postdoc to ask about co-first authorship.

The best way to move forward is to discuss their perspectives openly. Both the graduate student and postdoc want first authorship on papers to advance their careers. The postdoc feels they contributed more to the overall concept and design than the graduate student is recognizing, and the postdoc did contribute half of the data. This is likely frustrating and upsetting for the postdoc. On the other hand, perhaps the postdoc is forgetting how much a thesis becomes like “your baby,” so to speak. The work is the graduate student’s thesis, so it is easy to see why the graduate student would feel a sense of ownership of it. Given this fact, it may be hard for the graduate student to accept the idea that they would share first-author recognition for the work. Yet, the graduate student should consider that the manuscript would not be possible without the postdoc’s contribution. Further, if the postdoc was truly being unreasonable, then the postdoc could make the case for sole first authorship based on contributing the most data to the paper, in addition to contributing ideas and writing the paper. The graduate student should consider that the postdoc may be suggesting co-first authorship in good faith.

As with any interpersonal conflict, clear communication is key. While it might be temporarily uncomfortable to voice their views and address this disagreement, it is critical to avoiding permanent damage to their working relationship. The pair should consider each other’s perspectives and potential alternatives. For example, if the graduate student is first author and the postdoc second, at a minimum they could include an author note in the manuscript that describes the contribution of each author. This would make it clear the scope of the postdoc’s contribution, if they decided not to go with co-first authorship. Also, the graduate student should consider their assumptions about co-first authorship. Maybe they assume it makes it appear they contributed less, but instead, perhaps co-first authorship highlights their collaborative approach to science. Collaboration is a desirable quality many (although arguably not all) research organizations look for when they are hiring.

They will also need to speak with others for advice. The pair should definitely speak with the PI who could provide input about how these cases have been handled in the past. Ultimately, if they cannot reach an agreement, the PI, who is likely to be the last or “senior” author, may make the final decision. They should also speak to the other graduate student who is an author.

If either individual is upset with the situation, they will want to discuss it when they have had time to cool down. This might mean taking a day before discussing, or speaking with someone outside of the lab for support. Ideally, all authors on this paper would have initiated this conversation earlier, and the standards in the lab for first authorship would be discussed routinely. Clear communication may have avoided the conflict.

HOW TO USE DECISION-MAKING STRATEGIES TO NAVIGATE CHALLENGES

We have provided advice on some specific challenges you might encounter in research. This final section covers our overarching recommendation that you adopt a set of ethical decision-making strategies. These strategies help researchers address challenges by helping them think through a problem and possible alternatives ( McIntosh et al., 2020 ). The strategies encourage you to gather information, examine possible outcomes, consider your assumptions, and address emotional reactions before acting. They are especially helpful when you are uncertain how to proceed, face a new problem, or when the consequences of a decision could negatively impact you or others. The strategies also help people be honest with themselves, such as when they are discounting important factors or have competing goals, by encouraging them to identify outside perspectives and test their motivations. You can remember the strategies using the acronym SMART .

1. S eek Help

Obtain input from others who can be objective and that you trust. They can assist you with assessing the situation, predicting possible outcomes, and identifying potential options. They can also provide you with support. Individuals to consult may be peers, other faculty, or people in your personal life. It is important that you trust the people you talk with, but it is also good when they challenge your perspective, or encourage you to think in a new way about a problem. Keep in mind that people such as program directors and university ombudsmen are often available for confidential, objective advice.

2. M anage Emotions

Consider your emotional reaction to the situation and how it might influence your assessment of the situation, and your potential decisions and actions. In particular, identify negative emotions, like frustration, anxiety, fear, and anger, as they particularly tend to diminish decision-making and the quality of interactions with others. Take time to address these emotions before acting, for example, by exercising, listening to music, or simply taking a day before responding.

3. A nticipate Consequences

Think about how the situation could turn out. This includes for you, for the research team, and anyone else involved. Consider the short, middle-term, and longer-term impacts of the problem and your potential approach to addressing the situation. Ideally, it is possible to identify win-win outcomes. Often, however, in tough professional situations, you may need to select the best option from among several that are not ideal.

4. R ecognize Rules and Context

Determine if any ethical principles, professional policies, or rules apply that might help guide your choices. For instance, if the problem involves an authorship dispute, consider the authorship guidelines that apply. Recognizing the context means considering the situational factors that could impact your options and how you proceed. For example, factors such as the reality that ultimately the PI may have the final decision about authorship.

5. T est Assumptions and Motives

Examine your beliefs about the situation and whether any of your thoughts may not be justified. This includes critically examining the personal motivations and goals that are driving your interpretation of the problem and thoughts about how to resolve it.

These strategies do not have to be engaged in order, and they are interrelated. For example, seeking help can help you manage emotions, test assumptions, and anticipate consequences. Go back to the scenarios and our advice throughout this article, and you will see many of our suggestions align with these strategies. Practice applying SMART strategies when you encounter a problem and they will become more natural.

Learning practices for responsible research will be the foundation for your success in graduate school and your career. We encourage you to be reflective and intentional as you learn and hope that our advice helps you along the way.

ACKNOWLEDGEMENTS

This work was supported by the National Human Genome Research Institute (Antes, K01HG008990) and the National Center for Advancing Translational Sciences (UL1 TR002345).

LITERATURE CITED

• Anderson MS, Horn AS, Risbey KR, Ronning EA, De Vries R, & Martinson BC (2007). What Do Mentoring and Training in the Responsible Conduct of Research Have To Do with Scientists’ Misbehavior? Findings from a National Survey of NIH-Funded Scientists . Academic Medicine , 82 ( 9 ), 853–860. doi: 10.1097/ACM.0b013e31812f764c [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Antes AL, Brown RP, Murphy ST, Waples EP, Mumford MD, Connelly S, & Devenport LD (2007). Personality and Ethical Decision-Making in Research: The Role of Perceptions of Self and Others . Journal of Empirical Research on Human Research Ethics , 2 ( 4 ), 15–34. doi: 10.1525/jer.2007.2.4.15 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Antes AL, English T, Baldwin KA, & DuBois JM (2018). The Role of Culture and Acculturation in Researchers’ Perceptions of Rules in Science . Science and Engineering Ethics , 24 ( 2 ), 361–391. doi: 10.1007/s11948-017-9876-4 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Antes AL, Kuykendall A, & DuBois JM (2019a). The Lab Management Practices of “Research Exemplars” that Foster Research Rigor and Regulatory Compliance: A Qualitative Study of Successful Principal Investigators . PloS One , 14 ( 4 ), e0214595. doi: 10.1371/journal.pone.0214595 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Antes AL, Kuykendall A, & DuBois JM (2019b). Leading for Research Excellence and Integrity: A Qualitative Investigation of the Relationship-Building Practices of Exemplary Principal Investigators . Accountability in Research , 26 ( 3 ), 198–226. doi: 10.1080/08989621.2019.1611429 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Antes AL, & Maggi LB Jr. (2021). How to Navigate Trainee-Mentor Relationships and Interpersonal Dynamics in the Lab . Current Protocols Essential Laboratory Techniques. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Asplund M, & Welle CG (2018). Advancing Science: How Bias Holds Us Back . Neuron , 99 ( 4 ), 635–639. doi: 10.1016/j.neuron.2018.07.045 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Baker M (2016). Is There a Reproducibility Crisis? Nature , 533 , 452–454. doi: 10.1038/533452a [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Barba LA (2016). The Hard Road to Reproducibility . Science , 354 ( 6308 ), 142. doi: 10.1126/science.354.6308.142 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Beall’s List of Potential Predatory Journals and Publishers . (2020). Retrieved from https://beallslist.net/#update [ Google Scholar ]
• Carroll HA, Toumpakari Z, Johnson L, & Betts JA (2017). The Perceived Feasibility of Methods to Reduce Publication Bias . PloS One , 12 ( 10 ), e0186472–e0186472. doi: 10.1371/journal.pone.0186472 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Chappell B (2019). Duke Whistleblower Gets More Than $33 Million in Research Fraud Settlement . NPR. Retrieved from https://www.npr.org/2019/03/25/706604033/duke-whistleblower-gets-more-than-33-million-in-research-fraud-settlement [ Google Scholar ]
• Davis MS, Riske-Morris M, & Diaz SR (2007). Causal Factors Implicated in Research Misconduct: Evidence from ORI Case Files . Science and Engineering Ethics , 13 ( 4 ), 395–414. doi: 10.1007/s11948-007-9045-2 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• DeMets DL (1999). Statistics and Ethics in Medical Research . Science and Engineering Ethics , 5 ( 1 ), 97–117. doi: 10.1007/s11948-999-0059-9 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Department of Health and Human Services. (2005). 42 CFR Parts 50 and 93 Public Health Service Policies on Research Misconduct; Final Rule. Retrieved from https://ori.hhs.gov/sites/default/files/42_cfr_parts_50_and_93_2005.pdf [ Google Scholar ]
• DePellegrin TA, & Johnston M (2015). An Arbitrary Line in the Sand: Rising Scientists Confront the Impact Factor . Genetics , 201 ( 3 ), 811–813. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• DuBois JM, Anderson EE, Chibnall J, Carroll K, Gibb T, Ogbuka C, & Rubbelke T (2013). Understanding Research Misconduct: A Comparative Analysis of 120 Cases of Professional Wrongdoing . Account Res , 20 ( 5–6 ), 320–338. doi: 10.1080/08989621.2013.822248 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• DuBois JM, & Antes AL (2018). Five Dimensions of Research Ethics: A Stakeholder Framework for Creating a Climate of Research Integrity . Academic Medicine , 93 ( 4 ), 550–555. doi: 10.1097/ACM.0000000000001966 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Else H (2018). Does Science have a Bullying Problem? Nature , 563 , 616–618. doi: 10.1038/d41586-018-07532-5 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Emanuel EJ, Wendler D, & Grady C (2000). What Makes Clinical Research Ethical ? Journal of the American Medical Association , 283 ( 20 ), 2701–2711. doi:jsc90374 [pii] [ PubMed ] [ Google Scholar ]
• Evans TM, Bira L, Gastelum JB, Weiss LT, & Vanderford NL (2018). Evidence for a Mental Health Crisis in Graduate Education . Nature Biotechnology , 36 ( 3 ), 282–284. doi: 10.1038/nbt.4089 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Frank DJ (2018). How to Write a Research Manuscript . Current Protocols Essential Laboratory Techniques , 16 ( 1 ), e20. doi: 10.1002/cpet.20 [ CrossRef ] [ Google Scholar ]
• Goodman SN, Fanelli D, & Ioannidis JPA (2016). What Does Research Reproducibility Mean? Science Translational Medicine , 8 ( 341 ), 341ps312. doi: 10.1126/scitranslmed.aaf5027 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Grudniewicz A, Moher D, Cobey KD, Bryson GL, Cukier S, Allen K, … Lalu MM (2019). Predatory journals: no definition, no defence . Nature , 576 ( 7786 ), 210–212. doi: 10.1038/d41586-019-03759-y [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Head ML, Holman L, Lanfear R, Kahn AT, & Jennions MD (2015). The Extent and Consequences of P-Hacking in Science . PLoS Biology , 13 ( 3 ), e1002106. doi: 10.1371/journal.pbio.1002106 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Hofstra B, Kulkarni VV, Munoz-Najar Galvez S, He B, Jurafsky D, & McFarland DA (2020). The Diversity–Innovation Paradox in Science . Proceedings of the National Academy of Sciences , 117 ( 17 ), 9284. doi: 10.1073/pnas.1915378117 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• International Committee of Medical Journal Editors. (2020). Defining the Role of Authors and Contributors . Retrieved from http://www.icmje.org/recommendations/browse/roles-and-responsibilities/defining-the-role-of-authors-and-contributors.html
• Keith-Spiegel P, Sieber J, & Koocher GP (2010). Responding to Research Wrongdoing: A User-Friendly Guide . Retrieved from http://users.neo.registeredsite.com/1/4/0/20883041/assets/RRW_11-10.pdf
• McIntosh T, Antes AL, & DuBois JM (2020). Navigating Complex, Ethical Problems in Professional Life: A Guide to Teaching SMART Strategies for Decision-Making . Journal of Academic Ethics . doi: 10.1007/s10805-020-09369-y [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Meyers LC, Brown AM, Moneta-Koehler L, & Chalkley R (2018). Survey of Checkpoints along the Pathway to Diverse Biomedical Research Faculty . PloS One , 13 ( 1 ), e0190606–e0190606. doi: 10.1371/journal.pone.0190606 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Munafò MR, Nosek BA, Bishop DVM, Button KS, Chambers CD, Percie du Sert N, … Ioannidis JPA (2017). A manifesto for reproducible science . Nature Human Behaviour , 1 ( 1 ), 0021. doi: 10.1038/s41562-016-0021 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• National Academies of Science. (2009). On Being a Scientist: A Guide to Responsible Conduct in Research . Washington DC: National Academics Press. [ PubMed ] [ Google Scholar ]
• National Academies of Sciences Engineering and Medicine. (2017). Fostering Integrity in Research . Washington, DC: The National Academies Press [ PubMed ] [ Google Scholar ]
• National Academies of Sciences Engineering and Medicine. (2018a). An American Crisis: The Growing Absence of Black Men in Medicine and Science: Proceedings of a Joint Workshop . Washington, DC: The National Academies Press. [ PubMed ] [ Google Scholar ]
• National Academies of Sciences Engineering and Medicine. (2018b). Sexual harassment of women: climate, culture, and consequences in academic sciences, engineering, and medicine : National Academies Press. [ PubMed ] [ Google Scholar ]
• National Institutes of Health. (2009). Update on the Requirement for Instruction in the Responsible Conduct of Research . NOT-OD-10-019 . Retrieved from https://grants.nih.gov/grants/guide/notice-files/NOT-OD-10-019.html
• National Science Foundation. (2017). Important Notice No. 140 Training in Responsible Conduct of Research – A Reminder of the NSF Requirement . Retrieved from https://www.nsf.gov/pubs/issuances/in140.jsp
• No Place for Bullies in Science. (2018). Nature , 559 ( 7713 ), 151. doi: 10.1038/d41586-018-05683-z [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Norris D, Dirnagl U, Zigmond MJ, Thompson-Peer K, & Chow TT (2018). Health Tips for Research Groups . Nature , 557 , 302–304. doi: 10.1038/d41586-018-05146-5 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Nosek BA, Alter G, Banks GC, Borsboom D, Bowman SD, Breckler SJ, … Yarkoni T (2015). Scientific Standards . Promoting an Open Research Culture. Science , 348 ( 6242 ), 1422–1425. doi: 10.1126/science.aab2374 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Nuzzo R (2015). How Scientists Fool Themselves - and How They Can Stop . Nature , 526 , 182–185. [ PubMed ] [ Google Scholar ]
• O’Connor A (2018). More Evidence that Nutrition Studies Don’t Always Add Up . The New York Times. Retrieved from https://www.nytimes.com/2018/09/29/sunday-review/cornell-food-scientist-wansink-misconduct.html [ Google Scholar ]
• Park A (2012). Great Science Frauds . Time. Retrieved from https://healthland.time.com/2012/01/13/great-science-frauds/slide/the-baltimore-case/ [ Google Scholar ]
• Plemmons DK, Baranski EN, Harp K, Lo DD, Soderberg CK, Errington TM, … Esterling KM (2020). A Randomized Trial of a Lab-embedded Discourse Intervention to Improve Research Ethics . Proceedings of the National Academy of Sciences , 117 ( 3 ), 1389. doi: 10.1073/pnas.1917848117 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Research Institutions Must Put the Health of Labs First. (2018). Nature , 557 ( 7705 ), 279–280. doi: 10.1038/d41586-018-05159-0 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Research Integrity is Much More Than Misconduct . (2019). ( 570 ). doi: 10.1038/d41586-019-01727-0 [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Resnik DB (2011). Scientific Research and the Public Trust . Science and Engineering Ethics , 17 ( 3 ), 399–409. doi: 10.1007/s11948-010-9210-x [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Roper RL (2019). Does Gender Bias Still Affect Women in Science? Microbiology and Molecular Biology Reviews , 83 ( 3 ), e00018–00019. doi: 10.1128/MMBR.00018-19 [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Shamoo AE, & Resnik DB (2015). Responsible Conduct of Research (3rd ed.). New York: Oxford University Press. [ Google Scholar ]
• Steneck NH (2007). ORI Introduction to the Responsible Conduct of Research (Updated ed.). Washington, D.C.: U.S. Government Printing Office. [ Google Scholar ]
• Winchester C (2018). Give Every Paper a Read for Reproducibility . Nature , 557 , 281. doi: 10.1038/d41586-018-05140-x [ PubMed ] [ CrossRef ] [ Google Scholar ]

Research Project Guidelines

For the purposes of the traditional (non-basic science track) degree, clinical research is defined as research conducted with human subjects or patients’ samples in which an investigator makes direct measurements or observations on the subjects. It includes clinical trials, studies of the mechanism of human disease, epidemiological and behavioral studies, outcomes research and health services research. For examples of mechanistic studies, correlating serum biomarkers or genetic variants with clinical phenotypes would be acceptable. For information specific to the Research Project for the  Basic Science Research Track , please refer to that page.  Studies based entirely in animal models or in vitro systems would only be appropriate for students on the Basic Science Research Track.

Although it is desirable the research project be one in which the student is involved throughout from the concept and design stage, this is not mandatory. A secondary data analysis of an existing data set might be acceptable, for example, as might a project that the student designs and conducts, but from which only preliminary results are reported.  Students unsure if a potential project satisfies this definition should email a brief description to John Williams, MD, MHSc.

Three faculty members constitute an examining committee to certify the student has successfully completed the research project requirement for the degree. The committee must include a clinical investigator and a statistician, both of whom are on the faculty of the Clinical Research Training Program (CRTP). The third member of the committee should be a faculty member who has substantive knowledge in the area in which the clinical research project is conducted; for clinical fellows, this committee member is often the student's mentor. The chair of the committee must be a member of the CRTP faculty.

Important Note : Committees formed for other purposes (e.g., Fellowship committees, CTSA committees, etc.) cannot substitute for the CRTP Research Committee. You must have an approved CRTP committee for your research project.

A full year before you want to graduate: 

• Develop your research proposal.
• Complete Research Project Proposal (RP1) Form and submit to Gail Ladd
• John  Williams or Scott Palmer will contact you if changes are needed.  They will identify your Committee Chair, Committee Member and Mentor.  They will send an official notification (RP2) to you that the proposal is approved.

When your proposal is approved: 

• Convene your Committee within the month your proposal is approved.  If you are unable to meet in person, work with Gail Ladd  to set up an on line meeting.
• Within a week of your initial meeting, submit Initial Meeting Form (RP3) with summary of key discussion items and any planned changes to the protocol to your Committee members and Gail. 
•  Keep in regular contact with your Committee.
• Begin work on your Research Project.

Two months before you plan to defend:

• Submit a forma written progress report (RP5) to your Committee members and Gail Ladd .  Use RP Mid-Point Progress Report form.

One month before you plan to defend:

• Submit final manuscript to Committee and to Gail Ladd .
• Set oral defense date and location.

After you have defended: 

• Submit the RP Contribution checklist form (RP6) detailing your roles in the project.
• If necessary, complete any required modifications and resubmit to the Committee Chair and Gail Ladd.

While you can start your project at any point during your time in CRTP, you must register for the 12 credits the semester before you plan to graduate.

Illustration by James Round

How to plan a research project

Whether for a paper or a thesis, define your question, review the work of others – and leave yourself open to discovery.

by Brooke Harrington   + BIO

is professor of sociology at Dartmouth College in New Hampshire. Her research has won international awards both for scholarly quality and impact on public life. She has published dozens of articles and three books, most recently the bestseller Capital without Borders (2016), now translated into five languages.

Edited by Sam Haselby

Need to know

‘When curiosity turns to serious matters, it’s called research.’ – From Aphorisms (1880-1905) by Marie von Ebner-Eschenbach

Planning research projects is a time-honoured intellectual exercise: one that requires both creativity and sharp analytical skills. The purpose of this Guide is to make the process systematic and easy to understand. While there is a great deal of freedom and discovery involved – from the topics you choose, to the data and methods you apply – there are also some norms and constraints that obtain, no matter what your academic level or field of study. For those in high school through to doctoral students, and from art history to archaeology, research planning involves broadly similar steps, including: formulating a question, developing an argument or predictions based on previous research, then selecting the information needed to answer your question.

Some of this might sound self-evident but, as you’ll find, research requires a different way of approaching and using information than most of us are accustomed to in everyday life. That is why I include orienting yourself to knowledge-creation as an initial step in the process. This is a crucial and underappreciated phase in education, akin to making the transition from salaried employment to entrepreneurship: suddenly, you’re on your own, and that requires a new way of thinking about your work.

What follows is a distillation of what I’ve learned about this process over 27 years as a professional social scientist. It reflects the skills that my own professors imparted in the sociology doctoral programme at Harvard, as well as what I learned later on as a research supervisor for Ivy League PhD and MA students, and then as the author of award-winning scholarly books and articles. It can be adapted to the demands of both short projects (such as course term papers) and long ones, such as a thesis.

At its simplest, research planning involves the four distinct steps outlined below: orienting yourself to knowledge-creation; defining your research question; reviewing previous research on your question; and then choosing relevant data to formulate your own answers. Because the focus of this Guide is on planning a research project, as opposed to conducting a research project, this section won’t delve into the details of data-collection or analysis; those steps happen after you plan the project. In addition, the topic is vast: year-long doctoral courses are devoted to data and analysis. Instead, the fourth part of this section will outline some basic strategies you could use in planning a data-selection and analysis process appropriate to your research question.

Step 1: Orient yourself

Planning and conducting research requires you to make a transition, from thinking like a consumer of information to thinking like a producer of information. That sounds simple, but it’s actually a complex task. As a practical matter, this means putting aside the mindset of a student, which treats knowledge as something created by other people. As students, we are often passive receivers of knowledge: asked to do a specified set of readings, then graded on how well we reproduce what we’ve read.

Researchers, however, must take on an active role as knowledge producers . Doing research requires more of you than reading and absorbing what other people have written: you have to engage in a dialogue with it. That includes arguing with previous knowledge and perhaps trying to show that ideas we have accepted as given are actually wrong or incomplete. For example, rather than simply taking in the claims of an author you read, you’ll need to draw out the implications of those claims: if what the author is saying is true, what else does that suggest must be true? What predictions could you make based on the author’s claims?

In other words, rather than treating a reading as a source of truth – even if it comes from a revered source, such as Plato or Marie Curie – this orientation step asks you to treat the claims you read as provisional and subject to interrogation. That is one of the great pieces of wisdom that science and philosophy can teach us: that the biggest advances in human understanding have been made not by being correct about trivial things, but by being wrong in an interesting way . For example, Albert Einstein was wrong about quantum mechanics, but his arguments about it with his fellow physicist Niels Bohr have led to some of the biggest breakthroughs in science, even a century later.

Step 2: Define your research question

Students often give this step cursory attention, but experienced researchers know that formulating a good question is sometimes the most difficult part of the research planning process. That is because the precise language of the question frames the rest of the project. It’s therefore important to pose the question carefully, in a way that’s both possible to answer and likely to yield interesting results. Of course, you must choose a question that interests you, but that’s only the beginning of what’s likely to be an iterative process: most researchers come back to this step repeatedly, modifying their questions in light of previous research, resource limitations and other considerations.

Researchers face limits in terms of time and money. They, like everyone else, have to pose research questions that they can plausibly answer given the constraints they face. For example, it would be inadvisable to frame a project around the question ‘What are the roots of the Arab-Israeli conflict?’ if you have only a week to develop an answer and no background on that topic. That’s not to limit your imagination: you can come up with any question you’d like. But it typically does require some creativity to frame a question that you can answer well – that is, by investigating thoroughly and providing new insights – within the limits you face.

In addition to being interesting to you, and feasible within your resource constraints, the third and most important characteristic of a ‘good’ research topic is whether it allows you to create new knowledge. It might turn out that your question has already been asked and answered to your satisfaction: if so, you’ll find out in the next step of this process. On the other hand, you might come up with a research question that hasn’t been addressed previously. Before you get too excited about breaking uncharted ground, consider this: a lot of potentially researchable questions haven’t been studied for good reason ; they might have answers that are trivial or of very limited interest. This could include questions such as ‘Why does the area of a circle equal π r²?’ or ‘Did winter conditions affect Napoleon’s plans to invade Russia?’ Of course, you might be able to make the argument that a seemingly trivial question is actually vitally important, but you must be prepared to back that up with convincing evidence. The exercise in the ‘Learn More’ section below will help you think through some of these issues.

Finally, scholarly research questions must in some way lead to new and distinctive insights. For example, lots of people have studied gender roles in sports teams; what can you ask that hasn’t been asked before? Reinventing the wheel is the number-one no-no in this endeavour. That’s why the next step is so important: reviewing previous research on your topic. Depending on what you find in that step, you might need to revise your research question; iterating between your question and the existing literature is a normal process. But don’t worry: it doesn’t go on forever. In fact, the iterations taper off – and your research question stabilises – as you develop a firm grasp of the current state of knowledge on your topic.

Step 3: Review previous research

In academic research, from articles to books, it’s common to find a section called a ‘literature review’. The purpose of that section is to describe the state of the art in knowledge on the research question that a project has posed. It demonstrates that researchers have thoroughly and systematically reviewed the relevant findings of previous studies on their topic, and that they have something novel to contribute.

Your own research project should include something like this, even if it’s a high-school term paper. In the research planning process, you’ll want to list at least half a dozen bullet points stating the major findings on your topic by other people. In relation to those findings, you should be able to specify where your project could provide new and necessary insights. There are two basic rhetorical positions one can take in framing the novelty-plus-importance argument required of academic research:

• Position 1 requires you to build on or extend a set of existing ideas; that means saying something like: ‘Person A has argued that X is true about gender; this implies Y, which has not yet been tested. My project will test Y, and if I find evidence to support it, that will change the way we understand gender.’
• Position 2 is to argue that there is a gap in existing knowledge, either because previous research has reached conflicting conclusions or has failed to consider something important. For example, one could say that research on middle schoolers and gender has been limited by being conducted primarily in coeducational environments, and that findings might differ dramatically if research were conducted in more schools where the student body was all-male or all-female.

Your overall goal in this step of the process is to show that your research will be part of a larger conversation: that is, how your project flows from what’s already known, and how it advances, extends or challenges that existing body of knowledge. That will be the contribution of your project, and it constitutes the motivation for your research.

Two things are worth mentioning about your search for sources of relevant previous research. First, you needn’t look only at studies on your precise topic. For example, if you want to study gender-identity formation in schools, you shouldn’t restrict yourself to studies of schools; the empirical setting (schools) is secondary to the larger social process that interests you (how people form gender identity). That process occurs in many different settings, so cast a wide net. Second, be sure to use legitimate sources – meaning publications that have been through some sort of vetting process, whether that involves peer review (as with academic journal articles you might find via Google Scholar) or editorial review (as you’d find in well-known mass media publications, such as The Economist or The Washington Post ). What you’ll want to avoid is using unvetted sources such as personal blogs or Wikipedia. Why? Because anybody can write anything in those forums, and there is no way to know – unless you’re already an expert – if the claims you find there are accurate. Often, they’re not.

Step 4: Choose your data and methods

Whatever your research question is, eventually you’ll need to consider which data source and analytical strategy are most likely to provide the answers you’re seeking. One starting point is to consider whether your question would be best addressed by qualitative data (such as interviews, observations or historical records), quantitative data (such as surveys or census records) or some combination of both. Your ideas about data sources will, in turn, suggest options for analytical methods.

You might need to collect your own data, or you might find everything you need readily available in an existing dataset someone else has created. A great place to start is with a research librarian: university libraries always have them and, at public universities, those librarians can work with the public, including people who aren’t affiliated with the university. If you don’t happen to have a public university and its library close at hand, an ordinary public library can still be a good place to start: the librarians are often well versed in accessing data sources that might be relevant to your study, such as the census, or historical archives, or the Survey of Consumer Finances.

Because your task at this point is to plan research, rather than conduct it, the purpose of this step is not to commit you irrevocably to a course of action. Instead, your goal here is to think through a feasible approach to answering your research question. You’ll need to find out, for example, whether the data you want exist; if not, do you have a realistic chance of gathering the data yourself, or would it be better to modify your research question? In terms of analysis, would your strategy require you to apply statistical methods? If so, do you have those skills? If not, do you have time to learn them, or money to hire a research assistant to run the analysis for you?

Please be aware that qualitative methods in particular are not the casual undertaking they might appear to be. Many people make the mistake of thinking that only quantitative data and methods are scientific and systematic, while qualitative methods are just a fancy way of saying: ‘I talked to some people, read some old newspapers, and drew my own conclusions.’ Nothing could be further from the truth. In the final section of this guide, you’ll find some links to resources that will provide more insight on standards and procedures governing qualitative research, but suffice it to say: there are rules about what constitutes legitimate evidence and valid analytical procedure for qualitative data, just as there are for quantitative data.

Circle back and consider revising your initial plans

As you work through these four steps in planning your project, it’s perfectly normal to circle back and revise. Research planning is rarely a linear process. It’s also common for new and unexpected avenues to suggest themselves. As the sociologist Thorstein Veblen wrote in 1908 : ‘The outcome of any serious research can only be to make two questions grow where only one grew before.’ That’s as true of research planning as it is of a completed project. Try to enjoy the horizons that open up for you in this process, rather than becoming overwhelmed; the four steps, along with the two exercises that follow, will help you focus your plan and make it manageable.

Key points – How to plan a research project

• Planning a research project is essential no matter your academic level or field of study. There is no one ‘best’ way to design research, but there are certain guidelines that can be helpfully applied across disciplines.
• Orient yourself to knowledge-creation. Make the shift from being a consumer of information to being a producer of information.
• Define your research question. Your question frames the rest of your project, sets the scope, and determines the kinds of answers you can find.
• Review previous research on your question. Survey the existing body of relevant knowledge to ensure that your research will be part of a larger conversation.
• Choose your data and methods. For instance, will you be collecting qualitative data, via interviews, or numerical data, via surveys?
• Circle back and consider revising your initial plans. Expect your research question in particular to undergo multiple rounds of refinement as you learn more about your topic.

Good research questions tend to beget more questions. This can be frustrating for those who want to get down to business right away. Try to make room for the unexpected: this is usually how knowledge advances. Many of the most significant discoveries in human history have been made by people who were looking for something else entirely. There are ways to structure your research planning process without over-constraining yourself; the two exercises below are a start, and you can find further methods in the Links and Books section.

The following exercise provides a structured process for advancing your research project planning. After completing it, you’ll be able to do the following:

• describe clearly and concisely the question you’ve chosen to study
• summarise the state of the art in knowledge about the question, and where your project could contribute new insight
• identify the best strategy for gathering and analysing relevant data

In other words, the following provides a systematic means to establish the building blocks of your research project.

Exercise 1: Definition of research question and sources

This exercise prompts you to select and clarify your general interest area, develop a research question, and investigate sources of information. The annotated bibliography will also help you refine your research question so that you can begin the second assignment, a description of the phenomenon you wish to study.

Jot down a few bullet points in response to these two questions, with the understanding that you’ll probably go back and modify your answers as you begin reading other studies relevant to your topic:

• What will be the general topic of your paper?
• What will be the specific topic of your paper?

b) Research question(s)

Use the following guidelines to frame a research question – or questions – that will drive your analysis. As with Part 1 above, you’ll probably find it necessary to change or refine your research question(s) as you complete future assignments.

• Your question should be phrased so that it can’t be answered with a simple ‘yes’ or ‘no’.
• Your question should have more than one plausible answer.
• Your question should draw relationships between two or more concepts; framing the question in terms of How? or What? often works better than asking Why ?

c) Annotated bibliography

Most or all of your background information should come from two sources: scholarly books and journals, or reputable mass media sources. You might be able to access journal articles electronically through your library, using search engines such as JSTOR and Google Scholar. This can save you a great deal of time compared with going to the library in person to search periodicals. General news sources, such as those accessible through LexisNexis, are acceptable, but should be cited sparingly, since they don’t carry the same level of credibility as scholarly sources. As discussed above, unvetted sources such as blogs and Wikipedia should be avoided, because the quality of the information they provide is unreliable and often misleading.

To create an annotated bibliography, provide the following information for at least 10 sources relevant to your specific topic, using the format suggested below.

Name of author(s):

Publication date:

Title of book, chapter, or article:

If a chapter or article, title of journal or book where they appear:

Brief description of this work, including main findings and methods ( c 75 words):

Summary of how this work contributes to your project ( c 75 words):

Brief description of the implications of this work ( c 25 words):

Identify any gap or controversy in knowledge this work points up, and how your project could address those problems ( c 50 words):

Exercise 2: Towards an analysis

Develop a short statement ( c 250 words) about the kind of data that would be useful to address your research question, and how you’d analyse it. Some questions to consider in writing this statement include:

• What are the central concepts or variables in your project? Offer a brief definition of each.
• Do any data sources exist on those concepts or variables, or would you need to collect data?
• Of the analytical strategies you could apply to that data, which would be the most appropriate to answer your question? Which would be the most feasible for you? Consider at least two methods, noting their advantages or disadvantages for your project.

Links & books

One of the best texts ever written about planning and executing research comes from a source that might be unexpected: a 60-year-old work on urban planning by a self-trained scholar. The classic book The Death and Life of Great American Cities (1961) by Jane Jacobs (available complete and free of charge via this link ) is worth reading in its entirety just for the pleasure of it. But the final 20 pages – a concluding chapter titled ‘The Kind of Problem a City Is’ – are really about the process of thinking through and investigating a problem. Highly recommended as a window into the craft of research.

Jacobs’s text references an essay on advancing human knowledge by the mathematician Warren Weaver. At the time, Weaver was director of the Rockefeller Foundation, in charge of funding basic research in the natural and medical sciences. Although the essay is titled ‘A Quarter Century in the Natural Sciences’ (1960) and appears at first blush to be merely a summation of one man’s career, it turns out to be something much bigger and more interesting: a meditation on the history of human beings seeking answers to big questions about the world. Weaver goes back to the 17th century to trace the origins of systematic research thinking, with enthusiasm and vivid anecdotes that make the process come alive. The essay is worth reading in its entirety, and is available free of charge via this link .

For those seeking a more in-depth, professional-level discussion of the logic of research design, the political scientist Harvey Starr provides insight in a compact format in the article ‘Cumulation from Proper Specification: Theory, Logic, Research Design, and “Nice” Laws’ (2005). Starr reviews the ‘research triad’, consisting of the interlinked considerations of formulating a question, selecting relevant theories and applying appropriate methods. The full text of the article, published in the scholarly journal Conflict Management and Peace Science , is available, free of charge, via this link .

Finally, the book Getting What You Came For (1992) by Robert Peters is not only an outstanding guide for anyone contemplating graduate school – from the application process onward – but it also includes several excellent chapters on planning and executing research, applicable across a wide variety of subject areas. It was an invaluable resource for me 25 years ago, and it remains in print with good reason; I recommend it to all my students, particularly Chapter 16 (‘The Thesis Topic: Finding It’), Chapter 17 (‘The Thesis Proposal’) and Chapter 18 (‘The Thesis: Writing It’).

Emerging therapies

How to use cooking as a form of therapy

No matter your culinary skills, spend some reflective time in the kitchen to nourish and renew your sense of self

by Charlotte Hastings

Meaning and the good life

How to appreciate what you have

To better face an imperfect world, try a deeper reflection on the things, people and legacies that make your life possible

by Avram Alpert

How to use ‘possibility thinking’

Have you hit an impasse in your personal or professional life? Answer these questions to open your mind to what’s possible

by Constance de Saint Laurent & Vlad Glăveanu

Writing a Project Proposal

Main navigation, a good proposal describes....

• what you hope to accomplish
• why those objectives are important to your academic or artistic field
• how you intend to achieve your objectives

Your original project proposal is the core of your grant application. 

Detailed Proposal Requirements

• General guidelines for all grant proposals
• Additional specific guidelines for  Research, Arts/Design, and Senior Synthesis  project proposals -- please follow carefully!
• Ways to turn your good proposal into a great one
• Sample Project Proposals : Check out exemplars of past student project proposals.

Connect with Faculty Mentors and UADs

• Faculty Mentors should meet required eligibility criteria .
• Students should  schedule a meeting with their Undergraduate Advising Director (UAD)  as they write their proposal. UADs are well-versed with all VPUE Undergraduate Research Grants!

Watch a 3-minute overview of the VPUE Student Grant application process.

Watch a 2-minute video on how to write the critical dialogue section of a creative arts project proposal.

• Latest CDC Covid-19 Info
• CSR Extranet
• Staff Directory
• Contact CSR

For Applicants

CSR’s primary role is to handle the receipt and review of ~ 75% of the grant applications that NIH receives. NIH separates the review process from funding decisions.

• For Reviewers

Reviewers are critical to our mission to see that NIH grant applications receive, fair, independent, expert, and timely scientific reviews. We appreciate the generosity with which reviewers give their time.

News & Policy

The latest news and policy updates from CSR. Read about our outreach programs and publications.

Study Sections

Applications are reviewed in study sections (Scientific Review Group, SRG). Review Branches (RBs) are clusters of study sections based on scientific discipline.

Review Panels & Dates

Guidelines & templates, general guidelines.

• Guidelines, Critique Templates & Review Criteria
• Conflict of Interest
• Working with the Critique Templates
• Rigor and Reproducibility
• Consolidated List of Reviewer Documents

Specific Review Guidelines

• R Awards (Research Project Grants: R01, R03, R21, SBIR/STTR, etc.)
• K Awards (Career Development)
• F Awards (Fellowships)
• T Awards (Training)
• C06, UC6 & G20 Awards
• Administrative Centers

Last updated: 08/10/2021 10:21

• More Social Media from NIH

Geography Department Penn State

• ORIENTATION
• INSTRUCTOR INFORMATION
• PROGRAM HOME PAGE
• LIBRARY RESOURCES
• GETTING HELP

Research Project Guidelines

A central part of earning a master's degree is demonstrating the ability to conduct research and to analyze real-world phenomena. The value of doing research and analysis is not limited to academia; these are skills that have immense practical value in both personal and professional spheres, especially within the fields of intelligence and human security, where analysis is the heart of the work. It takes training and practice to learn and refine research and analysis skills. This is why the successful completion of a well-written and well-documented research project is one of the requirements to pass GEOG 571. This research project is worth 230 points total.

What Is a Research Project?

We want to make an important distinction between research projects and reports.

A report is a project where a student chooses a topic, consults relevant sources from which they collect and collate facts about that topic, composes a carefully organized summary or discussion of those facts, and delivers all of this as a paper or presentation.

By contrast, a research project is a project in which a student formulates a central question, consults relevant sources from which they collect and collate information (which can include basic facts, quantitative data, and qualitative data) relating to that question, analyzes and interprets the information they have found, and develops a defensible argument that supports their interpretation. The argument is usually presented in its most basic form as a thesis statement, which appears in the introduction to the completed project, and the completed project is often delivered as a paper or presentation.

Reports and research projects have some things in common: they require a student to identify a question or topic worthy of study; identify, vet, and consult a number and variety of relevant sources; synthesize information from those sources; and compose some kind of deliverable that presents their work. Yet there are a couple of crucial differences that we want to highlight here.

First, while a report centers on a topic , a research project centers on a question . Second, while a report presents a synthesis of information on its topic, a research project makes an argument based on the researcher’s analysis of the information. Both a report and a research project can be a lot of work — but the difference in approach up front (i.e., choosing a question rather than a topic) means that a research project entails a level of critical thinking and analytical skills that a report does not.

Ultimately, where a report simply presents existing knowledge, the goal of a research project is to present new knowledge (or a new interpretation) about something. This is what makes research projects common requirements of graduate-level courses — including this one.

Project Medium:

Research projects are traditionally delivered as research papers. Yet we now have technology that enables us to present research effectively across a variety of media. For this course, you may choose one of the following options for presenting your research:

• a traditional research paper
• ArcGIS Online StoryMap

You may also suggest another medium. If you choose to present your project in a medium other than a research paper, please make sure to discuss your vision with your instructor.

Make sure that you choose a medium that is appropriate to your project. Certain kinds of projects lend themselves better to some formats than others. If your evidence is best presented in a set of interactive maps, or if your project includes lots of visuals (still or video), StoryMaps is a good option. If your project revolves around interviews that you are conducting, or if there is a significant audio component, a podcast might be appropriate. If your project is purely or primarily text-driven (possibly with a map or table or two), you should present it as a traditional research paper.

Regardless of the medium, the research project must result in the equivalent of 12 to 15 pages of text (not including title, reference, graphics, or figures). Note that for podcasts, this translates to 20 to 30 minutes of airtime. The project must also be documented with the APA citation guidelines . Please visit the Penn State University Libraries APA Quick Citation Guide to make sure you are using the proper citations.

Note for students submitting podcasts : your podcast must be accompanied by two things:

• either a script or transcript (some students have had good luck with Welder’s free audio-to-text transcriptions , though you will have to edit the output for spelling and format)
• a separate document containing your works cited, properly formatted in APA style.

Reminder: you are expected to incorporate citations in the podcast itself (e.g., by conversationally stating the author and year of a study). If you are not sure how best to do this, contact your instructor.

If you choose to present your project as a paper, it must use these formatting guidelines:

• 12 point font
• double-spaced
• 1” margins on all sides

Research Project Deliverables:  

• Week 2 - Project Proposal (20 points) Students will submit via Canvas a primary research question that will be the focal point of their project. In addition to this, students will provide a statement justifying the merits of completing this research, a list of three potential sources, a set of keywords or search terms they will use to search for additional sources, and the medium they intend to use to convey their research. Note that the medium is not set in stone and can be changed later.
• Week 4 - Project Outline and Annotated Bibliography (50 points) Students will submit an outline that includes the main topic areas of their research project along with an annotated bibliography.
• Week 7 - Rough Draft (30 points) Students will submit a draft of the project.
• Week 9 - Final Draft (130 points) Students will submit the final project (130 points.) Note: In addition to the final draft, a one-page Executive Summary of the project (worth 75 points) is due as part of Lesson 9.

Choosing a Research Question:

The key to completing a successful research project is identifying and constructing a good research question. For a research question to be good, it needs to fulfill some basic criteria (the following bullet points come from the George Mason University Writing Center website):

• clear: it provides enough specifics that one’s audience can easily understand its purpose without needing additional explanation.
• focused: it is narrow enough that it can be answered thoroughly in the space the writing task allows.
• concise: it is expressed in the fewest possible words.
• complex: it is not answerable with a simple “yes” or “no,” but rather requires synthesis and analysis of ideas and sources prior to composition of an answer.
• arguable: its potential answers are open to debate rather than accepted facts.

All of these points are important for generating a good research question. Given that this is a graduate level course, we want to emphasize the last two points here. The topics and concepts that we use in this course are complex, and your research is meant to generate new knowledge or a new interpretation of existing knowledge. Be mindful of this as you generate your research question. If you can answer your research question with an unqualified “yes” or “no” after a few minutes’ research, or if your sources provide you with a clear-cut answer to your question, it means that your question is not deep enough to merit a research project.

Your research project should sit at the nexus of cultural/political geography and intelligence or human security. That is, your project should consider an issue related to intelligence or human security from a geographical perspective . We encourage you to develop your primary research question around your professional interests; we expect you to take a geographical approach to answering that question.

Your research project should be focused on one or more specific places or regions within the world, and on a specific issue with respect to intelligence or human security. Below are some examples of general research questions that you can use as a starting point to build a more specific question tailored to your interests:

• How does the creation of border walls impact conflict?
• What role do GIS and cartography play in the geographic separation of people?
• What role(s) does ethnicity play in the development of separatist movements?
• How well do disaster relief policies accommodate people’s attachments to their homes?
• How well do refugee resettlement programs handle cultural differences between local and refugee populations?
• What practices or policies can be established to mitigate ethical differences between the disciplines of human geography and intelligence?
• How are geography, immigration, and marginalization related with regard to radicalization?
• How does the distribution of natural resources impact conflict?
• How does the distribution of natural resources impact migration?
• How are territorial control and the geographic distribution of terrorist attacks related in conflict zones?

Remember: the questions above are meant to get you thinking. Many of the questions posed above are far too broad for you to conduct a research project within the scope of this assignment. You will likely need to hone one of these questions to focus on a more specific question. Note that many of these could be designed to focus on a single place, or to present two or more case studies in comparison. Bear in mind that any research question that you propose should be as specific as possible with regard to place or region, actors, conflict, and other relevant factors.

To help you understand what we mean by “specific,” here are some examples of research questions around which students in other terms have based their research projects:

• What role has ethnic identity played in the ongoing conflict in the Donbas region of Ukraine?
• How have place identity and human geography shaped support for paramilitary organizations in Northern Ireland?
• Do incidents of domestic terrorism and international terrorism display different geographic patterns? [This project presented and compared several case studies.]
• What was the relationship between nationalism and religion in the Northern Ireland Conflict?
• How have the PLO and Hamas differed in their approach to Palestinian identity, and what are the geopolitical impacts of these differences?
• What is the significance of natural resources in the conflict over the Grand Ethiopian Renaissance Dam?
• How do geography and identity influence radicalization among ethnic minorities? [This project presented and compared several case studies.]

A Note on Sources:

A good project will incorporate at least 16-20 vetted sources. These should be primarily scholarly sources, though some projects will also rely on gray literature. Projects that use GIS data should list all data sources in their works cited, but only two of these will count toward the source minimum. Students should not use Britannica, Wikipedia, or other general tertiary sources (though it is okay to use specialized encyclopedias such as the Dictionary of Human Geography ). See below for more information.

Projects will be graded on strength and sophistication of analysis, clarity of presentation, logic, reliance on accurate information and facts, integration of reading materials (including online lectures), and attention to detail. The grading rubric is provided in the assignment dropbox.

The grade will not be based upon the position taken relative to the issue, but instead will rest upon the accuracy and effectiveness of applying geographic thought. Simply arguing that you "feel" a certain way about something is not a reasonable defense of your position. Instead, you will need to cite relevant sources to support your assertions, with the majority of these sources being scholarly sources. The following video from the University of South Australia describes which sources are considered scholarly.

• Privacy Policy

Home » How To Write A Research Proposal – Step-by-Step [Template]

How To Write A Research Proposal – Step-by-Step [Template]

Table of Contents

How To Write a Research Proposal

Writing a Research proposal involves several steps to ensure a well-structured and comprehensive document. Here is an explanation of each step:

1. Title and Abstract

• Choose a concise and descriptive title that reflects the essence of your research.
• Write an abstract summarizing your research question, objectives, methodology, and expected outcomes. It should provide a brief overview of your proposal.

2. Introduction:

• Provide an introduction to your research topic, highlighting its significance and relevance.
• Clearly state the research problem or question you aim to address.
• Discuss the background and context of the study, including previous research in the field.

3. Research Objectives

• Outline the specific objectives or aims of your research. These objectives should be clear, achievable, and aligned with the research problem.

4. Literature Review:

• Conduct a comprehensive review of relevant literature and studies related to your research topic.
• Summarize key findings, identify gaps, and highlight how your research will contribute to the existing knowledge.

5. Methodology:

• Describe the research design and methodology you plan to employ to address your research objectives.
• Explain the data collection methods, instruments, and analysis techniques you will use.
• Justify why the chosen methods are appropriate and suitable for your research.

6. Timeline:

• Create a timeline or schedule that outlines the major milestones and activities of your research project.
• Break down the research process into smaller tasks and estimate the time required for each task.

7. Resources:

• Identify the resources needed for your research, such as access to specific databases, equipment, or funding.
• Explain how you will acquire or utilize these resources to carry out your research effectively.

8. Ethical Considerations:

• Discuss any ethical issues that may arise during your research and explain how you plan to address them.
• If your research involves human subjects, explain how you will ensure their informed consent and privacy.

9. Expected Outcomes and Significance:

• Clearly state the expected outcomes or results of your research.
• Highlight the potential impact and significance of your research in advancing knowledge or addressing practical issues.

10. References:

• Provide a list of all the references cited in your proposal, following a consistent citation style (e.g., APA, MLA).

11. Appendices:

• Include any additional supporting materials, such as survey questionnaires, interview guides, or data analysis plans.

Research Proposal Format

The format of a research proposal may vary depending on the specific requirements of the institution or funding agency. However, the following is a commonly used format for a research proposal:

1. Title Page:

• Include the title of your research proposal, your name, your affiliation or institution, and the date.

2. Abstract:

• Provide a brief summary of your research proposal, highlighting the research problem, objectives, methodology, and expected outcomes.

3. Introduction:

• Introduce the research topic and provide background information.
• State the research problem or question you aim to address.
• Explain the significance and relevance of the research.
• Review relevant literature and studies related to your research topic.
• Summarize key findings and identify gaps in the existing knowledge.
• Explain how your research will contribute to filling those gaps.

5. Research Objectives:

• Clearly state the specific objectives or aims of your research.
• Ensure that the objectives are clear, focused, and aligned with the research problem.

6. Methodology:

• Describe the research design and methodology you plan to use.
• Explain the data collection methods, instruments, and analysis techniques.
• Justify why the chosen methods are appropriate for your research.

7. Timeline:

8. Resources:

• Explain how you will acquire or utilize these resources effectively.

9. Ethical Considerations:

• If applicable, explain how you will ensure informed consent and protect the privacy of research participants.

10. Expected Outcomes and Significance:

11. References:

12. Appendices:

Research Proposal Template

Here’s a template for a research proposal:

1. Introduction:

2. Literature Review:

3. Research Objectives:

4. Methodology:

5. Timeline:

6. Resources:

7. Ethical Considerations:

8. Expected Outcomes and Significance:

9. References:

10. Appendices:

Research Proposal Sample

Title: The Impact of Online Education on Student Learning Outcomes: A Comparative Study

1. Introduction

Online education has gained significant prominence in recent years, especially due to the COVID-19 pandemic. This research proposal aims to investigate the impact of online education on student learning outcomes by comparing them with traditional face-to-face instruction. The study will explore various aspects of online education, such as instructional methods, student engagement, and academic performance, to provide insights into the effectiveness of online learning.

2. Objectives

The main objectives of this research are as follows:

• To compare student learning outcomes between online and traditional face-to-face education.
• To examine the factors influencing student engagement in online learning environments.
• To assess the effectiveness of different instructional methods employed in online education.
• To identify challenges and opportunities associated with online education and suggest recommendations for improvement.

3. Methodology

3.1 Study Design

This research will utilize a mixed-methods approach to gather both quantitative and qualitative data. The study will include the following components:

3.2 Participants

The research will involve undergraduate students from two universities, one offering online education and the other providing face-to-face instruction. A total of 500 students (250 from each university) will be selected randomly to participate in the study.

3.3 Data Collection

The research will employ the following data collection methods:

• Quantitative: Pre- and post-assessments will be conducted to measure students’ learning outcomes. Data on student demographics and academic performance will also be collected from university records.
• Qualitative: Focus group discussions and individual interviews will be conducted with students to gather their perceptions and experiences regarding online education.

3.4 Data Analysis

Quantitative data will be analyzed using statistical software, employing descriptive statistics, t-tests, and regression analysis. Qualitative data will be transcribed, coded, and analyzed thematically to identify recurring patterns and themes.

4. Ethical Considerations

The study will adhere to ethical guidelines, ensuring the privacy and confidentiality of participants. Informed consent will be obtained, and participants will have the right to withdraw from the study at any time.

5. Significance and Expected Outcomes

This research will contribute to the existing literature by providing empirical evidence on the impact of online education on student learning outcomes. The findings will help educational institutions and policymakers make informed decisions about incorporating online learning methods and improving the quality of online education. Moreover, the study will identify potential challenges and opportunities related to online education and offer recommendations for enhancing student engagement and overall learning outcomes.

6. Timeline

The proposed research will be conducted over a period of 12 months, including data collection, analysis, and report writing.

The estimated budget for this research includes expenses related to data collection, software licenses, participant compensation, and research assistance. A detailed budget breakdown will be provided in the final research plan.

8. Conclusion

This research proposal aims to investigate the impact of online education on student learning outcomes through a comparative study with traditional face-to-face instruction. By exploring various dimensions of online education, this research will provide valuable insights into the effectiveness and challenges associated with online learning. The findings will contribute to the ongoing discourse on educational practices and help shape future strategies for maximizing student learning outcomes in online education settings.

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

How To Write A Proposal – Step By Step Guide...

Grant Proposal – Example, Template and Guide

How To Write A Business Proposal – Step-by-Step...

Business Proposal – Templates, Examples and Guide

Proposal – Types, Examples, and Writing Guide

How to choose an Appropriate Method for Research?

• Getting Started
• Hearst URCC Faculty Research Assistant Program
• URCC Peer Research Mentors
• Find Brandeis Research Opportunities and Faculty Mentors
• Frequently Asked Questions
• Research By Class Year
• Guidelines for Undergraduate Research Participation
• Research Advice from Brandeis Undergraduates
• Undergraduate Research Spotlights
• Faculty-Undergraduate Research Partnership Spotlight
• Graduate Research Mentor Spotlight
• Alumni Spotlights
• Past Spotlights
• Brandeis Funding Opportunities
• Dr. John R. Hose P’97, P’00 Research Fellowship
• Provost's Undergraduate Research Fund
• Jerome A. Schiff Undergraduate Fellows Program
• Ariel Weissmann Undergraduate Research Fund
• David & Lola Swede P'89 Research Fellowship for Jewish Studies
• Phoebe Rothman Giddon Brandeis Alumna Undergraduate Research Fellowship
• Fisher Explorer Grants
• Lurie Undergraduate Fellowship in Disability Policy
• Science Research
• External Funding Opportunities
• Recent Awards
• Research Support
• Spring 2023 Undergraduate Research Symposium
• Symposium Key Dates and Deadlines
• Oral Presentation Tips
• Presentation Templates
• URCC Abstract writing guidelines
• Tips for Faculty
• Suggestions for Faculty on Getting Started
• Connect with Undergraduate and Faculty Research Collaborators
• Hearst URCC Undergraduate Faculty Research Assistant Program
• Faculty Resources for Undergraduate Research
• Faculty-Staff URCC Advisory Group
• Graduate Researcher and Mentor Spotlight
• Undergraduate Research News
• URCC Newsletters
• Degree Programs
• Majors and Minors
• Graduate Programs
• The Brandeis Core
• School of Arts and Sciences
• Brandeis Online
• Brandeis International Business School
• Graduate School of Arts and Sciences
• Heller School for Social Policy and Management
• Rabb School of Continuing Studies
• Precollege Programs
• Faculty and Researcher Directory
• Brandeis Library
• Academic Calendar
• Undergraduate Admissions
• Summer School
• Financial Aid
• Research that Matters
• Resources for Researchers
• Brandeis Researchers in the News
• Provost Research Grants
• Faculty Research
• Student Research
• Centers and Institutes
• Office of the Vice Provost for Research
• Office of the Provost
• Housing/Community Living
• Campus Calendar
• Student Engagement
• Clubs and Organizations
• Community Service
• Dean of Students Office
• Orientation
• Hiatt Career Center
• Spiritual Life
• Graduate Student Affairs
• Directory of Campus Contacts
• Division of Creative Arts
• Brandeis Arts Engagement
• Rose Art Museum
• Bernstein Festival of the Creative Arts
• Theater Arts Productions
• Brandeis Concert Series
• Public Sculpture at Brandeis
• Women's Studies Research Center
• Creative Arts Award
• Our Jewish Roots
• The Framework for the Future
• Mission and Diversity Statements
• Distinguished Faculty
• Nobel Prize 2017
• Notable Alumni
• Administration
• Working at Brandeis
• Commencement
• Offices Directory
• Faculty & Staff
• Alumni & Friends
• Parents & Families
• 75th Anniversary
• New Students
• Shuttle Schedules
• Support at Brandeis

Undergraduate Research and Creative Collaborations

Guidelines for undergraduate participation in faculty research projects, including independent laboratory research and research volunteers.

Approved Oct 27, 2021

Participating in independent research as an undergraduate within laboratories and other research and creative project opportunities directed by Brandeis faculty is a valuable educational experience encouraged across the academic disciplines. The following guidelines are based on historical practices at Brandeis and provide a basic outline for the process.

Students must have permission from the faculty mentor (principal investigator [PI] responsible for operation of the group or leading the research project) to participate in the research or lab project.

A member of the research team should be designated to act as a direct supervisor (grad student, post-doc, research staff member and/or the faculty mentor/PI themselves).

Student researchers must comply with all applicable safety, confidentiality and compliance rules, documented through completing mandatory online or in-person training, including human subjects policies (IRB) or animal research rules (IACUC). The faculty PI is responsible for ensuring that all students have completed all required training before commencing research, and that they comply with all relevant rules and protocols.

In normal circumstances, students either should be paid or receive academic credit for independent research.

Students will normally register for a course during the academic year. The one-credit courses numbered 91g (“Introduction to Research Practice”) will cover most students starting on a research project.

In exceptional circumstances, students may petition to volunteer outside of the employment and course structures for a period of up to four months. Petitions should originate with the faculty mentor and be approved first by the department chair or their designate, and then by the Undergraduate Research and Creative Collaborations Office ( URCC). Volunteers are not allowed to participate until the petition is approved.

Situations for which a volunteer position may be appropriate might include:

Starting research during the middle of a semester.

Working during the summer if fellowship or grant funding is unavailable.

Volunteering in a lab for more than one summer or semester should be avoided.

Only projects with a research or learning component are governed by these rules. Students whose only task is support (e.g., making media, flipping flies, copying documents, etc.) should always be paid as hourly student employees. Faculty can review the defining characteristics of an unpaid volunteer position as described in the U.S. Department of Labor Fact Sheet on unpaid internships .

All students are strongly encouraged to contact the URCC which can assist in finding appropriate research opportunities or fellowships to fund their research participation.

• Research Spotlights
• Funding Opportunities
• Undergraduate Research Symposium
• For Faculty
• News and Events

Contact For More Information

Margaret Lynch, PhD Director of Undergraduate-Faculty Research Partnerships [email protected]

Purdue Online Writing Lab Purdue OWL® College of Liberal Arts

Welcome to the Purdue Online Writing Lab

Welcome to the Purdue OWL

This page is brought to you by the OWL at Purdue University. When printing this page, you must include the entire legal notice.

Copyright ©1995-2018 by The Writing Lab & The OWL at Purdue and Purdue University. All rights reserved. This material may not be published, reproduced, broadcast, rewritten, or redistributed without permission. Use of this site constitutes acceptance of our terms and conditions of fair use.

The Online Writing Lab at Purdue University houses writing resources and instructional material, and we provide these as a free service of the Writing Lab at Purdue. Students, members of the community, and users worldwide will find information to assist with many writing projects. Teachers and trainers may use this material for in-class and out-of-class instruction.

The Purdue On-Campus Writing Lab and Purdue Online Writing Lab assist clients in their development as writers—no matter what their skill level—with on-campus consultations, online participation, and community engagement. The Purdue Writing Lab serves the Purdue, West Lafayette, campus and coordinates with local literacy initiatives. The Purdue OWL offers global support through online reference materials and services.

A Message From the Assistant Director of Content Development 

The Purdue OWL® is committed to supporting  students, instructors, and writers by offering a wide range of resources that are developed and revised with them in mind. To do this, the OWL team is always exploring possibilties for a better design, allowing accessibility and user experience to guide our process. As the OWL undergoes some changes, we welcome your feedback and suggestions by email at any time.

Please don't hesitate to contact us via our contact page  if you have any questions or comments.

All the best,

Social Media

Facebook twitter.

• Open access
• Published: 09 May 2024

Facilitators of and barriers to County Behavioral Health System Transformation and Innovation: an interview study

• Xin Zhao 1 , 2 ,
• Rachel Varisco 3 ,
• Judith Borghouts 3 ,
• Elizabeth V. Eikey 4 , 5 ,
• David Safani 6 ,
• Dana B. Mukamel 3 ,
• Stephen M. Schueller 7 &
• Dara H. Sorkin 3  

BMC Health Services Research volume  24 , Article number:  604 ( 2024 ) Cite this article

112 Accesses

2 Altmetric

Metrics details

Inadequate and inequitable access to quality behavioral health services and high costs within the mental health systems are long-standing problems. System-level (e.g., fee-for-service payment model, lack of a universal payor) and individual factors (e.g., lack of knowledge of existing resources) contribute to difficulties in accessing resources and services. Patients are underserved in County behavioral health systems in the United States. Orange County’s (California) Behavioral Health System Transformation project sought to improve access by addressing two parts of their system: developing a template for value-based contracts that promote payor-agnostic care (Part 1); developing a digital platform to support resource navigation (Part 2). Our aim was to evaluate facilitators of and barriers to each of these system changes.

We collected interview data from County or health care agency leaders, contracted partners, and community stakeholders. Themes were informed by the Consolidated Framework for Implementation Research.

Five themes were identified related to behavioral health system transformation, including 1) aligning goals and values, 2) addressing fit, 3) fostering engagement and partnership, 4) being aware of implementation contexts, and 5) promoting communication. A lack of fit into incentive structures and changing state guidelines and priorities were barriers to contract development. Involving diverse communities to inform design and content facilitated the process of developing digital tools.

Conclusions

The study highlights the multifaceted factors that help facilitate or hinder behavioral health system transformation, such as the need for addressing systematic and process behaviors, leveraging the knowledge of leadership and community stakeholders, fostering collaboration, and adapting to implementation contexts.

Peer Review reports

In the United States, the system of providing and coordinating behavioral health services is inefficient. Health care systems are largely paid via a “fee-for-service” model that incentivizes increasing the number of billable hours rather than improving patient outcomes and quality of care [ 1 ]. At the system level, provider shortage, disparities in insurance coverage, and the existing fee-for-service reimbursement model contributed to longstanding unmet service needs [ 2 , 3 , 4 , 5 ]. At the individual level, behavioral health stigma, limited mental health literacy, and lack of knowledge about appropriate resources challenge people’s ability to navigate and access behavioral health resources, especially among historically marginalized and uninsured groups [ 3 ].

In light of the multilevel barriers hindering access to behavioral health services, system transformation is needed. According to California Health Interview Survey reports, one in five Orange County residents reported they needed, but did not receive, behavioral health support [ 6 ]. Those from disadvantaged socioeconomic backgrounds were less likely to access behavioral health support than those with more resources. The unmet need within the County behavioral health system can be improved if providers and patients have access to information about care (efficient resource navigation) and patients get linked with value-based behavioral health services regardless of their insurance status (payor-agnostic care). Orange County’s Behavioral Health System Transformation (BHST) Innovation Project in California aims to create a patient-centered system where all residents in Orange County can be served regardless of their insurance status and clinical needs. An innovation project introduces a new practice or approach in the field of behavioral health with a primary focus on learning or process change. The BHST Innovation Project includes two parts: developing a template for value-based payment contracts that promote payor-agnostic care (Part 1) and developing a public-facing digital platform (OC Navigator) to increase access to information and support resource navigation (Part 2) [ 6 ]. Innovation projects are limited to a maximum of five years, with the expectation that successful projects should transition to integration into standard practices and sustainment. The data for this paper come from the first half of a five-year innovation project. Thus, the current paper focuses on early lessons learned regarding facilitators and barriers.

Backgrounds for components of the BHST Innovation Project

Part 1. developing a template for value-based payment contracts that promote payor-agnostic care.

Value-based payment models tie payments for services to the quality of care and patients’ clinical outcomes rather than the volume of services delivered [ 7 ]. Since 2016, several states in the US, such as Washington, New York, Minnesota, Maine, and Massachusetts, made attempts to implement value-based care [ 1 , 8 , 9 , 10 , 11 , 12 ]. In Washington and New York, transitioning to value-based care improved the quality of behavioral health services. In Washington State, a value-based care initiative targeted the implementation of the Collaborative Care Model, an evidence-based team approach to behavioral health interventions in primary care. In this program, 25% of funding to participating community clinics was contingent on meeting value-based payment targets (i.e., providing evidence-based care) and active participation in the program [ 8 ]. Compared to patients enrolling in the program without the value-based components, those enrolling in the program with value-based components were more likely to have depression outcomes that improved in a shorter amount of time (Bao et al., 2017). New York State Collaborative Care Medicaid also used a value-based program. In this program, 25% of the monthly, patient-level case-rate payment was withheld each month and was paid retroactively after six months for patients who had clinical improvement or had their treatment plan adjusted in response to a lack of clinical improvement [ 12 , 13 ]. This program led to an increase in the proportion of patients screened for depression and patients who showed clinical improvement after 10 weeks of treatment at participating sites, compared to before launching the program. In addition to improving the quality of care, value-based payment models also have the potential to help address challenges faced by the traditional fee-for-service model, such as overutilization of services and high costs [ 7 , 14 ].

Despite the reported positive implementation outcomes of the value-based payment initiatives, the process of implementing and sustaining value-based payment models is often challenging and varies largely by state [ 7 , 10 ]. In the United States, buy-in of value-based payment models from commercial payors is challenging. For example, qualitative analyses of interview data revealed limited interest in adopting value-based contracts with providers among commercial payors in Arkansas, Maine, and Minnesota [ 10 ]. Specifically, commercial payors expressed concerns about (1) the need for tailoring a value-based contract to align with just one state when they have business in multiple states; and (2) market competition, such as subsidizing the care of patients covered by other payors who did not make similar investments to adopt value-based payment models [ 10 ].

Payor-agnostic care allows all patients to be served regardless of their insurance status and clinical needs, as it prioritizes patient needs and outcomes above financial profitability. A payor-agnostic model typically includes supporting uninsured individuals. Funding sources for uninsured individuals might include self-pay options, philanthropic donations, and government grants. The barriers to and facilitators of multi-payor alignment have been more studied in primary care settings. One notable effort is the Comprehensive Primary Care (CPC) initiative launched by Centers for Medicare & Medicaid Services (CMS), which is one of the largest multi-payor initiatives [ 15 ]. Interviews from CMS staff, CPC-participating payors, and stakeholder organizations described that competitive market dynamics and competing institutional priorities were barriers to multi-payor and multi-sector collaboration. Leveraging champion support and seeking input on decisions related to system transformation from key community stakeholders helped build trusting relationships and align different payors. In the sphere of behavioral health, emerging efforts of moving towards payor-agnostic care hold great potential in ensuring equitable health care access. However, barriers to and facilitators of implementing payor agnostic behavioral health care are less known. One example was the Blue Shield Health Reimagined pilot program. In this program, Blue Shield embedded Community Health Advocates in ten primary and specialty care practices to provide payor-agnostic care to support individuals who were not Blue Shield members to receive services [ 16 ]. In the first fourteen months of the program, a large and diverse population was served (i.e., N  > 1,900 patients, > 30% Latinx/Hispanic) across four participating regions in California (Paulson et al., 2021). Paulson et al. (2021) analyzed focus group and interview data to identify facilitators of and barriers to embedding Community Health Advocates within primary and specialty care practices to provide payor-agnostic care, with a focus on intervention implementation. Overall, to improve access to behavioral health care access, additional work is needed to understand facilitators of and barriers to promoting payor-agnostic care.

Part 2. Developing a digital platform to increase access to information and support resource navigation

A digital resource navigator is a public-facing digital platform that serves as a resource directory. A great digital resource navigator can improve the efficiency of resource navigation, care coordination, and knowledge sharing by speeding up communication among different sectors and reducing the need for human labor. Past work on digital resource navigation has mostly focused on supporting care coordination for patients and providers who are already situated in the care system, such as through the use of the electronic health record and web-based communications [ 17 ]. Much less work has focused on knowledge sharing and information exchange of service options before patients connect with a provider in the behavioral health sphere. One exception was the M ental health I ntelligent information R esource A ssistant (MIRA), a web-based conversational chatbot developed in Canada during the COVID-19 global pandemic [ 18 ]. MIRA was developed to provide individuals with (1) information on substance use and mental health and (2) information on behavioral health services in Canada. This digital resource navigation tool is publicly available and informed by subject experts. As described by their published protocol, data collection was anticipated to take place from May 2022 to May 2023 [ 18 ]. However, no published work is available regarding provider and stakeholder perceptions of such tools. A digital resource navigator presents a scalable opportunity to streamline information sharing and improve access to care, although further exploration of facilitators of and barriers to developing and implementing such tools is needed.

Consolidated Framework for Implementation Research (CFIR) framework

CFIR is a comprehensive framework that can capture innovation-related factors and the complicated contextual factors that may influence implementation of an innovation. The initial version of CFIR comprised 39 subdomains grouped into the following five broad domains, including (1) innovation characteristics, (2) inner setting, (3) outer setting, (4) individual characteristics, and (5) process factors [ 19 ]. Innovation characteristics refer to characteristics of the template for value-based contracts (Part 1) and the digital resource navigator (Part 2), such as the perceived innovation source, complexity, evidence strength and quality, and relative advantage. Inner setting refers to the context in which the innovation takes place (in this study the Orange County Public Behavioral Healthcare System) including factors such as compatibility, leadership engagement, and networks and communication. Outer setting refers to the wider economic and social context that influences the innovation, such as federal and state policies and external incentives. Process refers to the steps taken during the innovation and implementation process, such as engaging and planning. Individual characteristics refer to the values and views of individual users of the innovation. Adapted definitions of CFIR constructs are presented in Table  1 .

Researchers have used CFIR to understand complicated system transformation efforts within organizations and health care systems [ 20 , 21 , 22 ]. For example, Kilaru et al. (2022) interviewed regulators and health care agency leaders about the all-payor global budget system in Maryland; their analyses using CFIR revealed factors that facilitated the design, implementation, and sustainability of system transformation efforts, such as clear and reasonable expectations, the appropriate amount of autonomy within the global budget, close communication, actionable data, and shared commitment and readiness for change. As such, CFIR has demonstrated its applicability when evaluating system transformation in different settings.

Evaluation context and aims

The BHST Innovation Project, approved by the Mental Health Services Oversight and Accountability Commission (MHSOAC), is a five-year Mental Health Services Act (MHSA) Project with a total budget of approximately $18 million. California uses innovation projects as part of their Mental Health Services Act (MHSA) program to provide resources for mental health. The goal of Orange County’s BHST Innovation Project is a system transformation effort to enable access to behavioral health services regardless of insurance status, insurance type, and/or level of clinical need. Specifically, this project included two parts:

Part 1: leveraging a value-based contract to align legal, fiscal, and regulatory requirements to improve the quality of behavioral health services, and implementing payor-agnostic care to improve access to care; and.

Part 2: developing a digital resource navigator to improve resource sharing and behavioral health service navigation.

The aim of this paper is to use the CFIR framework to evaluate facilitators of and barriers to the success of a behavioral health system transformation project. This paper fills a gap in knowledge by sharing learnings from the early stages of innovation in behavioral health payment and care and organizing these learnings in the CFIR model to promote their application to other projects. Although many systems are exploring such models, the learnings in county behavioral health settings from such explorations are too rarely shared. Moving towards value-based payor agnostic behavioral health care (Part 1) and improving access to information about care (Part 2) can help alleviate the unmet need within the County behavioral health system.

Participants

As part of an evaluation of the BHST Innovation Project, 29 individuals participated in key informant interviews between May and August 2022. Participant information is available in Table  2 . Seven individuals who had leadership roles at the County or a participating health care agency (L) participated in interviews that included both Part 1 and Part 2. Staff who only have knowledge about one part of the project participated in part-specific interviews, including eight contracted partners in Part 1 (CP1), four contracted partners in Part 2 (CP2), three community members and County stakeholders in Part 1 (CS1), and seven community members and County stakeholders in Part 2 (CS2). These interviewees were recruited due to their knowledge and involvement in the BHST Innovation Project.

The Consolidated Framework for Implementation Research (CFIR) was selected to guide the evaluation of both Part 1 and Part 2 in a consistent and systematic way.

Data collection

Our institutional review board deemed that this work was exempt from human participant research approval (University of California, Irvine Institutional Review Board (IRB)# #20,195,406). All participants provided verbal consent prior to participating in the interviews.

We developed a semi-structured interview guide based on relevant constructs from the CFIR model (Damschroder et al., 2009). The interview guide included a set of general questions for all interviewees with additional tailored questions for interviewees with different project roles (e.g., CP1, CP2, CS1, CS2, L). Interview guides are available in the supplementary material (Supplement 1 ). Each interview question was anchored to a CFIR construct. All interviewers had expertise in program evaluation or implementation science (DS, RV, SMS). Interviews were a mix of one interviewer, two interviewers, and two interviewers and a notetaker. Each interview lasted approximately 30 to 60 min. A total of 29 interviews were conducted, auto-transcribed by Zoom, and then the transcripts were verified and cleaned by the evaluation team.

Data analyses

We conducted thematic analyses following Braun and Clarke’s recommendation (2006). We outline how we follow their 6 proposed phases of analysis below.

Phase 1–2 (being familiar with data and initial coding)

XZ and RV were both trained in the CFIR framework, qualitative coding best practices, and use of the coding software (ATLAS.ti, version 22) prior to conducting data analyses. All data cleaning and analyses were completed using ATLAS.ti. We developed an initial draft of the codebook with adapted definitions of the CFIR constructs (Table  1 ). We used the five broad CFIR domains (intervention characteristics, outer setting, inner setting, characteristics of individuals, and process) and identified relevant subdomains. We completed initial coding, adapted the general CFIR definitions to be project-specific, and added a subdomain (i.e., COVID-19 as a factor in the outer setting). A total of 29 codes derived from the CFIR, including CFIR domains and subdomains, were included in the final codebook. Adapted definitions of the CFIR codes for both Part 1 and Part 2 of the project are presented in Table  2 .

Phase 3–5 (searching for, reviewing, defining, and naming themes)

XZ and RV double coded all transcripts. Initial percentages of agreement between two coders at the transcript level ranged from 46 to 74%. XZ and RV met weekly to review discrepancies and discuss revisions of the codebook (e.g., clarification of domain and subdomain definitions, addition of relevant subdomains). Coding was discussed during weekly team meetings to support consistency and resolve any discrepancies. These meetings were attended by the two coders and two other members of our research team (DS and SMS). Through discussion, final codes were decided for any discrepancies. Thus, our codes used for data analysis were codes with initial agreement or codes with discrepancies resolved through discussion with the broader research team.

We used ATLAS.ti software to calculate the frequency of the codes by CFIR domain and project aspect (Part 1 vs. Part 2) to obtain an overview of code distribution. This allowed an initial overview of codes and identification of which codes were more common for Part 1 and/or Part 2. We followed best practices in qualitative analyses mentioned by Braun and Clarke (2006) and constructed salient themes that “capture something important about the data in relation to the research and represents some level of patterned response or meaning within the data set.”

Phase 6 (locating exemplars and producing the report)

XZ, RV, and SMS engaged in documenting the themes described in this paper. XZ built a narrative of the data and selected illustrative example quotes under each theme. XZ labeled individual participants; for example, an example quote from the first contracted partner in Part 1 (CP1) was labeled “CP1.1”. RV and SMS reviewed the themes and examples and provided feedback.

Guided by CFIR, we examined facilitators and barriers related to behavioral health care system transformation efforts in Orange County separately for each of the two parts: (Part 1) developing a template for value-based payment contracts that promote payor-agnostic care, and (Part 2) creating a digital resource navigator. Overall, five themes were identified from the key informant interviews including (1) aligning goals and values (2) assessing and addressing fit, (3) fostering partnership and engagement, (4) being aware of implementation contexts, and (5) promoting communication. In Table  3 , we presented barriers and facilitators along with their CFIR domains related to each of the five themes. Different barriers and facilitators were identified for Part 1 and Part 2. Some barriers in the outer setting, such as changing state guidelines and priorities and fostering partnerships with private and nonprofit sectors, were unique to developing a template for value-based contracts that move toward payor-agnostic care. Engaging diverse communities to inform the design and content, mostly innovation characteristics, was a key facilitator for developing the digital resource navigator.

Part 1: Develop a Template for Value-Based Payment Contracts That Promote Payor-Agnostic Care

Themes and example quotes for facilitators of and barriers to developing a template for value-based payment contracts that promote payor-agnostic care are presented in Table  4 .

Aligning goals and values

Despite shared enthusiasm about value-based payment models that promote payor-agnostic care, misalignment in vision and scope was a barrier (inner setting, compatibility). For example, CP1.1 shared their excitement for increasing access to care and expressed a desire for payor-agnostic care (e.g., “From a clinician standpoint, it’s so much easier when a clinician can just treat the client and not have to worry about what type of insurance do they have, what can I not and what can I, and can I not do. What can or can they not receive for resources referrals”). Despite the shared enthusiasm among contracted partners and County health care agency leaders about increasing access to care via payor-agnostic care, perceptions of vision and scope of the contract varied, posing barriers in the inner setting. A County health care agency leader (L.1) described this barrier: “I would say that I think that there has not been alignment and agreement on the focus or the vision or the purpose and it’s felt like a kind of ongoing debate in terms of whether we want a liberal or conservative interpretation of the Constitution. It’s just fundamental disagreement on how to come to what that approved proposal was and how lenient and open to interpretation that approval is, and therefore we have not been able to get on the same page.” Confusion about the scope of the current project vision (e.g., L.2: “How it’s going to happen, I have no idea”) and skepticism about its feasibility (e.g., L.1: “Payor-agnostic… too ambitious and it’s certainly not doable or feasible in the time left on the project”) were barriers that tempered the enthusiasm for the project. Leveraging strong management and leadership (inner setting, leadership) as internal champions facilitated the process of aligning visions within the organization (e.g., “… the previous Health Care Agency Director was a champion and then the previous Behavioral Health Director was a champion… just having external subcontractors moving it forward isn’t enough to be able to realize the full value of the planning project or… to be able to support what the resulting plan would be.”).

Assessing and addressing fit

Lack of fit with existing health care system infrastructure was identified as a barrier to developing a template for value-based payment contracts that move toward payor-agnostic care. County health care agency leaders mentioned that private and public payors had different priorities and incentives within their organizations (outer setting, external policies and incentives). When describing challenges of bringing the private sector to the table, county health care agency leaders (L.2, L.4) used words such as “profit” and “return on investment.” L.2 described that a lack of incentives for commercial plans and private companies was a key challenge to engaging commercial payors: “It’s really hard to bring all the insurance companies to the table and say, ‘hey forget your profits, let’s just provide services at any cost’… the number one obstacle is getting those people to the table and questioning their profit level”. In contrast, the public sector had a bigger focus on compliance. For example, certified public expenditures (CPE) in the public sector were described as very specific (L.3). As described by L.5, the lack of flexibility of CPE suggested a poor fit between the value-based contracting and public funding structure: “So, I think for the Medi-Cal payment, I don’t think that we’re there, and we can’t gift public funds as a reward or an incentive to providers. It’s not laid out there. I know there are conversations at the state, but I think we’re [Orange County] so far ahead, as I understand it, and we don’t have the ability to just pay people extra, let them keep things that …there’s not a cost to it.”

Foster partnership

Strong cross-sector partnerships facilitated the process of braiding different funding streams. The importance of private-public payor partnership was recognized, especially related to factors in the outer setting. For instance, staff members and community stakeholders reported successful buy-in from commercial plans (outer setting, cosmopolitanism). CP1.2, CP1.3, and CS1.1 mentioned Kaiser Permanente as an example. CP1.2 stated: “Some of them [Commercial Plans] were already there. I mean Kaiser was a very early participant. They were an investor…They’re a big component of the… ecosystem, and they’re very much there”). This indicated a clear need for more efforts to facilitate the partnership with private insurance companies. CP1.2 also shared that their team’s cross-sector background and expertise facilitated establishing relationships and building cross-sector partnerships: “I come from a place of cross-sector, cross-organizational collaboration, and I think we can only improve what we’re doing if we learn what’s happening in other people’s backyards in like… how hard their jobs are”. Additionally, L.3 mentioned partnering with philanthropic organizations to obtain funding that aligns with the project mission (i.e., being able to serve everyone regardless of insurance status and clinical needs) could facilitate moving the County behavioral health system towards payor-agnostic care: “bring philanthropy to the table as well, because we really have a lot of wealth in our county. Philanthropy and some sort of a fund that accrues good interest, and we could utilize that as a stopgap between someone who doesn’t have payment and someone who does.” Participants with different roles on the project (CP1.1, CP1.3, L.3, L.4, CS1.2) described the COVID-19 global pandemic as disruptive to partnership relationship building and capacity of community members and staff members (outer setting, COVID-19). For example, CP1.3 stated: “We also had COVID, a lot of that [collaboration] got disrupted.”

Being aware of implementation contexts

A multitude of barriers influenced implementation contexts, including workforce challenges, the impact of COVID-19, and state-level policies. Workforce challenges were related to barriers in both inner and outer settings. Staff turnover and limited time and bandwidth were noted challenges in the inner setting and oftentimes led to de-prioritization of value-based contracting over other initiatives, particularly within the context of various state regulations (outer setting, external incentives and policies). For example, CP1.3 stated: “When there was turnover… that’s where kind of some of the thread may have gotten lost a little bit…. the focus of [value-based contracting] …was diminished…”. Relatedly, health care agencies had to shift their priorities due to COVID-19 related disruptions (e.g., CP1.3: “COVID changed things…there’s been a huge amount of distraction from the focus on COVID, and COVID response and vaccine, and response to COVID response… the transformation in the Community that happened from the CARES [The Coronavirus Aid, Relief, and Economic Security] Act dollars and other things getting poured in. It really took up a whole lot of time and space from elected officials to County staff to providers to take that in beyond what the usual kind of extravaganza of MHSA funding does every year. It really took all of that and poured gasoline on that fire, so it really changed the capacity of folks to engage in the work that we were trying to do.”). In addition to COVID-19 related disruptions and initiatives, state-level policies and the current CA health care infrastructure, such as the “carve-out” (the separation of mental health and substance use treatment services from the broader health care system), were mentioned by staff members and community stakeholders as barriers (CS1.3, CP1.2, CP1.3, CP1.4). CP1.2 described: “…in California, as long as the carve-out remains, there’re only so many levers you can pull.”

Promoting transparent and efficient communication

The complexity of developing a template for value-based contracts requires transparent and efficient communication. The need for promoting communication was identified as a theme related to building trust and relationships within a team (inner setting, networks and communication) and cultivating external partnerships (outer setting, cosmopolitism). A lack of transparent communication was identified as a barrier in both inner and outer settings. For example, L.2 described that the lack of communication could lead to a lack of shared understanding and trust within the project team (inner setting). L.4 mentioned that open communication with the state about the project progress was necessary to obtain state support and guidance (outer setting). To enhance communication among diverse stakeholders, the need for tailoring communication styles was mentioned. Participants (CP1.5, CS1.2, CS1.3) implied that using academic and technical jargon could be a barrier to communicating with community members and the lay workforce. For example, CS1.2 stated: “I looked at the summary of the survey and I said ‘…we are not baking a cake, so stop using the word measure. This is technical jargon; this is behind-the-scenes jargon. If you’re giving this to the Community, it should be as simple as I’m talking to you right now’.

Part 2: develop a Digital Resource Navigator (OC Navigator)

Themes and example quotes for facilitators of and barriers to developing a digital resource navigator to improve resource sharing and behavioral health service navigation are presented in Table  5 .

Shared enthusiasm was a facilitator for the development of the digital resource navigator. Specifically, its clear fit with County values and workflows in the inner setting and its relative advantages (innovation characteristics) contributed to the shared enthusiasm. Participants with different roles shared the same goal of improving existing workflows and increasing patient care in Orange County (inner setting, compatibility). As described by CP2.1: “It’s like knowing that the people that I’m interfacing with, the people that I’m like bothering and requesting meetings for, they all have the same like… we all share the same goal of like wanting to help people, and improve services, and improve access to services.” Multiple community stakeholders (CS2.1, CS2.5) described the strong fit between the digital resource navigator and the County’s equity-driven values. For instance, CS2.1 described the digital resource navigator as in line with the equity-driven values of the County: “One of our [Orange County Health Care Agency’s] key focuses was decreasing inequity, increasing equity… this is going to the next step on bringing more resources… I would say absolutely [the digital resource navigator] fits with Orange County’s values and workflows”. In addition to the clear fit between the digital resource navigator and County values, the strong fit between the digital resource navigator and existing workflows also contributed to enthusiasm among providers and community members. L.2 described that the digital resource navigator improved the efficiency of day-to-day tasks of County staff: “It’s helping the workflows [at OCHCA] be more efficient and cut out the extra unnecessary steps. It definitely is working to improve the system at the County.” Despite its fit into the larger values and workflows of Orange County Health Care Agency, not everyone deemed the digital resource navigator as a current need. For example, CP2.2 raised the question about whether the digital resource navigator was a redundant resource in the community: “We got a lot of comments and there’s a lot of chatter in the Community about is this [digital resource navigator] a waste of money because it’s a redundant resource?”.

The clear relative advantages (innovation characteristics) of the digital resource navigator contributed to shared enthusiasm among stakeholders and contracted partners. One relative advantage (innovation characteristics) was implementing a more centralized information-sharing, compared to the traditional paper-pencil format and other existing online tools. For example, L.1 stated: “ I would say it’s changed workflows in terms of centralizing and digitizing what used to be manual paper notes, post-it notes, some documentation here, some documentation there. They’ve digitized and in some cases automated a lot of the workflows for our telephone-based navigation line”. The increased efficiency in referral workflow was described as another relative advantage of the digital resource navigator. CP2.2 stated: “…you get to a point [when using other applications to find behavioral health resources] and you’d be stuck, and you just have to call the agency. You might have a list of twenty agencies, and you have to call them all before you can get some real basic information. But ours has… the way that they’re [the digital resource navigator] setting up the information cards. They give a lot of information, and then it just seems to be more robust and user-friendly.”

Fostering Engagement

Community engagement was identified as a process factor that facilitated the development and improvement of the digital resource navigator. Outreach efforts to community organizations facilitated the process of gathering feedback from community members and raising awareness about the digital resource navigator, as described by a community stakeholder (CS2.2) and multiple contracted partners (e.g., CP2.1, CP2.2, CP2.3). CP2.2 described: “… we have been successful in this sense where we’ve been talking to organizations and collaboratives and coalitions. Our pitch has been… give us your resource directory. We will highlight it on the website [the digital resource navigator]. You can correct it. We’ll tell you who authored it. We will give you a link that highlights your website, and these resource directories on the site and just let us help us help you, and then do your job better.” CP2.3 and CS2.2 identified connections with external networks as an important innovation source (e.g., National Council of Negro Women, National Alliance on Mental Illness (NAMI)) because they increased the team’s knowledge base about available resources and community needs. For example, CS2.2 described that conversations with NAMI informed the design, content, and implementation of the digital resource navigator: “So it was really great to be able to sit with [the technology vendor team] and have them ask sort of what [NAMI’s] vision was for the database and search platform to be, how we wanted it implemented, and just even how it looks so it’s not so scary” (innovation source, innovation characteristics).

Engaging community members, such as individuals with lived experiences and from marginalized groups, facilitated the improvement of design features. CP2.4 recommended leveraging community champions as a way of reaching historically marginalized groups: “Have [County leaders] help us identify champions in the community that might be good representatives for different sorts of things. So that kind of got us started. I remember a couple of years ago… they said, ‘this group is working on this, and this group is working on that’. They kind of helped with some initial introductions.” Despite the support from champions, engaging community members was especially challenging in the context of the COVID-19 global pandemic. COVID-19 lockdowns disrupted engagement with consumers who needed or preferred in-person engagement, as described by CP2.1: “At the beginning of the project, it was definitely like come one come all because we were getting started, it was COVID and really hard to engage.” (COVID, outer setting).

Lack of connections with historically marginalized communities was a barrier to curating multilinguistic and culturally relevant content on the digital resource navigator. The contracted partner’s team (e.g., CP2.3) commented on the earlier challenges of connecting with the Spanish-speaking community: “…not having been able to connect with more Spanish-speaking populations than we have. Along with that, having all the other languages that we represent on the Navigator… Not really having that much direct access to folks in that community”. Additionally, CP2.2 described that community engagement could have been more helpful in the early planning process of the project: “…we recently had a couple work groups with… a group of Latinas. They were Spanish-speaking only and they actually had a lot of suggestions because they actually have their own Facebook group, but these are the conversations I wish we’d had a little bit earlier. But better late than never.” In addition to engaging Latinx and Spanish-speaking populations through Spanish work groups (CS2.3, CS2.4), multiple participants (CP2.2, CP2.3, CS2.3, CS2.5) also mentioned that effective marketing and outreach efforts might facilitate the process of connecting with other historically underrepresented groups, such as veteran communities, faith-based organizations, individuals with special needs, and Korean and Chinese communities.

Transparent and open communication between County, contracted partners, and community partners was the most highlighted facilitator. CP2.1 described that a trusting relationship allowed more time and resources to be allocated to innovation development and community engagement: “There’s a very wide level of trust within everybody in the team to making the best decisions as possible for the project and for staff, so less checks and balances there, and more like this is what the Community wants. This is what we’re going to try to do as much as possible.” Multiple participants across different roles on the project (CP2.1, CP2.4, L.6) expressed that they were able to build trusting and collaborative relationships with proactive communication. Regularly scheduled meetings with clear agenda items, openness to feedback, active incorporation of feedback from community members, and discussions about specific design features and usefulness of the content improved design and feature development of the digital resource navigator. For example, CP2.2 described the process of engaging community co-chairs in the decision making around the need for broader community outreach, “for short things we run it by them [community co-chairs] and sort of get their temperature about if we should ask a broader group. Then, of course, we asked the County…it’s not a perfect process…and it’s iterative but we try our best….”.

This paper fills a literature gap by disseminating insights garnered from the initial phases of innovation in behavioral health payment and care. Applying the CFIR framework, we identified important facilitators of and barriers to Orange County’s behavioral health system transformation efforts under several key themes in the context of an innovation project. Specifically, aligning goals and values, fostering engagement and partnership, and promoting communication were all highlighted as important factors related to developing a template for value-based contracts that promote payor-agnostic care (Part 1) and creating a digital resource navigator (Part 2). Changing state guidelines and priorities, different incentive structures within the US health care system, and difficulties in braiding public and private funds (e.g., private insurance companies, philanthropic organizations) were unique barriers to developing a template for value-based contracts that promote payor-agnostic care. Leveraging diverse communities to inform the design and content of the digital platform, mostly in the domain of innovation characteristics, was a facilitator of creating the resource navigator.

Part 1. Develop a template for value-based payment contracts that promote payor-agnostic care

Misalignment in incentives, values, and goals posed barriers to developing a template for value-based contracts. Value-based payment models and payor-agnostic care are a disruption to the status quo of fee-for-service predominance and service fragmentation by payor source. Thus, the change to value-based payments could be perceived as adding regulatory and financial risks for both public and private sectors. Similar to challenges identified in our study, in a study of value-based care for substance use disorder treatment, researchers identified providers’ buy-in to value-based concepts as a key workforce challenge [ 11 ]. Additionally, we found developing a template for value-based contracts was perceived as a lower priority compared to other organizational and state initiatives due to limited agency bandwidth and competing priorities during the COVID-19 global pandemic. This was not unique in this implementation context. Delayed transition to value-based payments has been common in many hospital settings. For example, the Centers for Medicare and Medicaid Services (CMS) stopped accepting applications for new Accountable Care Organizations (a type of value-based contract) in 2021 [ 23 ].

Providing financial incentives may increase the acceptability of value-based contracts that promote payor-agnostic care. Existing financial strategies often rely on public funds, such as increased pay-for-service, grants, and cost-sharing, which may not be a sustainable and scalable solution [ 1 ]. Braiding public and private funding streams may help change the current incentive structure and provide sustainable resources for implementing value-based payment and payor-agnostic care. Funding from philanthropic organizations may be a potential funding source to fill some gaps in the behavioral healthcare sphere, and venture capital firms are getting involved in creating new innovations that may increase access to behavioral health services [ 24 ]. With both private and public sectors around the table, long-term cost-saving potentials of a value-based contract may act as a shared motive towards creating momentum that is needed to facilitate system transformation efforts [ 25 ].

Our work furthers the understanding of factors influencing the development of a digital resource navigator. We find that aligning goals and values, fostering engagement, and promoting transparent and efficient communication were important to the development and implementation of a digital resource navigator in Orange County. In our analyses, the perceived compatibility between the digital resource navigator and the extent to which it improved the current referral process workflow impacted the enthusiasm about the digital resource navigator. Consistent with our findings, past research has also found that providers were more likely to implement a technology-enabled tool when the tool could fit into or improve the existing workflow [ 26 , 27 ]. Our data also revealed that some interviewees did not deem the digital resource navigator as a priority. Similarly, Zhao et al. (2022) found that some providers perceived implementing a technology-enabled tool as a low priority, especially considering that there were limited resources in their organizations.

Inclusive design, communication and engagement strategies allowed the contracted partners to better align a digital resource navigator with the needs and priorities of the diverse communities in Orange County. In particular, it is important to understand the unique needs for information and resources among marginalized and underrepresented populations (e.g., monolingual communities, faith-based organizations) [ 28 ]. The design of technology-enabled behavioral health tools, including the digital resource navigator that was evaluated in this study, needs feedback from diverse community stakeholders. In our analyses, staff members and community champions shared their wishes for engaging diverse community members early in the iterative design process of the digital resource navigator. This cross-sector collaboration process requires inclusive communication strategies, as contracted partners, community stakeholders, and academic evaluators often speak different languages and have different levels of technological understanding. It was highlighted in our data that leveraging connections and knowledge of community champions not only facilitated communication but also outreach to the broader community. However, it is also worth noting that centering community voice and consistent outreach activities often required additional staff time and bandwidth, which could be challenging within the constraints of the County’s resources and regulations. Additionally, ongoing tailoring and adaptation of existing resources on the digital resource navigator are necessary. For example, community members shared frustration when resources on the platform were outdated. The process of sustaining timely updates of platform content may be costly and create workforce challenges.

Limitations

As found in the current study, external state policies and financial incentives in the outer setting were barriers to the acceptance of value-based and payor-agnostic care; although, government initiatives, such as California Advancing and Innovating Medi-Cal (CalAIM), could also facilitate the process. Attitudes and approaches towards system transformation may differ by state due to different state initiatives. It is important to note this project, along with a few mentioned past studies [ 8 , 9 ] relied on public funding, which could be even more limited in lower-resourced states and counties. This project was conducted in the state of California, which has more resources than other states, as evidenced by higher income and higher GDP. Substantial variations in resources (e.g., provider availability, funding) at the County and state level may also impact agency bandwidth to pilot value-based payment contracts [ 25 ] and develop and sustain a digital resource navigator. Additionally, we did not conduct consumer interviews and observations. Consumer perspectives and outcomes can be particularly helpful in curating inclusive content and improving the user interface of the digital resource navigator. However, it is worth noting that we included a diverse group of interviewees, including staff members, leadership, and community stakeholders. Some of the interviewees worked closely with the community being served in this early stage of the grant. Additionally, the data were from an early stage (first 2.5 years) of a five-year grant-funded project in Orange County; some identified facilitators (e.g., shared enthusiasm about a new and exciting project) and barriers (e.g., COVID-19) may be related to the time point.

Future directions

First, given the influence of external state policies, financial incentives, and County resources on the acceptability of value-based and payor-agnostic care, further investigation is needed to understand the impact of a specific County health care initiative on behavioral health care system transformation efforts. Comparative evaluation of barriers to and facilitators of various system transformation efforts across states and counties can provide valuable insights into policy implications and guide tailored support for local stakeholders. Second, centering the voices and experiences of consumers can be particularly helpful in ensuring the inclusivity of the content and design of the digital resource navigator. Collecting data from consumers through interviews and surveys can facilitate understanding the perceived usefulness and usability of the platform and content on the digital resource navigator.

We analyzed 29 key informant interviews to provide insight into the barriers and facilitators related to County behavioral health system transformation in a state-funded project. Overall, aligning goals and values, fostering engagement and partnership, and promoting communication were important factors to consider when developing a template for value-based contracts that promote payor-agnostic care (Part 1) and developing a digital resource navigator (Part 2). Being aware of changing state guidelines and priorities, having cross-sector specialty knowledge about incentive structures in the public and private sectors, and braiding public and private funds were important to developing a template for value-based contracts that promote payor-agnostic care. Leveraging diverse communities to inform the design and content and incorporating their timely feedback was particularly important to the development of the resource navigator. As these insights were drawn from diverse perspectives within the County Behavioral Health system, we hope that our research will prove invaluable to similar transformation endeavors in the future.

Availability of data and materials

The data collected and analyzed for this study are not publicly available because this was not a requirement of the project’s funder, and the verbal consent process did not elicit participants’ consent for their data to be publicly shared.

Dopp AR, Narcisse MR, Mundey P, et al. A scoping review of strategies for financing the implementation of evidence-based practices in behavioral health systems: state of the literature and future directions. Implement Res Pract. 2020;1:263348952093998. https://doi.org/10.1177/2633489520939980 .

Article   Google Scholar  

Porter ME, Kaplan RS. How to pay for health care. Harv Bus Rev. 2016;94(7–8):88–98.

PubMed   Google Scholar  

Priester MA, Browne T, Iachini A, Clone S, DeHart D, Seay KD. Treatment access barriers and disparities among individuals with co-occurring mental health and substance use disorders: an integrative literature review. J Subst Abuse Treat. 2016;61:47–59. https://doi.org/10.1016/j.jsat.2015.09.006 .

Article   PubMed   Google Scholar  

U.S. Department of Health and Human Services, Health Resources and Services Administration, National Center for Health Workforce Analysis. State-level projections of supply and demand for behavioral health occupations: 2016–2030. Published online 2018. https://bhw.hrsa.gov/sites/default/files/bureau-health-workforce/data-research/state-level-estimates-report-2018.pdf . Accessed 15 Aug 2023.

Walker ER, Cummings JR, Hockenberry JM, Druss BG. Insurance status, use of mental health services, and unmet need for mental health care in the United States. Psychiatr Serv. 2015;66(6):578–84. https://doi.org/10.1176/appi.ps.201400248 .

Article   PubMed   PubMed Central   Google Scholar  

Sorkin DH, Mukamel DB, Eikey EV, et al. Behavioral health system transformation innovation project – MHSA innovation report. In: Program of research in translational technology enabling high quality care. Irvine: University of California; 2022.

Google Scholar  

Hyatt AS, Tepper MC, O’Brien CJ. Recognizing and seizing the opportunities that value-based payment models offer behavioral health care. Psychiatr Serv. 2021;72(6):732–5. https://doi.org/10.1176/appi.ps.202000044 .

Bao Y, McGuire TG, Chan YF, et al. Value-based payment in implementing evidence-based care: the mental health integration program in Washington state. Am J Manag Care. 2017;23(1):48–53.

PubMed   PubMed Central   Google Scholar  

Goff SL, Gurewich D, Alcusky M, Kachoria AG, Nicholson J, Himmelstein J. Barriers and facilitators to implementation of value-based care models in new Medicaid accountable care organizations in Massachusetts: a study protocol. Front Public Health. 2021;9:645665. https://doi.org/10.3389/fpubh.2021.645665 .

Kissam SM, Beil H, Cousart C, Greenwald LM, Lloyd JT. States encouraging value-based payment: lessons from CMS’s state innovation models initiative. Milbank Q. 2019;97(2):506–42. https://doi.org/10.1111/1468-0009.12380 .

O’Grady MA, Lincourt P, Gilmer E, et al. How are substance use disorder treatment programs adjusting to value-based payment? A statewide qualitative study. Subst Abuse Res Treat. 2020;14:117822182092402. https://doi.org/10.1177/1178221820924026 .

Sederer LI, Derman M, Carruthers J, Wall M. The New York state collaborative care initiative: 2012–2014. Psychiatr Q. 2016;87(1):1–23. https://doi.org/10.1007/s11126-015-9375-1 .

American Psychiatric Association. Case studies: New York State Collaborative Care Initiative: 2012–20141. https://www.psychiatry.org/getmedia/b684932c-c994-4125-abee-165d7082ed1a/Case-Study-Collaborative-Care-Model-NewYork.pdf . Accessed 1 Aug 2023.

Cattel D, Eijkenaar F. Value-based provider payment initiatives combining global payments with explicit quality incentives: a systematic review. Med Care Res Rev. 2020;77(6):511–37. https://doi.org/10.1177/1077558719856775 .

Anglin G, Tu H, Liao K, Sessums L, Taylor EF. Strengthening multipayer collaboration: lessons from the comprehensive primary care initiative. Milbank Q. 2017;95(3):602–33. https://doi.org/10.1111/1468-0009.12280 .

Paulson CA, Durazo EM, Purry LD, et al. Adding a seat at the table: a case study of the provider’s perspective on integrating community health workers at provider practices in California. Front Public Health. 2021;9:690067. https://doi.org/10.3389/fpubh.2021.690067 .

Falconer E, Kho D, Docherty JP. Use of technology for care coordination initiatives for patients with mental health issues: a systematic literature review. Neuropsychiatr Dis Treat. 2018;14:2337–49. https://doi.org/10.2147/NDT.S172810 .

Noble JM, Zamani A, Gharaat M, et al. Developing, implementing, and evaluating an artificial intelligence–guided mental health resource navigation chatbot for health care workers and their families during and following the COVID-19 pandemic: protocol for a cross-sectional study. JMIR Res Protoc. 2022;11(7):e33717. https://doi.org/10.2196/33717 .

Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009;4(1):50. https://doi.org/10.1186/1748-5908-4-50 .

Hill JN, Locatelli SM, Bokhour BG, et al. Evaluating broad-scale system change using the consolidated framework for implementation research: challenges and strategies to overcome them. BMC Res Notes. 2018;11(1):560. https://doi.org/10.1186/s13104-018-3650-9 .

Kilaru AS, Crider CR, Chiang J, Fassas E, Sapra KJ. Health care leaders’ perspectives on the Maryland All-Payer Model. JAMA Health Forum. 2022;3(2):e214920. https://doi.org/10.1001/jamahealthforum.2021.4920 .

Safaeinili N, Brown-Johnson C, Shaw JG, Mahoney M, Winget M. CFIR simplified: Pragmatic application of and adaptations to the Consolidated Framework for Implementation Research (CFIR) for evaluation of a patient‐centered care transformation within a learning health system. Learn Health Syst. 2020;4(1). https://doi.org/10.1002/lrh2.10201 .

Feeley TW. Covid-19 hasn’t been a tipping point for value-based care, but it should be. NEJM Catal. 2021;2(1):CAT.20.0641. https://doi.org/10.1056/CAT.20.0641 .

Shah RN, Berry OO. The rise of venture capital investing in mental health. JAMA Psychiat. 2021;78(4):351. https://doi.org/10.1001/jamapsychiatry.2020.2847 .

Conrad DA, Grembowski D, Hernandez SE, Lau B, Marcus-Smith M. Emerging lessons from Regional and State Innovation in Value-based payment reform: balancing collaboration and disruptive innovation: emerging lessons from Innovation in Value-based payment reform. Milbank Q. 2014;92(3):568–623. https://doi.org/10.1111/1468-0009.12078 .

Lattie EG, Nicholas J, Knapp AA, Skerl JJ, Kaiser SM, Mohr DC. Opportunities for and tensions surrounding the use of technology-enabled mental health services in community mental health care. Adm Policy Ment Health Ment Health Serv Res. 2020;47(1):138–49. https://doi.org/10.1007/s10488-019-00979-2 .

Zhao X, Stadnick NA, Ceballos-Corro E, et al. Facilitators of and barriers to integrating digital mental health into county mental health services: qualitative interview analyses. JMIR Form Res. 2023;7:e45718. https://doi.org/10.2196/45718 .

Ramos G, Ponting C, Labao JP, Sobowale K. Considerations of diversity, equity, and inclusion in mental health apps: a scoping review of evaluation frameworks. Behav Res Ther. 2021;147:103990. https://doi.org/10.1016/j.brat.2021.103990 .

Download references

This work was funded by the Orange County Health Care Agency, Mental Health and Recovery Services, Innovation Projects, Mental health Services Act/Prop 63 (contract number MA-042-21011324). This work was also supported by the Institute for Clinical and Translational Sciences (ICTS) under Grant (UL1TR001414). The information or content and conclusions presented here are those of the authors and should not be construed as official position or policy of, nor should any endorsements be inferred by, the participating partners and/or the Orange County Health Care Agency.

Author information

Authors and affiliations.

Department of Psychiatry and Behavioral Sciences, University of Washington, Seattle, USA

Seattle Children’s Research Institute, Seattle, USA

Department of Medicine, University of California, Irvine, USA

Rachel Varisco, Judith Borghouts, Dana B. Mukamel & Dara H. Sorkin

Herbert Wertheim School of Public Health and Human Longevity Science, University of California San Diego, La Jolla, USA

Elizabeth V. Eikey

The Design Lab, University of California San Diego, La Jolla, USA

Department of Psychiatry and Human Behavior, University of California, Irvine, USA

David Safani

Department of Psychological Science, University of California, Irvine, USA

Stephen M. Schueller

You can also search for this author in PubMed   Google Scholar

Contributions

XZ contributed to the conceptualization of the work, analysis and interpretation of data, and writing the original draft, and writing-review and editing. RV contributed to conceptualization of the work, collection analysis, interpretation of the data and writing-review and editing. JB, EE, DS, and DM contributed to writing-review and editing. SS contributed to the conceptualization of the work, collection and interpretation of the data, and writing-review and editing. DHS contributed to funding acquisition, conceptualization of the work, collection and interpretation of the data, and writing-review and editing.

Corresponding author

Correspondence to Xin Zhao .

Ethics declarations

Ethics approval and consent to participate.

Our institutional review board deemed that this work was exempt from human participant research approval (University of California, Irvine Institutional Review Board# #20195406). Key informant interviews are considered non-human subject research because results are intended to describe the specific context under which the data is collected, and will not be used to generalize beyond this context. Verbal informed consent was obtained from all participants.

Consent for publication

Not applicable.

Competing interests

SMS has received consulting payments from Otsuka Pharmaceuticals and Boehringer Ingelheim, and is a member of the Headspace Scientific Board, for which he receives compensation. XZ has received consulting payments from FirstThen Inc for work unrelated to this manuscript. The authors have no further interests to declare.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary material 1., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Zhao, X., Varisco, R., Borghouts, J. et al. Facilitators of and barriers to County Behavioral Health System Transformation and Innovation: an interview study. BMC Health Serv Res 24 , 604 (2024). https://doi.org/10.1186/s12913-024-11041-9

Download citation

Received : 08 February 2024

Accepted : 24 April 2024

Published : 09 May 2024

DOI : https://doi.org/10.1186/s12913-024-11041-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• System transformation
• Behavioral health
• Value-based payment
• Payor-agnostic care
• Digital resource navigation

BMC Health Services Research

ISSN: 1472-6963

• General enquiries: [email protected]

Prospective neuroimaging and neuropsychological evaluation in adults with newly diagnosed focal epilepsy

• Find this author on Google Scholar
• Find this author on PubMed
• Search for this author on this site
• For correspondence: [email protected]
• ORCID record for Simon S Keller
• Info/History
• Preview PDF

Objective: There are few prospective longitudinal studies in patients with newly diagnosed epilepsy (NDE) despite that this is a key time point to understand the underlying biology of epilepsy and to identify potential interventions and biomarkers for seizure and cognitive outcomes. Here we have performed a prospective combined neuroimaging and neuropsychological study in a cohort of patients with focal NDE and healthy controls. Methods: We recruited 104 patients with NDE and 45 healthy controls for research-grade 3 Tesla MRI (diagnostic and structural imaging, diffusional kurtosis imaging, resting-state functional MRI, task-based functional MRI), EEG, comprehensive neuropsychological, and blood biomarker investigations. We report here on the baseline clinical, neuroradiological, MRI morphometric, and neuropsychological findings in this cohort. Results: 38% of patients had unremarkable MRI features, 12% had lesions of known significance in epilepsy, 49% with abnormalities of unknown significance in epilepsy, and 23% with incidental findings. In comparison, 56% of controls had unremarkable MRI features, 7% had lesions of known significance in epilepsy, 33% with abnormalities of unknown significance in epilepsy, and 16% had incidental findings. Patients had a higher incidence of white matter hyperintensities compared to controls. Reduced bihemispheric frontal lobe cortical thickness and thalamic volumes were observed in patients with moderate effect sizes. Patients scored significantly lower on tasks of executive function, processing speed, and visual, delayed, and immediate memory, and significantly higher on depression and anxiety assessments compared to controls. Patient neuropsychological performance was related to various brain morphometric features. Significance: People with adult focal NDE have a greater proportion of MRI-positive findings than previously reported. Subtle white matter lesions may represent an important diagnostic criterion and have a pathophysiological basis in focal epilepsy. Morphometric and neuropsychological alterations are present at the point of diagnosis of epilepsy, which suggests that brain and cognitive changes are not exclusively due to the deleterious impact of chronic epilepsy.

Competing Interest Statement

The authors have declared no competing interest.

Funding Statement

This work was funded by a UK Medical Research Council (MRC) research grant (MR/S00355X/1).

Author Declarations

I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.

The details of the IRB/oversight body that provided approval or exemption for the research described are given below:

This study was approved by the Northwest Liverpool East Research Ethics Committee (19/NW/0384) through the Integrated Research Application System (Project ID 260623). The research is sponsored by the University of Liverpool (UoL001449) and UK Health Research Authority approval was provided on 22nd August 2019.

I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.

Data Availability

All data produced in the present study are available upon reasonable request to the authors

View the discussion thread.

Thank you for your interest in spreading the word about medRxiv.

NOTE: Your email address is requested solely to identify you as the sender of this article.

Citation Manager Formats

• EndNote (tagged)
• EndNote 8 (xml)
• RefWorks Tagged
• Ref Manager
• Tweet Widget
• Facebook Like
• Google Plus One
• Addiction Medicine (323)
• Allergy and Immunology (627)
• Anesthesia (163)
• Cardiovascular Medicine (2365)
• Dentistry and Oral Medicine (287)
• Dermatology (206)
• Emergency Medicine (378)
• Endocrinology (including Diabetes Mellitus and Metabolic Disease) (833)
• Epidemiology (11758)
• Forensic Medicine (10)
• Gastroenterology (702)
• Genetic and Genomic Medicine (3724)
• Geriatric Medicine (348)
• Health Economics (632)
• Health Informatics (2388)
• Health Policy (929)
• Health Systems and Quality Improvement (895)
• Hematology (340)
• HIV/AIDS (780)
• Infectious Diseases (except HIV/AIDS) (13300)
• Intensive Care and Critical Care Medicine (767)
• Medical Education (365)
• Medical Ethics (104)
• Nephrology (398)
• Neurology (3485)
• Nursing (198)
• Nutrition (522)
• Obstetrics and Gynecology (673)
• Occupational and Environmental Health (661)
• Oncology (1818)
• Ophthalmology (535)
• Orthopedics (218)
• Otolaryngology (286)
• Pain Medicine (232)
• Palliative Medicine (66)
• Pathology (445)
• Pediatrics (1031)
• Pharmacology and Therapeutics (426)
• Primary Care Research (420)
• Psychiatry and Clinical Psychology (3171)
• Public and Global Health (6131)
• Radiology and Imaging (1276)
• Rehabilitation Medicine and Physical Therapy (744)
• Respiratory Medicine (825)
• Rheumatology (379)
• Sexual and Reproductive Health (372)
• Sports Medicine (322)
• Surgery (400)
• Toxicology (50)
• Transplantation (172)
• Urology (145)

More From Forbes

5 highest paying skills for project managers in 2024, from research.

• Share to Facebook
• Share to Twitter
• Share to Linkedin

To remain competitive and gain an edge as a project management professional, it helps to learn one ... [+] or more of these five critical skills this year

Within a competitive job market in which 25 million project managers are projected to be in demand over the next few years, it's critical for aspiring and existing project management professionals to stay ahead of the curve so they can secure the highest-paying roles. While the project management profession in itself is a well-remunerated industry, with U.S. median salaries jumping to as high as $120,000, that doesn't necessarily mean that you can sit back, relax, and assume you're worthy of a six-figure salary.

There are things that lie within your power, which you can do to dramatically skyrocket your earnings.

When exploring how to increase and even multiply your salary as a project manager, it's best to look into what are the highest-paying, in-demand skills within the project management industry, so that you can position yourself as best suited for a promotion, or as a star candidate when job-searching.

Highest-Paying In-Demand Skills For Project Managers

The compensation and salary data company, PayScale, compiled a list of high-paying skills which include a combination of hard, technical skills, and soft or power skills for those within the project industry.

It's worth noting that there are some industries in which the impact of these skills on your pay increase will be more visible than others. For example, considering programme/project management within the technology industry, you can expect to earn more for your skill set, as the tech industry is renowned for offering the highest salaries to project and program management professionals. As such, some of the skills listed below are more specific to the technology industry.

Best High-Yield Savings Accounts Of 2024

Best 5% interest savings accounts of 2024.

However, many of these skills are equally in demand in other industries as well, so they can still be applicable and relevant to you.

These competencies include:

1. Agile Software Development

Using the agile software development approach for building and delivering software products entails prioritising flexibility responsiveness and adaptability to change as well as collaboration. Following this framework, you and your team would be working in sprints, (short iterations) which allows you to make incremental progress and quickly respond to stakeholder requirements and feedback to meet their needs.

This is the highest-paying skill for project managers, with Payscale data revealing that it can boost your earnings by 47%. Coursera and Skillsoft are two of the amazing resources that are available for you to learn this skill.

The tech industry offers the highest salaries for PMs

2. Business Analysis

Analytical thinking is listed in the World Economic Forum's Future of Jobs Report 2023 as the number one skill needed by professionals over the next few years.

Business analysis, in simple terms, is when you identify business needs and research solutions for them. This skill comes handy in your PM role, as you will naturally be working with stakeholders on a regular basis to gather and analyze data on their needs and define requirements, and then propose the best solution with your program or project. This skill improves salary prospects by up to 43%.

3. Engineering Design

Next on the list is engineering design, which is of course more specifically focused on the technology industry. This involves taking the results of your business analysis, and creating detailed plans and specifications, including prototyping and testing, to ensure you deliver a high-quality product that satisfies your stakeholders' needs. This skill is more relatable to product managers, and provides a 14% uptake in salary.

4. Risk Management/Risk Control

With all the changes that are occurring to disrupt industries in 2024, it's no wonder that risk management and risk control are highly in-demand skills for project and program managers to possess. This skill also has the potential for a 14% salary boost, according to Payscale analysis.

You can learn more about how to comprehensively manage and plan for risk by undertaking courses and certifications such as those offered by the Institute of Risk Management, which are globally recognized.

5. Strategy

Strategic thinking is a trademark of leadership, and of exceptional program and project management. You need to be able to steer your project team in the right direction to achieve successful project outcomes, and this means you'll need to have a solid strategy that is clearly articulated so that everyone is on the same page. Setting clear goals and objectives, and assigning the right team members to fulfil each task according to their strengths requires some practice to get it right, but it can be achieved if you're persistent and intentional. Possessing and showcasing this skill results in a 12% salary increase.

Risk management skills helps ensure your project is prepared for worst-case scenarios and minimizes ... [+] negative outcomes, including those that could affect your job directly

By investing in yourself this year by upskilling in one or more of these five essential competencies, you can position yourself to successfully meet the evolving demands of your stakeholders and the project management industry, and unlock numerous career-building and salary-boosting opportunities.

• Editorial Standards
• Reprints & Permissions

Join The Conversation

One Community. Many Voices. Create a free account to share your thoughts. 

Forbes Community Guidelines

Our community is about connecting people through open and thoughtful conversations. We want our readers to share their views and exchange ideas and facts in a safe space.

In order to do so, please follow the posting rules in our site's  Terms of Service.   We've summarized some of those key rules below. Simply put, keep it civil.

Your post will be rejected if we notice that it seems to contain:

• False or intentionally out-of-context or misleading information
• Insults, profanity, incoherent, obscene or inflammatory language or threats of any kind
• Attacks on the identity of other commenters or the article's author
• Content that otherwise violates our site's  terms.

User accounts will be blocked if we notice or believe that users are engaged in:

• Continuous attempts to re-post comments that have been previously moderated/rejected
• Racist, sexist, homophobic or other discriminatory comments
• Attempts or tactics that put the site security at risk
• Actions that otherwise violate our site's  terms.

So, how can you be a power user?

• Stay on topic and share your insights
• Feel free to be clear and thoughtful to get your point across
• ‘Like’ or ‘Dislike’ to show your point of view.
• Protect your community.
• Use the report tool to alert us when someone breaks the rules.

Thanks for reading our community guidelines. Please read the full list of posting rules found in our site's  Terms of Service.