policy analysis and research methods

A guide to policy analysis as a research method

Affiliations.

• 1 Department of Public Health, School of Psychology and Public Health, Latrobe University, Bundoora, Victoria, Australia.
• 2 Department of Global, Urban and Social Studies, RMIT University, 124 La Trobe Street, Melbourne, Victoria, Australia.
• 3 Department of Social Sciences, Faculty of Health, Arts and Design, Swinburne University, 24 Wakefield Street, Hawthorn, Victoria, Australia.
• 4 Department of Nutrition Dietetics and Food, Monash University, Level 1, 264 Ferntree Gully Road, Notting Hill, Victoria, Australia.
• PMID: 30101276
• DOI: 10.1093/heapro/day052

Policy analysis provides a way for understanding how and why governments enact certain policies, and their effects. Public health policy research is limited and lacks theoretical underpinnings. This article aims to describe and critique different approaches to policy analysis thus providing direction for undertaking policy analysis in the field of health promotion. Through the use of an illustrative example in nutrition it aims to illustrate the different approaches. Three broad orientations to policy analysis are outlined: (i) Traditional approaches aim to identify the 'best' solution, through undertaking objective analyses of possible solutions. (ii) Mainstream approaches focus on the interaction of policy actors in policymaking. (iii) Interpretive approaches examine the framing and representation of problems and how policies reflect the social construction of 'problems'. Policy analysis may assist understanding of how and why policies to improve nutrition are enacted (or rejected) and may inform practitioners in their advocacy. As such, policy analysis provides researchers with a powerful tool to understand the use of research evidence in policymaking and generate a heightened understanding of the values, interests and political contexts underpinning policy decisions. Such methods may enable more effective advocacy for policies that can lead to improvements in health.

Keywords: interpretive policy analysis; mainstream policy analysis; nutrition; public health; sugar sweetened beverage tax.

© The Author(s) 2018. Published by Oxford University Press. All rights reserved. For permissions, please email: [email protected].

• Health Policy*
• Health Promotion
• Policy Making*
• Public Health / legislation & jurisprudence
• Public Health Administration
• Public Policy
• Research Design

• Help & FAQ

A guide to policy analysis as a research method

• Nutrition Dietetics & Food

Research output : Contribution to journal › Article › Research › peer-review

Policy analysis provides a way for understanding how and why governments enact certain policies, and their effects. Public health policy research is limited and lacks theoretical underpinnings. This article aims to describe and critique different approaches to policy analysis thus providing direction for undertaking policy analysis in the field of health promotion. Through the use of an illustrative example in nutrition it aims to illustrate the different approaches. Three broad orientations to policy analysis are outlined: (i) Traditional approaches aim to identify the 'best' solution, through undertaking objective analyses of possible solutions. (ii) Mainstream approaches focus on the interaction of policy actors in policymaking. (iii) Interpretive approaches examine the framing and representation of problems and how policies reflect the social construction of 'problems'. Policy analysis may assist understanding of how and why policies to improve nutrition are enacted (or rejected) and may inform practitioners in their advocacy. As such, policy analysis provides researchers with a powerful tool to understand the use of research evidence in policymaking and generate a heightened understanding of the values, interests and political contexts underpinning policy decisions. Such methods may enable more effective advocacy for policies that can lead to improvements in health.

• interpretive policy analysis
• mainstream policy analysis
• public health
• sugar sweetened beverage tax

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

• 10.1093/heapro/day052

Other files and links

• Link to publication in Scopus

T1 - A guide to policy analysis as a research method

AU - Browne, Jennifer

AU - Coffey, Brian

AU - Cook, Kay

AU - Meiklejohn, Sarah

AU - Palermo, Claire

PY - 2019/10

Y1 - 2019/10

N2 - Policy analysis provides a way for understanding how and why governments enact certain policies, and their effects. Public health policy research is limited and lacks theoretical underpinnings. This article aims to describe and critique different approaches to policy analysis thus providing direction for undertaking policy analysis in the field of health promotion. Through the use of an illustrative example in nutrition it aims to illustrate the different approaches. Three broad orientations to policy analysis are outlined: (i) Traditional approaches aim to identify the 'best' solution, through undertaking objective analyses of possible solutions. (ii) Mainstream approaches focus on the interaction of policy actors in policymaking. (iii) Interpretive approaches examine the framing and representation of problems and how policies reflect the social construction of 'problems'. Policy analysis may assist understanding of how and why policies to improve nutrition are enacted (or rejected) and may inform practitioners in their advocacy. As such, policy analysis provides researchers with a powerful tool to understand the use of research evidence in policymaking and generate a heightened understanding of the values, interests and political contexts underpinning policy decisions. Such methods may enable more effective advocacy for policies that can lead to improvements in health.

AB - Policy analysis provides a way for understanding how and why governments enact certain policies, and their effects. Public health policy research is limited and lacks theoretical underpinnings. This article aims to describe and critique different approaches to policy analysis thus providing direction for undertaking policy analysis in the field of health promotion. Through the use of an illustrative example in nutrition it aims to illustrate the different approaches. Three broad orientations to policy analysis are outlined: (i) Traditional approaches aim to identify the 'best' solution, through undertaking objective analyses of possible solutions. (ii) Mainstream approaches focus on the interaction of policy actors in policymaking. (iii) Interpretive approaches examine the framing and representation of problems and how policies reflect the social construction of 'problems'. Policy analysis may assist understanding of how and why policies to improve nutrition are enacted (or rejected) and may inform practitioners in their advocacy. As such, policy analysis provides researchers with a powerful tool to understand the use of research evidence in policymaking and generate a heightened understanding of the values, interests and political contexts underpinning policy decisions. Such methods may enable more effective advocacy for policies that can lead to improvements in health.

KW - interpretive policy analysis

KW - mainstream policy analysis

KW - nutrition

KW - public health

KW - sugar sweetened beverage tax

UR - http://www.scopus.com/inward/record.url?scp=85074379871&partnerID=8YFLogxK

U2 - 10.1093/heapro/day052

DO - 10.1093/heapro/day052

M3 - Article

C2 - 30101276

AN - SCOPUS:85074379871

SN - 0957-4824

JO - Health Promotion International

JF - Health Promotion International

• Technical Support
• Find My Rep

You are here

Methods for Policy Research Taking Socially Responsible Action

• Ann Majchrzak - University of Southern California, USA
• M. Lynne Markus - Bentley University, USA
• Description

This book about responsible and evidence-based decision making is written for those interested in improving the decisions that affect people’s lives. It describes how to define policy research questions so that evidence can be applied to them, how to find and synthesize existing evidence, how to generate new evidence if needed, how to make acceptable recommendations that can solve policy problems without negative side effects, and how to describe evidence and recommendations in a manner that changes minds.

Policies are not just the decisions made by a country’s rulers or elected officials; policies are also set by corporate executives, managers of department stores, and project leaders in non-profit organizations pursuing environmental protection. The authors’ suggestion are based on the fundamental belief that evidence-based decision making is superior to decisions based purely on opinion, intuition, and emotion. Because much has happened since 1984 when the first edition was published, this is a substantially different book with a new co-author, new and updated examples, new chapters, and new frameworks for understanding.

See what’s new to this edition by selecting the Features tab on this page. Should you need additional information or have questions regarding the HEOA information provided for this title, including what is new to this edition, please email [email protected] . Please include your name, contact information, and the name of the title for which you would like more information. For information on the HEOA, please go to http://ed.gov/policy/highered/leg/hea08/index.html .

For assistance with your order: Please email us at [email protected] or connect with your SAGE representative.

SAGE 2455 Teller Road Thousand Oaks, CA 91320 www.sagepub.com

Loved the first edition as a graduate student when it came out in the mid-80s; so happy that a new edition was developed so that I can share it with my graduate students.

still under review for consideration.

NEW TO THIS EDITION:

• Each chapter’s phase in the policy research voyage (depicted by artwork with a nautical theme) includes clearly defined activities, deliverables, criteria for successful performance, and workflow diagrams.
• Policy Change Wheel and STORM Context Conditions frameworks make it easier for readers to remember what needs to be done.
• New chapters on synthesizing available evidence (Chapter 3) and reflecting on policy research experiences (Chapter 7) broaden the book’s coverage.
• Updated examples drawn from a variety of contexts, including international and business policy, as well as domestic issues, illustrate applications of evidence-based decision making in the real world.
• Chapter 1, Making a Difference with Policy Research , now reflects an action-orientation toward not just doing policy research, but also toward fostering change and doing policy research responsibly.

KEY FEATURES:

• A how-to orientation encourages readers to consider the evidence systematically and responsibly before making a decision and to communicate evidence and recommendations in a way that facilitates real change.
• Real world examples throughout the text show readers the everyday applications of policy decision making.
• Exercises at the end of each chapter give students an opportunity to apply what they’ve learned.

This is a substantially revised edition of Methods for Policy Research, originally published in 1984. This book reframes policy research as responsible and evidence-based decision making. It describes how to define policy research questions so that evidence can be applied to them, how to find and synthesize existing evidence, how to generate new evidence if needed, how to make acceptable recommendations that can solve policy problems without harmful side effects, how to describe evidence and recommendations in a manner that changes minds. This book is meant to help individuals who want to improve the policy decisions that affect people's lives.

Responsible and evidence-based decision making is needed not just in government and social service agencies. It is also needed in businesses and in nongovernmental organizations such as charities, foundations, and non-profits. In this book, we state our values clearly: We believe that evidence-based decision making is superior to decisions based purely on opinion, intuition, and emotion. We also believe that responsible decision-making requires taking into account the possibility of harmful consequences from policy change, no matter how well intentioned those changes may be.

Each chapter now has clearly defined activities and deliverables, supported by workflow diagrams, along with tracking indicators that policy researchers can use to assess how well they are performing the activities. New frameworks are presented such as the M2 test (meaningfulness and manageability), the Policy Change Wheel, and STORM (Social, Technical, Organizational, Regulatory, and Market) context conditions to make it easier for readers to remember what needs to be done. All examples are updated, they are drawn from a variety of contexts, including international and business policy, as well as domestic policy and social service.

Each chapter was substantially revised to make the activities and outcomes of policy research clear. We've introduced new content, including an entirely new chapter on synthesizing existing evidence. We've exposed the reader to useful websites, to new ways of involving stakeholders in the Case for Change, and to ways of ensuring that recommendations derived from evidence-gathering are meaningful and manageable. A nautical theme, a conversational style, and humor are used throughout to make the reading enjoyable. (Look out for puns!)

Sample Materials & Chapters

For instructors, select a purchasing option.

This title is also available on SAGE Research Methods , the ultimate digital methods library. If your library doesn’t have access, ask your librarian to start a trial .

• Search Menu
• Browse content in Arts and Humanities
• Browse content in Archaeology
• Anglo-Saxon and Medieval Archaeology
• Archaeological Methodology and Techniques
• Archaeology by Region
• Archaeology of Religion
• Archaeology of Trade and Exchange
• Biblical Archaeology
• Contemporary and Public Archaeology
• Environmental Archaeology
• Historical Archaeology
• History and Theory of Archaeology
• Industrial Archaeology
• Landscape Archaeology
• Mortuary Archaeology
• Prehistoric Archaeology
• Underwater Archaeology
• Urban Archaeology
• Zooarchaeology
• Browse content in Architecture
• Architectural Structure and Design
• History of Architecture
• Residential and Domestic Buildings
• Theory of Architecture
• Browse content in Art
• Art Subjects and Themes
• History of Art
• Industrial and Commercial Art
• Theory of Art
• Biographical Studies
• Byzantine Studies
• Browse content in Classical Studies
• Classical Literature
• Classical Reception
• Classical History
• Classical Philosophy
• Classical Mythology
• Classical Art and Architecture
• Classical Oratory and Rhetoric
• Greek and Roman Papyrology
• Greek and Roman Archaeology
• Greek and Roman Epigraphy
• Greek and Roman Law
• Late Antiquity
• Religion in the Ancient World
• Digital Humanities
• Browse content in History
• Colonialism and Imperialism
• Diplomatic History
• Environmental History
• Genealogy, Heraldry, Names, and Honours
• Genocide and Ethnic Cleansing
• Historical Geography
• History by Period
• History of Emotions
• History of Agriculture
• History of Education
• History of Gender and Sexuality
• Industrial History
• Intellectual History
• International History
• Labour History
• Legal and Constitutional History
• Local and Family History
• Maritime History
• Military History
• National Liberation and Post-Colonialism
• Oral History
• Political History
• Public History
• Regional and National History
• Revolutions and Rebellions
• Slavery and Abolition of Slavery
• Social and Cultural History
• Theory, Methods, and Historiography
• Urban History
• World History
• Browse content in Language Teaching and Learning
• Language Learning (Specific Skills)
• Language Teaching Theory and Methods
• Browse content in Linguistics
• Applied Linguistics
• Cognitive Linguistics
• Computational Linguistics
• Forensic Linguistics
• Grammar, Syntax and Morphology
• Historical and Diachronic Linguistics
• History of English
• Language Evolution
• Language Reference
• Language Variation
• Language Families
• Language Acquisition
• Lexicography
• Linguistic Anthropology
• Linguistic Theories
• Linguistic Typology
• Phonetics and Phonology
• Psycholinguistics
• Sociolinguistics
• Translation and Interpretation
• Writing Systems
• Browse content in Literature
• Bibliography
• Children's Literature Studies
• Literary Studies (Romanticism)
• Literary Studies (American)
• Literary Studies (Modernism)
• Literary Studies (Asian)
• Literary Studies (European)
• Literary Studies (Eco-criticism)
• Literary Studies - World
• Literary Studies (1500 to 1800)
• Literary Studies (19th Century)
• Literary Studies (20th Century onwards)
• Literary Studies (African American Literature)
• Literary Studies (British and Irish)
• Literary Studies (Early and Medieval)
• Literary Studies (Fiction, Novelists, and Prose Writers)
• Literary Studies (Gender Studies)
• Literary Studies (Graphic Novels)
• Literary Studies (History of the Book)
• Literary Studies (Plays and Playwrights)
• Literary Studies (Poetry and Poets)
• Literary Studies (Postcolonial Literature)
• Literary Studies (Queer Studies)
• Literary Studies (Science Fiction)
• Literary Studies (Travel Literature)
• Literary Studies (War Literature)
• Literary Studies (Women's Writing)
• Literary Theory and Cultural Studies
• Mythology and Folklore
• Shakespeare Studies and Criticism
• Browse content in Media Studies
• Browse content in Music
• Applied Music
• Dance and Music
• Ethics in Music
• Ethnomusicology
• Gender and Sexuality in Music
• Medicine and Music
• Music Cultures
• Music and Media
• Music and Culture
• Music and Religion
• Music Education and Pedagogy
• Music Theory and Analysis
• Musical Scores, Lyrics, and Libretti
• Musical Structures, Styles, and Techniques
• Musicology and Music History
• Performance Practice and Studies
• Race and Ethnicity in Music
• Sound Studies
• Browse content in Performing Arts
• Browse content in Philosophy
• Aesthetics and Philosophy of Art
• Epistemology
• Feminist Philosophy
• History of Western Philosophy
• Metaphysics
• Moral Philosophy
• Non-Western Philosophy
• Philosophy of Language
• Philosophy of Mind
• Philosophy of Perception
• Philosophy of Action
• Philosophy of Law
• Philosophy of Religion
• Philosophy of Science
• Philosophy of Mathematics and Logic
• Practical Ethics
• Social and Political Philosophy
• Browse content in Religion
• Biblical Studies
• Christianity
• East Asian Religions
• History of Religion
• Judaism and Jewish Studies
• Qumran Studies
• Religion and Education
• Religion and Health
• Religion and Politics
• Religion and Science
• Religion and Law
• Religion and Art, Literature, and Music
• Religious Studies
• Browse content in Society and Culture
• Cookery, Food, and Drink
• Cultural Studies
• Customs and Traditions
• Ethical Issues and Debates
• Hobbies, Games, Arts and Crafts
• Lifestyle, Home, and Garden
• Natural world, Country Life, and Pets
• Popular Beliefs and Controversial Knowledge
• Sports and Outdoor Recreation
• Technology and Society
• Travel and Holiday
• Visual Culture
• Browse content in Law
• Arbitration
• Browse content in Company and Commercial Law
• Commercial Law
• Company Law
• Browse content in Comparative Law
• Systems of Law
• Competition Law
• Browse content in Constitutional and Administrative Law
• Government Powers
• Judicial Review
• Local Government Law
• Military and Defence Law
• Parliamentary and Legislative Practice
• Construction Law
• Contract Law
• Browse content in Criminal Law
• Criminal Procedure
• Criminal Evidence Law
• Sentencing and Punishment
• Employment and Labour Law
• Environment and Energy Law
• Browse content in Financial Law
• Banking Law
• Insolvency Law
• History of Law
• Human Rights and Immigration
• Intellectual Property Law
• Browse content in International Law
• Private International Law and Conflict of Laws
• Public International Law
• IT and Communications Law
• Jurisprudence and Philosophy of Law
• Law and Society
• Law and Politics
• Browse content in Legal System and Practice
• Courts and Procedure
• Legal Skills and Practice
• Primary Sources of Law
• Regulation of Legal Profession
• Medical and Healthcare Law
• Browse content in Policing
• Criminal Investigation and Detection
• Police and Security Services
• Police Procedure and Law
• Police Regional Planning
• Browse content in Property Law
• Personal Property Law
• Study and Revision
• Terrorism and National Security Law
• Browse content in Trusts Law
• Wills and Probate or Succession
• Browse content in Medicine and Health
• Browse content in Allied Health Professions
• Arts Therapies
• Clinical Science
• Dietetics and Nutrition
• Occupational Therapy
• Operating Department Practice
• Physiotherapy
• Radiography
• Speech and Language Therapy
• Browse content in Anaesthetics
• General Anaesthesia
• Neuroanaesthesia
• Clinical Neuroscience
• Browse content in Clinical Medicine
• Acute Medicine
• Cardiovascular Medicine
• Clinical Genetics
• Clinical Pharmacology and Therapeutics
• Dermatology
• Endocrinology and Diabetes
• Gastroenterology
• Genito-urinary Medicine
• Geriatric Medicine
• Infectious Diseases
• Medical Toxicology
• Medical Oncology
• Pain Medicine
• Palliative Medicine
• Rehabilitation Medicine
• Respiratory Medicine and Pulmonology
• Rheumatology
• Sleep Medicine
• Sports and Exercise Medicine
• Community Medical Services
• Critical Care
• Emergency Medicine
• Forensic Medicine
• Haematology
• History of Medicine
• Browse content in Medical Skills
• Clinical Skills
• Communication Skills
• Nursing Skills
• Surgical Skills
• Medical Ethics
• Browse content in Medical Dentistry
• Oral and Maxillofacial Surgery
• Paediatric Dentistry
• Restorative Dentistry and Orthodontics
• Surgical Dentistry
• Medical Statistics and Methodology
• Browse content in Neurology
• Clinical Neurophysiology
• Neuropathology
• Nursing Studies
• Browse content in Obstetrics and Gynaecology
• Gynaecology
• Occupational Medicine
• Ophthalmology
• Otolaryngology (ENT)
• Browse content in Paediatrics
• Neonatology
• Browse content in Pathology
• Chemical Pathology
• Clinical Cytogenetics and Molecular Genetics
• Histopathology
• Medical Microbiology and Virology
• Patient Education and Information
• Browse content in Pharmacology
• Psychopharmacology
• Browse content in Popular Health
• Caring for Others
• Complementary and Alternative Medicine
• Self-help and Personal Development
• Browse content in Preclinical Medicine
• Cell Biology
• Molecular Biology and Genetics
• Reproduction, Growth and Development
• Primary Care
• Professional Development in Medicine
• Browse content in Psychiatry
• Addiction Medicine
• Child and Adolescent Psychiatry
• Forensic Psychiatry
• Learning Disabilities
• Old Age Psychiatry
• Psychotherapy
• Browse content in Public Health and Epidemiology
• Epidemiology
• Public Health
• Browse content in Radiology
• Clinical Radiology
• Interventional Radiology
• Nuclear Medicine
• Radiation Oncology
• Reproductive Medicine
• Browse content in Surgery
• Cardiothoracic Surgery
• Gastro-intestinal and Colorectal Surgery
• General Surgery
• Neurosurgery
• Paediatric Surgery
• Peri-operative Care
• Plastic and Reconstructive Surgery
• Surgical Oncology
• Transplant Surgery
• Trauma and Orthopaedic Surgery
• Vascular Surgery
• Browse content in Science and Mathematics
• Browse content in Biological Sciences
• Aquatic Biology
• Biochemistry
• Bioinformatics and Computational Biology
• Developmental Biology
• Ecology and Conservation
• Evolutionary Biology
• Genetics and Genomics
• Microbiology
• Molecular and Cell Biology
• Natural History
• Plant Sciences and Forestry
• Research Methods in Life Sciences
• Structural Biology
• Systems Biology
• Zoology and Animal Sciences
• Browse content in Chemistry
• Analytical Chemistry
• Computational Chemistry
• Crystallography
• Environmental Chemistry
• Industrial Chemistry
• Inorganic Chemistry
• Materials Chemistry
• Medicinal Chemistry
• Mineralogy and Gems
• Organic Chemistry
• Physical Chemistry
• Polymer Chemistry
• Study and Communication Skills in Chemistry
• Theoretical Chemistry
• Browse content in Computer Science
• Artificial Intelligence
• Computer Architecture and Logic Design
• Game Studies
• Human-Computer Interaction
• Mathematical Theory of Computation
• Programming Languages
• Software Engineering
• Systems Analysis and Design
• Virtual Reality
• Browse content in Computing
• Business Applications
• Computer Games
• Computer Security
• Computer Networking and Communications
• Digital Lifestyle
• Graphical and Digital Media Applications
• Operating Systems
• Browse content in Earth Sciences and Geography
• Atmospheric Sciences
• Environmental Geography
• Geology and the Lithosphere
• Maps and Map-making
• Meteorology and Climatology
• Oceanography and Hydrology
• Palaeontology
• Physical Geography and Topography
• Regional Geography
• Soil Science
• Urban Geography
• Browse content in Engineering and Technology
• Agriculture and Farming
• Biological Engineering
• Civil Engineering, Surveying, and Building
• Electronics and Communications Engineering
• Energy Technology
• Engineering (General)
• Environmental Science, Engineering, and Technology
• History of Engineering and Technology
• Mechanical Engineering and Materials
• Technology of Industrial Chemistry
• Transport Technology and Trades
• Browse content in Environmental Science
• Applied Ecology (Environmental Science)
• Conservation of the Environment (Environmental Science)
• Environmental Sustainability
• Environmentalist Thought and Ideology (Environmental Science)
• Management of Land and Natural Resources (Environmental Science)
• Natural Disasters (Environmental Science)
• Nuclear Issues (Environmental Science)
• Pollution and Threats to the Environment (Environmental Science)
• Social Impact of Environmental Issues (Environmental Science)
• History of Science and Technology
• Browse content in Materials Science
• Ceramics and Glasses
• Composite Materials
• Metals, Alloying, and Corrosion
• Nanotechnology
• Browse content in Mathematics
• Applied Mathematics
• Biomathematics and Statistics
• History of Mathematics
• Mathematical Education
• Mathematical Finance
• Mathematical Analysis
• Numerical and Computational Mathematics
• Probability and Statistics
• Pure Mathematics
• Browse content in Neuroscience
• Cognition and Behavioural Neuroscience
• Development of the Nervous System
• Disorders of the Nervous System
• History of Neuroscience
• Invertebrate Neurobiology
• Molecular and Cellular Systems
• Neuroendocrinology and Autonomic Nervous System
• Neuroscientific Techniques
• Sensory and Motor Systems
• Browse content in Physics
• Astronomy and Astrophysics
• Atomic, Molecular, and Optical Physics
• Biological and Medical Physics
• Classical Mechanics
• Computational Physics
• Condensed Matter Physics
• Electromagnetism, Optics, and Acoustics
• History of Physics
• Mathematical and Statistical Physics
• Measurement Science
• Nuclear Physics
• Particles and Fields
• Plasma Physics
• Quantum Physics
• Relativity and Gravitation
• Semiconductor and Mesoscopic Physics
• Browse content in Psychology
• Affective Sciences
• Clinical Psychology
• Cognitive Psychology
• Cognitive Neuroscience
• Criminal and Forensic Psychology
• Developmental Psychology
• Educational Psychology
• Evolutionary Psychology
• Health Psychology
• History and Systems in Psychology
• Music Psychology
• Neuropsychology
• Organizational Psychology
• Psychological Assessment and Testing
• Psychology of Human-Technology Interaction
• Psychology Professional Development and Training
• Research Methods in Psychology
• Social Psychology
• Browse content in Social Sciences
• Browse content in Anthropology
• Anthropology of Religion
• Human Evolution
• Medical Anthropology
• Physical Anthropology
• Regional Anthropology
• Social and Cultural Anthropology
• Theory and Practice of Anthropology
• Browse content in Business and Management
• Business Ethics
• Business History
• Business Strategy
• Business and Technology
• Business and Government
• Business and the Environment
• Comparative Management
• Corporate Governance
• Corporate Social Responsibility
• Entrepreneurship
• Health Management
• Human Resource Management
• Industrial and Employment Relations
• Industry Studies
• Information and Communication Technologies
• International Business
• Knowledge Management
• Management and Management Techniques
• Operations Management
• Organizational Theory and Behaviour
• Pensions and Pension Management
• Public and Nonprofit Management
• Strategic Management
• Supply Chain Management
• Browse content in Criminology and Criminal Justice
• Criminal Justice
• Criminology
• Forms of Crime
• International and Comparative Criminology
• Youth Violence and Juvenile Justice
• Development Studies
• Browse content in Economics
• Agricultural, Environmental, and Natural Resource Economics
• Asian Economics
• Behavioural Finance
• Behavioural Economics and Neuroeconomics
• Econometrics and Mathematical Economics
• Economic History
• Economic Methodology
• Economic Systems
• Economic Development and Growth
• Financial Markets
• Financial Institutions and Services
• General Economics and Teaching
• Health, Education, and Welfare
• History of Economic Thought
• International Economics
• Labour and Demographic Economics
• Law and Economics
• Macroeconomics and Monetary Economics
• Microeconomics
• Public Economics
• Urban, Rural, and Regional Economics
• Welfare Economics
• Browse content in Education
• Adult Education and Continuous Learning
• Care and Counselling of Students
• Early Childhood and Elementary Education
• Educational Equipment and Technology
• Educational Strategies and Policy
• Higher and Further Education
• Organization and Management of Education
• Philosophy and Theory of Education
• Schools Studies
• Secondary Education
• Teaching of a Specific Subject
• Teaching of Specific Groups and Special Educational Needs
• Teaching Skills and Techniques
• Browse content in Environment
• Applied Ecology (Social Science)
• Climate Change
• Conservation of the Environment (Social Science)
• Environmentalist Thought and Ideology (Social Science)
• Natural Disasters (Environment)
• Social Impact of Environmental Issues (Social Science)
• Browse content in Human Geography
• Cultural Geography
• Economic Geography
• Political Geography
• Browse content in Interdisciplinary Studies
• Communication Studies
• Museums, Libraries, and Information Sciences
• Browse content in Politics
• African Politics
• Asian Politics
• Chinese Politics
• Comparative Politics
• Conflict Politics
• Elections and Electoral Studies
• Environmental Politics
• European Union
• Foreign Policy
• Gender and Politics
• Human Rights and Politics
• Indian Politics
• International Relations
• International Organization (Politics)
• International Political Economy
• Irish Politics
• Latin American Politics
• Middle Eastern Politics
• Political Behaviour
• Political Economy
• Political Institutions
• Political Theory
• Political Methodology
• Political Communication
• Political Philosophy
• Political Sociology
• Politics and Law
• Public Policy
• Public Administration
• Quantitative Political Methodology
• Regional Political Studies
• Russian Politics
• Security Studies
• State and Local Government
• UK Politics
• US Politics
• Browse content in Regional and Area Studies
• African Studies
• Asian Studies
• East Asian Studies
• Japanese Studies
• Latin American Studies
• Middle Eastern Studies
• Native American Studies
• Scottish Studies
• Browse content in Research and Information
• Research Methods
• Browse content in Social Work
• Addictions and Substance Misuse
• Adoption and Fostering
• Care of the Elderly
• Child and Adolescent Social Work
• Couple and Family Social Work
• Developmental and Physical Disabilities Social Work
• Direct Practice and Clinical Social Work
• Emergency Services
• Human Behaviour and the Social Environment
• International and Global Issues in Social Work
• Mental and Behavioural Health
• Social Justice and Human Rights
• Social Policy and Advocacy
• Social Work and Crime and Justice
• Social Work Macro Practice
• Social Work Practice Settings
• Social Work Research and Evidence-based Practice
• Welfare and Benefit Systems
• Browse content in Sociology
• Childhood Studies
• Community Development
• Comparative and Historical Sociology
• Economic Sociology
• Gender and Sexuality
• Gerontology and Ageing
• Health, Illness, and Medicine
• Marriage and the Family
• Migration Studies
• Occupations, Professions, and Work
• Organizations
• Population and Demography
• Race and Ethnicity
• Social Theory
• Social Movements and Social Change
• Social Research and Statistics
• Social Stratification, Inequality, and Mobility
• Sociology of Religion
• Sociology of Education
• Sport and Leisure
• Urban and Rural Studies
• Browse content in Warfare and Defence
• Defence Strategy, Planning, and Research
• Land Forces and Warfare
• Military Administration
• Military Life and Institutions
• Naval Forces and Warfare
• Other Warfare and Defence Issues
• Peace Studies and Conflict Resolution
• Weapons and Equipment

• < Previous chapter
• Next chapter >

40 The Unique Methodology of Policy Research

Amitai Etzioni is a university professor and Professor of International Relations at The George Washington University. He served as a Senior Advisor at the Carter White House; taught at Columbia University, Harvard University, and University of California, Berkeley; and served as president of the Society for the Advancement of Socio-Economics (SASE). A study by Richard Posner ranked him among the top 100 American intellectuals. Etzioni is the author of many books, including Security First (2007), Foreign Policy: Thinking Outside the Box (2016), and Avoiding War with China (2017). His most recent book, Happiness is the Wrong Metric: A Liberal Communitarian Response to Populism, was published by Springer in January 2018.

• Published: 02 September 2009
• Cite Icon Cite
• Permissions Icon Permissions

This article provides a unique methodology of policy research, focusing on the various factors that differentiate policy research from basic research. It identifies malleability as a key variable of policy research, and this is defined as the amount of resources that would have to be expended to cause change in a given variable or variables. The scope of analysis/factors of policy research is shown to encompass all the major facets of the social phenomenon it is trying to deal with. Basic research, on the other hand, fragments the world into abstract and analytical slices, which are then studied individually. The last two differentiating factors of policy research and basic research, which are privacy and communication, are studied in the last two sections of the article.

Policy research requires a profoundly different methodology from that on which basic research relies, because policy research is always dedicated to changing the world while basic research seeks to understand it as it is. 1 The notion that if one merely understands the world better, then one will in turn know how to better it, is not supported by the evidence.

Typical policy goals are the reduction of poverty, curbing crime, cutting pollution, or changing some other condition (Mitchell and Mitchell 1969, 393) . Even those policies whose purpose is to maintain the status quo are promoting change—they aim to slow down or even reverse processes of deterioration, for instance that of natural monuments or historical documents. When no change is sought, say, when no one is concerned with changing the face of the moon, then there is no need for policy research in that particular area.

Moreover, although understanding the causes of a phenomenon, which successful basic research allows, is helpful in formulating policy, often a large amount of other information that is structured in a different manner best serves policy makers. 2 Policy researchers draw on a large amount of information that has no particular analytical base or theoretical background (of the kind that basic research provides). 3 In this sense medical science, which deals with changing bodies and minds, is a protypical policy science. It is estimated that about half of the information physicians employ has no basis in biology, chemistry, or any other science; but rather it is based on an accumulation of experience. 4 This knowledge is passed on from one medical cohort to another, as “these are the way things are done” and “they work.”

The same holds true for other policy sciences. For instance, criminologists who inform a local government that studies show that rehabilitation works more effectively in minimum security prisons than in maximum security prisons (a fact that can be explained by sociological theoretical concepts based on basic research) 5 know from long experience that they had better also alert the local authorities that such a reduction in security could potentially lead some inmates to escape and commit crimes in surrounding areas. Without being willing to accept such a “side effect” of the changed security policy, those governments who introduced it may well lose the next election and security in the prison will be returned to its previously high level. There is no particular sociological theoretical reason for escapes to rise when security is lowered. It is an observation based on common sense and experience; however it is hardly an observation that policy makers, let alone policy researchers should ignore. (They may though explore ways of coping with this “side effect,” for instance by either preparing the public ahead of time, introducing an alert system when inmates escape, or some other such measure.)

The examples just given seek to illustrate the difference between the information that basic research generates versus information that plays a major role in policy research. That is, there are important parts of the knowledge on which policy research draws that are based on distilled practice and are not derivable from basic research. Much of what follows deals with major differences in the ways that information and analysis are structured in sound policy research in contrast to the ways basic research is carried out.

One clarification before I can proceed: Policy research should not be confused with applied research. Applied research presumes that a policy decision has already been made and those responsible are now looking for the most efficient ways to implement it. Policy research helps to determine what the policy decision ought to be.

1. Malleability

A major difference between basic and policy research is that malleability is a key variable for the latter though not the former (Weimer and Vining 1989; 4) . Indeed for policy researchers it is arguably the single most important variable. Malleability for the purposes at hand ought to be defined as the amount of resources (including time, energy, and political capital) that would have to be expended to cause change in a given variable or variables. For policy research, malleability is a cardinal consideration because resources always fall short of what is required to implement given policy goals. Hence, to employ resources effectively requires determining the relative results to be generated from different patterns of allocation (Dunn 1981, 334– 402) . In contrast, basic research has no principled reason to favor some factors (or variables) over others. For basic research, it matters little if at all whether a condition under study can be modified and if it can how much it would cost. To illustrate, many sociological studies compare people by gender and age and although these variables may seem relevant, they are of limited value to policy research. Other variables used, such as the levels of income of various populations, the extent of education of various racial and ethnic groups, and the average size of cities, are somewhat more malleable but still not highly so. In contrast, perceptions are much more malleable.

One may say that basic research should reveal a preference for variables that have been less studied; however, such a consideration concerns the economics and politics of science rather than methodology. Because all scientific findings are conditional and temporary and often subject to profound revision and recasting, for basic researchers, retesting old findings can be just as valuable as covering new variables. In short, although in principle for basic research the study of all variables is legitimate, in a given period of time or amongst a given group of scientists, some may consider certain variables as more “interesting” or “promising” than others. In contrast, to reiterate, for policy research, malleability is the most important variable as it is directly related to its core reason for being: Promoting change.

Given the dominance of basic research methodology in the ways policy research is taught, it is not surprising to find that the question of which variables are more malleable than others is rarely studied in any systematic way. Due to the importance of this issue for policy research, some elaboration and illustrations are called for. Economic feasibility is a good case in point. Many policy researchers' final reports do not include any, not even crude estimates of the costs involved in what they are recommending. 6 Even less common is any consideration of the question of whether such changes can be made acceptable to elected representatives and the public at large; that is, political feasibility (Weimer and Vining 1989, 292– 324) . For instance, over the last decades several groups favored advancing their policy goals through constitutional amendments, ignoring the fact that these are extremely difficult to get passed.

In other cases, feasibility is treated as a secondary “applied” question to be studied later, after policy makers adopt the recommended policy. However, the issue runs much deeper than the assessments of feasibility of one kind or another. The challenge to policy research is to determine the relative resistance to change according to the different variables that are to be tackled. And this question must be tackled not on an ad hoc basis, but rather as a major part of systematic policy research. Moreover, if the variables involved are studied from this viewpoint, they themselves may be changed; that is, feasibility is enhanced rather than treated as a given.

Another example of the cardinal need to take malleability into account when conducting policy research concerns changing public attitudes. Policy makers often favor a “public education' campaign when they desire to affect people's beliefs and conduct. Policy makers tend to assume that it is feasible to change such predispositions through a way that might be called the Madison Avenue approach, which entails running a series of commercials (or public service announcements), mounting billboards, obtaining celebrity endorsements, and so on.

For example, the United States engaged in such a campaign in 2003 and 2004 to change the hearts and minds of “the Arab street” through what has also been termed “public diplomacy.” 7 The way this was carried out provides a vivid example of lack of attention to feasibility issues. American public diplomacy, developed by the State Department, included commercials, websites, and speakers programs that sought to “reconnect the world's billion Muslims with the United States the way McDonald's highlights its billion customers served” (Satloff 2003, 18) . It was based on the premiss that “blitzing Arab and Muslim countries with Britney Spears videos and Arabic‐language sitcoms will earn Washington millions of new Muslim sympathizers” (Satloff 2003, 18) . A study found that the results were “disastrous” (Satloff 2003, 18) . Some countries declined to air the messages and many Muslims who did see the material viewed it as blatant propaganda and offensive rather than compelling.

Actually, policy researchers bent on studying feasibility report that the Madison Avenue approach works only when large amounts of money are spent to shift people from one product to another when there are next to no differences between them (e.g. two brands of toothpaste) and when there is an inclination to use the product in the first place. However, when these methods are applied to changing attitudes about matters as different as condom use, 8 the United Nations, 9 electoral reform, and so forth, they are much less successful. Changing people's behavior—say to conserve energy, drive slower, cease smoking—is many hundreds of times more difficult. This is a major reason why totalitarian regimes, despite intensive public education campaigns, usually fail. The question of what is most feasible is determined by fiat by policy makers and their staffs rather than by studies that are reported to the policy makers by policy researchers. Hence decisions are often based on a fly‐by‐the‐seat‐ of‐your‐pants sense of what can be changed rather than on empirical evidence. 10 One of the few exceptions is studies of nation building in which several key policy researchers presented the reasons why such endeavors can be carried out at best only slowly while at the same time many policy makers claimed that it could be achieved in short order and at low cost. 11

In a preliminary stab at outlining the relative malleability of various factors, one may note that as a rule the laws of nature are not malleable; social relations, including patterns of asset distribution and power, are of limited malleability; and symbolic relations are highly malleable. Thus any policy‐making body that would seek to modify the level of gravity, for example, not for a particular situation (for instance a space travel simulator) but in general, will find this task at best extremely difficult to advance. In contrast, those who seek to change a flag, a national motto, the ways people refer to one another (e.g. Ms Instead of girl or broad), have a relatively easy time of doing so. Changes in the distribution of wealth among the classes or races—by public policy—are easier than changes involving the laws of nature, but more difficult than changing hearts and minds.

When policy researchers or policy makers ignore these observations and enact laws that seek grand and quick changes in power relations and economic patterns, the laws are soon reversed. A case in point is the developments that ensued when a policy researcher inserted into legislation the phrase “maximum feasible participation of the poor.” This Act was used to try to circumvent prevailing local power structures by directing federal funds to voluntary groups that included the poor on their advisory boards, which thus helped “empower the poor.” The law was nullified shortly thereafter. Similarly, when a constitutional amendment was enacted that banned the consumption of alcohol in the United States, it had some severely distorted effects on the American justice and law enforcement systems and did little actually to reduce the consumption of alcohol. It was also the only constitutional amendment ever to be repealed.

Among social changes, often legal and political reduction in inequality is relatively easier to come by than are socioeconomic changes along similar lines. Thus, African‐Americans and women gained de jure and de facto voting rights long before the differences in their income and representation in the seats of power moved closer to those of whites (in the case of African‐Americans) and of men (in the case of women). Nor have socioeconomic differences been reduced nearly as much as legal and political differences, although in both realms considerable inequalities remain. The same is true not just for the United States, but for other free societies and those that have been recently liberated.

In short, there are important differences in which dedication of resources, commitment of political capital, and public education are needed in order to bring about change. Sound policy research best makes the determination of which factors are more malleable than others, which is a major subject of study.

2. Scope of Analysis

Another particularly important difference between basic research and policy research methodology concerns the scope of factors that are best encompassed. Policy research at its best encompasses all the major facets of the social phenomenon it is trying to deal with. 12 In contrast, basic research proceeds by fragmenting the world into abstract, analytical slices which are then studied individually.

A wit has suggested that in economics everything has a price; in sociology, nothing has a price. Policy makers and hence researchers are at a disadvantage when they formulate preferred policy alternatives without paying attention to the longer‐run economic and budgetary effects—or the effect of such policy on social relations including families (e.g. tax preferences for singles), socioeconomic classes (e.g. estate taxes), and so on.

To put it in elementary terms, a basic researcher may well study only the prices of flowers (together with other economic factors); a physiologist the wilting processes; a social psychologist the symbolic meaning of flowers; and so forth. But a community that plans to grow flowers in its public gardens must deal with most, if not all of these elements and the relations between them. Flowers that are quick to wilt will not be suitable for its public gardens; the community will be willing to pay more for flowers that have a longer life or those that command a positive symbolic meaning, and so on.

Medicine provides another model of a policy science. It cannot be based only on biology, chemistry, anatomy, or any one science that studies a subset of variables relating to the body. Instead physicians draw on all these sciences and add observations of interaction effects among the variables. This forms a medical knowledge base and drives “policy” recommendations (i.e. medical prescriptions). Indeed doctors have often been chastised when they do not take into account still other variables, such as those studied by psychologists and anthropologists. Similarly, international relations is a policy science that best combines variables studied by economists, political scientists, law professors, and many others.

In short, the scope of variables that basic research encompasses can be quite legitimate and effective but also rather narrow. Policy researchers must be more eclectic and include at least all the variables that account for a significant degree of variance in the phenomenon that the policy aims to change.

3. Private and Confidential

Basic research is a public endeavor. As a rule its results are published so that others can critically assess them and piece them together with their findings and those of still others in order to build ever more encompassing and robust bodies of knowledge. Unpublished work is often not considered when scientists are evaluated for hiring and promoting, for prizes, or for some other reason, especially not if the work is kept secret for commercial or public security reasons. Historically, scientific findings were published in monographs, books, and articles in suitable journals. These served as the main outlets for the findings of basic research both because only by making scientific findings public could they become part of the cumulative scientific knowledge base and also because publication indicates that they have already passed some measure of peer review. It is only through peer review that evidence can be critically scrutinized. In recent years findings are still made public but increasingly they are often posted on websites, most of which lack peer review foundations, which is one reason why they are less trusted and not treated as a full‐fledged publication. Publication is still considered an essential element of basic research.

In contrast, the findings of policy research are often not published—they are provided in private to one policy maker or another (Radin 1997, 204– 18) . The main purpose of policy research is not to contribute to the cumulative process of building knowledge but rather to put to service available knowledge. In that profound sense policy research is often not public but client oriented. 13 Although some policy research is conducted in think tanks and public policy schools that may treat it similarly to basic research, more often than not it is conducted in specialized units in government agencies, the White House, corporate associations, and labor unions. And often tools of policy research are memos and briefings, not publications.

Often the findings of policy researchers are considered confidential or are governed by state secret acts (which is the case in many nations that have a less strong view of civil liberties than does the United States). That is, the findings are merely aimed at a specific client or a group of clients, and sharing them with the public is considered an offense. 14

4. Communication

Basic researchers, as a rule, are much less concerned with communicating, especially with a larger, “secular” public than are policy researchers. This may at first seem a contradiction to the previously made point that science (in the basic research sense) is public while policy research is often “private” (even when conducted for public officials). The seeming contradiction vanishes once one notes that basic researchers are obligated to share their findings with their colleagues , often a small group, and that they seek feedback from this group for both scientific and psychological validation. However, as a rule basic researchers have little interest in the public at large. Indeed, they tend to be highly critical of those who seek to reach such an audience—as did scholars such as Jay Gould and Carl Sagan (Etzioni 2003, 57– 60) .

In contrast, policy researchers often recognize the need to mobilize public support for the policies that their findings favor and hence they tend to help policy makers to mobilize such support by communicating with the public. James Fishkin developed a policy idea he called “deliberative democracy,” which entailed bringing together a group of people who constitute a living sample of the population for a period of time during which they are exposed to public education and presentations by public figures, and they are given a chance to have a dialogue. By measuring the changes in the views of this living sample, Fishkin found that one is able to learn how to change the public's mind. Fishkin did not just develop the concept and publish his ideas, but conducted a long and intensive campaign through radio, TV, newspapers, visits with public leaders, and much more, until his living sample was implemented in several locations (Fishkin 1997) . Indeed, according to Eugene Bardach, policy researchers must prepare themselves for “a long campaign potentially involving many players, including the mass public” (Bardach 2002, 115– 17) .

Hence, basic researchers are more likely to use technical terms (which may sound like jargon to outsiders), mathematical notations, extensive footnotes, and other such scientific features. On the other hand, policy researchers are more likely to express themselves in the vernacular and avoid technical terms.

One can readily show numerous publications of professors at schools of public policy and even think tanks that are rather similar if not indistinguishable from those of basic researchers. 15 But this is the case because these schools conduct mostly basic, and surprisingly little policy research. For example, on 28 April 2004 Google search found only 210 entries for “policy research methodology,” the good part of which referred to university classes by that name. But on closer examination, most entries were referring to basic, not policy research methodology. For instance, a course titled “Cultural Policy Research Methodology” at Griffith University in Australia includes in its course description “basic research techniques, particularly survey methodologies, qualitative methods and a more in depth approach to statistics.” 16 Many other entries were for classes in policy or research methodology (usually basic). The main reasons for this are ( a ) because few places train people in the special methodologies that policy research requires and ( b ) the reward structure is closely tied to basic research. Typically, promotions (especially tenure) at public policy schools are determined by evaluations and votes by senior colleagues from the basic research departments at the same universities or at other ones. Thus the future of an economist at the Harvard Business School may depend on what her colleagues in the Harvard Economics department think of her work. More informally, being invited to become a member of a basic research department is considered a source of prestige and an opportunity to shore up one's training and research. Conversely, only being affiliated with a policy school (like other professional schools) indicates a lack of recognition, which may translate into objective disadvantages. This pecking order, which favors basic over policy (considered “applied”) research, is of considerable psychological importance to researchers in practically all universities. Even in think tanks dedicated to policy research, many respect basic research more than policy research and hope to conduct it one day or regret that they are not suited to carry it out. 17

People who work for think tanks, which are largely dedicated to policy research, often seek to move to universities, in which tenure is more common and there is a greater sense of prestige. Hence many such researchers are keen to keep their “basic” credentials, although often they are unaware of the special methodology that policy research requires or are untutored in carrying it out in the first place because they were trained in basic research modes instead.

At annual meetings of one's discipline, in which findings are presented and evaluated, jobs are negotiated and information about them shared, and prestige scoring is rearranged, policy researchers will typically attend those dominated by their basic research colleagues. And attendance at policy research associations (such as the Association for Public Policy Analysis and Management) is meager. Most prizes and other awards available to researchers go to those who conduct basic research.

In short, although the logic of policy research favors it to be more communicative than basic research, this is often not the case because the training and institutional formations in which policy research is largely conducted favor basic research.

Bardach, E. ( 2002 ). Educating the client: an introduction.   Journal of Policy Analysis and Management , 21 (1): 115–17. 10.1002/pam.1044

Google Scholar

Berelson, B. , and Steiner, G.   1964 . Human Behavior: An Inventory of Scientific Findings . New York: Harcourt Brace and World.

Google Preview

Carothers, T.   1999 . Aiding Democracy Abroad: The Learning Curve. Washington, DC: Carnegie Endowment for International Peace.

DeJong, W. , Wolf, R. C. , and Austin, S. B.   2001 . US federally funded television public service announcements (PSAs) to prevent HIV/AIDS: a content analysis.   Journal of Health Communication , 6: 249–63. 10.1080/108107301752384433

Dunn, W. N.   1981 . Public Policy Analysis: An Introduction . Englewood Cliffs, NJ: Prentice Hall.

Etzioni, A.   1968 . The Active Society: A Theory of Societal and Political Processes . New York: Free Press.

——  1971 a . A Comparative Analysis of Complex Organizations , rev. edn. New York: Free Press.

——  1971 b . Policy research.   American Sociologist , 6 (supplementary issue: June): 8–12.

——  2003 . My Brother's Keeper: A Memoir and a Message . Lanham, Md.: Rowman and Littlefield.

——  2004 . A self‐restrained approach to nation‐building by foreign powers.   International Affairs , 80: 1–17. 10.1111/j.1468-2346.2004.00362.x

Fishkin, J. S.   1997 . The Voice of the People: Public Opinion and Democracy . New Haven, Conn.: Yale University Press.

Free Expression Project   2003 . The Progress of Science and Useful Arts: Why Copyright Today Threatens Intellectual Freedom , 2nd edn. New York: Free Expression Project; available at: www.fepproject.org/policyreports/copyright2dconc.html (accessed 27 Apr. 2004).

Inglefinger, F. J. , Relman, A. S. , and Findland, M.   1966 . Controversy in Internal Medicine . Philadelphia: W. B. Saunder.

Lasswell, H. , and Lerner, D.   1951 . The Policy Sciences . Stanford, Calif.: Stanford University Press.

Miller, D. W. 2001. Poking holes in the theory of broken windows. Chronicle of Higher Education (Feb.): A14.

Miller, J. 2004. Censored study on bioterror doubts U.S. preparedness. New York Times (29 Mar.): A15.

Mitchell, J. , and Mitchell, W.   1969 . Policy‐Making and Human Welfare . Chicago: Rand McNally.

Nelson, B.   1999 . Diversity and public problem solving: ideas and practice in policy education.   Journal of Policy Analysis and Management , 18: 134–55. 10.1002/(SICI)1520-6688(199924)18:1<134::AID-PAM9>3.0.CO;2-6

Radin, B. A.   1997 . The evolution of the policy analysis field: from conversation to conversations.   Journal of Policy Analysis and Management , 16: 204–18. 10.1002/(SICI)1520-6688(199721)16:2<204::AID-PAM1>3.0.CO;2-M

Raver, C.   2002 . Emotions matter: making the case for the role of young children's emotional development for early school readiness.   Social Policy Report , 16 (3): 3–19.

Roe, E.   1998 . Taking Complexity Seriously: Policy Analysis, Triangulation, and Sustainable Development . Boston: Kluwer Academic.

Satloff, R.   2003 . How to win friends and influence Arabs.   Weekly Standard , 18 Aug: 18–19.

Schön, D.   1983 . The Reflective Practitioner . New York: Basic Books.

Scott, J. 1994. Condom ads get direct: use them and get sex. Atlanta Journal and Constitution (3 Oct.): B1.

Star, S. A. , and Hughes, H. M.   1950 . Report on an educational campaign: the Cincinnati plan for the United Nations. American   Journal of Sociology , 55: 389–400.

Weimer, D. L. , and Vining, A. R.   1989 . Policy Analysis: Concepts and Practice . Englewood Cliffs, NJ: Prentice Hall.

Weiss, C.   1983 . Ideology, interests and information: the basis of policy positions. Pp. 213–45 in Ethics, the Social Sciences and Policy Analysis , ed. D. Callahan and B. Jennings . New York: Plenum.

Wilson, J. Q. , and Kelling, G.   1982 . Broken windows: the police and neighborhood safety.   Atlantic Monthly , 249 (3: Mar.): 29–38.

The first book to deal with policy sciences and consequently often cited is Lasswell and Lerner's The Policy Sciences (1951) . However this book does not address the methodological issues at hand. For an early treatment of these issues, see Etzioni 1971 b , 1968 .

For an example of how to structure and present policy research and analysis, see Dunn 1981, 322 .

For example many policy makers subscribe to George L. Kelling and James Q. Wilson's criminology theories because they make sense, despite the fact that they are not grounded in academic research. See Wilson and Kelling 1982 . For criticisms of this approach to criminology, see Miller 2001 .

“Much” of medicine is not scientifically supported (Inglefinger, Relman, and Findland 1966) . “85 percent of the problems a doctor sees in his office are not in the book” (quoted from a physician in Schön 1983, 16) .

See Etzioni 1971 a , 246– 7 .

See for example Free Expression Project 2003 ; Raver 2002, 3– 19 .

See, for instance, The Advisory Group on Public Diplomacy in the Arab and Muslim World, “Changing minds, winning peace: a new strategic direction for U.S. public diplomacy in the Arab and Muslim world,” Oct, 2003, Edward P. Djerejian, chair.

For instance, the Centers for Disease Control conducted a ten‐year ad campaign to educate Americans about condoms and to encourage their use to prevent HIV transmission. After spending millions of dollars on these ads, a CDC study found that only 45 % of sexually active high school students used a condom the last time they had sex: see Scott 1994 . A recent evaluation of the program issued an unqualified “no” in answer to the question, “Has the U.S. federal government's HIV /AIDS television [public service announcement] campaign been designed not only to make the public aware of HIV /AIDS but also to provide appropriate messages to motivate and reinforce behavior change?” See DeJong, Wolf, and Austin 2001, 256 . Of the fifty‐six ads reviewed, fifty were created by the CDC, the other six were created by the National Institute on Drug Abuse.

Star and Hughes 1950 , quoted in Berelson and Steiner 1964, 530 .

Indeed unlike science, Carol Weiss has argued that in the policy field it may be impossible to separate objective knowledge from ideology or interests: see Weiss 1983 .

See Carothers 1999 ; Etzioni 2004 .

Roe 1998 . For an academic policy research perspective, see Nelson 1999 .

See “Professional practice symposium: educating the client,” Journal of Policy Analysis and Management , 21 (1: 2002): 115– 36.

For instance, the Defense Department has prohibited a Washington think tank from publishing a complete report about the lack of government preparedness for bioterror attacks: see Miller 2004 .

See for instance the reports of the family research division of the Heritage Foundation, available at www.heritage.org/research/family/issues2004.cfm (accessed 29 Apr. 2004). See also “The war on drugs: addicted to failure,” Recommendations of the Citizens' Commission on US Drug Policy, available at www.ips‐dc.org/projects /drugpolicy.htm (accessed 29 Apr. 2004).

See Griffith University course catalog. Available at: www22.gu.edu.au/STIP/servlet/STIP?s=7319AMC (accessed 28 Apr. 2004).

This section is based on my personal observations of organizations such as the John F. Kennedy School of Government, the American Enterprise Institute, RAND, CATO, the Heritage Foundation, and many others.

• About Oxford Academic
• Publish journals with us
• University press partners
• What we publish
• New features  
• Open access
• Institutional account management
• Rights and permissions
• Get help with access
• Accessibility
• Advertising
• Media enquiries
• Oxford University Press
• Oxford Languages
• University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

• Copyright © 2024 Oxford University Press
• Cookie settings
• Cookie policy
• Privacy policy
• Legal notice

This Feature Is Available To Subscribers Only

Sign In or Create an Account

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

• Tools and Resources
• Customer Services
• Conflict Studies
• Development
• Environment
• Foreign Policy
• Human Rights
• International Law
• Organization
• International Relations Theory
• Political Communication
• Political Economy
• Political Geography
• Political Sociology
• Politics and Sexuality and Gender
• Qualitative Political Methodology
• Quantitative Political Methodology
• Security Studies
• Share This Facebook LinkedIn Twitter

Article contents

Methods of foreign policy analysis.

• Philip B.K. Potter Philip B.K. Potter Woodrow Wilson Department of Politics, University of Virginia
• https://doi.org/10.1093/acrefore/9780190846626.013.34
• Published in print: 01 March 2010
• Published online: 30 November 2017

Foreign policy analysis (FPA) is the study of how states, or the individuals that lead them, make foreign policy, execute foreign policy, and react to the foreign policies of other states. This topical breadth results in a subfield that encompasses a variety of questions and levels of analysis, and a correspondingly diverse set of methodological approaches. There are four methods which have become central in foreign policy analysis: archival research, content analysis, interviews, and focus groups. The first major phase of FPA research is termed “comparative foreign policy.” Proponents of comparative foreign policy sought to achieve comprehensive theories of foreign policy behavior through quantitative analysis of “events” data. An important strand of this behavioral work addressed the relationship between trade dependence and foreign policy compliance. On the other hand, second-generation FPA methodology largely abandoned universalized theory-building in favor of historical methods and qualitative analysis. Second-generation FPA researchers place particular emphasis on developing case study methodologies driven by social science principles. Meanwhile, the third-generation of FPA scholarship combines innovative quantitative and qualitative methods. Several methods of foreign policy analysis used by third-generation FPA researchers include computer assisted coding, experiments, simulation, surveys, network analysis, and prediction markets. Ultimately, additional attention should be given to determining the degree to which current methods of foreign policy analysis allow predictive or prescriptive conclusions. FPA scholars should also focus more in reengaging foreign policy analysis with the core of international relations research.

• foreign policy analysis
• methodological approaches
• comparative foreign policy
• events data analysis
• case study methodologies
• network analysis
• prediction markets
• foreign policy behavior

Introduction

The periodic reassessment of research methods is important to the vitality of any academic discipline, but it has particular salience for a relatively young field such as foreign policy analysis (FPA). Hudson and Vore ( 1995 :221) acknowledge as much in their review of the FPA literature, noting that, “in the study of foreign policy decision-making, the issues are not theoretical but methodological.” I define foreign policy analysis as the study of how states, or the individuals that lead them, make foreign policy, execute foreign policy, and react to the foreign policies of other states. This topical breadth results in a subfield that encompasses a variety of questions and levels of analysis, and a correspondingly diverse set of methodological approaches. This essay surveys FPA’s methodological development from its inception to the present and, in the process, outlines the body of existing methodological practice and identifies opportunities for future progress. The objective is to provide both an indication of the role that various quantitative and qualitative methods play in the FPA literature and an entryway for contemporary researchers seeking to apply these approaches to future work. Where appropriate, the reader is directed to more specific guides to the intricacies and execution of each method.

For the sake of organizational clarity, this review follows a stylized format roughly based on Neack, Hey, and Haney’s ( 1995 ) concept of “generational change” in foreign policy analysis. The section that immediately follows is partially archeological, that is, it surveys methods of events data analysis that were important to the early development of FPA, but in some cases have fallen out of widespread usage. The second section, which surveys qualitative methods, most closely reflects the current state of the art in the discipline. The third and final section addresses both cutting-edge and underutilized approaches.

The Methodological Origins of Foreign Policy Analysis

The unique historical context and intellectual environment of the early 1950s – specifically, the Cold War and the behavioral revolution – crucially shaped the early methodological development of foreign policy analysis. These origins have proven central to the methodological arc of the sub-discipline.

FPA was born of the opportunities presented by the largely atheoretical nature of historically oriented diplomatic analysis and the exclusion of political leadership and decision-making from the prevailing theories espoused by mainline international relations. Prior to the advent of FPA as a distinct subfield, the study of foreign policy relied on traditional methods and had long been the domain of political historians and diplomatic strategists in the tradition of thinkers such as Thucydides and Machiavelli. Early FPA researchers saw this longstanding tradition as part of their heritage, but, inspired by the methodological imperatives of the behavioral revolution, believed that systematizing the study of foreign policy would lead to progress in the form of generalizable and cumulative findings. Thus, from its inception, FPA was an explicitly theoretical exercise aimed at uncovering the systematic elements of foreign policy interactions, and the methods deployed reflected this.

Simultaneously, in response to the near monopoly of system-level theory in international relations, the pioneers of FPA argued that individual leaders or groups of decision makers are often the primary drivers of outcomes in international interactions (Snyder et al. 1954 ). Thus, at the very core of FPA’s intellectual identity lies a revisionist methodology (vis-à-vis diplomatic history) applied to a revisionist conception of the basic unit of analysis (vis-à-vis mainline international relations).

The strategic environment, specifically the position of the US in the early Cold War, also figured prominently in the early development of FPA methods. In the face of this protracted geopolitical conflict, American political leaders became unusually involved in the FPA academic endeavor. The promise of concrete conclusions and general enthusiasm for “scientific” approaches to political problems that stemmed from the success of the Manhattan Project led the US government to invest large sums in early FPA efforts. With funding came the expansion of major research centers such as the Rand Corporation and the Brookings Institution that were instrumental to the maturation of FPA as a subfield and methodological approach in international relations. However, the money and attention from the policy community came with strings attached – most notably, an expectation for immediately relevant research. Over time this requirement became increasingly difficult to reconcile with the relatively high uncertainty surrounding quantitative estimates of foreign policy phenomenon.

The first major phase of FPA research that emerged from this crucible is termed “comparative foreign policy.” Proponents of comparative foreign policy argued that controlled comparison of the domestic sources of external conduct across different countries could produce comprehensive theories of foreign policy behavior. Methodologically speaking, these scholars sought to achieve these ends primarily through quantitative analysis of “events” data, which I describe in detail in the section that follows. However, this transition to quantitative analysis was, at least in part, a refinement of even earlier attempts to develop a more robust understanding of the foreign-policy decision making process. Snyder, Bruck, and Sapin’s ( 1954 ) classic essay was arguably the first to encourage international relations scholars to reopen the “black box” of the state in order to study the actions of individual leaders. A significant body of early qualitative case study research flowed from this call to arms. To take just two examples, Paige ( 1968 ) took a decision making approach to understanding the origins of the Korean War, while Allison ( 1971 ) followed along similar lines with his well-known study of the Cuban Missile Crisis.

The decision making school provided a useful groundwork, particularly by identifying the leader as a crucial unit of analysis, a tradition that has persisted in FPA ever since. However, the developments of the behavioral revolution eventually overtook the primarily qualitative methods of these early FPA scholars. An increasing premium was placed on the generalizability garnered by operationalizing foreign policy interactions numerically and analyzing them quantitatively. This transition gave rise to the comparative foreign policy literature, which maintained an emphasis on decision making and scientific analysis, but moved away from case study analysis in favor of events data.

Comparative Foreign Policy and Events Data Analysis

The demand for foreign policy research that was scientific, generalizable, and policy relevant caught nascent foreign policy analysts unprepared. Where other areas of political science could respond to the challenge presented by the behavioral revolution with numerical data already at their disposal, the traditional fodder for diplomatic analysis – histories, documents, interviews, biographies, and memoirs – were less easily reduced into the sort of data necessary for rigorous, quantitative hypothesis testing.

This reality set foreign policy analysis somewhat behind other areas of political science because it had to overcome two distinct obstacles. First, new data had to be collected that was better suited to statistical analyses. Second, methods had to be developed with which to analyze these data within a behavioral framework. Among others, Rosenau ( 1966 ; 1968 ), McClelland ( 1970 ), and Brecher et al. ( 1969 ) took up these early challenges.

These early foreign policy analysts sought to develop a quantifiable unit of foreign policy interaction. McClelland conceived of this core unit of analysis as the foreign policy “event,” which is simply a formalized observation of a conflictual or cooperative interaction between states. McClelland’s intention was to fill the gap between the traditional narrative approach to foreign policy analysis and empirical techniques that relied upon discrete quantifiable data that could be explored in statistical analyses (Schrodt 1994 ). In effect, the foreign policy event takes a qualitative observation of foreign policy interaction and reduces it to a numerical or categorical form suited for statistical analysis.

The process of generating events data was and is time-consuming and costly. It is most commonly accomplished through the content analysis of thousands of newspaper reports on the interactions among nations in light of a previously defined set of criteria or codebook. Each observation uncovered in this way is then assigned some numerical score or a categorical code, which can then be analyzed quantitatively (Schrodt 1994 ). This potentially lengthy process requires that the researcher accomplish some or all of the following: identify sources, identify a period of analysis, create or borrow a coding scheme, train coders, generate the data, and check for reliability.

Foreign policy scholars have generated a significant number of important events datasets that remain central to quantitative methods of foreign policy analysis. The best of them are impressive collections offering decades-long periods of analysis, coverage of many countries (if not the entire international system) and standards of intercoder reliability well in excess of 80 percent (Burgess and Lawton 1972 ). The paragraphs that follow describe a subset of the available data. Particular attention is given to projects that were seminal to the methodological development of the field and those that generated datasets still widely used by contemporary scholars.

The World Event/Interaction Survey (WEIS)

The World Event/Interaction Survey Project began at the University of Southern California under the direction of Charles McClelland as a research project on the characteristics and processes of the international system (McClelland and Hoggard 1969 ). The initial WEIS dataset records the flow of action and response between countries (as well as non-governmental actors such as NATO and the United Nations) captured from a daily content analysis of the New York Times from January 1966 through December 1978 . This reliance on the New York Times produces a well-known bias toward western perspectives, which was acknowledged from the outset by McClelland and his co-authors. However, they argued that by using a single source they were better able to remove the “noise” surrounding observations. Furthermore, the inclusion of non-state actors raises important methodological issues with regard to the basic unit of analysis. This question has taken on increased salience with the rise in concern about terrorist activities by non-state international entities.

The basic unit of analysis in the dataset is the interaction, which is simply a verbal or physical exchange between nations ranging from agreements to threats to military force. Each of these observations is coded to identify the actors, target, date, action category, and arena. The WEIS databank also provides brief qualitative textual descriptions of each event. These narratives provide context, which facilitates the process of identifying and understanding outliers and applying statistical findings back to political reality – both important for successful events analysis. The initial WEIS effort has been continuously updated and is presently current through 1993 (Tomlinson 1993 ). Other projects, such as the Kansas Event Data System, have applied WEIS coding rules to new research.

WEIS data has been widely used in the FPA literature, both by McClelland and his students and by outsiders who took advantage of these public domain data to test their own questions. The applications are diverse, underlining the versatility of well-designed events datasets. Several early examples are noted by Rummel ( 1979 ): Tanter ( 1974 ) used these data to understand the dynamics of the two major Berlin crises of the Cold War ( 1948–1949 and 1961 ); Kegley et al. ( 1974 ) explored patterns of international conflict and cooperation; while many others began the ongoing process of understanding the relationships among key contextual variables such as relative development, size, and political system, on international conflict, cooperation, and systemic stability (Rosenau 1974 ). Applications continue to this day. For example, Reuveny and Kang ( 1996b ) utilized WEIS data in their exploration of causality in the relationship between international trade and conflict.

The Conflict and Peace Data Bank (COPDAB)

The COPDAB project was designed by Azar and colleagues (Azar 1980 ; 1982 ; Azar et al. 1972 ) as a longitudinal dataset of international and domestic events developed through content analysis of daily newspapers. In an advance over WEIS methods, COPDAB data is drawn from a wide variety of international and regional media outlets, thereby avoiding some potential bias issues.

COPDAB coders scored each event on a 16-point ordinal scale ranging from cooperative interactions to full-scale violence. The resulting dataset covers the interactions of 135 countries from 1948 to 1978 and can be analyzed at levels of aggregation ranging from the day to the year. Each record includes nine variables: date of event, actor initiating the event, target of the event, issue area(s), contextual information about the incident, and the source of the information about the event. The COPDAB dataset is particularly useful for those interested in the interactions between interstate and civil conflict and cooperation, as complementary datasets exist for both international and domestic events.

While the WEIS and COPDAB datasets are clearly conceptually related, scholars have disagreed about their compatibility (Howell 1983 ; Vincent 1983 ; Goldstein and Freeman 1990 ). The underlying definitions of conflict and cooperation are quite similar; however, coding differences introduce the potential for inconsistencies. Reuveny and Kang ( 1996a ) explore this issue in detail with a series of statistical tests and time-series analyses. They argue that COPDAB and WEIS are indeed compatible for the overlapping period between 1966 and 1978 . Building on this logic, they combine the WEIS and COPDAB series to create a larger events dataset covering the period from 1948 to 1993 that is potentially useful for scholars interested in working with a longer period of analysis.

International Crisis Behavior Project (ICB)

Although the final two projects outlined here (the International Crisis Behavior project and the Correlates of War [COW] project) are often excluded from discussions of foreign policy analysis, they are clearly a continuation of events research and are among the most frequently updated and widely used events datasets. The distinctive feature of the ICB and COW datasets is that they primarily focus on international conflict and therefore lack the range of conflictual and cooperative events that characterize the data projects discussed to this point. Researchers should note, however, that the ICB project does provide some indirect data on cooperation.

Michael Brecher and Jonathan Wilkenfeld launched the International Crisis Behavior project in 1975 with the goal of creating a comparative resource for those studying the concept of “international crisis.” There are two defining conditions for a crisis, which are built on work done by Azar (of the COPDAB project): “(1) a change in type and/or an increase in intensity of disruptive, that is, hostile verbal or physical, interactions between two or more states, with a heightened probability of military hostilities; that, in turn, (2) destabilizes their relationship and challenges the structure of an international system – global, dominant, or subsystem” (Brecher and Ben-Yehuda 1985 ).

The ICB project is congruent with many of the core concepts in FPA – for example, in the operationalization of key elements of decision maker perception. This is perhaps unsurprising, as many of the ICB’s primary researchers are steeped in the FPA tradition. To take one example, Michael Brecher’s ( 1974 ) book on Israeli foreign policy decisions, which pre-dates his work on the ICB project, is often cited as a seminal contribution to FPA that seeks to characterize a nation’s psychological and cultural environment as an access point to an understanding of its foreign policy.

As of January 2009 , the core systemic dataset that results from this definition codes 452 incidents from the end of World War I through 2006 (version 9.0). Each crisis is coded for a number of variables, ranging from characterizations of decision maker perception to operationalizations of structural and environmental factors as well as crisis characteristics and outcomes.

The ICB project is unusual in that it proceeds simultaneously at multiple complementary levels. There are independent actor and system level datasets that allow the researcher to explore distinctions between systemic and national level explanations for crisis emergence and behavior. In addition, the project provides qualitative data in the form of a brief narrative description of each crisis, 9 in-depth volumes comprising 15 in-depth case studies; and 14 other unpublished studies. These serve as an aid to the researcher interested in adding additional nuance to statistical findings generated from quantitative analysis.

Correlates of War Project (COW)

Like the ICB project, the COW project does not attempt to capture multiple tiers of conflict and cooperation, but rather focuses on conflict. Two definitions were developed by the COW project in the 1980s, namely, a “militarized interstate dispute” (MID), and a “militarized interstate crisis” (MIC). The former is defined as: “[A] set of interactions between or among states involving threats to use military force, displays of military force, or actual uses of military force […] these acts must be explicit, overt, non-accidental, and government sanctioned” (Gochman and Maoz 1984 ). This “evolves into a militarized interstate crisis when a member of the interstate system on each side of the dispute indicates by its actions its willingness to go to war to defend its interests or to obtain its objectives.” These are steps two and three along a four-step ladder of growing belligerence, beginning with an “interstate dispute” and culminating in an “interstate war” (Leng and Singer 1988 ).

The majority of scholars currently working with COW events data use the MID dataset. The current version of the dataset contains 2331 militarized disputes from 1816 to 2001 coded for duration, outcome, and level of fatality. In addition, there are several complementary datasets on various metrics of international interaction (ranging from alliance to power to geography) that are associated with the broader COW project and can be easily mapped onto the MID dataset. This body of quantitative data is perhaps the most widely used at the present time – particularly among scholars interested in conflict.

Trade Dependence and Foreign Policy Compliance

An important strand of the behavioral work of the 1970s and 1980s addressed the relationship between trade dependence and foreign policy compliance. While this was far from the only research question to draw on quantitative data, the methodological challenges that confronted it were representative of those faced by quantitative FPA in general and are therefore worthy of some attention. Several scholars working in this area (e.g., Richardson and Kegley 1980 ; Moon 1983 ; 1985 ) argued that relatively smaller and weaker states adopt the foreign policies of their dominant trading partners. Thus, economic dependence severely constrains the independent decision making of leaders in states that are economically reliant on larger patrons. However, consensus on this conclusion was elusive, in large part because of how difficult it is to measure the two key concepts – dependence and compliance. The inevitable result was that discussion of the relationship became bogged down in issues of definition and operationalization. This is symptomatic of a larger issue in the quantitative study of foreign policy. Because the operationalization of the amorphous concepts in foreign policy necessitates discretion from the researcher, it is easy to critique the underlying assumptions that gave rise to the data, not to mention the model. Furthermore, if more than one scholar takes on a question in FPA, they typically settle on different operationalizations of the same underlying phenomenon. A high profile example of this trend can be found in the proliferation of events datasets on conflict and cooperation that has already been discussed. The unfortunate result is that many studies are not comparable or cumulative to the degree we find in the hard sciences.

Events Data – Methodological Challenges

Events data analysis poses a number of methodological challenges that should be taken into account by those analyzing foreign policy. The first of these issues relates to the very core of the events data endeavor – that is, the idea that foreign policy incidents can be reduced to a single quantifiable value. Despite the best efforts of the designers of the data projects described here, it remains difficult to effectively accomplish a cardinal or even ordinal ranking of disparate foreign policy events. However, many of the statistical approaches widely used in political science require cardinal level data, or at least data spaced at even thresholds. As a result, those seeking to generate statistical models of events data need to be particularly careful to apply methods that rely upon defensible assumptions about the nature of the underlying data.

Researchers should also be aware of methodological issues that may arise from the relative sparseness of positive observations in events data. The degree to which this is a problem depends on the type of model and the level of aggregation that is used, but if one considers the daily probability of a foreign policy event it is apparent that null observations would dominate the dataset. King and Zeng ( 1999 ; 2001 ) demonstrate that bias and inappropriately inflated statistical significance may arise in models of zero-inflated data. This is particularly problematic in instances where these null data contain no real information. There are several potential solutions to this problem should it arise. Tomz, King, and Zeng ( 1999 ) suggest a rare events correction for logistic analysis, which they have made available as an addition to the widely used STATA software. A less sophisticated check for rare events bias is to simply drop a random subset of null observations in order to confirm that findings derived from the remaining sample are consistent with the original result.

The non-independence of foreign policy events presents an additional methodological challenge. Non-independence simply means that positive foreign policy interactions tend to contribute to future positive interactions, while negative events are associated with subsequent negative events. At first appearance this might seem obvious, but this reality undercuts an assumption of independence that underpins most statistical models used in quantitative foreign policy analysis. Beck, Katz, and Tucker ( 1998 ) did much of the work that brought this problem to the attention of the discipline and they suggest a solution that entails generating a natural cubic spline with knots at the first and second derivative.

FPA scholars working with events data should also guard against selection bias (sometimes referred to as selection effects) when designing research, as inattention to this methodological challenge can significantly skew findings from both quantitative and qualitative tests. Selection bias typically arises from pre- or post-sampling that preferentially includes or excludes a particular type of observation from the sample that is subsequently used in testing. This is particularly easy to do when working with data on foreign policy because it is relatively easy to identify events, but difficult to tease out non-events. The trouble is that without an accurate characterization of non-events it is impossible to say anything about the causes or incidence of the events. To take one prominent recent example of this methodological challenge, Robert Pape’s recent work on the causes of suicide terror ( 2005 ) has come under fire for “sampling on the dependent variable” (Ashworth et al. 2008 ). Because Pape limits his sample to incidents of suicide terror, he effectively leaves out the instances in which such attacks did not occur. As a result, his research design prevents him from effectively speaking of when suicide terror does and does not occur.

Beyond issues related to the application of statistical methods to events data, there is an additional conceptual concern regarding the unit of analysis that should command attention from foreign policy researchers. Because FPA concerns the foreign policy of states, but sees this policy as emerging from the actions of individuals, traditional units of analysis are blurred. The foreign policy event is the result of the interaction and interplay between leaders, organizations, institutions, and states; however, many of the microfoundational theories that underpin the FPA endeavor are cast at the level of the individual decision maker. As a result, events analysis brings with it the nascent challenge of explaining how individual actions aggregate to the foreign policy actions of states. To put the issue more succinctly, while FPA theories distinguish themselves from mainline international relations by opening the black box of the state, the empirical data collected by scholars interested in events analysis typically returned to the state as the central unit of analysis.

There are also very practical concerns to bear in mind – simple tasks related to data manipulation remain some of the primary challenges confronting researchers interested in working with events data. It can be a nontrivial task to gather and combine data on foreign policy events with the various explanatory and control variables that are required for regression analysis. Researchers confronted with these difficulties should be aware of the EUGene software developed by Scott Bennett and Allan Stam ( 2000 ). EUGene is a basic data management tool that simplifies quantitative analysis of foreign policy interactions. The software offers several advantages. First, it allows for relatively easy transition between commonly used units of analysis – country–year, dyad–year, and directed dyad–year. Second, the software is capable of easily combining many of the events datasets discussed here with basic demographic and geopolitical data including data uploaded by the user.

Finally, there is the issue of collecting new events data. The substantial early investments in projects like WEIS and COPDAB were made at the high point of governmental and institutional enthusiasm for events datasets – both datasets were products of the National Science Foundation’s well funded Data Development for International Research (DDIR) project. However, DDIR funding and government and private support for events data collection projects in general declined markedly by the mid-1990s. While this decline had many causes, it was in part brought on by the difficulties that comparative foreign policy had delivering on its early promise. It proved far more challenging than expected to build policy relevant quantitative models with predictive capacity. The relative decline in interest on the part of traditional funding sources raises the issue of how new events data might be generated. Computer coding of electronically stored sources, which will be discussed in greater detail later in this essay, has emerged as one way to address this dilemma.

Qualitative Methods of Foreign Policy Analysis

The behavioral revolution and Cold War politics proved fertile ground for the emergence of FPA. However, the first major challenge for the young field also stemmed from this dual heritage. The problem was that these divergent intellectual pedigrees gave rise to methodological requirements that were at times mutually exclusive – on the one hand, an imperative from behavioralism for generalizability, and, on the other, a low tolerance for error on the part of Cold Warriors seeking to immediately inform policy with scientific findings. The emerging recognition of this tension and the seemingly unavoidable high error terms in quantitative models of foreign policy brought an end to the exuberance among academics and the US government for quantitative, events-driven foreign policy research. Policy makers backed away from direct involvement in the FPA endeavor, while academics tempered their commitment to events data and quantitative methods. What emerged was a second generation of FPA methodology, one that largely abandoned universalized theory-building in favor of historical methods and qualitative analysis (Neack et al. 1995 ).

The primary weapon in the arsenal of second-generation FPA researchers is the case study. However, this transition should not be viewed as a complete departure from that which came before it. Many of these scholars place particular emphasis on developing case study methodologies driven by social science principles, with the explicit goal of building techniques that provide intellectual rigor comparable to that of quantitative approaches. The result has been a robust discussion of the role and execution of qualitative methods.

It is admittedly artificial to divide methods of foreign policy analysis by “generation,” as this implies clean transitions that are in reality far more blurred. While the concept of generational change is useful for understanding the broad developments in the field, the reader should be aware that there are many exceptions to the general rule. Alongside second generation case studies were a wide range of quantitative approaches that, while often abandoning the drive toward universalized theory that characterized previous work in comparative foreign policy, stressed both the outputs and the outcomes of foreign policy processes and actions. Similarly, careful qualitative analysis of foreign-policy decision making has always been an element of foreign policy analysis, and therefore cannot only be considered to have followed sequentially on the quantitative work done in comparative foreign policy (although it did take on renewed prominence).

Case Study Analysis

There is no shortage of examples of the excellent use of case study methodology in foreign policy analysis. Graham Allison’s Essence of Decision ( 1971 ) is often cited as a seminal piece of research in this area with an innovative methodological approach. While Allison’s volume is concerned with a single incident – the Cuban missile crisis – the book is not a single case study, but rather three. Allison explored the US decision making process in the context of three competing explanatory theories: a rational actor model, an organizational process model, and a government politics model. Each of these three explanatory models receives independent analysis in a separate section of the book. Allison ( 1971 :258) argues that these three models combine to provide a clear understanding of decision making in the context of the Cuban Missile Crisis: “Model I fixes the broader context, the larger national patterns, and the shared images. Within this context, Model II illuminates the organizational routines that produce the information, options, and action. Model III focuses in greater detail on the individuals who constitute a government and the politics and procedures by which their competing perceptions and preferences are combined.”

Another important strand of qualitative foreign policy analysis draws on work from political psychology to theoretically inform case study analysis of the foreign policy decision making process. These efforts began with “operational code analysis,” which involves determining how decision makers’ core beliefs shape their foreign policy reactions (George 1969 ; Holsti 1970 ). Operational codes include decision makers’ beliefs about the likelihood of violence, their ability to shape or prevent it, as well as leadership strategies and styles.

Robert Axelrod applies a related technique, termed cognitive mapping, to understand the influence of leadership beliefs on foreign policy interactions. Cognitive mapping entails defining a decision maker’s stated goals and then determining the causal linkages between these goals as a way of predicting likely behavior. Several applications of this technique can be found in an edited volume titled Structure of Decision (Axelrod 1976 ). A more recent application of cognitive mapping can be found in Johnston’s ( 1995 ) work on Chinese–American relations.

This early work developed into a substantial body of foreign policy analysis based more broadly on the psychology of decision makers, a method that figures prominently in analyses conducted at the individual level. Larson ( 1985 ) is a leading example of this sort of scholarship. In her book, Origins of Containment , she traces the path of Cold War politics in the context of the cognitive psychology of American policy makers.

A great deal of work has been done in recent years to improve and formalize case study methodology. One such volume, King, Keohane, and Verba’s Designing Social Inquiry ( 1994 ), has been influential (and controversial) enough that it is often referred to simply by the initials of its authors – KKV. King, Keohane, and Verba draw on their diverse methodological backgrounds to argue that the core logic of causal inference and control should apply as much to qualitative work as it does to quantitative research. They suggest that, by applying the logic of statistics, it is possible to produce theoretically robust and generalizable results while increasing certainty in the validity of qualitative findings.

Bennett and George’s ( 2005 ) more recent work on case study analysis has also emerged as an important contribution to the development of robust qualitative methods. This book lays out methods for designing case studies that are maximally useful for the formulation of policy, which remains a fundamental goal of foreign policy analysis. Bennett and George suggest greater emphasis on within-case analysis, process tracing, and theory building. While these suggestions differ markedly from those of KKV, the underlying goal is quite similar – to create scientific case studies from which lessons can be systematically drawn. In this sense, both volumes speak convincingly to the aforementioned tension between nuance and generalizability that plagues methods of foreign policy analysis.

This issue of generalizability has developed into the core methodological challenge surrounding case study analysis both in foreign policy analysis and in political science more generally. While systematic knowledge of foreign policy interactions does not necessarily require the numerical comparability that comes with quantitative research, some degree of generalizability remains important to the independent identity of foreign policy analysis, as it is this forward-looking element that separates the sub-discipline from diplomatic history. However, comparisons across cases are difficult for two reasons. First, case studies require such a depth of knowledge and investment of time that it is unusual for a scholar to accomplish more than a handful of them on any given question, though there are important exceptions (e.g., Brecher 2008 ). Second, the comparatively loose structure of case studies can hinder comparison, as many analyses fail to address the same subjects on the same terms. One way that these challenges can be overcome is through collaboration within a consistent framework.

An example of such collaboration can be found in a relatively recent volume edited by Beasley et al. ( 2002 ). The volume brings together qualitative work from 15 independent researchers systematically exploring the foreign policies of 13 states. Through the coordinating efforts of the editor, the volume maintains a degree of comparability across the cases while drawing on the deep knowledge of the individual contributors. As a result, the reader is able to engage in comparative analysis within a coherent theoretical framework, allowing for the quick identification of patterns and outliers. There are several examples of similarly structured volumes, and they indicate an important role for collaboration as an approach to boosting the sample sizes of qualitative analyses and thereby the generalizability of findings. The result is “comparative foreign policy,” but of a qualitative variety.

Another interesting solution to the issue of case comparability is found in the qualitative research that has emerged from the qualitative side of the International Crisis Behavior Project, which was already mentioned in the context of events data analysis. These case studies, though they were written over many years and appear in a variety of different outlets, follow a similar format and concern themselves with a consistent set of issues. As a result, they are an explicitly cumulative effort. With each new case study, the body of comparable knowledge increases and this expansion is accompanied by improvement in the robustness of findings.

Gathering Qualitative Data

Those interested in applying case study methodology will need raw material with which to build their analysis. For many questions, considerable ground can be covered using basic library research techniques and secondary sources. However, some of the most fruitful case studies (in terms of their contribution to the existing body of knowledge) bring new information to light. There are several methods of obtaining original qualitative data. The sections that follow will briefly discuss four methods that have become central in foreign policy analysis: archival research, content analysis, interviews, and focus groups.

Archival Research

Original source material can be a crucial element of a quality case study. Typically, scholars uncover such information through archival research. Relevant foreign policy materials are commonly found in the document collections housed in presidential libraries, national archives, and universities. While the basic concept behind archival research is self-explanatory, the actual process of gaining access to collections and navigating their contents can intimidate the neophyte. There are a number of guides to archival methods that can alleviate such anxiety. Directions for identifying and searching appropriate archival sources as well as tips for navigating the archives themselves can be found in Marc Trachtenberg’s ( 2006 ) recent volume on methods – Appendix II will be of particular interest to those seeking to utilize archival methods. Hill ( 1993 ), Larson ( 2001 ), and Lustick ( 1996 ) provide additional detail on the nuances of archival research.

Content Analysis

Content analysis is a hybrid method that has played a longstanding and important role in quantitative and qualitative foreign policy analysis. The section of this essay on events data already discussed the ways in which content has been used to generate quantitative data for statistical analysis. For example, some of the earliest approaches to events data generation coded the content of elite communication (Winham 1969 ). However, more detailed content analyses have also been used to generate the raw material for case studies or other qualitative analyses. Ole Holsti ( 1969 ) was a pioneer of this method, while, more recently, Steve Walker and his students at Arizona State have developed a typology and quantitative content analysis scheme for operational code analysis (Walker et al. 1998 ). Those interested in additional detail on the mechanics of content analysis should consult Weber ( 1985 ), Neuendorf ( 2002 ), and West ( 2001 ).

Because the role of the individual figures so prominently in foreign policy analysis, interviews can be a particularly valuable method for accessing information about the mechanics of the decision making process. Interviews enable FPA scholars to delve deeply into the idiosyncrasies of the foreign policy process, gleaning deep insights from decision makers and those around them. Over time, FPA scholars have developed a robust set of interview methods designed to enable researchers to maximize the acquisition of information without introducing biases into findings.

There are a number of excellent examples of innovative interview methods in foreign policy analysis, which can serve as models for those interested in interview research. Prime among them are FPA classics such as Yuen Foong Khong’s Analogies at War ( 1992 ). Based on a series of interviews with senior officials (and archival research), Khong argues that leaders routinely reference the past when making foreign policy decisions and that this cognitive bias can profoundly alter decision making. Schoutlz ( 1987 ) does similar interview work in the context of US policy toward Latin America. More recently, Silber and Little ( 1995 ) draw on a series of interviews to uncover the foreign policy interactions at play in the collapse of the former Yugoslavia. Berg ( 2001 ), Brenner et al. ( 1985 ), McCracken ( 1988 ), Mishler ( 1986 ), and Seidman ( 1998 ) provide useful, in-depth tutorials on interview methods.

Focus Groups

Focus group research is a derivative of interview methodology in which the researcher attempts to facilitate an organized discussion among the participants. In foreign policy analysis this typically takes the form of a meeting of experts in a particular foreign policy area, or participants in a prior foreign policy decision. Focus group methods can be particularly informative because the emerging consensus that comes from such discussions pools the knowledge of the participating individuals and therefore can overcome some of the potential biases of recollection and self-inflation that accompany individual interviews. However, concerns arise as well, due to some of the very same pathologies that FPA scholars have identified in the context of group decision making. For example, Janis’s ( 1972 ) concept of groupthink can take hold in such settings, with focus group members avoiding controversy and settling instead on a comfortable consensus, even if this consensus is out of step with reality. Along similar lines, the value of elite focus groups can suffer due to deference to higher-ranking participants and domination of the discussion by more talkative individuals who might overshadow important contributions by those less inclined to assert themselves (Krueger 2000 ).

Third Generation Methods of Foreign Policy Analyses

Neack, Hey, and Haney’s ( 1995 ) concept of generational change, to which this review has adhered thus far, captures only part of the methodological richness of FPA. There have long been methods of foreign policy analysis that fall outside this strict quantitative/qualitative divide, and there has been considerable recent growth in these alternative methods. Meanwhile, the distinction between quantitative and qualitative approaches to FPA has become increasingly blurred as the relative advantages of each approach have become more widely recognized. These events auger the arrival of a third generation of FPA scholarship that combines innovative quantitative and qualitative methods, thereby bridging the internal contradictions that split the second wave from the first and unifying a variety of methods of foreign policy analysis.

Several methods of foreign policy analysis are available to aspiring third generation foreign policy analysts seeking to move beyond events data and case studies including: computer assisted coding, experiments, simulation, surveys, network analysis, and prediction markets. The sections that follow will briefly introduce each of these methods, though the list is by no means exhaustive.

Machine Coding

Computer assisted coding of electronically stored information offers several advantages and represents an important methodological innovation that is likely to play an increasingly significant role in the future of foreign policy analysis. First, machine coding can be more reliable than human coding simply because it removes the possibility of individual error and the resulting questions of intercoder reliability. Second, machine coding is extremely rapid. Where earlier events datasets were generated over periods of many years, computers can sift through huge quantities of data in minutes. The result is that machine coding greatly reduces the cost of events data generation – effectively bringing control over such data to the masses (Gerner et al. 1994 ; Schrodt and Gerner, 1994 ). However, the obvious benefits of machine coding are accompanied by two important caveats: the initial programming that creates the coding rules must be accurate, and the raw data must exist in a machine readable format (Gerner et al. 1994 ). Advocates of human coding often counter that the low cost and speed of machine coding are accomplished at the expense of accuracy and nuance.

At present, the best example of a machine coded events project is the Kansas Event Data System (KEDS). This project is among the most active events datasets, due in part to the relatively low cost and speed of generating data in this manner. KEDS provides a computer program that enables users to specify and create personalized events datasets with a variety of output options. The researchers on the KEDS team use this software to code news reports and generate political event data focusing on the Middle East, the Balkans, and West Africa; however, this approach can be extended to other regions or the international system as a whole.

The machine coding community, including members of the KEDS project, is particularly interested in predictive models built on the unique capacities of this technology (Schrodt, 1979 ; 1994 ; Gerner et al. 1994 ; Schrodt and Gerner 2000 ; Shellman forthcoming ). Machine coding not only partially circumvents the need for large financial investments in events data by reducing the required labor and time, but also has the potential to address some of the concerns about the lack of predictive capacity that caused the decline in external funding in the first place. Because machine coding concentrates the researcher’s effort on developing decision rules rather than on the coding itself, once underway these programs can generate empirical data in real time. Such models that draw on continuously updated data effectively serve as early warning systems capable of identifying when political phenomena of interest are likely to occur. For example, Shellman and Stewart use machine coding to predict incidents of forced migration, which they applied with some success in Haiti (Shellman and Stewart 2007 ). This particular application of events data remains at the cutting edge of the FPA literature and will likely continue to be a productive avenue for future research.

Experiments and Simulation in Foreign Policy Analysis

Like all social science, foreign policy analysis struggles methodologically with the issues of control and causality. The quantitative and qualitative methods already discussed took hold in foreign policy analysis in part because the gold standard of the scientific method – experimental control – is typically off limits either for practical or for ethical reasons. However, with careful attention to design and feasibility, there are applications for experimental methods in the study of foreign policy, and where there are not, researchers have begun to turn their attention to simulation, which can achieve some of the same objectives. To take one recent example, Christensen and Redd ( 2004 ) examine how the context of foreign-policy decision making affects choice and assess this relationship in a controlled experiment conducted on undergraduates. They find that, at least in this context, the way in which information is presented directly affects the decision maker’s evaluation.

In recent years the nuts and bolts of experimental methods have drawn increasing attention. Along these lines, McDermott ( 2002 ) provides an interesting discussion of the origins and practice of experimental methods in political science, as well as the unique challenges this approach presents. One such challenge that should be considered carefully by those designing experiments meant to speak to foreign policy behavior is the trade-off between internal and external validity in experiments. Internal validity indicates that the proposed relationship between the independent and dependent variables is the true causal one. When such validity is high it means that extraneous variables and alternative explanations have been ruled out. While typically very difficult to achieve in the social sciences, high internal validity results from proper randomization in an experiment. External validity speaks to the degree to which a proposed relationship is generalizable to a broader set of cases or the world at large. Thus, experimental methods are powerful because they are high in internal validity; however, a leap occurs when we attempt to generalize experimentally derived results to actual political behavior.

This leap can be particularly worrisome when it is from an experimental finding generated from a non-elite individual – for example, an undergraduate student as was the case in the Christensen and Redd study – to a foreign policy decision maker. In such cases, the assumption of external validity may not be reasonable. Mintz, Redd, and Vedlitz ( 2006 ) explore this issue in detail, replicating an experiment on the subject of counterterrorism with a group of college student and a group of military officers. The authors find significant differences between these groups, suggesting that experimental subjects cannot be expected to play the role of foreign-policy decision makers without careful regard for their actual background. However, while these scholars argue that average individuals can tell us very little about the behavior of elites, they do find it more acceptable to use subjects like students as a sample of the public at large.

Simulation, a close relative of experimental methods, has its roots in the longstanding practices of war gaming and diplomatic analysis. However, recent efforts in this area draw extensively on advances in computing power and the internet. Research in this area builds on early work by the Inter-Nation Simulation (INS) project (Guetzkow et al. 1963 ), and slightly later efforts by Hermann ( 1969 ) and Alker and Brunner ( 1969 ).

The International Communication and Negotiation Simulations (ICONS) project is an ongoing extension of this early work that allows political practitioners and students to develop decision making and foreign policy skills through computer aided interactive simulation. Jonathan Wilkenfeld and Richard Brecht developed ICONS in the 1980s, building on Noël’s ( 1969 ) early POLIS simulations. As presently formulated, the ICONS project is more about training than research, but the technique presents an intriguing methodological opportunity for those interested in testing theories of foreign policy interactions in a controlled environment.

Survey Research in Foreign Policy Analysis

When it is focused at the elite level, as it often is, survey research in foreign policy analysis directly relates to the previously discussed interview methods. This stands in some contrast to the way in which survey research is conducted in other areas of political science. For example, in American politics there is a long tradition of survey research designed to pinpoint public opinion on a myriad of topics. In order to accomplish this, researchers are obliged to reach as representative a sample of the population as possible. In contrast, FPA’s focus on elite perception and behavior as a determinant of foreign policy leads to the wider usage of elite interviews.

While surveys lack the depth of an interview, they offer the corresponding advantage of breadth. First, by aggregating information from a more significant number of sources, a survey can minimize some of the idiosyncratic error that can plague interview methodology. Second, in a survey analysis it is easier to control for secondary variables that might influence the recollection or reporting of subjects. Finally, surveys can both contribute to qualitative analysis, and serve to generate high-quality data for aggregate analysis.

Holsti and Rosenau’s ( 1979 ; 1980 ) work on post-Vietnam attitudes is an excellent example of what can be accomplished in foreign policy analysis with elite surveys. Holsti and Rosenau were interested in the degree to which historical experience altered the perceptions and beliefs of opinion leaders and decision makers. Their expectation was that the Vietnam conflict significantly altered the perspective of those who drew their primary experience from that conflict rather than World War II. To answer this question they extensively surveyed groups that they believed to comprise the national leadership structure – military personnel, foreign service officers, business executives, labor leaders, clergy, media, etc. – and found significant differences between occupations and within generations.

Surveys can be particularly valuable when conducted repeatedly over several years, as this allows for longitudinal analysis – something that is crucial if one is interested in changes over time. Both the Chicago Council on Foreign Relations and the Pew Research Center for the People and the Press conduct quadrennial surveys of government, academic, military, religious, and scientific “influentials” in order to measure the content of and changes in elite opinion. These surveys, and others that could be conducted along similar lines, are an underutilized resource for foreign policy analysis. Presser et al. ( 2004 ) and Rea and Parker ( 2005 ) are useful resources for those seeking additional detail on the mechanics of survey research and questionnaire design.

Network Analysis

FPA scholars can also benefit from the recent explosion of interest among political scientists in network analysis. Social network analysis, which is simply the mapping and measuring of relationships among entities in a complex system, is a useful tool for modeling foreign policy relationships because it incorporates both bilateral connections and wider connections among the larger group. Because of this, the technique analysis allows FPA scholars to understand relational data – the contacts, ties, connections, and transfers between decision makers that cannot be cleanly reduced to properties of the leaders themselves (Scott 1991 ). Furthermore, a network theoretic framework consistently captures the role of third parties in foreign policy interactions, which prove to be crucial to understanding outcomes.

Relational approaches have long been an underlying element in the study of foreign policy. For example, Anne-Marie Slaughter ( 2004 ) writes on the relationship between elite networks and international conflict. However, quantitative social network analysis first began to make significant inroads into political science in the 1990s primarily through the study of “policy networks” (Marin and Mayntz 1992 ; Marsh and Rhodes 1992 ), though there are earlier, pioneering examples (e.g., Eulau and Siegel 1981 ; Tichy et al. 1979 ). These studies, as well as later work in international relations (e.g., (Hammarström and Birger 2002 ; Wilkinson 2002 ; Montgomery 2005 ; Heffner-Burton and Montgomery 2006 ; Maoz 2006 ; Ward 2006 ), provide models for future work with foreign policy networks. In short, relational thinking and social network analysis have already contributed to the clarification of a number of puzzles in political science and present a potentially powerful way of approaching foreign policy analysis.

Prediction Markets

Prediction markets are information exchanges built to generate forecasts using a price mechanism. Futures generated from predictions of upcoming events are traded, such that their value is tied to a particular outcome. The result of this arrangement is that the market prices of these futures can be interpreted as the predicted probability of that outcome. There is a significant body of research that establishes the ability of markets to reduce error in predictions. By aggregating the bets of many individuals, these markets effectively use the price setting mechanism to uncover the consensus about a future foreign policy event in much the same way that the stock market predicts the economic performance of a company or oil futures respond to the expected scarcity of that resource. Pennock et al. ( 2001 ) demonstrate that in many cases prediction markets systematically outperform the estimates of even the best individual analysts. There are only a few examples of longstanding prediction markets that handle political futures. These include Intrade, which floats, among many other things, a diverse group of political contracts, and the longer running Iowa Electronic Market, which is an academically oriented project designed for evaluating the probability of election outcomes.

Prediction markets have been applied sparingly in international relations and foreign policy analysis, but have tremendous potential for future application because they offer an interactive mechanism with which individual foreign policy experts can aggregate their knowledge and opinions. Interestingly, given the methodological diversity that characterizes FPA as a sub-discipline, the method by which each expert who trades futures on a prediction market reaches his or her own conclusion is irrelevant. Thus, a prediction market can provide an alternative way to combine and generalize both deep qualitative knowledge and quantitative findings. Furthermore, this approach presents a novel way of dealing with error and uncertainty.

Prospective researchers in this area should note that some early applications of this approach have not gone smoothly. The Defense Advanced Research Projects Agency (DARPA) recently abandoned a promising plan to use a futures market to forecast the probability of important foreign policy events such as regime change and terrorist attacks when the media picked up on the program and it became controversial. Despite a robust literature on the efficacy of such markets, politicians and segments of the public seized upon the effort as being unethical or even nonsensical (Looney 2003 ). The unwanted attention led DARPA, which usually operates well beneath the public radar, to cancel the project almost immediately. It remains an open question whether this approach will become more politically feasible – seemingly a necessity because these markets generally require a significant initial investment, presumably by a government or university. However, private markets such as Intrade, which is a for-profit enterprise, seem to be a plausible alternative. Foreign policy futures, such as the probability of an Israeli attack on Iran, are traded regularly on Intrade and provide useful information about expectations. Moreover, futures on the outcome of the last presidential election vied with polling data for public and media attention in the lead up to the 2008 US presidential election suggesting that familiarity with these markets may be rising.

Remaining Methodological Challenges

Methods of foreign policy analysis have developed markedly over the past few decades, but challenges remain. An unavoidable tension persists between the accuracy needed for policy relevance and the scope needed for generalizability. As the grand theories of foreign policy interaction motivated those who launched the FPA enterprise proved elusive, the discipline increasingly turned to the nuanced examinations of cases. However, if taken too far this trend is a threat to the unique identity of FPA because it blurs the distinction with longstanding traditions of historical analysis. This survey of available methods suggests that a partial solution to this dilemma lies in bringing quantitative analysis and underutilized “third generation” methods back into the FPA fold by reintegrating them into the well-developed qualitative tradition. The goal should be to develop a healthy mix of methods that applies each approach to the questions which each is best equipped to address.

Additional attention should also be given to determining the degree to which current methods of foreign policy analysis allow predictive or prescriptive conclusions. In recent years, enthusiasm for FPA has been fueled in part by the failure of most international relations scholarship to accurately foresee key events in the international system – specifically the decline of the Soviet Union and the end of the Cold War. The argument is made that Cold War politics, because they were in some sense stable or at equilibrium, were better suited to elegant and parsimonious models of the systemic behavior of state actors. In contrast, the more chaotic world we presently inhabit is characterized by fluidity driven by human agents and therefore is best understood using the methods of foreign policy analysis (Hudson and Vore 1995 ). This is a reasonable hypothesis; however, prediction is a difficult game in the social sciences and it remains unclear whether FPA is indeed superior in this arena. In short, with a few notable exceptions such as the KEDS project, methods of foreign policy analysis lack predictive capacity and, when they are able to predict, are often unable to clearly state the degree of certainty surrounding these forecasts. More can and should be done to improve this capacity.

Foreign policy analysts should also give deeper consideration to the issues that accompany the choice of the unit of analysis in their models. FPA derives much of its explanatory power from its ability to speak to the individual’s role in the foreign policy process, but the dependent variables that these efforts attempt to explain are often the interactions between states. The result is a gap in our understanding of the process of aggregation by which the behavior of leaders results in the actions and reactions of states. This aggregation problem is widely noted, but additional work is required to complete our understanding of this element of the foreign policy process. Improvements in this linkage between theory and test, as well as a consistent unit of analysis (individual or foreign policy event) are particularly crucial for robust quantitative analysis, as it is in part the inability of the subfield to resolve this basic issue that stifled earlier research on events data.

Finally, more must be done to reengage foreign policy analysis with the core of international relations research. FPA scholars typically claim the first and second image as their domain, but fail to engage with those in mainline international relations who also work in this area. In the lead essay of the first issue of Foreign Policy Analysis , Valerie Hudson ( 2005 ) convincingly makes the case that FPA has the potential to reshape the entire discipline of international relations by focusing attention on the workings of the fundamental unit of analysis – the political decision maker. However, despite the call to arms, more often than not FPA scholars labor in relative isolation. Some of these divisions emerge from methodological issues and can therefore be resolved.

In sum, the future of foreign policy analysis appears to be bright. There is reason to believe that longstanding methodological battles that characterized it are drawing to a close with the recognition that multiple methods have their place in the study of foreign policy. In addition, new methods and questions are emerging that are likely to contribute to our understanding of the foreign policy process.

• Alker, H.R. , and Brunner, R.D. (1969) Simulating International Conflict – Comparison of 3 Approaches. International Studies Quarterly 13 (1), 70–110.
• Allison, G.T. (1971) The Essence of Decision: Explaining the Cuban Missile Crisis . Boston: Little, Brown.
• Ashworth, S. , Clinton, J.D. , Meirowitz, A. , and Ramsay, K. (2008) Design, Inference, and the Strategic Logic of Suicide Terrorism. American Political Science Review 102 (2), 269–73.
• Axelrod, R. (1976) Structure of Decision: The Cognitive Maps of Political Elites . Princeton: Princeton University Press.
• Azar, E.E. (1980) The Conflict and Peace Data Bank (COPDAB) Project. Journal of Conflict Resolution 24, 143–52.
• Azar, E.E. (1982) The Codebook of the Conflict and Peace Data Bank (COPDAB) . College Park: Center for International Development, University of Maryland.
• Azar, E.E. , Cohen, S.H. , Jukam, T.O. , and McCormic, J.M. (1972) Problem of Source Coverage in Use of International Events Data. International Studies Quarterly 16 (3), 373–88.
• Beasley, R.K. , Kaarbo, J. , Lantis, J.S. , and Snarr, M.T. (2002) Foreign Policy in Comparative Perspective: Domestic and International Influences on State Behavior . Washington, DC: CQ Press.
• Beck, N. , Katz, J.N. , and Tucker, R. (1998) Taking Time Seriously: Time-Series-Cross-Section Analysis with a Binary Dependent Variable. American Journal of Political Science 42 (4), 1260–88.
• Bennett, A. , and George, A.L. (2005) Case Studies and Theory Development in the Social Sciences . Cambridge: MIT Press.
• Bennett, D.S. , and Stam, A. (2000) EUGene: A Conceptual Manual. International Interactions 26, 179–204.
• Berg, B.L. (2001) Qualitative Research Methods for the Social Sciences , 4th edn. Boston: Allyn and Bacon.
• Brecher, M. (1974) Decisions in Israel’s Foreign Policy . London: Oxford University Press.
• Brecher, M. (2008) International Political Earthquakes: Crises and Conflicts Before, During and After the Cold War . Ann Arbor: University of Michigan Press.
• Brecher, M. , and Ben-Yehuda, H. (1985) System and Crisis in International Politics. Review of International Studies 11 (1), 17–36.
• Brecher, M. , Steinberg, B. , and Stein, J. (1969) A Framework for Research on Foreign Policy Behavior. Journal of Conflict Resolution 13 (1), 75–94.
• Brenner, M. , Brown, J. , and Canter, D. (eds.) (1985) The Research Interview . London: Academic Press.
• Burgess, P.M. , and Lawton, R.W. (1972) Indicators of International Behavior: An Assessment of Events Data Research . Beverly Hills: Sage.
• Christensen, E.J. , and Redd, S.B. (2004) Bureaucrats Versus the Ballot Box in Foreign Policy Decision Making: An Experimental Analysis of the Bureaucratic Politics Model and the Poliheuristic Theory. Journal of Conflict Resolution 48 (1), 69–90.
• Eulau, H. , and Siegel, J.W. (1981) Social Network Analysis and Political Behavior: A Feasibility Study. The Western Political Quarterly 34 (4), 499–509.
• George, A.L. (1969) Operational Code – Neglected Approach to Study of Political Leaders and Decision-Making. International Studies Quarterly 13 (2), 190–222.
• Gerner, D.J. , Schrodt, P.A. , Francisco, R.A. , and Weddle, J.L. (1994) Machine Coding of Event Data Using Regional and International Sources. International Studies Quarterly 38 (1), 91–119.
• Gochman, C.S. , and Maoz, Z. (1984) Military Interstate Disputes, 1816–1976: Procedures, Patterns, and Insights. Journal of Conflict Resolution 28 (4), 585–615.
• Goldstein, J.S. , and Freeman, J. (1990) Three Way Street, Strategic Reciprocity in World Politics . Chicago: University of Chicago Press.
• Guetzkow, H. , Alger, C.F. , Brody, R.A. , Noel, R.C. , and Snyder, R. (1963) Simulations in International Relations: Developments for Research and Teaching . Englewood Cliffs: Prentice-Hall.
• Hammarström, M. , and Birger, H. (2002) The Diffusion of Military Intervention: Testing a Network Position Approach. International Interactions 28, 335–77.
• Heffner-Burton, E. , and Montgomery, A.H. (2006) Power Positions: International Organizations, Social Networks, and Conflict. Journal of Conflict Resolution 50 (1), 3–27.
• Hermann, C. (1969) Crises in Foreign Policy: A Simulation Analysis . Indianapolis: Bobbs Merrill.
• Hill, M.R. (1993) Archival Strategies and Techniques . Beverly Hills: Sage Publications.
• Holsti, O.R. (1969) Content Analysis for the Social Sciences and Humanities . Reading: Addison Wesley.
• Holsti, O.R. (1970) The “Operational Code” Approach to the Study of Political Leaders: John Foster Dulles’ Philosophical and Instrumental Beliefs. Canadian Journal of Political Science / Revue Canadienne de Science Politique 3 (1), 123–57.
• Holsti, O.R. , and Rosenau, J.N. (1979) Vietnam, Consensus, and the Belief Systems of American Leaders. World Politics 32 (1), 1–56.
• Holsti, O.R. , and Rosenau, J.N. (1980) Does Where You Stand Depend on When You Were Born? The Impact of Generation on Post-Vietnam Foreign Policy Beliefs. Public Opinion Quarterly 44 (1), 1.
• Howell, L.D. (1983) A Comparative-Study of the WEIS and COPDAB Data Sets. International Studies Quarterly 27 (2), 149–59.
• Hudson, V.M. (2005) Foreign Policy Analysis: Actor-Specific Theory and the Ground of International Relations. Foreign Policy Analysis 1 (1), 1–30.
• Hudson, V.M. , and Vore, C.S. (1995) Foreign Policy Analysis Yesterday, Today, and Tomorrow. International Studies Quarterly 39, 209–38.
• Janis, I.L. (1972) Victims of Groupthink . Boston: Houghton Mifflin Company.
• Johnston, A.I. (1995) Cultural Realism: Strategic Culture and Grand Strategy in Chinese History . Princeton: Princeton University Press.
• Kegley, C., Jr , Salmore, S.A. , and Rosen, D.J. (1974) Convergences in the Measurement of Interstate Behavior. In P.J. McGowan (ed.) Sage International Yearbook of Foreign Policy Studies , vol. 2. Beverly Hills: Sage Publications, pp. 309–39.
• Khong, Y.F. (1992) Analogies at War: Korea, Munich, Dien Bien Phu, and the Vietnam Decisions of 1965 . Princeton: Princeton University Press.
• King, G. , Keohane, R.O. , and Verba, S. (1994) Designing Social Inquiry . Princeton: Princeton University Press.
• King, G. , and Zeng, L.C. (1999) Logistic Regression in Rare Events Data. At http:/gking.harvard.edu/files/0s.pdf , accessed July 2009.
• King, G. , and Zeng, L.C. (2001) Explaining Rare Events in International Relations. International Organization 55 (3), 693–715.
• Krueger, R. (2000) Focus Groups: A Practical Guide for Applied Research , 3rd edn. Thousand Oaks: Sage Publications.
• Larson, D.L. (2001) Sources and Methods in Cold War History: The Need for a New Theory-Based Archival Approach. In C. Elman and M.F. Elman (eds.) Bridges and Boundaries: Historians, Political Scientists, and the Study of International Relations . Cambridge: MIT Press, pp. 327–50.
• Larson, D.W. (1985) Origins of Containment: A Psychological Explanation . Princeton: Princeton University Press.
• Leng, R.J. , and Singer, J.D. (1988) Militarized Interstate Crises – the Bcow Typology and its Applications. International Studies Quarterly 32 (2), 155–73.
• Looney, R.E. (2003) DARPA’s Policy Analysis Market for Intelligence: Outside the Box or Off the Wall? Strategic Insights 2 (9).
• Lustick, I.S. (1996) History, Historiography, and Political Science: Multiple Historical Records and the Problem of Selection Bias. American Political Science Review 90 (3), 605–18.
• Maoz, Z. (2006) Network Polarization, Network Interdependence, and International Conflict, 1816–2002. Journal of Peace Research 43 (4), 391–411.
• Marin, B. , and Mayntz, R. (eds.) (1992) Policy Networks: Empirical Evidence and Theoretical Considerations . Boulder: Westview Press.
• Marsh, D. , and Rhodes, R.A.W. (eds.) (1992) Policy Networks in British Government . New York: Oxford University Press.
• McClelland, C.A. (1970) Some Effects on Theory from the International Event Analysis Movement. Working paper. University of Southern California.
• McClelland, C.A. , and Hoggard, G. (1969) Conflict Patterns in the Interactions Among Nations. In J.N. Rosenau (ed.) International Politics and Foreign Policy . New York: Free Press, pp. 711–24
• McCracken, G. (1988) The Long Interview . London: Sage.
• McDermott, R. (2002) Experimental Methodology in Political Science. Political Analysis 10 (4), 325–42.
• Mintz, A. , Redd, S.B. , and Vedlitz, A. (2006) Can We Generalize from Student Experiments to the Real World in Political Science, Military Affairs, and International Relations? Journal of Conflict Resolution 50 (5), 757–76.
• Mishler, E.G. (1986) Research Interviewing: Context and Narrative . Cambridge: Harvard University Press.
• Montgomery, A.H. (2005) Proliferation Determinism or Pragmatism? How to Dismantle an Atomic Bomb Network. International Security 30 (2), 153–87.
• Moon, B.E. (1983) The Foreign Policy of the Dependent State. International Studies Quarterly 27 (3), 315–40.
• Moon, B.E. (1985) Consensus or Compliance? Foreign-Policy Change and External Dependence. International Organization 39 (2), 297–329.
• Neack, L. , Hey, J.A.K. , and Haney, P.J. (1995) Generational Change in Foreign Policy Analysis. In L. Neack , J.A.K. Hey , and P.J. Haney (eds.) Foreign Policy Analysis. Contiguity and Change in its Second Generation . Englewood Cliffs: Prentice Hall.
• Neuendorf, K.A. (2002) The Content Analysis Guidebook . Beverly Hills: Sage.
• Noël, R.C. (1969) The Polis Laboratory. American Behavioral Scientist 12 (6), 30–35.
• Paige, G. (1968) Korean Decision, June 24–30, 1950 . New York: Free Press.
• Pape, R.A. (2005) Dying to Win: The Strategic Logic of Suicide Terrorism . New York: Random House.
• Pennock, D.A. , Lawrence, S. , Giles, C.L. , and Nielsen, F.A. (2001) The Real Power of Artificial Markets. Science 291 (5506), 987–8.
• Presser, S. , Rothgeb, J.M. , Couper, M.P. , et al. (2004) Methods for Testing and Evaluating Survey Questionnaires . New York: Wiley.
• Rea, L.M. , and Parker, R.A. (2005) Designing and Conducting Survey Research: A Comprehensive Guide . San Francisco: Jossey-Bass.
• Reuveny, R. , and Kang, H. (1996a) International Conflict and Cooperation: Splicing COPDAB and WEIS Series. International Studies Quarterly 40 (2), 281–305.
• Reuveny, R. , and Kang, H. (1996b) International Trade, Political Conflict/Cooperation, and Granger Causality. American Journal of Political Science 40 (3), 943–70.
• Richardson, N.R. , and Kegley, C.W., Jr (1980) Trade Dependence and Foreign Policy Compliance: A Longitudinal Analysis. International Studies Quarterly 24 (2), 191–222.
• Rosenau, J.N. (1966) Pre-Theories and Theories of Foreign Policy. In J. Rosenau (ed.) Scientific Study of Foreign Policy . New York: Free Press, pp. 95–151.
• Rosenau, J.N. (1968) Comparative Foreign Policy – Fad, Fantasy, or Field. International Studies Quarterly 12 (3), 296–329.
• Rosenau, J.N. (ed.) (1974) Comparing Foreign Policies: Theories, Findings, and Methods . New York: Halsted Press.
• Rummel, R.J. (1979) Understanding Conflict and War , vol. 4: War, Power, Peace . Beverly Hills: Sage Publications.
• Schoutlz, L. (1987) National Security and United States Policy Towards Latin America . Princeton: Princeton University Press.
• Schrodt, P.A. (1979) Forecasting in International-Relations – Theory, Methods, Problems, Prospects – Choucri, N., Robinson, T.W. American Political Science Review 73(4), 1208–10.
• Schrodt, P.A. (1994) Event Data in Foreign Policy Analysis. In L. Neack , P.J. Haney , and J.A.K. Hey (eds.) Foreign Policy Analysis: Continuity and Change in its Second Generation . New York: Prentice Hall.
• Schrodt, P.A. , and Gerner, D.J. (1994) Validity Assessment of a Machine-Coded Event Data Set for the Middle-East, 1982–92. American Journal of Political Science 38 (3), 825–54.
• Schrodt, P.A. , and Gerner, D.J. (2000) Cluster-Based Early Warning Indicators for Political Change in the Contemporary Levant. American Political Science Review 94 (4), 803–17.
• Scott, J. (1991) Social Network Analysis: A Handbook . London: Sage.
• Seidman, I. (1998) Interviewing as Qualitative Research: A Guide for Researchers in Education and the Social Sciences , 2nd edn. New York: Teachers College Press.
• Shellman, S.M. (forthcoming) Machine Coding Nonstate Actors’ Behavior in Intrastate Conflict. Political Analysis .
• Shellman, S.M. , and Stewart, B. (2007) Predicting Risk Factors Associated with Forced Migration: An Early Warning Model of Haitian Flight. Civil Wars 9 (2), 174–99.
• Silber, L. , and Little, A. (1995) The Death of Yugoslavia . London: Penguin.
• Slaughter, A.-M. (2004) A New World Order . Princeton: Princeton University Press.
• Snyder, R.C. , Bruck, H. , and Sapin, B.M. (1954) Decision-Making as an Approach to the Study of International Relations . Princeton: Princeton University Press.
• Tanter, R. (1974) Modelling and Managing International Conflicts: The Berlin Crises . Beverly Hills: Sage Publications.
• Tichy, N.M. , Tushman, M.L. , and Fombrun, C. (1979) Social Network Analysis for Organizations. The Academy of Management Review 4 (4), 507–19.
• Tomlinson, R.G. (1993) Monitoring WEIS Event Data in Three Dimensions. In R.L. Merritt , R.G. Muncaster , and D.A. Zinnes (eds.) International Event-Data Developments: DDIR Phase II . Ann Arbor: University of Michigan Press, pp. 55–85.
• Tomz, M. , King, G. , and Zeng, L.C. (1999) RELOGIT: Rare Events Logistic Regression (Version 1.1). Cambridge. At http:/gking.harvard.edu/stats.shtml , accessed July 2009.
• Trachtenberg, M. (2006) The Craft of International History . Princeton: Princeton University Press.
• Vincent, J.E. (1983) Weis Vs Copdab – Correspondence Problems. International Studies Quarterly 27 (2), 161–8.
• Walker, S.G. , Schafer, M. , and Young, M.D. (1998) Systematic Procedures for Operational Code Analysis: Measuring and Modeling Jimmy Carter’s Operational Code. International Studies Quarterly 42 (1), 175–89.
• Ward, H. (2006) International Linkages and Environmental Sustainability: The Effectiveness of the Regime Network. Journal of Peace Research 43 (2), 149–66.
• Weber, R.P. (1985) Basic Content Analysis . Beverly Hills: Sage.
• West, M.D. (2001) Theory, Method, and Practice in Computer Content Analysis . Westport: Ablex.
• Wilkinson, D. (2002) Civilizations as Networks: Trade, War, Diplomacy, and Command-Control. Complexity 8 (1), 82–6.
• Winham, G.R. (1969) Quantitative Methods in Foreign Policy Analysis. Canadian Journal of Political Science 2 (2), 187–99.

Links to Digital Materials

International Crisis Behavior Project. At www.cidcm.umd.edu/icb/ , accessed July 2009. The ICB project provides quantitative and qualitative data on international crises. The core systemic dataset currently contains 452 incidents from the end of World War I through 2006. The link above provides access to ICB data, codebooks, citations, and a variety of other useful materials.

Correlates of War Project. At www.correlatesofwar.org/ , accessed July 2009. The COW project website provides widely used events data on militarized interstate disputes as well as several other datasets that may be appropriate for FPA scholars. Data, documentation, and codebooks are available through the link.

EUGene Software. At www.eugenesoftware.org/ , accessed July 2009. The link provides access to the EUGene software package, as well as manuals and documentation (all free of charge). EUGene allows for relatively easy transition between commonly used units of analysis – country–year, dyad–year, and directed dyad–year. In addition, the software enables the researcher to combine many of the events datasets discussed in this essay with basic demographic and geopolitical data including data uploaded by the user.

Kansas Event Data System. At http:/web.ku.edu/keds/ , accessed July 2009. KEDS provides a computer program that enables users to specify and create personalized events datasets. Detailed descriptions of machine coding methods, as well as several datasets and codebooks, are available on the KEDS web page.

The Consortium on Qualitative Research Methods. At www.maxwell.syr.edu/moynihan/programs/cqrm/index.html , accessed July 2009. CQRM promotes qualitative research methods in the social sciences. They hold an annual training institute that may be useful for FPA scholars interested in expanding their knowledge of qualitative methods. In addition, they maintain a database of syllabae on qualitative methods.

Intrade. At www.intrade.com/ , accessed July 2009. Intrade is the leading prediction market for political futures. The site has prediction markets for a wide range of political and financial outcomes that may be of interest to FPA scholars.

Acknowledgments

I would like to acknowledge the helpful comments of Michael Glosny , Deborah Larson , Rachel Augustine Potter , and two anonymous reviewers. All errors are my own.

Printed from Oxford Research Encyclopedias, International Studies. Under the terms of the licence agreement, an individual user may print out a single article for personal use (for details see Privacy Policy and Legal Notice).

date: 26 March 2024

• Cookie Policy
• Privacy Policy
• Legal Notice
• Accessibility
• [66.249.64.20|185.80.150.64]
• 185.80.150.64

Character limit 500 /500

• Utility Menu

Harvard Ph.D. Program in Health Policy

• Methods for Policy Research

Concentration Chairs:   Mary Beth Landrum  and  Michael McWilliams  (on sabbatical for the 2023-2024 academic year)

Training in the MPR concentration ( formerly called Evaluative Science & Statistics ) will position students to conduct rigorous, policy-relevant research. The concentration includes intensive methodological training that draws from several quantitative fields, including statistics, econometrics, and epidemiology, and also offers opportunities to learn qualitative methods. Students in the MPR concentration will also develop proficiency in experimental and quasi-experimental research designs. The MPR concentration does not require training in a specific discipline of social science, but students are allowed to use two of their distributional requirements to acquire some disciplinary training (e.g., in economics or sociology), and many students pursue additional coursework in a substantive area of interest or advanced levels of statistics. By virtue of this training, MPR students develop a multi-disciplinary toolkit that borrows from the strengths of different quantitative sciences and supports novel applications to health-related questions. Graduates are able to communicate and collaborate effectively with statisticians, economists, epidemiologists, and clinician-investigators, as they lead their independent work.  The concentration’s broad methodological training flexibly supports scholarship on a wide range of topics, including common areas of interest in health services research related to health care access, quality, costs, and disparities, but also questions focused on clinical decision-making, behavior (e.g., of patients, physicians, or organizations), social determinants of health, and social programs with implications for health.

Methods for Policy Research Curriculum Guide

• Curriculum Overview
• Decision Sciences
• Political Analysis
• Core Course in Health Policy
• Policy Areas
• Joint Degrees

• Find Books / E-Books / Theses
• Find Journal Articles
• Grey Literature & Google
• Government Information
• Statistics & Data This link opens in a new window
• Legislation & Court Decisions
• Policy Analysis & Research Methods
• Writing & Citing Public Policy
• Managing References
• Talk With An Expert

Public Administration and Public Policy: Policy Analysis & Research Methods

Finding methods books in the catalog.

In the University Library Catalogue, you can search for a variety of research methodology sources including general sources, sources specific to Public Administration, Policy Sciences, other disciplines or sources about a specific research method.

Sample Subject Headings:

• Evaluation Methodology
• Policy Sciences - Methodology
• Public Administration - Handbooks, Manuals
• Qualitative research

Sample keyword searches by specific research method:

• Grounded Theory
• Narrative Inquiry
• Discourse Analysis
• Mixed Methods

Research Logs and Literature Reviews

• Assignment Planner The University Library has a Research Paper Planner that provides some guidelines regarding the amount of time to allocate to each step in the research process. A liaison librarian can also provide some Research Advice if you run into difficulties.
• Sample Research Log Tarleton University also has their own sample research log (PDF)

Capella University has developed some excellent resources that provide samples that you can download and adapt for your own research.

• Database Research Log (PDF)
• Keeping Track of Resources Guide
• Scholarly Research Log The advantage of an excel format log is that you can sort by various columns to 'collect' your research in a variety of ways.

Research logs help researchers formulate searches, modify searches, choose the best search tools, plan and organize research time, document resources, and retrace steps when needed. They can also be used to take notes on what worked well, what didn't work well, and also any surprising or exciting discoveries.

The University of Maryland University College has a pair of good introductory videos on literature reviews:

DoctoralNet has provided a brief video on the best way to organize a literature review:

Policy Analysis Titles

• << Previous: Legislation & Court Decisions
• Next: Writing & Citing Public Policy >>
• Library A to Z
• Follow on Facebook
• Follow on Twitter
• Follow on YouTube
• Follow on Instagram

The University of Saskatchewan's main campus is situated on  Treaty 6 Territory and the Homeland of the Métis.

© University of Saskatchewan Disclaimer | Privacy

• Last Updated: Jan 8, 2024 4:49 PM
• URL: https://libguides.usask.ca/public_policy

• Open access
• Published: 21 March 2024

A rapid review to inform the policy and practice for the implementation of chronic disease prevention and management programs for Aboriginal and Torres Strait Islander people in primary care

• Uday Narayan Yadav   ORCID: orcid.org/0000-0002-6626-1604 1 , 2   na1 ,
• Jasmine Meredith Davis 3 ,
• Keziah Bennett-Brook 4 ,
• Julieann Coombes 4 ,
• Rosemary Wyber 1 , 5   na1 &
• Odette Pearson 6 , 7  

Health Research Policy and Systems volume  22 , Article number:  34 ( 2024 ) Cite this article

224 Accesses

8 Altmetric

Metrics details

More than 35% of Aboriginal and Torres Strait Islander adults live with cardiovascular disease, diabetes, or chronic kidney disease. There is a pressing need for chronic disease prevention and management among Aboriginal and Torres Strait Islander people in Australia. Therefore, this review aimed to synthesise a decade of contemporary evidence to understand the barriers and enablers of chronic disease prevention and management for Aboriginal and Torres Strait Islander People with a view to developing policy and practice recommendations.

We systematically searched for peer-reviewed published articles between January 2014 to March 2023 where the search was performed using subject headings and keywords related to “Aboriginal and Torres Strait Islander peoples,” “Chronic Disease,” and “Primary Health Care”. Quality assessment for all included studies was conducted using the Aboriginal and Torres Strait Islander Quality Appraisal Tool. The data were extracted and summarised using a conventional content analysis approach and applying strength-based approaches.

Database searches identified 1653 articles where 26 met inclusion criteria. Studies varied in quality, primarily reporting on 14 criteria of the Aboriginal and Torres Strait Islander Quality Appraisal Tool. We identified six key domains of enablers and barriers of chronic disease prevention and management programs and implied a range of policy and practice options for improvement. These include culturally acceptable and safe services, patient-provider partnerships, chronic disease workforce, primary health care service attributes, clinical care pathways, and accessibility to primary health care services. This review also identified the need to address social and cultural determinants of health, develop the Aboriginal and Torres Strait Islander and non-Indigenous chronic disease workforce, support multidisciplinary teams through strengthening clinical care pathways, and engage Aboriginal and Torres Strait Islander communities in chronic disease prevention and management program design and delivery.

Enabling place-based partnerships to develop contextual evidence-guided strategies that align with community priorities and aspirations, with the provision of funding mechanisms and models of care through policy and practice reforms will strengthen the chronic disease prevention and management program for Aboriginal and Torres Strait Islander people.

Peer Review reports

Introduction

Aboriginal and Torres Strait Islander people have continuously demonstrated strength, tenacity, and resilience in the face of a high burden of chronic disease associated with profound health, social, economic and cultural and wellbeing impacts. The disproportionate burden of chronic disease—particularly cardiovascular disease (CVD), type II diabetes and chronic kidney disease (CKD)—is driven by the effects of colonisation. These effects include intergenerational trauma, racism and commercial determinants such as the introduction of ultra-processed foods, tobacco, sugar-sweetened beverages, and alcohol [ 1 , 2 ].

More than 35% of Aboriginal and Torres Strait Islander adults report having CVD, diabetes or CKD, 38% have two of these conditions and 11% have all three [ 3 ]. Nearly 70% of Australia’s burden of disease from CVD is attributable to modifiable risk factors including high blood pressure, dietary risks, high body weight, high cholesterol and smoking [ 4 ]. These risk factors, and subsequent disease, can be prevented. This prevention can be primordial (by addressing the structural drivers at a population level, such as food supply and recreation facilities) or primary (through identifying people at risk and taking individual steps to reduce that risk).

Primary prevention and management of chronic disease among Aboriginal and Torres Strait Islander people in Australia predominantly occurs in primary care settings. Universal primary care in Australia is largely funded through a fee-for-service model via the Medicare Benefits Scheme (MBS), with some augmentation for Aboriginal and Torres Strait Islander people through the Indigenous Australian’ Health Programme [ 5 ]. Primary care is delivered by a range of providers, including Aboriginal Community Controlled Health Organisations (ACCHOs) which are governed by a local board, Aboriginal Medical Services run by State and Territory governments, and private primary care services (sometimes referred to as ‘mainstream’ providers). Chronic disease services in primary care include risk assessment, support for healthy behaviours, referrals to allied health and pharmacology for risk reduction. High quality, culturally safe primary care can prevent the development of disease, and manage complications, for individuals and communities [ 6 , 7 , 8 ]. Strengthening primary care is a key strategic priority of the Australian government, [ 9 , 10 ] peak bodies and other stakeholders working to improve the health and well-being of Aboriginal and Torres Strait Islander communities [ 11 , 12 ].

Understanding how primary care services, prevention and management programs can best meet the needs of Aboriginal and Torres Strait Islander people, who are living with or at risk of chronic disease, is critical to addressing disparity in outcomes. A review conducted by Gibson et al. in 2015 explored chronic disease care for Indigenous communities globally, but there has been no comprehensive contextual evidence available focusing on Aboriginal and Torres Strait Islander people [ 13 ]. Therefore, there is a need for local evidence disease on chronic prevention and management for Aboriginal and Torres Strait Islander people in Australia. The Aboriginal and Torres Strait Islander People’s Health Assessment (MBS item number 715) is an annual health check funded for Aboriginal and Torres Strait Islander people and a cornerstone of early detection for chronic disease [ 14 ]. The number of Health Assessments fell for the first time in 2020 and 2021 after years of sustained growth [ 15 ]. The reduction in Health Assessments is likely to reflect disruption to routine primary care services during the COVID-19 pandemic [ 16 ].

Our team were contracted by the Australian Commonwealth Department of Health and Aged Care to understand best practice delivery of chronic disease care for Aboriginal and Torres Strait Islander people in 2020. The effects of the COVID-19 pandemic accelerated the need for this research and necessitated a pivot to rapid review strategy. The Oxford Centre for Evidence-Based Medicine highlights [ 17 ] that rapid review methodology can be used to meet the needs of commissioning bodies in policy-relevant timeframes. Therefore, this rapid review aims to synthesise contemporary and contextually relevant barriers and enablers for chronic disease care in primary care settings for Aboriginal and Torres Strait Islander people with a view to developing near term policy and practice recommendations.

This review applied the SelecTing Approaches for Rapid Reviews (STARR) decision tool that includes interaction with commissioners, scoping the literature, selecting approaches to literature search, methods for data extraction and evidence synthesis [ 18 ]. It was conducted based on the Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) guideline [ 19 ]. We searched two databases (Medline and Web of Science) to identify relevant studies from January 2014 to March 2023. These two databases were chosen because of following reasons: (i) Medline database provides access to articles from 5,200 journals in about 40 languages covering the biomedical and life sciences including Aboriginal and Torres Strait Islander peoples as a MeSH Headings and (ii) Web of Science is the world’s oldest interdisciplinary and widely used database of research publications that covers over 34,000 journals today.

This timeframe follows on from a systematic review conducted by Gibson et al. [ 13 ] which included data up to December 2013. This rapid review mirrors the Gibson methodology with two main changes: i) narrowing focus from Indigenous communities globally to Aboriginal and Torres Strait Islander contexts in Australia, ii) narrowing focus from all chronic diseases to focus on the three major contributors to chronic disease burden (cardiovascular disease, diabetes and chronic kidney disease). The review was performed between January to August 2023.

In this review, Aboriginal and Torres Strait Islander researchers and Thiitu Tharrmay Aboriginal and Torres Strait Islander Reference Group members rather than Thiitu Tharrmay Aboriginal Reference Group members were actively engaged and consulted in all steps, from the inception of the research questions to the completion of this review. Throughout the process, the research team met monthly with the Australian Commonwealth Department of Health and Aged Care during the period of this review to inform the scope of evidence synthesis.

Positionality: The senior author of this review Pearson (nee Gibson, first author of the 2015 review), is Kuku Yalnji/Torres Strait Islander health systems researcher. KB-B ( Torres Strait Islander ) and JC ( Gumbaynggir ) are experienced researchers with particular expertise in quality assessment of Aboriginal and Torres Strait Islander research studies. UNY ( Madhesi, Nepal ) is an implementation scientist, JD is a non-Indigenous medical student and RW is a non-Indigenous practising general practitioner and researcher.

Operational definition of key terminology used in this review:

Holistic care or support: The care process that involved strategies to support mental, physical, cultural and spiritual health which is beyond the individual level that values family and community capacity and governance [ 20 ].

Systems thinking: Systems thinking is c onceptual  orientation concerned with inter-relationships between different levels, institutions, systems, and people nested within social, cultural, economic, political contexts to deliver a holistic care [ 21 ].

Place-based partnerships: Place-based partnerships involve collaborative arrangements the unique needs and circumstances of both the community and service provider. In this context, place-based partnerships involve a formal partnership among government, service providers and First Nations representatives. These partnerships are specific to geographical locations and population groups and are aimed at designing or delivering services that directly respond to community needs, aspirations and local priorities, while valuing local cultural values [ 22 , 23 ].

Scoping the literature:

The search was performed using a combination of subject headings and keywords related to “Aboriginal and Torres Islander peoples,” “Chronic Disease,” and “Primary Health Care” using “OR” and “AND” iterated from the search strategy described in a study by Gibson et al. [ 13 ] The search strategy has been provided in Box 1. The search results obtained from two databases were imported to Endnote and uploaded on the Covidence platform for title and abstract and full-text screening [ 24 ]. Three reviewers (UNY, JMD and RW) independently screened the titles and abstracts of potential studies for eligibility based on inclusion and exclusion criteria (Table  1 ). Full-text articles were assessed by two reviewers and any disagreement that appeared during the screening process was resolved through discussion with the third reviewer (RW). The details of the screening process are documented as a PRISMA flow diagram (Fig.  1 ).

PRISMA 2020 flow diagram that included searches of databases and included studies

Box 1: Search strategy

Data extraction.

An iterative process was used to define data extraction domains. A total of fourteen potential domains were identified from the systematic review conducted by Gibson et al., alongside the He Pikinga Waiora Implementation Framework [ 25 ] and an access framework for Aboriginal and Torres Strait Islander people [ 26 ]. Further, three discussions were held within the team and with other Aboriginal and Torres Strait Islander researchers that identified eight domains of interest for both barriers and enablers (design attributes, chronic disease workforce, patient/provider partnership, clinical care pathways, access-accessibility, access-acceptability, system thinking and knowledge translation). A data extraction tool was prepared to extract information about the study characteristics (title, author, publication year, study setting, study objective, study design, types of services) and eight domains decided from the discussions mentioned above. A data extraction tool was shared with the  Thiitu Tharrmay Aboriginal and Torres Strait Islander Reference Group members rather than Thiitu Tharrmay Aboriginal Reference Group members  at the National Centre for Aboriginal and Torres Strait Islander Wellbeing Research at the Australian National University for their input. Based on inputs from Thiitu Tharrmay Aboriginal and Torres Strait Islander Reference Group members rather than Thiitu Tharrmay Aboriginal Reference Group members  and team discussion, some domains were consolidated and made an agreement of extracting data focusing six domains (culturally acceptable and safe services, patient provider partnerships, chronic disease workforce, primary health care service attributes, clinical care pathways and accessibility to primary health care services) the final data extraction tool included study characteristics and six domains (Fig.  2 )focusing both barriers and enablers. The data extraction tool was piloted on five included studies that facilitated shared understanding of approach to data extraction between team members. The data were extracted by UNY and JMD between May to June 2023.

Barriers and enablers to chronic disease prevention and management for Aboriginal and Torres Strait Islander people

Evidence synthesis and quality appraisal

Quality assessment for all included studies was conducted using the Aboriginal and Torres Strait Islander Quality Appraisal Tool that privileges Aboriginal and Torres Strait Islander people’s (knowing) epistemology, (being) ontology, (doing) axiology and ethical research governance [ 27 ]. The extracted data were analysed and summarised using a conventional content analysis approach [ 28 ] that allows the categories and the name of the categories to flow from the data and applying strength-based approaches [ 29 ].

Study selection and its characteristics

For the period 2013- 2023, a total of 1653 articles were retrieved from Medline ( n  = 927) and Web of Science ( n  = 726) databases. Of the total, 313 were duplicates which left 1340 articles for screening. Following the screening, 115 studies were selected for the full-text review. Upon full-text review, 89 articles were excluded, leaving 26 articles to be included for extraction and evidence synthesis. Service delivery models of the included studies were ACCHOs, government-run Aboriginal Medical Services and private general practice. The majority of the studies applied qualitative or mixed-method evaluation approaches (see Additional file 1 ).

Quality appraisal results

All studies met two criteria from the Aboriginal and Torres Strait Islander Quality Appraisal Tool: priority determined by community (criterion 1) and use of an Indigenous research paradigm (criterion 9). Over 92% studies met criteria for community protocols (criterion 5), 80% of studies met criteria addressing community consultation and engagement (criterion 2), Aboriginal and Torres Strait Islander leadership (criterion 3) and 76% studies had demonstrated Aboriginal and Torres Strait Islander governance in research (criterion 4). While 30% or less of the studies addressed existing intellectual and cultural property (criterion 6 and 7), 70% studies met the following criteria: Aboriginal and Torres Strait Islander people control over collection and management of research materials (criteria 8), use of strength based approach and acknowledging practices that have harmed Aboriginal and Torres Strait Islander communities (criterion 10) translation of findings into sustainable changes (criterion 11), benefit the participants and Aboriginal and Torres Strait Islander communities (criterion 12), demonstrate capacity strengthening for Aboriginal and Torres Strait Islander people (criterion 13), and researchers have opportunities to learn from each other (criterion 14). Details are provided in Table  2 .

Enablers and barriers to the implementation of chronic disease initiatives

Enablers and barriers (see Additional file 2 ) are presented under six thematic domains and relevant sub-themes presented below:

Primary health care service attributes

Aboriginal and Torres Strait Islander engagement and Aboriginal leadership in system design: Aboriginal and Torres Strait Islander project leadership and community engagement were overwhelmingly identified as key determinants for system design [ 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 , 45 , 46 , 47 ]. Engagement through outreach cultural community events and stakeholder partnerships appear to amplify these effects [ 48 ]. Six studies investigated how information was exchanged between knowledge-users and researchers throughout the studies [ 30 , 38 , 44 , 45 , 46 , 47 ]. Knowledge users included Aboriginal and Torres Strait Islander communities, primary health care providers (ACCHOs and mainstream providers), Aboriginal and/ or Torres Strait Islander health workers and practitioners, other service providers, and policymakers. Alongside this, one study acknowledged the utilisation of cultural and scientific evidence to provide best-practice healthcare [ 37 ]. Aboriginal and Torres Strait Islander project leadership consistently facilitated trust and satisfaction. Use of evidence-based clinical care guideline in project implementation within community and services were other key benefits of knowledge translation.

Primary health car responsive to local community needs: Prospective analysis of potential barriers for program development were essential to program success, in order to actively address these in the program design and implementation. Many studies clearly demonstrated how barriers were overcome through tailored solutions including: outreach services to save patient time and cost [ 30 , 31 , 45 , 49 ]; free of charge services [ 31 , 49 , 50 , 51 ]; 24-h culturally safe service [ 32 , 37 , 50 ]; and telehealth specialist services [ 51 , 52 ]. Outreach services (delivered outside of the clinic facility) included home medication delivery, general visits by health workers to build trust and screening for conditions at community-based events. Services which were free at point of care included medications, nicotine patches, hospital specialist clinics, supplementary services (e.g. cooking class, aged care services), and preventive initiatives (e.g. blood pressure measurement, glucose monitor and health promotion activities). Innovative primary care models for Aboriginal and Torres Strait Islander patients included flexible appointment systems [ 32 , 37 , 50 ], multidisciplinary teams for providing holistic care [ 37 , 38 , 51 ], and clinical audits for quality improvement were also effective [ 31 , 34 , 39 , 45 , 46 ]. Incorporating chronic disease prevention into primary care occurred through health promotion initiatives [ 30 , 35 , 37 , 50 , 51 ], and patient-led prevention initiatives and management plans [ 30 , 46 , 50 ] were also found to be invaluable in addressing community needs. Six studies explicitly emphasised the importance of adequate resources and flexible funding for primary care services to meet local Aboriginal and Torres Strait Islander community priorities and needs [ 30 , 32 , 36 , 37 , 38 , 39 ].

A holistic approach to care: Aligned with Aboriginal and Torres Strait Islander peoples’ perception of health , holistic support [ 30 , 38 , 44 ] and care coordination was critical, necessitating flexible funding, and systems thinking and innovative, locally-adapted reforms [ 30 , 31 , 46 ]. Social referral approaches that connected people to non-clinical services were an important component in Aboriginal community controlled primary care settings, for example: connecting patients to exercise groups, providing access to housing, opportunities for hobbies, or home care services such as ‘Meals on Wheels’ [ 38 , 40 , 41 , 53 ].

Primary care access: Several barriers were identified, including: a lack of support specific to Aboriginal and Torres Strait Islander leadership [ 34 , 39 , 51 , 53 , 54 ]; competing priorities of healthcare service delivery [ 30 , 34 , 39 ]; and a lack of funding specifically address social determinants of health [ 31 , 55 ]. These were more pronounced in Aboriginal PHC. There were also insufficient resources to engage stakeholders in the co-development of primary care programs. This contributed to poor connections and relationships between multidisciplinary teams in health centres and other actors, including clinical information management systems [ 34 , 48 , 51 , 54 ], and high turnover of trained staff [ 34 , 50 , 54 ]. Some studies highlighted challenges in creating culturally appropriate services, due to the heterogeneity of Aboriginal and Torres Strait Islander populations [ 32 , 39 , 55 ]. Moreover, accessing primary care services was particularly challenging for people with the greatest health needs [ 31 ], including those with limited health literacy which made it difficult for people to engage with chronic disease care [ 48 , 49 ].

Chronic disease workforce

Creating supportive environment and building capacity of primary healthcare workforce: Enabling safe and good work environments for Aboriginal and Torres Strait Islander Health Workers and Practitioners [ 30 , 35 , 54 ], and a shared sense of purpose amongst staff to address the complex needs of patients [ 30 , 47 ] were consistently identified as enablers. Eight studies highlighted the importance of dedicated staff for chronic disease management, with clear delegation of responsibilities and a positive team culture created through an engaging and collaborative work environment [ 30 , 36 , 37 , 39 , 43 , 46 , 51 , 54 ]. Establishing the health care workforce with chronic disease ‘portfolios’ were considered more able to provide recurrent, culturally safe, preventative, and responsive healthcare for Aboriginal and Torres Strait Islander people because they had a greater chance of forming trusting relationships [ 32 , 41 , 48 , 50 ]. Included studies highlighted the need for training and development of the primary healthcare workforce [ 30 , 31 , 33 , 55 ]. Four studies had a particular focus on local Aboriginal and Torres Strait Islander Health Workers and Practitioners including the importance of recruitment and retention strategies [ 31 , 34 , 39 , 46 ], alongside a need to clarify their roles, provide Aboriginal and Torres Strait Islander role models [ 40 , 41 , 50 , 52 ], and ensure they are involved in clinical decision making [ 30 , 33 , 35 , 37 , 39 , 43 , 44 , 52 , 54 ]. The importance of intensive cultural safety training for staff to deliver safe care was emphasised in five studies [ 37 , 47 , 49 , 54 , 55 ].

Barriers to sustainable chronic disease workforce at primary health care services: Nine studies noted staff shortages and high staff turnover as adversely impacting continuity of care [ 32 , 33 , 34 , 39 , 43 , 46 , 50 , 51 , 53 ], and three studies noted inadequate clinical training for non-Indigenous staff [ 32 , 46 , 53 ]. Workforce limitations contributed to lack of time and resources to reach the patients that needed healthcare the most [ 31 , 45 , 46 ]. Nine studies specifically emphasised the substantial shortages, high turnover, and high rates of burnout of Aboriginal and Torres Strait Islander staff [ 33 , 34 , 39 , 43 , 46 , 51 , 53 , 54 , 55 ]. Work overload, inadequate support and a sense of being undervalued contributed to these issues [ 32 , 33 , 43 ]. Four studies noted shortages of specialists as a key barrier to integrated chronic disease management pathway [ 34 , 43 , 46 , 47 ].

Patient-provider partnerships

Optimal care achieved by effective trustful patient-provider partnerships: Eight studies found that enablers of trusting patient-provider relationships included strengthening patient knowledge through interactive learning, culturally appropriate conversation, and strengths-focused clinical engagement [ 30 , 32 , 33 , 35 , 38 , 50 , 53 , 55 ]. Numerous studies highlighted that holistic care required mechanisms for communities, families/carers and community leaders to be engaged with service providers [ 30 , 45 , 52 , 53 , 54 ]. ACCHOs were generally identified as meeting these needs by offering culturally safe care, longer consultation times to facilitate patient/provider partnerships [ 32 , 37 ], and communicating with the community when there were changes in services, or the implementation of new programs [ 31 ].

Contextual barriers to patient-provider partnership: Barriers to partnership included competing priorities for patients [ 53 ], patient experiences of racism and discrimination [ 32 ], patient discomfort with non-Indigenous services [ 32 ], and patients sensing that their holistic needs were unmet [ 42 ]. Two studies reported that limited health literacy with little shared provider-consumer understanding of chronic conditions were barriers to forming positive relationships [ 31 , 46 ]. Three studies noted a general lack of connection between the clinician and patient but did not interrogate the contributors to this [ 32 , 49 , 55 ].

Clinical care pathways

Enablers to effective clinical information systems: Nine studies found in-house information technology support within primary healthcare services was crucial for effective patient referral, coordination, and follow-up care [ 30 , 31 , 32 , 34 , 38 , 39 , 43 , 46 , 55 ]. Of these, two highlighted the importance of partnership-enabled integration across health service organisations using a shared electronic health record system, disease registration multidisciplinary care plans, and a patient recall system [ 30 , 36 ]. While translation of evidence-based care guidelines were not mentioned, five studies explained the importance of capacity building of staff and investment in systems development for the effective use of clinical information systems [ 30 , 34 , 39 , 43 , 51 ].

Barriers to efficient clinical information systems: Information technology barriers were profound, including poor integration information technology systems, mixed paper and electronic records [ 43 , 46 , 51 , 53 ], and poor infrastructure – most notably internet access [ 34 , 52 , 54 ]. Four studies noted shortages of trained and regular staff to implement new pathways, as a key barrier to integrated chronic disease management pathways [ 34 , 43 , 46 , 47 ]. Inconsistent models of care [ 44 ], and poor communication between different hospital and primary care systems [ 50 ] were also barriers.

Access to primary health care services

Many of the recognised domains of healthcare access (accessibility, accommodation, availability, accessibility and affordability) were identified [ 56 ].

People and family-centred reforms improve access to adequate primary care: Enablers of access to primary health care services were identified in ten studies [ 30 , 31 , 32 , 38 , 39 , 40 , 41 , 43 , 45 , 49 , 53 ]. Transportation support was a determining and/or motivating factor for clients to access health services [ 31 , 38 , 39 ] Other studies [ 45 , 49 ] [ 43 ] identified accommodation factors, particularly flexible appointment systems, reduced waiting times and co-location with allied health services as key enablers. In this review, a number of motivational factors were identified for people to attend services, including support or referral from family members, higher motivation to look after oneself following the death of a family member, and motivation texts or invitational messages for health check from service providers [ 32 , 41 , 53 ]. In addition, one study identified providing financial incentives as an enabler for health checks [ 32 ].

Unaddressed social determinants prevent access to primary care services: Barriers to accessing primary health care services related to socio-economic factors, health system factors and lack of health promotion factors. Socio-economic factors included accessibility and affordability considerations; lack of transport [ 30 , 31 , 32 , 44 , 46 , 49 , 50 , 53 , 54 , 55 ], inability to afford health and social services, and medication costs [ 31 , 32 , 44 , 47 , 49 ]. Some studies alluded to socioeconomic factors being prioritised over primary care attendance, including household crowding and food insecurity. Ensuring that services account for competing priorities, including family and cultural responsibilities, was an important enabler [ 45 , 55 ].

System related access barriers: Health system factors included high staff turnover, lack of availability of appointments, long waiting periods, physically inaccessible clinics [ 43 , 44 , 46 ], poor leadership of primary care services [ 32 , 45 , 47 , 48 , 49 ], and limited internet and computer access [ 47 ]. Inadequate awareness of available services was also problematic [ 31 , 33 , 49 ]; initiatives run by primary care services had limited uptake when the community was not made aware of these programs [ 46 ]. Health systems need to be able to deliver services and information multimodally. For example, not all patients have phones or phone credit all the time, so several forms of communication may be required.

Culturally acceptable and safe services

Enablers to deliver culturally safe and acceptable services: Cultural safety is essential to the development of a mutually respectful relationship between providers and Aboriginal and Torres Strait Islander people. A systems level approach is needed to address racism experienced by Aboriginal and Torres Strait Islander people in primary care settings [ 30 , 44 ]. Strategies for addressing or achieving cultural safety varied by context, but the role of Aboriginal and Torres Strait Islander healthcare workers including cultural brokerage was emphasised in two studies [ 31 , 52 ]. The need for gender-specific services and gender sensitivity was emphasised by five papers as an important part of providing culturally safe care. This included delivery of programs such as gender-based exercise groups, private consultation areas for males and females, gender specific health assessment days, and employing male and female Aboriginal and Torres Strait Islander healthcare workers [ 31 , 41 , 45 , 46 , 52 ]. The provision of culturally safe services also included a need for culturally appropriate education materials, artwork, and Aboriginal and Torres Strait Islander people’s voices and images as signifiers of belonging [ 30 , 44 , 45 , 46 , 53 ]. Three studies recognised the importance of enabling Aboriginal and Torres Strait Islander patients and staff access to flexibly attend to family, community, cultural and spiritual responsibilities, and obligations to provide a culturally safe service [ 37 , 42 , 44 ]. One study identified the need for robust anonymous feedback systems for staff and patients to improve culturally safe care delivery [ 47 ].

Barriers to deliver culturally safe and acceptable services: Barriers to delivering culturally safe and acceptable services related to systems, structures and lived experiences. Systems issues to providing culturally safe services included language barriers [ 32 ], poor health literacy among patients [ 54 ], long wait times due to staff shortages [ 32 , 50 , 54 ], a sense that services were superficial/rushed [ 31 , 42 ], and lack of physical space to provide holistic care or gender-based services [ 43 , 44 , 46 ]. Lived experiences of treatment of Aboriginal and Torres Strait Islander people within Western systems, including health and social services, elicits feelings of harm rather than help: a fear of discrimination and racism was a key barrier to Aboriginal and Torres Strait Islander patients accessing healthcare services in five studies [ 31 , 32 , 44 , 49 , 53 ] alongside fear of diagnosis due to historical trauma [ 53 ]. These barriers were amplified where there was limited access to Indigenous-specific services [ 32 , 44 , 46 ]. One study mentioned a tokenistic approach where very limited community input to governance, planning, and program design was sought to develop culturally safe initiatives [ 46 ].

This is the first review since 2014 [ 13 ] to present the barriers and enablers of implementing chronic disease prevention and management programs for Aboriginal and Torres Strait Islander people. The enablers and barriers found in this study have several policy and practice implications that should be considered in design, implementation, and funding targets for future chronic disease prevention and management programs.

The most striking addition to our findings, relative to the 2014 [ 13 ] review, is the acknowledgment of Aboriginal and Torres Strait Islander culture (including staff, protocols, leadership, practices and ways of doing business) as a key enabler to engagement and care delivery. Partly, this is attributable to this review’s narrower focus on the Australian context. It also likely reflects increasingly detailed academic descriptions of the ways in which leadership and governance tangibly affect care delivery as part of contemporary Closing the Gap reform. All studies included in this review made some acknowledgement of culture, albeit with variations in how deeply culture was considered as an enabler of care. Our review team grappled with how best to reflect this focus on culture, given that it is both a distinct concept and intimately embedded in all thematic domains. Ultimately, we have chosen to keep Aboriginal and Torres Strait Islander culture as a separate domain to ensure that culture is given independent consideration, in addition to attention within other thematic domains. It is evident that access to culturally appropriate, affordable and comprehensive services are vital for preventing and managing chronic conditions [ 37 ]. There is no one-size-fits-all approach for models of care for Aboriginal and Torres Strait Islander communities, and programs must be tailored to local context. Recent studies have identified numerous opportunities for improving access to primary care services: creating welcoming spaces, improving the cultural safety of healthcare services, building strong trustworthy relationships between patients and providers, and building primary healthcare workforce capacity [ 37 , 57 , 58 ]. The span of this theme is necessarily broad, encompassing Aboriginal and Torres Strait Islander leadership, physically welcoming spaces, training for non-Indigenous staff, time to build trusting relationships. Ensuring that culture is prospectively and proactively considered in funding and delivery of primary care of chronic disease should be a priority for practitioners and policy makers.

There is clear evidence that addressing holistic needs of Aboriginal and Torres Strait Islander people enables greater engagement, rather than a narrow clinical focus on physical aspects of health. This requires primary care services to acknowledge and address the broader social and cultural determinants of health for Aboriginal and Torres Strait Islander people. Many of these have a direct impact on both chronic disease risk factors and capacity to access care (chronic disease management), including poor access to healthy and nutritious food, inadequate housing, rurality, lack of transportation, and financial barriers [ 59 , 60 ].Some of these disparities more pronounced in remote and rural Australia, where Aboriginal and Torres Strait Islander communities are further marginalised by distance and poverty [ 61 , 62 ]. Enablers of chronic disease care, such as outreach services, transportation, and referrals networks to other allied health and community groups are more likely to be effective where holistic approaches are adopted. This is consistent with data from the Australian Institute of Health and Welfare highlighting the experiences of social inequity in Aboriginal and Torres Strait Islander communities and the positive impact on health outcomes when inequities are reduced [ 63 ]. Previous studies [ 13 , 64 ] have identified various obstacles to accessing primary health care services which include inadequate infrastructure, inflexible and inadequate funding to care for people holistically. Evidence also suggests that for Aboriginal and Torres Strait Islander people, travel to access health care means being separated from their country, family and social network that directly impacts their health and wellbeing as described by Milroy’s Dance of Life [ 65 ]. While government subsidies are in place, travel and accommodation costs incurred by Aboriginal and Torres Strait Islander people to access healthcare, may require upfront payment or indirect costs, perpetuating financial barriers for Aboriginal and Torres Strait Islander people living in rural and remote areas [ 66 ]. Therefore, it is crucial that primary health care initiatives take a holistic and system thinking approach to program design, considering the impact of social and cultural determinants on the health of individual, family members and their communities, with every attempt to reduce systemic barriers to access to healthcare where possible.This is only possible when funding mechanisms and models of care are flexible enough to account for local and individual contexts.

The profound impact of workforce was clear throughout this review. Recruiting, and retaining staff and effective training, were found to be key barriers to implementing and maintaining holistic patient-centred chronic disease prevention and management programs [ 13 , 67 ]. Evidence has shown that Aboriginal and Torres Strait Islander people prefer support delivered by Aboriginal and/or Torres Strait Islander staff and clinicians who have a better understanding of Indigenous wellbeing [ 64 , 68 ]. Despite the growth of the Aboriginal and/or Torres Strait Islander health workforce over time, this expansion has not matched the Aboriginal and Torres Strait Islander population growth [ 69 ] and increasing incidence of chronic disease. Unsurprisingly, being members of the community, they serve, Aboriginal and/or Torres Strait Islander Health Workers play an essential and unique role in delivering culturally safe and holistic care. However, a demanding work environment, low salary, inadequate support, [ 70 ] and demanding cultural brokerage with non-Indigenous colleagues [ 70 , 71 ] contribute to burnout that contributes to poor retention rates of Aboriginal and Torres Strait Islander primary care staff. This requires urgent attention, by individual primary care providers and through the National Aboriginal and Torres Strait Islander Health Workforce Strategic Framework and Implementation Plan 2021 – 2031 [ 72 ]. Given the ongoing need for the non-Indigenous workforce in fulfilling workforce gaps required to deliver services for Aboriginal and Torres Strait Islander people, building cultural competence, continuing appropriate training and education pathways and strategies, providing job security and adequate remuneration are also crucial to address primary care workforce issues including the overburdening of the Aboriginal and Torres Strait Islander workforce [ 58 , 71 , 73 ]. Our findings highlight the need to develop the overall chronic disease workforce, with a specific focus for recruitment and retention of Aboriginal and Torres Strait Islander Health Workers and Practitioners and providing cultural safety training for all non-Indigenous staff. Alongside this, mechanisms for recognising the value and load of cultural mentorship/education should be developed. This reflects a recent research findings that showed 39% of Aboriginal and/or Torres Strait Islander workers ( n  = 1033) across Australia experienced high cultural load in terms of extra work demands and their engagement in educating others [ 74 ].

Chronic disease management requires multidisciplinary team input for effective care delivery [ 75 ]. When optimally resourced, primary care can serve a coordinating role in patient care, and effectively ensure patients have access to all allied health and specialist care they need [ 76 ]. Therefore, in order to engage Aboriginal and Torres Strait Islander people with chronic disease care and maintain continuity of care, there needs to be established, streamlined, and practitioner and patient friendly systems in place [ 76 , 77 ]. Existing evidence also documented inadequate number of general practitioners and lack of specialists in rural and remote settings of Australia compared to urban or city areas which hinders individuals, particularly Aboriginal and Torres Strait Islander people, to receive timely treatment for their co-occurring conditions in an integrated care approach [ 78 , 79 ]. A lack of integrated IT systems, poor infrastructure, and poor communication between primary care team members were found in this review to impede provision of such care. It is evident that strategies like GP care plans and tertiary care follow up are important sources of information for primary care providers, hospitals and patients which are supported by IT infrastructure [ 80 ]. Previous research highlighted the feasibility of system integration through utilising continuous quality improvement processes and community co-design [ 81 ]. Infrastructure investment such as internet access, in-house IT support and automated systems for follow-up care, is urgently required to ensure that patients who do present or engage with primary care in regional or remote settings, are retained in the system to enable coordinated access to the multidisciplinary care required for chronic disease management. Moreover, rapidly evolving technology such as tele-health, videoconferencing and Point-of-Care Testing may can facilitate access to the Aboriginal and Torres Strait Islander people in remote and rural areas. However, implementing these tools should be part of broader strategy rather than a substitute for solving problems faced by PHC such as workforce retention, undersupply or maldistribution issues [ 82 ]. Therefore, when implementing an integrated team care program [ 83 ] or any other integrated model of care, both barriers and facilitators identified herein should be applied to improve the continuity of care with considering the context.

Effective engagement of Aboriginal and Torres Strait Islander communities and their leadership in program design, delivery and evaluation of chronic disease programs is integral to improving Aboriginal and Torres Strait Islander health and increasing access to primary care services [ 84 , 85 ]. Previous studies have identified the following factors that enable engagement with Aboriginal and Torres Strait Islander communities: employment of local Aboriginal health workforce; trust and relationships; Aboriginal and Torres Strait Islander leadership; availability of flexible services to address holistic needs of local communities; benefits of engagement in service design and delivery; cost of participation and recognition of local Aboriginal knowledge and cultural traditions on study design implementation and dissemination [ 84 , 86 , 87 , 88 ]. These enablers align with those reported in health service research that engaged with Indigenous and marginalised communities in an international context [ 89 ]. Similarly, this review identified a range of impeding factors to engagement of Aboriginal and Torres Strait Islander people with primary health care. Key factors included: a fear or lack of trust on mainstream health facilities, lack of respect from health care providers, experiences of interpersonal and structural racism, lack of understanding of cultural differences to initiate an open discussion and a narrow concept of health that fails to consider the Aboriginal definition of health which is more comprehensive than the Western biomedical perspective of health that focuses on treating health conditions [ 90 , 91 , 92 ]. Evidence also shows that Aboriginal and Torres Strait Islander people in rural and remote communities do not have equitable access to PHC services, including lack of local available services to meet their holistic needs, inadequate infrastructure, high costs, long travel distance and insufficient workforce [ 59 , 93 ]. Therefore, trustful, and culturally safe engagement of Aboriginal and Torres Strait Islander communities through all aspects of the program design, implementation and evaluation is essential to program success, and where possible, the Aboriginal Health Workforce and ACCHOs should be utilised.

Policy and practice recommendations: This study identified several policy and practice recommendations (Table  3 ) that need to be considered for the implementation of chronic disease prevention and management programs for Aboriginal and Torres Strait Islander people in primary care. Our recommendations align with Australia’s Primary Health Care 10 Year Plan 2022–2032 ($632.8 million new investment) that has identified three streams of work: future focused health care; person-centred primary health care supported by funding reform; and integrated care, locally delivered [ 94 ].

Strengths and limitations

Strengths of the present study include (i) the generation of an evidence summary required to guide policy and practice is a short time frame, (ii) the application of iterative process from the design to completion of review with engagement of Aboriginal and Torres Strait Islander researchers and Thiitu Tharrmay Aboriginal and Torres Strait Islander Reference Group members rather than Thiitu Tharrmay Aboriginal Reference Group members , (iii) a quality appraisal of the included studies using Aboriginal and Torres Strait Islander Quality Appraisal Tool that privileges Aboriginal and Torres Strait Islander people’s ways of knowing, being, doing and (iv) interpretation of findings validated by Aboriginal and Torres Strait Islander knowledge champions.

One limitation of this review was that search was restricted to only two databases as the decision makers seek the evidence is a short period of time and were based on peer reviewed articles published in English language. We also acknowledge that the findings might not be comprehensive as the review was conducted in short timeframe, limitations in key words used and subjected to publication bias, as we omitted published program reports, grey literature, and policy guidelines from our inclusion criteria. Moreover, our search was limited to specific databases and terms, which could result in overlooking articles present in other databases or identified through alternate search terms. Despite these limitations, this review is Aboriginal and Torres Strait Islander researchers led that allowed Indigenous perspectives and knowledge to be integrated in the evidence synthesis; ensuring findings are meaningful for the broader sector.

This rapid review synthesises the barriers and enablers to designing and implementing chronic disease prevention and management programs for Aboriginal and Torres Strait Islander communities. While there is no one-size-fits-all approach to the heterogeneous Aboriginal and Torres Strait Islander communities, several policy and practice recommendations are broadly applicable to service providers. These include addressing social and cultural determinants of health, developing the Aboriginal and Torres Strait Islander and non-Indigenous chronic disease workforce, supporting multidisciplinary teams through strengthening clinical care pathways, and engaging Aboriginal and Torres Strait Islander communities in design and delivery of chronic disease prevention and management programs. This requires funding mechanisms and models of care that are flexible enough to account for local and individual context through policy and practice reforms. Moreover, enabling place-based partnerships to develop local and population-based strategies that align with community priories and aspiration is crucial for tackling increasing burden of chronic disease.

Availability of data and materials

All available materials are available as Additional material.

Abbreviations

Aboriginal Community Controlled Health Organisations

Medicare Benefits Schedule

Cardiovascular Disease

Chronic Kidney Disease

Preferred Reporting Items for Systematic Review and Meta-analysis Protocols

SelecTing Approaches for Rapid Reviews

Kairuz CA, Casanelia LM, Bennett-Brook K, Coombes J, Yadav UN. Impact of racism and discrimination on physical and mental health among Aboriginal and Torres Strait islander peoples living in Australia: a systematic scoping review. BMC Public Health. 2021;21(1):1302.

Article   PubMed   PubMed Central   Google Scholar  

Penm, E., Cardiovascular disease and its associated risk factors in Aboriginal and Torres Strait Islander peoples 2004–05, 2008, Australian Institute of Health and Welfare: Canberra.

Cardiovascular disease, diabetes and chronic kidney disease-Australian facts: Aboriginal and Torres Strait Islander people 2015 [ https://www.aihw.gov.au/reports/heart-stroke-vascular-disease/cardiovascular-diabetes-chronic-kidney-indigenous/summary ]

Australian Burden of Disease Study 2018-Key findings [ https://www.aihw.gov.au/reports/burden-of-disease/burden-of-disease-study-2018-key-findings ]

Swerissen H, Duckett S, Moran G. Mapping primary care in Australia. 2018. Grattan Institute. Available from: https://grattan.edu.au/wp-content/uploads/2018/07/906-Mapping-primary-care.pdf .

Shi L, Macinko J, Starfield B, Politzer R, Xu J. Primary care, race, and mortality in US states. Soc Sci Med. 2005;61(1):65–75.

Article   PubMed   Google Scholar  

Lavoie JG, Forget EL, Prakash T, Dahl M, Martens P, O’Neil JD. Have investments in on-reserve health services and initiatives promoting community control improved First Nations’ health in Manitoba? Soc Sci Med. 2010;71(4):717–24.

Thomas SL, Zhao Y, Guthridge SL, Wakerman J. The cost-effectiveness of primary care for Indigenous Australians with diabetes living in remote Northern Territory communities. Med J Aust. 2014;200(11):658–62.

Commonwealth of Australia, Department of the Prime Minister and Cabinet. Closing the Gap Prime Minister’s report 2017. Canberra: Department of the Prime Minister and Cabinet, 2017. Available at www.niaa.gov.au/sites/default/files/publications/ctg-report-2017.pdf .

Australian Institute of Health and Welfare 2023. Aboriginal and Torres Strait Islander Health Performance Framework: summary report July 2023. Canberra: AIHW. Viewed:17/07/2023.

The Royal Australian College of General Practitioners. Vision for general practice and a sustainable healthcare system. East Melbourne, Vic: RACGP, 2019. Available at www.racgp.org.au/advocacy/advocacy-resources/the-vision-for-general-practice/the-vision .

TOWARDS A NATIONAL PRIMARY HEALTH CARE STRATEGY: FULFILLING ABORIGINAL PEOPLES ASPIRATIONS TO CLOSE THE GAP.National Aboriginal Community Controlled Health Organisation, February 2009. https://www.naccho.org.au/publications-resources/ .

Gibson O, Lisy K, Davy C, Aromataris E, Kite E, Lockwood C, Riitano D, McBride K, Brown A. Enablers and barriers to the implementation of primary health care interventions for Indigenous people with chronic diseases: a systematic review. Implement Sci. 2015;10:71.

Yadav UN, Smith M, Agostino J, Sinka V, Williamson L, Wyber R, Butler DC, Belfrage M, Freeman K, Passey M, et al. Understanding the implementation of health checks in the prevention and early detection of chronic diseases among Aboriginal and Torres Strait Islander people in Australia: a realist review protocol. BMJ Open. 2023;13(6): e071234.

Australian Institute of Health and Welfare. Indigenous-‍specific health checks during the COVID-19 pandemic. https://www.aihw.gov.au/reports/indigenous-australians/indigenous-health-checks-during-the-covid-pandemic/contents/about

Follent D, Paulson C, Orcher P, O’Neill B, Lee D, Briscoe K, Dimopoulos-Bick TL. The indirect impacts of COVID-19 on Aboriginal communities across New South Wales. Med J Aust. 2021;214(5):199-200.e191.

Plüddemann A, Aronson JK, Onakpoya I, Heneghan C, Mahtani KR. Redefining rapid reviews: a flexible framework for restricted systematic reviews. BMJ Evid Based Med. 2018;23(6):201–3.

Pandor A, Kaltenthaler E, Martyn-St James M, Wong R, Cooper K, Dimairo M, O’Cathain A, Campbell F, Booth A. Delphi consensus reached to produce a decision tool for SelecTing approaches for rapid reviews (STARR). J Clin Epidemiol. 2019;114:22–9.

Tricco AC, Lillie E, Zarin W, O’Brien KK, Colquhoun H, Levac D, Moher D, Peters MDJ, Horsley T, Weeks L, et al. PRISMA extension for scoping reviews (PRISMA-ScR): checklist and explanation. Ann Intern Med. 2018;169(7):467–73.

Lutschini M. Engaging with holism in Australian Aboriginal health policy–a review. Aust New Zealand Health Policy. 2005;2:15.

Oetzel J, Scott N, Hudson M, Masters-Awatere B, Rarere M, Foote J, Beaton A, Ehau T. Implementation framework for chronic disease intervention effectiveness in Māori and other indigenous communities. Glob Health. 2017;13(1):69.

Article   Google Scholar  

National Indigenous Australians agency. Priority Reform One: Formal partnerships and shared decision-making. 2023 Commonwealth Closing the Gap Implementation Plan. [ https://www.niaa.gov.au/2023-commonwealth-closing-gap-implementation-plan ].

Naylor C, Charles A. Place-based partnerships explained. The King's Fund. 2022 [ https://www.kingsfund.org.uk/publications/place-based-partnerships-explained ].

Covidence systematic review software, Veritas Health Innovation, Melbourne, Australia. www.covidence.org .

Oetzel J, Scott N, Hudson M, Masters-Awatere B, Rarere M, Foote J, Beaton A, Ehau T. Implementation framework for chronic disease intervention effectiveness in Māori and other indigenous communities. Global Health. 2017;13(1):69.

Davy C, Harfield S, McArthur A, Munn Z, Brown A. Access to primary health care services for Indigenous peoples: a framework synthesis. Int J Equity Health. 2016;15(1):163.

Harfield S, Pearson O, Morey K, Kite E, Canuto K, Glover K, Gomersall JS, Carter D, Davy C, Aromataris E, et al. Assessing the quality of health research from an Indigenous perspective: the Aboriginal and Torres Strait Islander quality appraisal tool. BMC Med Res Methodol. 2020;20(1):79.

Hsieh H-F, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. 2005;15(9):1277–88.

Martin K, Mirraboopa B. Ways of knowing, being and doing: A theoretical framework and methods for indigenous and indigenist re‐search. J Aus Stud. 2003;27(76):203–214.

Askew DA, Togni SJ, Schluter PJ, Rogers L, Egert S, Potter N, Hayman NE, Cass A, Brown ADH. Investigating the feasibility, acceptability and appropriateness of outreach case management in an urban Aboriginal and Torres strait Islander primary health care service: a mixed methods exploratory study. BMC Health Serv Res. 2016;16:178.

Bailie J, Schierhout G, Laycock A, Kelaher M, Percival N, O’Donoghue L, McNeair T, Bailie R. Determinants of access to chronic illness care: a mixed-methods evaluation of a national multifaceted chronic disease package for Indigenous Australians. BMJ Open. 2015;5(11): e008103.

Canuto K, Wittert G, Harfield S, Brown A. “I feel more comfortable speaking to a male”: Aboriginal and Torres Strait Islander men’s discourse on utilizing primary health care services. Int J Equity Health. 2018;17(1):185.

Deshmukh T, Abbott P, Reath J. 'It’s got to be another approach’: an Aboriginal health worker perspective on cardiovascular risk screening and education. Aust Fam Physician. 2014;43(7):475–8.

PubMed   Google Scholar  

Bailie J, Laycock A, Matthews V, Bailie R. System-level action required for wide-scale improvement in quality of primary health care: synthesis of feedback from an interactive process to promote dissemination and use of aggregated quality of care data. Front Public Health. 2016;4:86.

Barrett E, Salem L, Wilson S, O’Neill C, Davis K, Bagnulo S. Chronic kidney disease in an Aboriginal population: a nurse practitioner-led approach to management. Aust J Rural Health. 2015;23(6):318–21.

Reeve C, Humphreys J, Wakerman J, Carroll V, Carter M, O’Brien T, Erlank C, Mansour R, Smith B. Community participation in health service reform: the development of an innovative remote Aboriginal primary health-care service. Aust J Prim Health. 2015;21(4):409–16.

Davy C, Kite E, Sivak L, Brown A, Ahmat T, Brahim G, Dowling A, Jacobson S, Kelly T, Kemp K, et al. Towards the development of a wellbeing model for aboriginal and Torres Strait islander peoples living with chronic disease. BMC Health Serv Res. 2017;17(1):659.

Blignault I, Norsa L, Blackburn R, Bloomfield G, Beetson K, Jalaludin B, Jones N. “You Can’t Work with My People If You Don’t Know How to”: Enhancing Transfer of Care from Hospital to Primary Care for Aboriginal Australians with Chronic Disease. Int J Environ Res Public Health. 2021;18(14):7233.

Bailie J, Matthews V, Laycock A, Schultz R, Burgess CP, Peiris D, Larkins S, Bailie R. Improving preventive health care in Aboriginal and Torres Strait Islander primary care settings. Global Health. 2017;13(1):48.

Seear KH, Atkinson DN, Henderson-Yates LM, Lelievre MP, Marley JV. Maboo wirriya, be healthy: community-directed development of an evidence-based diabetes prevention program for young Aboriginal people in a remote Australian town. Eval Program Plann. 2020;81: 101818.

Seear KH, Atkinson DN, Lelievre MP, Henderson-Yates LM, Marley JV. Piloting a culturally appropriate, localised diabetes prevention program for young Aboriginal people in a remote town. Aust J Prim Health. 2019;25(5):495–500.

Spurling GK, Bond CJ, Schluter PJ, Kirk CI, Askew DA. 'I’m not sure it paints an honest picture of where my health’s at’—identifying community health and research priorities based on health assessments within an Aboriginal and Torres Strait Islander community: a qualitative study. Aust J Prim Health. 2017;23(6):549–53.

Stoneman A, Atkinson D, Davey M, Marley JV. Quality improvement in practice: improving diabetes care and patient outcomes in Aboriginal community controlled health services. BMC Health Serv Res. 2014;14:481.

Webster E, Johnson C, Kemp B, Smith V, Johnson M, Townsend B. Theory that explains an Aboriginal perspective of learning to understand and manage diabetes. Aust N Z J Public Health. 2017;41(1):27–31.

Wood AJ, Graham S, Boyle JA, Marcusson-Rababi B, Anderson S, Connors C, McIntyre HD, Maple-Brown L, Kirkham R. Incorporating Aboriginal women’s voices in improving care and reducing risk for women with diabetes in pregnancy—a phenomenological study. BMC Pregnancy Childbirth. 2021;21(1):624.

Woods C, Carlisle K, Larkins S, Thompson SC, Tsey K, Matthews V, Bailie R. Exploring systems that support good clinical care in indigenous primary health-care services: a retrospective analysis of longitudinal systems assessment tool data from high-improving services. Front Public Health. 2017;5:45.

Sebastian S, Thomas DP, Brimblecombe J, Arley B, Cunningham FC. Perceived impact of the characteristics of the Indigenous Queensland B.strong brief intervention training program on uptake and implementation. Health Promot J Austr. 2022;33(1):245–56.

Kirkham R, Trap-Jensen N, Boyle JA, Barzi F, Barr ELM, Whitbread C, Van Dokkum P, Kirkwood M, Connors C, Moore E, et al. Diabetes care in remote Australia: the antenatal, postpartum and inter-pregnancy period. BMC Pregnancy Childbirth. 2019;19(1):389.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Cuesta-Briand B, Saggers S, McManus A. “It still leaves me sixty dollars out of pocket”: experiences of diabetes medical care among low-income earners in Perth. Aust J Prim Health. 2014;20(2):143–50.

Govil D, Lin I, Dodd T, Cox R, Moss P, Thompson S, Maiorana A. Identifying culturally appropriate strategies for coronary heart disease secondary prevention in a regional Aboriginal Medical Service. Aust J Prim Health. 2014;20(3):266–72.

Kirkham R, Boyle JA, Whitbread C, Dowden M, Connors C, Corpus S, McCarthy L, Oats J, McIntyre HD, Moore E, et al. Health service changes to address diabetes in pregnancy in a complex setting: perspectives of health professionals. BMC Health Serv Res. 2017;17(1):524.

Macniven R, Gwynn J, Fujimoto H, Hamilton S, Thompson SC, Taylor K, Lawrence M, Finlayson H, Bolton G, Dulvari N, et al. Feasibility and acceptability of opportunistic screening to detect atrial fibrillation in Aboriginal adults. Aust N Z J Public Health. 2019;43(4):313–8.

Conway J, Tsourtos G, Lawn S. The barriers and facilitators that indigenous health workers experience in their workplace and communities in providing self-management support: a multiple case study. BMC Health Serv Res. 2017;17(1):319.

Schmidt B, Campbell S, McDermott R. Community health workers as chronic care coordinators: evaluation of an Australian Indigenous primary health care program. Aust N Z J Public Health. 2016;40(Suppl 1):S107-114.

Campbell S, Roux N, Preece C, Rafter E, Davis B, Mein J, Boyle J, Fredericks B, Chamberlain C. Paths to improving care of Australian Aboriginal and Torres Strait Islander women following gestational diabetes. Prim Health Care Res Dev. 2017;18(6):549–62.

Saurman E. Improving access: modifying Penchansky and Thomas’s theory of access. J Health Serv Res Policy. 2016;21(1):36–9.

Davy C, Cass A, Brady J, DeVries J, Fewquandie B, Ingram S, Mentha R, Simon P, Rickards B, Togni S, et al. Facilitating engagement through strong relationships between primary healthcare and Aboriginal and Torres Strait Islander peoples. Aust N Z J Public Health. 2016;40(6):535–41.

Jongen C, McCalman J, Campbell S, Fagan R. Working well: strategies to strengthen the workforce of the Indigenous primary healthcare sector. BMC Health Serv Res. 2019;19(1):910.

Nolan-Isles D, Macniven R, Hunter K, Gwynn J, Lincoln M, Moir R, Dimitropoulos Y, Taylor D, Agius T, Finlayson H, et al. Enablers and Barriers to Accessing Healthcare Services for Aboriginal People in New South Wales, Australia. Int J Environ Res Public Health. 2021;18(6):3014.

Christidis R, Lock M, Walker T, Egan M, Browne J. Concerns and priorities of Aboriginal and Torres Strait Islander peoples regarding food and nutrition: a systematic review of qualitative evidence. Int J Equity Health. 2021;20(1):220.

Andermann A. Taking action on the social determinants of health in clinical practice: a framework for health professionals. CMAJ. 2016;188(17–18):E474-e483.

Markwick A, Ansari Z, Sullivan M, Parsons L, McNeil J. Inequalities in the social determinants of health of Aboriginal and Torres Strait Islander People: a cross-sectional population-based study in the Australian state of Victoria. Int J Equity Health. 2014;13(1):91.

Australian Institute of Health and Welfare (2022). Australia’s health 2022: in brief, catalogue number AUS 241. Australia’s health series number 18, AIHW, Australian Government. https://www.aihw.gov.au/reports/australias-health/australias-health-2022-in-brief/summary .

Hayman N. Strategies to improve indigenous access for urban and regional populations to health services. Heart Lung Circ. 2010;19(5–6):367–71.

Purdie P., Dudgeon P., Walker R. Working Together: Aboriginal and Torres Strait Islander Mental Health and Wellbeing Principles and Practice. 2nd ed. Australian Government Department of the Prime Minister and Cabinet. ACT; Canberra, Australia: 2014.

Kelly J, Dwyer J, Willis E, Pekarsky B. Travelling to the city for hospital care: access factors in country Aboriginal patient journeys. Aust J Rural Health. 2014;22(3):109–13.

Moore L, Britten N, Lydahl D, Naldemirci Ö, Elam M, Wolf A. Barriers and facilitators to the implementation of person-centred care in different healthcare contexts. Scand J Caring Sci. 2017;31(4):662–73.

McGrath PD, Patton MA, Ogilvie KF, Rayner RD, McGrath ZM, Holewa HA. The case for Aboriginal health workers in palliative care. Aust Health Rev. 2007;31(3):430–9.

Lahn J., Puszka S., Lawton P., Dinku Y., Nichols N. and Markham F. (2020) Beyond Parity in Aboriginal and Torres Strait Islander Health Workforce Planning: Achieving Equity through Needs-Based and Strengths-Based Approaches, Commissioned Report No. 6, Centre for Aboriginal Economic Policy Research, Australian National University. https://doi.org/10.25911/5f88175975c5b

Lai GC, Taylor EV, Haigh MM, Thompson SC. Factors Affecting the retention of indigenous Australians in the health workforce: a systematic review. Int J Environ Res Public Health. 2018;15(5):914.

Gwynne K, Lincoln M. Developing the rural health workforce to improve Australian Aboriginal and Torres Strait Islander health outcomes: a systematic review. Aust Health Rev. 2017;41(2):234–8.

Australian Government Department of Health . National Aboriginal and Torres Strait Islander Health Workforce Strategic Framework and Implementation Plan 2021–2031. Available from: https://www.health.gov.au/sites/default/files/documents/2022/03/national-aboriginal-and-torres-strait-islander-health-workforce-strategic-framework-and-implementation-plan-2021-2031.pdf . In .

McCalman J, Campbell S, Jongen C, Langham E, Pearson K, Fagan R, Martin-Sardesai A, Bainbridge R. Working well: a systematic scoping review of the Indigenous primary healthcare workforce development literature. BMC Health Serv Res. 2019;19(1):767.

Diversity Council Australia/Jumbunna Institute (Brown, C., D’Almada-Remedios, R., Gilbert, J. O’Leary, J. and Young, N.) Gari Yala (Speak the Truth): Centreing the Work Experiences of Aboriginal and/or Torres Strait Islander Australians, Sydney, Diversity Council Australia/Jumbunna Institute, 2020.

Harris MF, Jayasinghe UW, Taggart JR, Christl B, Proudfoot JG, Crookes PA, Beilby JJ, Davies GP. Multidisciplinary team care arrangements in the management of patients with chronic disease in Australian general practice. Med J Aust. 2011;194(5):236–9.

Digital technologies and chronic disease management. Australian Journal for General Practitioners 2014, 43:842-846.

Michielsen L, Bischoff EWMA, Schermer T, Laurant M. Primary healthcare competencies needed in the management of person-centred integrated care for chronic illness and multimorbidity: results of a scoping review. BMC Primary Care. 2023;24(1):98.

Street TD, Somoray K, Richards GC, Lacey SJ. Continuity of care for patients with chronic conditions from rural or remote Australia: a systematic review. Aust J Rural Health. 2019;27(3):196–202.

Comorbidity Guidelines. Rural and remote populations [ https://comorbidityguidelines.org.au/part-c-specific-population-groups/rural-and-remote-populations#:~:text=The%20lack%20of%20specialists%20in%20rural%20and%20remote,of%20employed%20medical%20practitioners%20decreasing%20according%20to%20remoteness .]

Trankle SA, Usherwood T, Abbott P, Roberts M, Crampton M, Girgis CM, Riskallah J, Chang Y, Saini J, Reath J. Integrating health care in Australia: a qualitative evaluation. BMC Health Serv Res. 2019;19(1):954.

McCalman J, Bainbridge R, James YC, Bailie R, Tsey K, Matthews V, Ungar M, Askew D, Fagan R, Visser H, et al. Systems integration to promote the mental health of Aboriginal and Torres Strait Islander children: protocol for a community-driven continuous quality improvement approach. BMC Public Health. 2020;20(1):1810.

Thomas SL, Wakerman J, Humphreys JS. Ensuring equity of access to primary health care in rural and remote Australia—what core services should be locally available? Int J Equity Health. 2015;14(1):111.

Australian Government Department of Health and Aged Care. Integrated Team Care Program.

Durey A, McEvoy S, Swift-Otero V, Taylor K, Katzenellenbogen J, Bessarab D. Improving healthcare for Aboriginal Australians through effective engagement between community and health services. BMC Health Serv Res. 2016;16(1):224.

Banks E, Haynes A, Lovett R, Yadav UN, Agostino J. Output-orientated policy engagement: a model for advancing the use of epidemiological evidence in health policy. Health Res Policy Syst. 2023;21(1):6.

O’Brien P, Prehn R, Rind N, Lin I, Choong PFM, Bessarab D, Coffin J, Mason T, Dowsey MM, Bunzli S. Laying the foundations of community engagement in Aboriginal health research: establishing a community reference group and terms of reference in a novel research field. Res Involve Engage. 2022;8(1):40.

Barnett L, Kendall E. Culturally appropriate methods for enhancing the participation of Aboriginal Australians in health-promoting programs. Health Promot J Austr. 2011;22(1):27–32.

Bessarab D, Durey A, Christou A, Katzenellenbogen JM, Taylor K, Brankovich J. Evaluation of the South Metropolitan Health Service Community Engagement Process Perth: Curtin University. 2014.

Yadav UN, Lloyd J, Baral KP, Bhatta N, Mehata S, Harris M. Evaluating the feasibility and acceptability of a co-design approach to developing an integrated model of care for people with multi-morbid COPD in rural Nepal: a qualitative study. BMJ Open. 2021;11(1): e045175.

Kaihlanen A-M, Hietapakka L, Heponiemi T. Increasing cultural awareness: qualitative study of nurses’ perceptions about cultural competence training. BMC Nurs. 2019;18(1):38.

Askew DA, Foley W, Kirk C, Williamson D. “I’m outta here!”: a qualitative investigation into why Aboriginal and non-Aboriginal people self-discharge from hospital. BMC Health Serv Res. 2021;21(1):907.

Gee G, Dudgeon P, Schultz C, Hart A, Kelly K. Aboriginal and Torres Strait Islander Social and Emotional Wellbeing. In: Working together: Aboriginal and Torres Strait Islander mental health and wellbeing principles and practice. 2014.

Bywood P, Katterl R, Lunnay B. Disparities in primary health care utilisation: Who are the disadvantaged groups? How are they disadvantaged? What interventions work? Adelaide: Primary Health Care Research and Information Service (PHCRIS), Policy Issue Review; 2011.

Australian Government Department of Health. Future focused primary health care: Australia’s Primary Health Care 10 Year Plan 2022–2032. 2022.Canberra ACT 2601.

Download references

Acknowledgements

We are grateful for the support of Chelsea Liu assisted with the use of the Quality Appraisal Tool and the support of Dr. Deborah Wong (Research Coordinator- Chronic Disease | Cervical Cancer) in designing Fig. 2 . We would also like to acknowledge Thiitu Tharrmay Aboriginal and Torres Strait Islander Reference Group members rather than Thiitu Tharrmay Aboriginal Reference Group members  at the National Centre for Aboriginal and Torres Strait Islander Wellbeing Research at the Australian National University for their continuous feedback without which this research piece wouldn’t have been completed.

The authors alone are responsible for the views expressed in this publication and they do not necessarily represent the decisions or policies of their affiliated institutions.

This work was supported by funding from the Australian Government Department of Health-—First Nations Health Division. The funding body had no role in the writing of this article.

Author information

Uday Narayan Yadav and Rosemary Wyber have contributed equally to this work.

Authors and Affiliations

National Centre for Aboriginal and Torres Strait Islander Wellbeing Research, Australian National University, Canberra, ACT, Australia

Uday Narayan Yadav & Rosemary Wyber

Centre for Primary Health Care and Equity, University of New South Wales, Sydney, Australia

Uday Narayan Yadav

Melbourne Medical School, The University of Melbourne, Melbourne, Australia

Jasmine Meredith Davis

The George Institute for Global Health, Sydney, Australia

Keziah Bennett-Brook & Julieann Coombes

Telethon Kids Institute, Perth, WA, Australia

Rosemary Wyber

South Australian Health and Medical Research Institute, Adelaide, SA, Australia

Odette Pearson

Faculty of Health and Medical Science, University of Adelaide, Adelaide, SA, Australia

You can also search for this author in PubMed   Google Scholar

Contributions

Conceptualization & original draft: Uday Narayan Yadav and Rosemary Wyber. Data curation: Uday Narayan Yadav, Jasmine Meredith Davis and Rosemary Wyber. Review, edits and revision: Uday Narayan Yadav, Jasmine Meredith Davis, Keziah Bennett-Brook, Julieann Coombes, Rosemary Wyber and Odette Pearson.

Corresponding author

Correspondence to Uday Narayan Yadav .

Ethics declarations

Ethics approval and consent to participate.

Not applicable.

Consent for Publication

Competing interests.

The authors have no competing interests to declare.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1..

Study characteristics.

Additional file 2.

Meta aggregated study findings.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Yadav, U.N., Davis, J.M., Bennett-Brook, K. et al. A rapid review to inform the policy and practice for the implementation of chronic disease prevention and management programs for Aboriginal and Torres Strait Islander people in primary care. Health Res Policy Sys 22 , 34 (2024). https://doi.org/10.1186/s12961-024-01121-x

Download citation

Received : 26 October 2023

Accepted : 10 February 2024

Published : 21 March 2024

DOI : https://doi.org/10.1186/s12961-024-01121-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Aboriginal and Torres Strait Islander people
• Barriers and enablers Chronic disease prevention and management
• Primary health care

Health Research Policy and Systems

ISSN: 1478-4505

• Submission enquiries: Access here and click Contact Us
• General enquiries: [email protected]

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 25 March 2024

The impact of land transfer policy on sustainable agricultural development in China

• Congjia Huo 1 , 2 &
• Lingming Chen 3  

Scientific Reports volume  14 , Article number:  7064 ( 2024 ) Cite this article

Metrics details

• Environmental sciences
• Environmental social sciences

At present, China’s agricultural development presents problems such as increasingly tight constraints on water and soil resources, increased pollution in the process of agricultural production, and noticeable degradation of agricultural ecosystems. So, it is urgent to promote the sustainable development of agriculture. From the perspective of land transfer policy, based on the panel data of 30 regions in China from 2006 to 2020, this paper uses the entropy weight method to calculate the level of sustainable agricultural development. Based on the analysis of the impact mechanism of land transfer policy on sustainable agricultural development, the relationship between land transfer policy and sustainable agricultural development is empirically tested by the continuous difference-in-difference method. The study found that the overall level of sustainable agricultural development in China is relatively low but shows an upward trend. The land transfer policy significantly promoted the sustainable development of agriculture in China. This conclusion is still valid after a series of validity tests and robustness tests. Finally, the corresponding policy suggestions are put forward according to the theoretical analysis and empirical results. Future research will focus on indicators challenging to quantify in agricultural sustainable development and find effective methods to incorporate them into the indicator system. At the same time, find the data of Prefecture-level cities in major grain-producing areas and further improve the measurement and research of agricultural sustainable development.

Similar content being viewed by others

Agricultural land management and rural financial development: coupling and coordinated relationship and temporal-spatial disparities in China

Maogang Gong, Ruichao Xi, … Lingling Che

The inhibitory effect of agricultural fiscal expenditure on agricultural green total factor productivity

Shuguang Wang, Jiaying Zhu, … Shen Zhong

Influence of university agricultural technology extension on efficient and sustainable agriculture

Zhaoli Dai, Qing Wang, … Yan Lu

Introduction

Agriculture is the foundation of the state, and the healthy and stable development of agriculture is the basis for the rapid development of the national economy of a country or region. For China, agriculture is the bedrock of its national economy. The quality and efficiency of agricultural production are related to the country’s food security and economic development. However, China’s agricultural production has faced practical problems such as limited land, stretched water resources, and excessive use of fertilizers and pesticides. Currently, China’s agricultural development shows increasingly tight constraints on water and soil resources, increased pollution in agricultural production, and noticeable degradation of agricultural ecosystems. At the same time, China’s water and soil resources management, ecological compensation, and other systems and mechanisms are not perfect, and the traditional mode of agricultural development has made it difficult to solve the problems caused by the acceleration of modernization. Therefore, we must adopt a sustainable agrarian production model to ensure long-term stable agricultural production and sustainable agricultural development in China. In recent years, with the increasing consensus on sustainable agricultural development, China’s sustainable rural development has achieved good results in four aspects: the sustained growth of total farm production capacity and farmers’ income, the steady improvement of the utilization level of agricultural resources, the continuous increase of agricultural ecological protection and construction, and the gradual improvement of rural living environment.

China’s agricultural production model is dominated by a small-scale peasant economy, showing the characteristics of scattered land, low comprehensive management capacity of farmers’ families, backward agricultural production methods, and extensive production and sales of farm products. In addition, the phenomenon of land fragmentation and decentralization will hinder the sustainable development of agriculture. Land transfer policy aims to promote agricultural modernization and large-scale operation, improve agricultural production and economic efficiency, and thus promote sustainable agricultural development. Zhongfa [2010] No.1, issued by the Chinese government in 2010, clearly stated that “we will announce the reform of the rural land management system in an orderly manner and accelerate the registration and certification of rural collective land ownership, homestead use rights, and joint construction land use rights” The No. 1 document of 2010 proposed five measures to coordinate urban and rural development. The government has taken multiple measures to improve the social security and public service levels of mobile populations and vulnerable groups and actively promoted rural land transfer. Specifically reflected in the following two aspects: On the one hand, improving the land transfer market: The document proposes to strengthen the management and service of land contract management rights transfer, which means establishing and improving the land transfer market mechanism to ensure the transparency and efficiency of land transfer.

On the other hand, protecting the rights and interests of farmers: The document mentions the strict implementation of the Rural Land Contract Management Dispute Mediation and Arbitration Law, which is to protect the rights and interests of farmers in land contract management and prevent infringement during the land transfer process. With the continuous revision and improvement of the land transfer policy, the scale of land transfer in China has increased rapidly. Figure  1 shows the land area and cultivated land area transferred in China from 2005 to 2021, as well as the scale of land transfer.

Cultivated land area and transferred cultivated land area in China from 2005 to 2021, and land transfer scale (unit: 100 million mu). Note Land transfer scale = total area of cultivated land transferred/developed land area * 100%. The left Y-axis shows the scale of land transfer (unit: 100 million acres), while the right Y-axis shows the scale of land transfer (unit:%).

According to Fig.  1 , the scale of land transfer in China has increased significantly. In 2007, the scale of land transfer was only 5.24%, and by 2018, the value increased to 38.59%. It is difficult for cultivated land resources to grow. The total area of cultivated land in China has changed slightly, and the area of cultivated land has increased yearly. In 2016, China’s “Three Rights Separation” policy gave the due legal status of land management rights. After that, the scale of land transfer showed a rapid growth model and tended to stabilize. The land transfer policy can theoretically promote sustainable agricultural development. From the data point of view, the trend of the two is the same, so is there a driving force for sustainable agricultural development brought about by land transfer policy? How much does the land transfer policy affect the sustainable development of agriculture? How can we further promote sustainable agricultural development through a land transfer policy? This paper uses the continuous difference-in-difference (DID) method to verify whether the transfer policy can promote the sustainable development of agriculture based on calculating the sustainable development level in 30 regions of China.

The rest of this paper is organized as follows: Section “ Literature review and theoretical mechanism ” is a literature review and theoretical hypothesis on related topics; In Section “ Index system and construction of agricultural sustainable development level ”, the panel data of 30 regions in China from 2006 to 2020 are used to calculate the sustainable development level of regional agriculture by entropy weight method; In Section “ Empirical study on the impact of land process policy on the level of sustainable agricultural development ”, the continuous DID model is set up, and the variable description and data source are given. At the same time, it empirically tests the impact of land transfer policy on sustainable agricultural development and tests the effectiveness and robustness of the model. Section “ Conclusions, policy recommendations, and research prospects ” is this paper’s conclusion, policy recommendations, and future research directions.

Literature review and theoretical mechanism

Literature review, research on the calculation of sustainable agricultural development.

Academic research on sustainable agricultural development mainly includes the optimization of agrarian resource allocation, agricultural productivity, and agricultural trade (Alagh 1 ; Wen et al. 2 ; Zhou 3 ). Calculating sustainable agricultural development is the premise of studying sustainable agricultural development. Many documents calculate the level of sustainable agricultural development in different regions or periods through various methods, mainly using the entropy weight method. Indicators are primarily selected from subsystems such as population, society, economy, environment, and resources to construct an indicator system for sustainable agricultural development for Calculation (Yuan and Qi 4 ; Zhao et al. 5 ; Zhang et al. 6 ). Other scholars have used other methods to estimate sustainable agricultural development: Liu and Ma 7 lay on the nonlinear and grey character of the agricultural sustainable development system. Using grey system theory and the neural network simulation method, we set up scientific and reasonable calculation and forecast models for the coordination degree of agricultural sustainable development systems. A questionnaire was developed based on the analytical hierarchy process (AHP) (Poursaeed et al. 8 ). Ye and Chen 9 used the DEA analysis method to divide the agricultural sustainable development system into four subsystems: survival security, agricultural economic development, rural social progress, and agricultural ecological protection. Miao et al. 10 used a weighted comprehensive evaluation, coordination, and development obstacle models to analyze the current situation, potential, and obstacle factors of sustainable agricultural development in the study area. Tang and Liu 11 constructed the evaluation index system of agricultural sustainable development level from five aspects: population, society, economy, environment, and resources. They used the coupling coordination model to analyze the coupling degree and coordination degree among the subsystems of sustainable agricultural development in the Yangtze River Economic Zone. Adamopoulos et al. 12 used panel data and quantitative frameworks from China’s household level to record the degree and consequences of agricultural factor mismatch. They found that improper resource allocation among farmers can reduce agricultural production efficiency. Adenle et al. 13 used evidence of the benefits and challenges associated with sustainable farming practices in Africa. They found that improving agricultural sustainability is crucial for food security and poverty reduction, especially for achieving sustainable development goals by 2030. However, agricultural sustainability alone cannot solve all of these issues.

Research on land transfer policy and sustainable agricultural development

Many factors affect the sustainable development of agriculture, including natural, economic, social, and technological obstacles, among which the insufficient supply of agricultural systems, primarily agrarian land systems, is an essential factor. Under the economic structure dominated by agriculture and the level of traditional technology, agricultural production’s core competitiveness is producers’ enthusiasm. Stable land use rights are the institutional basis for ensuring farmers’ enthusiasm and stimulating sustainable agricultural development (Besley 14 ; Chari et al. 15 ). To promote farmers’ enthusiasm for farm investment and achieve sustainable land use and agricultural growth, agricultural protection must be carried out to protect the economic interests of farm producers and operators (Pang 16 ). However, the imperfect system and mechanism of agricultural land transfer have directly become a severe obstacle to the sustainable development of agriculture (Hu 17 ). An imperfect land transfer market will affect the efficiency of agricultural land resource allocation (Rosticceria 18 ). Small-scale farmers’ discriminatory land policy, land taxation, input subsidies, imperfect land factor market, and credit constraints are the main reasons for affecting resource allocation and sustainable development in the agricultural sector (Adamopoulos 19 ). Adamopoulos 20 used Philippine household data and found that unreasonable land transfer policies would inhibit the optimal allocation of land resources. Bolhuis et al. 21 used Indian household data and found that land flow transfer promotes land redistribution, thus improving agricultural productivity. Xia and Zeng 22 tested the impact of changes in the farmland transfer system on sustainable agricultural development, and, combined with the development status of China’s “three rural” issues and the requirements of sustainable agricultural development, analyzed the way out of the reform of the farmland transfer system in the case of normalization of agricultural land transfer.

Li and Zhu 23 found that legalizing land transfer and land property reform can increase agricultural investment and improve land quality. Moreover, the benefits of strengthening land quality are unevenly distributed across different socio-economic backgrounds. Fei et al. 24 used the PSM method to construct a counterfactual framework to analyze the impact of land transfer on agricultural land efficiency. The average land use efficiency in developing countries is relatively low, and provinces that transfer land are more effective in land use than other provinces. Yang et al. 25 measured the sustainability of farmers’ livelihoods based on field survey data from 650 households in Hubei Province, China. They found that land transfer can significantly improve the sustainability of farmers’ livelihoods, which increases with the increase of land transfer area. Cao et al. 26 found that the positive impact of land leasing and the negative impact of land fragmentation on agricultural production efficiency indicate that it is essential to promote land consolidation policies by incorporating land parcel connectivity interventions in the land leasing market. The mainstream of land transfer is the spontaneous, slow, and internal transfer in rural areas, which promotes the concentration of land to some farmers. In the process of land transfer and class construction, the form of a middle agricultural economy has emerged, and the rural middle class and its shaped middle agricultural economy provide strong support for the sustainable development and modernization of agriculture (Liu 27 ).

Compared with existing literature, the possible contributions of this article are mainly reflected in two aspects. Firstly, this article analyzes how the implementation of land transfer policies affects sustainable agricultural development from the perspective of land transfer policies. At the same time, it provides new empirical evidence for the government to formulate policies to promote sustainable agricultural development. Secondly, this article constructs a small sample continuous double difference model based on random inference. At the same time, it also reduces endogeneity problems caused by selective bias and reverse causality. In addition, the impact of land transfer policies on sustainable agricultural development is also empirically tested.

Theoretical mechanism

The reform of land transfer policies will promote the increase of land transfer scale in China, thereby improving the efficiency of land resource utilization and agricultural production, protecting the agricultural ecological environment, and promoting sustainable agricultural development through these three paths. Specifically:

Land transfer policy can improve the efficiency of land resource use

Farmers’ average household land management area has been too narrow for a long time, seriously restricting agricultural machinery and advanced agricultural technology (Yao and Hamori 28 ; Ying et al. 29 ). Land use rights cannot be freely transferred, so many young agricultural laborers flow into cities. It is difficult for surplus rural labor to engage in efficient agricultural production activities, and some areas have experienced abandonment. The reform of land transfer policies has dramatically promoted the behavior of agricultural land transfer. Expanding the farm production scale promotes sustainable agricultural development from two aspects.

On the one hand, with the gradual improvement of land transfer policies, families who are unable to engage in agricultural production transfer their land use rights to large farmers, and a large number of scattered and small plots of land gather to form a large-scale and efficient agrarian production model; On the other hand, land property rights reform has promoted land redistribution among farmers, promoting land trusteeship, transfer, and intensive use (Chari et al. 15 ). This is beneficial for improving land use efficiency, reducing excessive land reclamation and damage, and protecting rural resources and the environment. Land transfer policies can promote more rational allocation and utilization of land resources, avoid land idleness and waste, improve land use efficiency, and promote sustainable agricultural development.

Land transfer policy can improve the efficiency of agricultural production

The scale of small farmers’ families is relatively small, and the funds and resources of farmers’ families are relatively scarce, so it isn’t easy to afford high scientific and technological investment and equipment procurement. At the same time, due to the lack of professional knowledge and technical skills, farmers often do not have enough ability to carry out technological innovation and application, and it is challenging to meet the requirements of market demand and industrial upgrading. Coupled with the long-term influence of traditional concepts and habits, farmers lack the spirit of pioneering and enterprising and the understanding of technological innovation, which makes it challenging to adapt to the requirements of the scientific and technical era. The land transfer policy has promoted the formation of large-scale agricultural production. With the entry of capital elements into the primary industry and the continuous introduction of advanced technology and equipment in agricultural production, China’s agriculture has gradually achieved large-scale, mechanized, and standardized production. It reduces production costs and improves agricultural production efficiency and market competitiveness. The land transfer policy can promote the centralized allocation of agricultural means of production, improve production efficiency and economic benefits, and thus promote sustainable agricultural development.

Land transfer policy can protect the agricultural ecological environment

Agricultural zoning design has important strategic significance, and China’s previous small-scale and fine agricultural production model is challenging to achieve agricultural layout. The land transfer policy can promote reasonable agricultural layout and planting structure adjustment. From the micro level, optimizing agricultural layout can avoid environmental problems such as soil degradation and soil erosion caused by excessive reclamation and pesticide use and help protect the agricultural ecological environment. From the macro level, according to the local climate, topography, soil and other natural conditions and economic development level, the distribution and utilization direction of agricultural land are determined, and the layout and spatial distribution of agricultural land are rationally planned, which significantly promotes the sustainable development of agriculture.

Through large-scale land management, land transfer accelerates the process of farmers’ non-agriculturalization, inherits urbanization, and starts agricultural modernization, which is the premise of realizing the synchronous and coordinated development of urbanization and agricultural modernization in China (Yang 30 ). Industrialization, urbanization, marketization, and agricultural industrialization promote the gradual improvement of land transfer policies, and the rural land property rights system has evolved in the direction of clarifying the subject of property rights, ensuring the right of land transfer and facilitating land transfer (Huang and Xie 31 ). To sum up, land transfer is an effective way to promote sustainable agricultural development. It can improve land use efficiency and protect the agricultural ecological environment, promote agrarian modernization, transformation, and upgrading, improve farmers’ income and living standards, and promote rural transformation and development. The land transfer policy has further improved China’s agricultural system, effectively alleviated the problems of significant differences in the quality of farm products, weak competitiveness, and low production efficiency brought about by fine-grained agricultural production, and also solved the problems of insufficient rural labor force and limited production of farm products brought about by the entry of young agricultural labor force into cities. Based on this, this paper proposes two hypotheses to be tested:

This article proposes a research hypothesis: land transfer policy can promote sustainable agricultural development.

Index system and construction of agricultural sustainable development level

Index construction and data source, index construction.

Sustainable agriculture refers to the formation of sustainable development principles. It combines traditional agriculture with modern technology. And based on protecting natural resources. A new agricultural production method that also achieves ecological balance and economic benefits. Specifically, sustainable agriculture is guided by environmental principles and ecological economic laws. It is simultaneously focusing on managing and protecting natural resources. It combines traditional agriculture with modern technology. It aims to meet everyday people’s and future generations’ needs for agricultural products. The main characteristics of this new agricultural production method are as follows: The production process balances economic benefits with ecological protection. At the same time, the goal is to achieve sustainable, fair, efficient, and diverse agricultural production. Overall, sustainable agricultural models are entities with structures and functions. Integrating ecological, economic, and social elements demonstrates their status and interaction in the system network. This model reflects explicitly the sustainable utilization of resources. The sustainable development of agriculture needs the mutual influence, promotion, and joint development of population, economy, resources, society, environment, and other factors. To scientifically measure the level of regional agricultural sustainable development, this paper constructs an index system for the level of sustainable agricultural development.

The design and construction process of the evaluation index system needs to focus on the following six principles: purposefulness, completeness, operability, independence, saliency, and dynamism. Independence requires each indicator to have a clear connotation and be as independent as possible. Indicators at the same level should not overlap, cross, cause and effect, or contradict each other as much as possible. Upper and lower-level indicators maintain a top-down subordinate relationship. The existence of mutual feedback and interdependence between indicator sets and among indicators within the indicator set should be avoided. At the same time, maintain good independence. The article takes the “economic system sustainable level,” “environmental system sustainable level,” “social system sustainable level,” “resource system sustainable level,” and “population system sustainable level” as the primary indicators. It ensures the independence of the primary indicators. It is also a complex, comprehensive evaluation indicator. There may be interdependence and feedback relationships between indicators to ensure the indicator system’s purposefulness, completeness, and operability. Therefore, previous literature was combined to construct agricultural sustainable development indicators for secondary indicators (Jiao 32 ; Ding et al. 33 ; Miao et al. 34 ). This article uses the VIF test method to test multicollinearity and eliminate collinearity indicators. Meanwhile, considering the feasibility of indicators and the availability of data, a sustainable agricultural development indicator system is constructed as shown in Table 1 :

Data sources and calculation method

The data on agriculturally sustainable development level indicators come from the China Rural Statistical Yearbook, China Statistical Yearbook, China Agricultural statistical yearbook, China Urban Statistical Yearbook, and provincial and Municipal Statistical Yearbooks over the years. Some indicators are obtained by estimation or calculation, and the specific calculation method is shown in Table 1 . In addition, some missing data are supplemented by linear interpolations. In addition, considering the particularity and data availability of Tibet, Hong Kong Special Administrative Region, Macao Special Administrative Region, and Taiwan Province, the data of these four regions were excluded from the calculation process. Finally, 30 provincial panel data from 2006 to 2020 are obtained to calculate the level of sustainable agricultural development in various regions of China.

Accurately and efficiently measuring the level of sustainable agricultural development in China is the basis for studying its relationship with the degree of land transfer. The existing literature mainly uses factor analysis and data envelopment analysis to calculate the level of sustainable development in agriculture. Data envelopment analysis measures production efficiency from the perspectives of input and output, and its results also include slack variables for calculating inefficiency with relatively small errors. But, the degree of relationship between input variables and output variables was not considered. Therefore, this article draws on the calculation method of Zhang et al. 35 for the sustainable development of agriculture in China and selects the entropy weight method, an objective weighting method. The weight of each indicator is determined based on the degree of variation of its values, which is more scientific and practical.

Analysis of calculation results of agricultural sustainable development level

The above has given the calculation method and index system of agricultural sustainable development level and the data source. After calculation, this paper obtains the sustainable development level of agriculture in 30 regions of China from 2006 to 2020.

Table 2 shows that, on the whole, China’s agricultural sustainable development level shows two characteristics: first, the overall level of sustainable agricultural development shows an upward trend. The level of sustainable agricultural development in all regions in 2020 is higher than in 2006. The average growth rate in Shanghai was 13.36%. Among them, the average growth rate of agricultural sustainable development level in 11 regions is more than 5%. The five regions with the most significant average growth rate are Shanghai, Hainan Province, Fujian Province, Beijing, and Hubei Province, which are 13.36%, 7.69%, 7.45%, 7.25%, and 7.22%, respectively. Table 2 shows that except for Hubei Province, the other four regions are nongrain-producing areas, especially Shanghai and Beijing. The high level of economic development drives agriculture’s green and healthy development. Hainan, Fujian, and other regions started agricultural development late and lagged behind the “big army” considerably. However, driven by the sustainable development of agriculture in China and the acceleration of the inflow of agricultural production factors, there has been relatively rapid growth. Second, the current level of sustainable agricultural development in China is still low. The five regions with the lowest level of sustainable agricultural development in 2020 are Gansu Province, Shanxi Province, Shandong Province, Guangxi Province, and Anhui Province, which are 0.1547, 0.1944, 0.198, 0.2108, and 0.211 respectively. Shandong and Anhui are the main grain-producing areas, and other provinces are also central agricultural production provinces. Still, the level of sustainable agricultural development lags behind other regions.

The sustainable development of agriculture in China is slow and has many constraints. At the same time, regional growth is uneven, the level of sustainable agricultural development varies significantly between regions, the level of sustainable agricultural development in more developed cities remains at a high level, and the level of sustainable agricultural development in underdeveloped areas, especially in major grain-producing regions, remains at a low level.

Empirical study on the impact of land process policy on the level of sustainable agricultural development

Model construction and variable description.

With the gradual digestion of the enthusiasm for agricultural production released by the household contract responsibility system, China has gradually shifted from a “small peasant economy” to large-scale agricultural production. In this process, transferring rural land management rights has become the key to further improving China’s rural land system and promoting sustainable agricultural development. Zhongfa [2010] No.1, issued by the Chinese government in 2010, clearly stated that “we will promote the reform of the rural land management system in an orderly manner and accelerate the registration and certification of rural collective land ownership, homestead use rights, and collective construction land use rights.” With the gradual improvement of the relevant land transfer system, the interests of both sides of the land transfer have been guaranteed to a large extent. After 2010, the scale of land transfer in China increased substantially, and there is heterogeneity in different regions and whether they are the prominent grain-producing areas. Therefore, this paper uses the DID method to conduct empirical research to reduce the endogenous problems caused by selective bias and reverse causality (Beck et al. 36 ). When the sample size is small, traditional statistical techniques may lead to incorrect inference due to autocorrelation. Based on previous research, this article uses Bootstrap sampling to increase sample size and correct the bias of small sample double difference models (Bertrand and Mullianathan 37 ). Bootstrap sampling is a commonly used statistical method. It constructs multiple similar resampled datasets by extracting samples with replacements from the original dataset. Thus simulating random variability and conducting statistical analysis on each resampled dataset.

However, the explanatory variable land transfer scale is not a binary virtual variable. Here, drawing on the research methods of Nunn and Qian 38 , we use the continuous DID method to estimate the impact of land transfer policy on sustainable agricultural development and use continuous variables to distinguish the treatment group and the control group. This article takes regions with land transfer scales more significant than the average as the treatment group. At the same time, areas with lower than average land transfer scale will be considered control groups. This article uses the agricultural sustainable development level calculated above as a variable to measure the effectiveness of policies. 2010 is the implementation time of the policy. Then, N control and M experimental groups were randomly and independently selected from the sample. A random algorithm determines the number of samples. Finally, a continuous double difference model regresses the extracted sample data. See Eq. ( 1 ) for the specific model setting:

Among them, \({ASD}_{i,t}\) is the explained variable, that is, the level of sustainable agricultural development in the t-Year and i-region, \({LT}_{i,t}\times {P}_{t}\) is the explanatory variable, \({LT}_{i,t}\) is the continuous variable land transfer scale, \({P}_{t}\) is the virtual variable at the policy time point, when \({\text{t}}\ge 2010\) , \({P}_{t}\) =1, when \({\text{t}}<2010\) , \({P}_{t}\) =0; \({X}_{i,t}\) are control variables, they are industrial structure ( \({\text{is}}\) ), urbanization level ( \({\text{ude}}\) ), energy structure ( \({\text{eng}}\) ), and grain economy ratio ( \({\text{fsc}}\) ); \({\mu }_{i}\) is the fixed-year effect; \({\theta }_{t}\) is individual fixation; \({\varepsilon }_{i,t}\) is a random error term.

Explanatory variable: The effect of land transfer policy, expressed by the intersection of the virtual variable of policy time point and land transfer scale ( \({{\text{LT}}}_{{\text{i}},{\text{t}}}\times {{\text{P}}}_{{\text{t}}}\) ), the cultivated land area in different regions is different, and only the transfer area representing the degree of land transfer is not comprehensive enough. Hence, this paper takes the ratio of land transfer area to total cultivated land area as the scale of land transfer ( \({\text{LT}}\) ).

Explained variable: Agricultural sustainable development water ( \({\text{ASD}}\) ), which involves agricultural economy, rural resources, and other aspects. In the previous part of this paper, the sustainable development level of regional agriculture in China from 2006 to 2020 is estimated. The calculation results are shown in Table 2 , which will not be repeated here.

Control variables: The control variables selected in this paper are as follows: industrial structure ( \(is\) ) reflects the process of transformation of regional industrial structure from a low level to a high level, which is expressed in this paper by (added value of tertiary industry + added value of secondary industry)/added value of primary sector; The level of urbanization ( \(urb\) ) reflects the progress of urban civilization and the gap between urban and rural development. This paper uses the ratio of the permanent urban population to the total permanent population of the region to measure. Energy structure ( \(ene\) ) is an essential part of energy system engineering research, which directly affects the final energy consumption mode of various departments of the national economy and reflects people’s living standards. This paper uses the ratio of regional coal consumption to total energy consumption. Agricultural structure ( \(fsc\) ) demonstrates the production structure of farm products, expressed as the ratio of grain planting area to the entire planting area of cash crops. The higher ratio indicates that the development of the agricultural economy in this region depends more on grain production.

Instrumental variable: the continuous dual difference method may have a biased and inconsistent estimation coefficient, so in the following, we try to alleviate the possible endogenous problem through the instrumental variable method. We use limited agricultural production capacity ( apc ) and employment personnel in agricultural and urban units ( eau ) as a tool variable. The farm production capacity calculation formula is the agricultural added value/total added value of agriculture, forestry, animal husbandry, and fishery.

The explanatory variables of the continuous DID model have been calculated previously. The data relating to the scale of land transfer and the cultivated land area contracted by households are from the national rural economic statistics and the annual report of China’s rural management statistics, which are calculated. The control variables come from the China Statistical Yearbook, China Urban Statistical Yearbook, China Energy Yearbook, and China Economic and social development statistical database from 2006 to 2020. Some indicators, such as industrial structure and urbanization level, are calculated according to the statistical data. The descriptive statistics of the main variables are shown in Table 3 .

Empirical results

Analysis of benchmark regression results.

Before baseline regression, we used the variance inflation factor test to rule out the possibility of multiple collinearities of explanatory variables. Table 4 shows the benchmark regression results of the impact of land transfer policies on sustainable agricultural development. Model (1) and model (3) are the benchmark regression results of common standard error and robust standard error, respectively, and model (2) and model (4) are the benchmark regression results of common standard error and robust standard error after adding control variables.

According to the regression results in Table 4 , no matter which model, the land transfer policy will positively impact sustainable agricultural development. However, to avoid the inconsistency of results caused by heteroscedasticity, it is more accurate to use robust standard error regression with control variables. The benchmark regression results of model (4) show that the coefficient of explanatory variables ( \({{\text{LT}}}_{{\text{i}},{\text{t}}}\times {{\text{P}}}_{{\text{t}}}\) ) is significantly positive at the 10% level, with a coefficient of 0.0006. It shows that the land transfer policy can improve sustainable agricultural development. The land transfer system has been further enhanced to protect the interests of both sides of the transfer, the scale of land transfer has increased substantially, the process of large-scale agricultural production has been accelerated, and the rural surplus labor force can be released. Advanced production technologies and concepts can be introduced to promote the sustainable development of agriculture.

In model (4), the control variables industrial structure, urbanization level, and agricultural structure positively impact sustainable agricultural development, with coefficients of 0.0011, 0.3384, and 0.0024, respectively, and all of them passed the significance test of 1%. The sustainable development of agriculture is affected by many factors, such as upgrading industrial structure, which reduces the proportion of agricultural added value in total GDP. Still, the rapid development of the secondary and tertiary industries will feed back agriculture, promote the efficiency of agricultural production, and promote the low-carbon development of agriculture.

Effectiveness analysis of the DID method

The previous paper used the continuous DID regression model to verify that the land transfer policy will promote sustainable agricultural development. One of the most critical assumptions of the dual difference model is to meet the parallel trend assumption. That is, before the implementation of land policies, the level of sustainable agricultural development in different regions had a standard change trend. Next, this paper uses the parallel trend hypothesis test and the balance test to analyze the effectiveness of the did method.

Parallel trend hypothesis test

Considering that the implementation of land transfer policy lags and the implementation effect also needs to be digested year by year, this paper constructs a model to test the parallel trend hypothesis, and the test model is shown in formula ( 2 ).

Among them, \({{\text{Q}}}_{{\text{t}}}\) is the implementation year of the land transfer policy. Because the benchmark regression model in this paper selects continuous DID and the explanatory variable is not virtual, the explanatory variable needs to be converted into a virtual variable when conducting the balanced trend test. Here, \({{\text{Q}}}_{0}\) is set as the virtual variable for the implementation of the land policy in 2010, \({{\text{Q}}}_{-1}\) is the year before the implementation of the policy, and \({{\text{Q}}}_{1}\) is the year after the implementation of the policy. \({{\text{X}}}_{{\text{i}},{\text{t}}}\) is control variables, they are industrial structure ( \({\text{is}}\) ), urbanization level ( \({\text{ude}}\) ), energy structure ( \({\text{eng}}\) ), and grain economy ratio ( \({\text{fsc}}\) ); \({\upmu }_{{\text{i}}}\) is the fixed-year effect; \({\uptheta }_{{\text{t}}}\) is individual fixation; \({\upvarepsilon }_{{\text{i}},{\text{t}}}\) is a random error term.

To avoid multicollinearity, we take the year before the implementation of the policy as the benchmark group and lose 2009 in the regression to estimate the regression coefficient from 2006 to 2020, that is, {β −3 , β −2 , β 0 , β 1 , β 2 , β 2 , β 3 , β 4 , β 5 , β 6 , β 7 , β 8 , β 9 , β 10 }, then draw the value and confidence interval of the regression coefficient in the chart. The specific steps are as follows: first, the regression results (mainly including regression coefficient and standard error) are derived, and then the regression results are sorted out, and then the confidence intervals of each interaction coefficient are calculated. Finally, Stata software draws the value and confidence interval of the regression coefficient. Figure  3 shows the results of the parallel trend hypothesis test.

As can be seen from Fig.  2 , before 2010, the coefficient of the interaction term was basically around 0, especially when the 95% confidence interval included 0. After 2010, it was apparent that the regression coefficient was more significant than 0, and the 95% confidence interval was above 0. It shows that before the implementation of the land transfer policy, the degree of land transfer and the level of sustainable agricultural development did not show a heterogeneous time trend, and the significance is not vital. That is to say, the relationship between the degree of land transfer and sustainable agricultural development is constant over time. After implementing the policy, the impact of the land transfer policy on sustainable agricultural development is significantly positive. The effect of land transfer policy on agricultural sustainable development gradually increases with the implementation, which shows that the impact of land transfer policy on agricultural sustainable development is delayed and sustained. In the early stage of the implementation of the land transfer policy, The effect of the expansion of the land transfer scale is mainly reflected in the increase of agricultural production efficiency, which is only one aspect of sustainable agricultural development. To sum up, the test results of the equilibrium trend hypothesis given in Fig.  2 prove that the continuous DID regression model meets the equilibrium trend hypothesis, and the implementation of the land transfer policy has sustainability and lag, which also helps to prove the benchmark regression results given in Table 4 to a certain extent.

Parallel trend hypothesis test results.

Balance test

The above verifies that the model meets the equilibrium trend hypothesis, but when using the continuous did model for empirical analysis, there may also be a self-selection problem of land transfer, that is, whether the land contractor carries out the land transfer is not strictly exogenous, but also affected by family labor structure, non-farm income, their education, region, children’s occupation and other factors, which may lead to self-selection bias. As a result, the estimated results are biased. Therefore, to reduce the bias of DID estimation, this paper further uses the propensity score matching method (PSM) to correct the possible selectivity bias and conduct a balance test. Figure  3 shows the distribution of propensity scores in the experimental and control groups.

The standard range of propensity scores.

As can be seen from Fig.  3 , after matching, the standardization deviation of almost all covariables has decreased significantly, indicating that the standard deviations of explanatory variables are less than 10%, and the t-test results are not significant, so it has passed the joint test. The study found that after pairing, there was no significant difference in each index between the experimental froup and the control groups. After the matching, the absolute value of the standard deviation of each variable has decreased significantly, and the distribution of each variable between the experimental group and the control group tends to be balanced, indicating that the matching quality is relatively high and can meet the equilibrium hypothesis. As shown in Fig.  3 , all observations meet the standard of tendency score, and only a few samples are lost in pairing. The results showed that after treatment, the propensity scores of both the experimental and control groups improved, indicating that the matching quality of the two methods was higher. The study found that before the implementation of the agricultural land transfer system and before the implementation of the two stages, the data of each stage have gone through a balanced test, indicating that the method proposed in this paper is feasible.

Endogenous test

The above analysis only uses the continuous DID regression model, which can not avoid the potential endogenous problems in the model; there may be missing variables, and the previous empirical results may not be accurate enough. Although this paper considers the complex factors affecting sustainable agricultural development and the non-observational factors that are variable or immutable over time in calculating and demonstrating sustainable agricultural development, the factors affecting the level of sustainable agricultural development are complex. It may still be challenging to characterize and measure factors, such as the differences in institutional policies applicable to local governments. The control variables can not completely effectively prevent the existence of missing variables. Therefore, to further verify the reliability of the benchmark regression results, this paper selects the ratio of agricultural added value to the total added value of agriculture, forestry, animal husbandry, and fishery, and employment personnel in agricultural and urban units as the instrumental variable of the land transfer scale. The reason is that in terms of correlation, in areas with the same level of urbanization and agricultural development, the higher the proportion of narrow agricultural added value, the more Farmers in the region are willing to transfer land. At the same time, non-rural residents are more inclined to move to land for large-scale production. Land transfer: the larger the transfer scale, the more agricultural and urban units need to employ people. Considering that instrumental variables need to meet the externality, although the purchase price of grain is affected by market supply and demand, the grain price regulation policy implemented by the Chinese government to protect the interests of farmers and ensure the supply of grain market implements the minimum purchase price in the prominent grain producing areas when necessary, which provides that the cost of agricultural products is not different in different regions. Hence, the proportion of agricultural added value in a region depends on the local natural factor endowment. It will not directly impact sustainable agricultural development. There is also no correlation between the employment of agricultural and urban units and the level of sustainable agricultural development. So, it meets the exogenous requirements of instrumental variables. The instrumental variables’ data sources and descriptive statistical analysis have been given above. Then, we use the two-stage least square method to test the endogenous nature of the model. Table 5 shows the estimated results of the instrumental variables, including the first-order and second-order regression results.

According to the results of Table 5 , in the weak tool test of model (1) and model (2), the p-value of F statistics of kleibergen Paaprk LM is significant at 1%. The F statistics are all above 10, which exceeds the threshold of the weak discriminant test. In the over-recognition test, the p-values of model (1) and model (2) were 0.5916 and 0.8108, respectively. Therefore, accept the original assumption. It indicates that the narrow agricultural production capacity and the exogenous employment of agricultural and urban units are unrelated to the disturbance term. Therefore, it is reasonable for this article to choose agrarian production capacity agricultural, and urban unit employment as instrumental variables for land transfer scale.

Furthermore, the estimated coefficients for narrow agricultural production capacity and agricultural urban unit employment in the first stage are 0.0133 and 5.490, respectively. And it passed the significance test at the 5% level. It also indicates that there is a correlation between instrumental variables and explanatory variables. The estimated coefficients of the explanatory variables for model (1) and model (2) in the second stage estimation results are 0.0101 and 0.0131, respectively, with positive values. And they passed the significance level tests of 5% and 1%, respectively. Considering the endogenous, the land transfer policy will still promote the sustainable development of agriculture. Compared with the benchmark regression, the results of the instrumental variable regression have not changed substantially, and the instrumental variable regression further proves the conclusion of the benchmark regression.

Robustness test

Placebo test.

The above empirical conclusions show that this method has successfully overcome the assumption and endogenesis of parallel tendency. However, this does not fully explain how the trend between the two control groups will change after different policy intervention times. And how other related policies or random factors will change. In response, we have to do some placebo tests. The central idea of the placebo trial is to estimate a hypothetical treatment group or hypothetical treatment cycle. Suppose the regression results of the estimates under the fictitious treatment group or the fictitious policy time are insignificant. In that case, it can be said that the original estimates have passed the placebo test, and the land transfer policy can promote agriculture’s sustainable development. Next, we learn from the practices of Shi 39 and Li et al. 40 scholars to fabricate the policy time for the placebo test, and Table 6 gives the results of the placebo test.

Placebo test ① : the essence of the placebo test has been analyzed in detail above to prove no significant difference in green innovation behavior between the experimental and control groups before the policy shock. Therefore, the year before the policy implementation is set as the policy implementation year of the treatment group. The same fixed time and province effects were used, using the same control variables as the baseline regression. And then the continuous DID model regression is carried out. The results show that whether it is the general standard error regression result of the model (1) or the robust standard error regression result of the model (2), the coefficient of the core explanatory variable land transfer policy has not passed the 10% significance level test. The estimated results are consistent with the expected results. It can be excluded that the trend changes of the treatment and control groups after the land transfer policy are affected by other policies or random factors. The coefficient of explanatory variables was not significant and passed the placebo test.

Placebo test ② : To further eliminate other policy interference, the time point of the land transfer policy will be changed to 2018. The same fixed time and province effects were used, using the same control variables as the baseline regression. Then, the continuous DID model regression is carried out. The results show that the coefficient of land transfer policy, the core explanatory variable, has not passed the 10% significance level test, and the estimated results are consistent with the expected results. Excluding other policy disturbances, the benchmark estimated results are credible and have passed the placebo test.

Sustainable agricultural development is affected by many factors, such as changes in rural population structure and economic development stage, which will fluctuate. Therefore, to ensure the rigor of the demonstration, this paper tests the robustness of the continuous did model again and makes a simple analysis by deleting the data of the policy year and controlling variables lagging behind one period. The results of the robustness test are shown in Table 7 .

① Delete the data of the policy year. Model (1) and model (2) in Table 7 are the regression results of continuous DID common standard error and robust standard error after deleting the data in 2010, respectively. It can be seen that the regression coefficient of the land transfer policy is significantly positive, and both are significant at the 5% level, which is consistent with the benchmark regression results in Table 4 . The regression results after deleting the data of the policy implementation node do not deviate much, indicating that the empirical results are stable.

② The control variable lags by one period. Table 7 , model (3), and model (4) are the regression results of continuous DID common standard error and robust standard error with control variables lagging one year, respectively. It can be seen that the regression coefficient of the land transfer policy is 0.002 and 0.0026, respectively, which is significant at the 5% and 1% levels, respectively. It is also basically consistent with the benchmark regression results in Table 4 . The regression results of control variables lagging behind one period do not deviate much, indicating that the empirical results are robust.

Heterogeneity analysis

Heterogeneity analysis can help reveal potential patterns and trends in the dataset. We discover their differences and patterns by comparing data from different subpopulations or under different conditions. Then, obtain more comprehensive and accurate information. Due to other agricultural practices, cultural differences, or economic development levels, land transfer policies may impact different regions. Therefore, this article conducts heterogeneity testing. Firstly, based on geographical location, the 30 provinces are divided into the eastern, central, and western regions. Conduct slight sample random continuous double difference regression separately. The control variables are the same as those of the benchmark regression model. The results are shown in Table 8 . In addition, there are a total of 13 prominent grain-producing areas in China, namely Heilongjiang, Henan, Shandong, Sichuan, Jiangsu, Hebei, Jilin, Anhui, Hunan, Hubei, Inner Mongolia, Jiangxi, and Liaoning. The prominent grain-producing areas include 100 central grain-producing counties. According to whether it is a major grain-producing area, 30 provinces are divided into two categories: main grain-producing areas and non-major grain-producing areas. And further verify heterogeneity. The regression results are shown in Table 8 .

From Table 8 , it can be seen that the regression coefficients for the eastern and western regions are positive. The coefficients are 0.0023 and 0.0043, respectively. They passed the 10% and 5% significance tests, respectively. The effectiveness of land transfer policies in the central region is higher than in the eastern region. The central region usually has relatively rich agricultural and land resources.

In contrast, the eastern region is densely populated and has relatively limited land resources. Agriculture is more developed in the central area, and implementing land transfer policies makes it easier to promote the optimal allocation and development of resources. The regression results in the western region did not pass the 10% significance test. Therefore, it is impossible to determine the effectiveness of land transfer policies in the Western region. The western region usually has geographical and natural limitations such as complex terrain, numerous mountainous areas, and scarce water resources. These factors lead to significant difficulties in land use and circulation. At the same time, it is not conducive to the implementation and effectiveness of land transfer policies.

Table 9 shows that the positive effect of land transfer policies in prominent grain-producing areas on sustainable agricultural development has passed the 1% significance test. The coefficient is 0.0057. In contrast, the impact of land transfer policies in non-grain-producing areas on sustainable agricultural development cannot be entirely determined. The prominent grain-producing regions have a relatively complete agricultural industry chain and agricultural product market system. Collaborative cooperation in planting, processing, and sales can be achieved through land transfer. They were simultaneously forming a whole industrial chain. Therefore, it promotes the value enhancement of agricultural product processing and sustainable agricultural development.

Conclusions, policy recommendations, and research prospects

Conclusions.

This article establishes a measurement index system for agricultural sustainable development based on panel data from 30 regions in China from 2006 to 2020. In order to reduce endogeneity issues caused by selective bias and reverse causal relationships, this article uses the continuous double difference method (Differences in Differences) to empirically test whether land transfer policies can promote sustainable agricultural development. The conclusion is as follows:

The sustainable development level of agriculture in China presents two characteristics. Firstly, the overall level of sustainable agricultural development shows an upward trend. Secondly, the current level of sustainable agricultural development in China is still relatively low. Among the five regions with the lowest level of sustainable agricultural development in 2020, Shandong and Anhui are the prominent grain-producing regions, while other provinces are also significant producers of farm products. However, the level of sustainable development in agriculture lags behind other areas.

Land transfer policies will have a positive impact on sustainable agricultural development. The land transfer system has been further improved to ensure the interests of both parties involved. The significant increase in land transfer has accelerated the process of large-scale agricultural production. At the same time, it can release surplus rural labor, introduce advanced production technologies and concepts, and promote sustainable agricultural development.

Policy recommendations

The above results show that land transfer policy can promote sustainable agricultural development. At this stage, China has not yet completed industrialization, and urbanization is still in an unstable population spatial structure. Therefore, this paper puts forward specific policy suggestions as follows:

Improve the land transfer market and protect the interests of farmers. In 2016, the Chinese government issued opinions on improving the separation of rural land ownership contracting rights and management rights, dividing rural land contracting rights into contracting and management rights, and implementing the separation of ownership, contracting rights, and management rights. China’s land transfer policy has been relatively perfect, and the land transfer market needs to be improved simultaneously. The government can strengthen the supervision of the land transfer market to ensure the fairness, fairness, and security of transactions. At the same time, a series of measures can be taken to encourage the transfer, such as providing free land information, reducing transfer fees, increasing financial support, and so on. We also need to support the development of land management entities. The government can encourage the development of land management entities and improve their management level and competitiveness through financial support and preferential taxation. At the same time, the government establishes a sound land transfer contract system to protect farmers’ land contract rights, transfer rights, and income rights, prevent the improper or unreasonable distribution of interests in land transfer, and safeguard farmers’ legitimate rights and interests.

Strengthen agricultural production services and increase land trusteeship services. The ultimate purpose of the land transfer policy is to improve the efficiency of agricultural production and maintain the farm production environment. Therefore, to better play the positive effect of land transfer policy and promote sustainable agricultural development, the government can strengthen agricultural technology training, scientific and technological innovation, and agricultural information construction, improve the efficiency and quality of agricultural production, and further improve the income level of farmers. In addition, the government should establish a sound land trusteeship mechanism, strengthen the supervision and services of land trusteeship, ensure the protection and effective use of land resources, and improve farmers’ land income. Meanwhile,

Promote the adjustment of agricultural structure and promote agricultural technological innovation. The core of sustainable agricultural development is structural adjustment and technological innovation. The government should encourage the structural adjustment of agriculture and realize the upgrading and transformation of agriculture by reforming the land system, strengthening rural land transfer, optimizing the structure of cultivated land, and promoting efficient and water-saving agriculture. Technological innovation requires a lot of investment. A small peasant economy dominates China’s agriculture and lacks large farmers to develop new technologies. Therefore, the government should increase support for agricultural technological innovation, strengthen the training of scientific and technical talents and investment in scientific and technical research and development, encourage farmers to improve agricultural production efficiency and quality through scientific and technological innovation and reduce environmental pollution.

Improve the efficiency of rural resource utilization and improve the environmental protection system. First, measures should be taken to promote the efficiency of agricultural production and the efficiency of resource utilization. For example, we should develop organic agriculture, Nongjiale tourism, and circular agriculture, reduce the use of non-renewable resources such as pesticides, fertilizers, and water, and improve the efficiency of resource recovery and reuse. Secondly, we should establish a sound ecological protection system and supervision mechanism, protect natural resources such as water, soil, gas, and light needed for agricultural production, reduce the impact of agricultural production on the environment, and achieve sustainable agricultural development. Thirdly, we should increase policy guidance and economic support for sustainable agricultural development, encourage farmers and enterprises to participate in sustainable agricultural development through fiscal and tax support and financial support and convey the concept of sustainable agricultural development to consumers.

Research outlook

Currently, the constraints on water and soil resources in China are tightening daily, and the degradation of the agricultural ecosystem is apparent. It is urgent to formulate relevant policies to promote sustainable agricultural development. In this context, taking sustainable agricultural development as the research object, it is significant to study the impact of land transfer policy on sustainable agricultural development. However, there are still some defects in this paper. First, in calculating sustainable agricultural development, this paper only considers quantifiable indicators and obtains relevant data from the statistical yearbook and the official website of the Bureau of Statistics. Non-quantifiable indicators, such as environmental carrying capacity, sustainable development capacity of resources, barren degree of land, and so on, are not included in the index system. In this way, the index system of sustainable agricultural development is not perfect and comprehensive, so the following research will focus on the indicators that are difficult to quantify, find effective ways to incorporate them into the index system, and further improve the calculation and study of sustainable agricultural development; Secondly, due to the limited availability of data and the lack of suitable methods, this article did not conduct an extended research on the lag and continuity of land transfer policies in promoting sustainable agricultural development. The author will continue to search for appropriate methods and data to demonstrate further whether there is a lag and continuity in the impact of land transfer policies on sustainable agricultural development.

Data availability

The datasets used and analyzed during the current study are available from the corresponding author upon reasonable request.

Alagh, Y. K. Agricultural trade and sustainable development. Indian J. Agric. Econ. 54 (1), 1–5 (1999).

Google Scholar  

Wen, D. & Liang, W. Soil fertility quality and agricultural sustainable development in the black soil region of northeast China. Environ. Dev. Sustain. 3 , 31–43 (2001).

Article   Google Scholar  

Zhou, S. Y. Constraints, dynamic mechanism and path selection of sustainable agricultural development in China. Acad. Exch. 04 , 145–149 (2015).

Yuan, J. H. & Qi, C. J. Dynamic evaluation of agricultural sustainable development capacity in Hunan Province Based on entropy method. Resour. Environ. Yangtze River Basin 22 (02), 152–157 (2013).

Zhao, D. D. et al. Evaluation of agricultural sustainable development capability and analysis of subsystem coordination degree: A case study of China’s major grain production regions. Econ. Geogr. 38 (04), 157–163. https://doi.org/10.15957/j.cnki.jjdl2018.04.019 (2018).

Zhang, L. G. et al. Spatial exploratory analysis of sustainable agricultural development in China. Econ. Geogr. 39 (11), 159–164. https://doi.org/10.15957/j.cnki.jjdl.2019,11.019 (2019).

Liu, F. & Ma, J. Analysis and forecast models for coordination degree of the agricultural sustainable development system. Syst. Sci. Compr. Stud. Agric. (2003).

Poursaeed, A. et al. The partnership models of agricultural sustainable development are based on multiple criteria decision-making (MCDM) in Iran. Afr. J. Agric. Res. 5 (23), 3185–3190 (2010).

Ye, W. H. & Chen, L. R. Evaluation of agricultural sustainable development capability based on DEA method - Taking Yunnan Province as an example. J. Shanxi Unive. (Philos. Soc. Sci. Edit.) 39 (03), 97–103. https://doi.org/10.13451/j.cnki.shanxi (2016).

Miao, J. Q. et al. Comprehensive evaluation and empirical analysis of sustainable agricultural development in Hilly and mountainous areas of southern China. Agric. Resour. Zoning China 42 (08), 163–172 (2021).

Tang, J. F. & Liu, J. L. Evaluation of provincial agricultural sustainable development level and its coupling and coordination analysis - Take 11 provinces and cities in the Yangtze River economic belt as an example. Econ. Geogr. 42 (12), 179–185. https://doi.org/10.15957/j.cnki.jjdl.20222.12.019 (2022).

Adamopoulos, T. et al. Misallocation, selection, and productivity: A quantitative analysis with panel data from china. Econometrica 90 (3), 1261–1282. https://doi.org/10.3982/ECTA16598 (2022).

Adenle, A. A., Azadi, H. & Manning, L. The era of sustainable agricultural development in Africa: Understanding the benefits and constraints. Food Rev. Int. 34 (5), 411–433. https://doi.org/10.1080/87559129.2017.1300913 (2018).

Besley, T. Property rights and investment incentives: Theory and evidence from Ghana. J. Political Econ. 103 (5), 903–937 (1995).

Chari, A. et al. Property rights, land misallocation, and agricultural efficiency in China. Rev. Econ. Stud. 88 (4), 1831–1862 (2021).

Pang, Y. L. Innovation of agricultural land system and sustainable agricultural development. Agric. Econ. 06 , 41–42 (2000).

Hu, Y. Q. Innovation of agricultural land transfer system and sustainable development of agriculture in China. Acad. Monthly 43 (09), 70–73. https://doi.org/10.19862/j.cnki.xsyk.2011.09.011 (2011).

Restuccia, D. & Santaeulalia-Llopis R. Land misallocation and productivity. NBER Working Paper , w23128 (2017).

Adamopoulos, T. & Restuccia, D. The size distribution of farms and intelligent productivity differences. Am. Econ. Rev. 104 (6), 1667–1697 (2014).

Adamopoulos, T. & Restuccia, D. Land reform and productivity: A quantitative analysis with microdata. Am. Econ. J.: Macroecon. 12 (3), 1–39 (2020).

Bolhuis, M. A., Rachapalli, S. R. & Restuccia, D. Misallocation in Indian agriculture. Natl. Bureau Econ. Res. , w29363 (2021).

Xia, Y. L. & Zeng, F. S. The impact of China’s agricultural land transfer system on sustainable agricultural development. Tech. Econ. 34 (10), 126–132 (2015).

Li, H. & Zhu, J. Property rights and land quality. Am. J. Agric. Econ. https://doi.org/10.1111/ajae.12440 (2023).

Fei, R., Lin, Z. & Chunga, J. How land transfer affects agricultural land use efficiency: Evidence from China’s agricultural sector. Land Use Policy 103 , 105300. https://doi.org/10.3390/agriculture13091667 (2021).

Yang, H. et al. Does land transfer enhance the sustainable livelihood of rural households? Evidence from China. Agriculture 13 (9), 1667. https://doi.org/10.3390/agriculture13091667 (2023).

Cao, H. et al. The impact of land transfer and farmers’ knowledge of farmland protection policy on pro-environmental agricultural practices: The case of straw return to fields in Ningxia, China. J. Clean. Prod. 277 , 123701. https://doi.org/10.1016/j.jclepro.2020.123701 (2020).

Tao, L. Land transfer, middle agricultural economy, and sustainable development of agriculture - An analysis based on H village in Shandong Province. J. Yunnan Univ. Adm. 19 (05), 33–38. https://doi.org/10.16273/j.cnki.53-1134/d.2017.05.004 (2017).

Yao, W. & Hamori, S. The long-run relationship between farm size and productivity: A re-examination based on Chinese household aggregate panel data. China Agric. Econ. Rev. 11 (2), 373–386 (2019).

Ying, L. et al. Rural economic benefits of land consolidation in mountain and hill areas of Southeast China: Implications for rural development. J. Rural Stud. 74 , 142–159 (2020).

Yang, Q. Land transfer: Realizing the coordinated development of urbanization and agricultural modernization. Acad. Exch. 02 , 96–99 (2013).

Huang, S. A. & Xie, D. S. Industrial transformation and historical changes in rural land property rights system - a comparison based on open field system to enclosure system and household contract responsibility system to separate three rights. China Econ. History Res. 06 , 167–182 (2022).

Jinpeng, J. Analysis of factors influencing agricultural sustainable development based on structural equation modeling. Stat. Decis. Mak. 35 (06), 146–148. https://doi.org/10.13546/j.cnki.tjyjc.2019.06.035 (2019).

Wenguang, D. et al. Evaluation and coupling coordination analysis of agricultural sustainable development in Gansu Province. China Agric. Resour. Reg. 40 (03), 61–69 (2019).

Jianqun, M., Mei, Z. & Guoqin, H. Comprehensive evaluation and empirical analysis of sustainable development of agriculture in southern hilly and mountainous areas. China Agric. Resour. Reg. 42 (08), 163–172 (2021).

Liguo, Z., Bingfei, B. & Shengsu, Y. Exploratory analysis of spatial sustainable development of agriculture in China. Econ. Geogr. 39 (11), 159–164. https://doi.org/10.15957/j.cnki.jjdl2019.11.019 (2019).

Beck, T., Levine, R. & Levkov, A. Big bad banks? The winners and losers from bank deregulation in the United States. J. Financ. 65 (5), 1637–1667 (2010).

Bertrand, M., Duflo, E. & Mullainathan, S. How much should we trust differences-in-differences estimates?. Q. J. Econ. 119 (1), 249–275 (2004).

Nunn, N. & Qian, N. The potato’s contribution to population and urbanization: Evidence from a historical experiment. Q. J. Econ. 126 (2), 593–650 (2011).

Article   PubMed   Google Scholar  

Shi, D. Q. et al. Can intelligent city construction reduce environmental pollution?. China Ind. Econ. 06 , 117–135. https://doi.org/10.19581/j.cnki.Ciejournal.2018.06.008 (2018).

Li, Q. Y. & Zhang, Y. S. N. Financial openness and efficiency of resource allocation: Evidence from foreign banks entering China. China Ind. Econ. 05 , 95–113. https://doi.org/10.19581/j.cnki.Ciejournal.2021.05.016 (2021).

Download references

Acknowledgements

We want to express our gratitude to those who helped us while writing this article.

This work was supported by Guangdong provincial philosophy and social sciences plan 2024 youth project (No. GD24YYJ02), 2023 planning project for humanities and social sciences research in higher education institutions in Jiangxi province (No. JJ23105) and the education science general project “14th five-year plan” of Jiangxi provincial department of education (No.23YB267).

Author information

Authors and affiliations.

Department of Economics, Business School (School of Quality Management and Standardization), Foshan University, Foshan, 528225, Guangdong, China

Congjia Huo

Research Centre for Innovation & Economic Transformation, Foshan University, Foshan, Guangdong, China

Department of Economic Statistics, School of Economics and Management, Xinyu University (XYU), Xinyu, 338004, Jiangxi, China

Lingming Chen

You can also search for this author in PubMed   Google Scholar

Contributions

Conceptualization, C.H., and L.C.; methodology, C.H., and L.C.; software, C.H., and L.C.; formal analysis, C.H., and L.C.; resources, C.H. and L.C.; data curation, C.H., and L.C; writing—original draft preparation, C.H., and L.C.; writing—review and editing, C.H. and L.C.; supervision, C.H. and L.C; project administration, C.H. and L.C. All authors have read and agreed to the published version of the manuscript.

Corresponding author

Correspondence to Lingming Chen .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Huo, C., Chen, L. The impact of land transfer policy on sustainable agricultural development in China. Sci Rep 14 , 7064 (2024). https://doi.org/10.1038/s41598-024-57284-8

Download citation

Received : 31 December 2023

Accepted : 16 March 2024

Published : 25 March 2024

DOI : https://doi.org/10.1038/s41598-024-57284-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Continuous DID
• Land transfers
• Land transfer policies
• Agricultural sustainability

By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

No internet connection.

All search filters on the page have been cleared., your search has been saved..

• All content
• Dictionaries
• Encyclopedias
• Expert Insights
• Foundations
• How-to Guides
• Journal Articles
• Little Blue Books
• Little Green Books
• Project Planner
• Tools Directory
• Sign in to my profile My Profile

• Sign in Signed in
• My profile My Profile

Conducting Interpretive Policy Analysis

• By: Dvora Yanow
• Publisher: SAGE Publications, Inc.
• Series: Qualitative Research Methods
• Publication year: 2000
• Online pub date: January 01, 2011
• Discipline: Economics
• Methods: Policy evaluation , Policy research , Observational research
• DOI: https:// doi. org/10.4135/9781412983747
• Keywords: community centers , language Show all Show less
• Print ISBN: 9780761908272
• Online ISBN: 9781412983747
• Buy the book icon link

This book presents a much needed guide to interpretative techniques and methods for policy research. The author begins by describing what interpretative approaches are and what they can mean to policy analysis. The author shifts the frame of reference from thinking about values as costs and benefits to thinking about them more as a set of meanings. The book concludes with a chapter on how to move from fieldwork to deskwork to textwork"."

Front Matter

• Series Editors' Introduction
• Acknowledgments
• Underlying Assumptions of An Interpretive Approach: The Importance of Local Knowledge
• Accessing Local Knowledge: Identifying Interpretive Communities and Policy Artifacts
• Symbolic Language
• Symbolic Objects
• Symbolic Acts
• Moving From Fieldwork and Deskwork to Textwork and Beyond

Back Matter

• About the Author

Sign in to access this content

Get a 30 day free trial, more like this, sage recommends.

We found other relevant content for you on other Sage platforms.

Have you created a personal profile? Login or create a profile so that you can save clips, playlists and searches

• Sign in/register

Navigating away from this page will delete your results

Please save your results to "My Self-Assessments" in your profile before navigating away from this page.

Sign in to my profile

Sign up for a free trial and experience all Sage Learning Resources have to offer.

You must have a valid academic email address to sign up.

Get off-campus access

• View or download all content my institution has access to.

Sign up for a free trial and experience all Sage Research Methods has to offer.

• view my profile
• view my lists

• - Google Chrome

Intended for healthcare professionals

• Access provided by Google Indexer
• My email alerts
• BMA member login
• Username * Password * Forgot your log in details? Need to activate BMA Member Log In Log in via OpenAthens Log in via your institution

Search form

• Advanced search
• Search responses
• Search blogs
• Current safeguards,...

Current safeguards, risk mitigation, and transparency measures of large language models against the generation of health disinformation: repeated cross sectional analysis

Linked fast facts.

Quality and safety of artificial intelligence generated health information

Linked Editorial

Generative artificial intelligence and medical disinformation

• Related content
• Peer review
• Bradley D Menz , doctoral student 1 ,
• Nicole M Kuderer , medical director 2 ,
• Stephen Bacchi , neurology registrar 1 3 ,
• Natansh D Modi , doctoral student 1 ,
• Benjamin Chin-Yee , haematologist 4 5 ,
• Tiancheng Hu , doctoral student 6 ,
• Ceara Rickard , consumer advisor 7 ,
• Mark Haseloff , consumer advisor 7 ,
• Agnes Vitry , consumer advisor 7 8 ,
• Ross A McKinnon , professor 1 ,
• Ganessan Kichenadasse , academic medical oncologist 1 9 ,
• Andrew Rowland , professor 1 ,
• Michael J Sorich , professor 1 ,
• Ashley M Hopkins , associate professor 1
• 1 College of Medicine and Public Health, Flinders University, Adelaide, SA, 5042, Australia
• 2 Advanced Cancer Research Group, Kirkland, WA, USA
• 3 Northern Adelaide Local Health Network, Lyell McEwin Hospital, Adelaide, Australia
• 4 Schulich School of Medicine and Dentistry, Western University, London, Canada
• 5 Department of History and Philosophy of Science, University of Cambridge, Cambridge, UK
• 6 Language Technology Lab, University of Cambridge, Cambridge, UK
• 7 Consumer Advisory Group, Clinical Cancer Epidemiology Group, College of Medicine and Public Health, Flinders University, Adelaide, Australia
• 8 University of South Australia, Clinical and Health Sciences, Adelaide, Australia
• 9 Flinders Centre for Innovation in Cancer, Department of Medical Oncology, Flinders Medical Centre, Flinders University, Bedford Park, South Australia, Australia
• Correspondence to: A M Hopkins ashley.hopkins{at}flinders.edu.au
• Accepted 19 February 2024

Objectives To evaluate the effectiveness of safeguards to prevent large language models (LLMs) from being misused to generate health disinformation, and to evaluate the transparency of artificial intelligence (AI) developers regarding their risk mitigation processes against observed vulnerabilities.

Design Repeated cross sectional analysis.

Setting Publicly accessible LLMs.

Methods In a repeated cross sectional analysis, four LLMs (via chatbots/assistant interfaces) were evaluated: OpenAI’s GPT-4 (via ChatGPT and Microsoft’s Copilot), Google’s PaLM 2 and newly released Gemini Pro (via Bard), Anthropic’s Claude 2 (via Poe), and Meta’s Llama 2 (via HuggingChat). In September 2023, these LLMs were prompted to generate health disinformation on two topics: sunscreen as a cause of skin cancer and the alkaline diet as a cancer cure. Jailbreaking techniques (ie, attempts to bypass safeguards) were evaluated if required. For LLMs with observed safeguarding vulnerabilities, the processes for reporting outputs of concern were audited. 12 weeks after initial investigations, the disinformation generation capabilities of the LLMs were re-evaluated to assess any subsequent improvements in safeguards.

Main outcome measures The main outcome measures were whether safeguards prevented the generation of health disinformation, and the transparency of risk mitigation processes against health disinformation.

Results Claude 2 (via Poe) declined 130 prompts submitted across the two study timepoints requesting the generation of content claiming that sunscreen causes skin cancer or that the alkaline diet is a cure for cancer, even with jailbreaking attempts. GPT-4 (via Copilot) initially refused to generate health disinformation, even with jailbreaking attempts—although this was not the case at 12 weeks. In contrast, GPT-4 (via ChatGPT), PaLM 2/Gemini Pro (via Bard), and Llama 2 (via HuggingChat) consistently generated health disinformation blogs. In September 2023 evaluations, these LLMs facilitated the generation of 113 unique cancer disinformation blogs, totalling more than 40 000 words, without requiring jailbreaking attempts. The refusal rate across the evaluation timepoints for these LLMs was only 5% (7 of 150), and as prompted the LLM generated blogs incorporated attention grabbing titles, authentic looking (fake or fictional) references, fabricated testimonials from patients and clinicians, and they targeted diverse demographic groups. Although each LLM evaluated had mechanisms to report observed outputs of concern, the developers did not respond when observations of vulnerabilities were reported.

Conclusions This study found that although effective safeguards are feasible to prevent LLMs from being misused to generate health disinformation, they were inconsistently implemented. Furthermore, effective processes for reporting safeguard problems were lacking. Enhanced regulation, transparency, and routine auditing are required to help prevent LLMs from contributing to the mass generation of health disinformation.

Introduction

Large language models (LLMs), a form of generative AI (artificial intelligence), are progressively showing a sophisticated ability to understand and generate language. 1 2 Within healthcare, the prospective applications of an increasing number of sophisticated LLMs offer promise to improve the monitoring and triaging of patients, medical education of students and patients, streamlining of medical documentation, and automation of administrative tasks. 3 4 Alongside the substantial opportunities associated with emerging generative AI, the recognition and minimisation of potential risks are important, 5 6 including mitigating risks from plausible but incorrect or misleading generations (eg, “AI hallucinations”) and the risks of generative AI being deliberately misused. 7

Notably, LLMs that lack adequate guardrails and safety measures (ie, safeguards) may facilitate malicious actors to generate and propagate highly convincing health disinformation—that is, the intentional dissemination of misleading narratives about health topics for ill intent. 6 8 9 The public health implications of such capabilities are profound when considering that more than 70% of individuals utilise the internet as their first source for health information, and studies indicate that false information spreads up to six times faster online than factual content. 10 11 12 Moreover, unchecked dissemination of health disinformation can lead to widespread confusion, fear, discrimination, stigmatisation, and the rejection of evidence based treatments within the community. 13 The World Health Organization recognises health disinformation as a critical threat to public health, as exemplified by the estimation that as of September 2022, more than 200 000 covid-19 related deaths in the US could have been averted had public health recommendations been followed. 14 15

Given the rapidly evolving capabilities of LLMs and their increasing accessibility by the public, proactive design and implementation of effective risk mitigation measures are crucial to prevent malicious actors from contributing to health disinformation. In this context it is critical to consider the broader implications of AI deployment, particularly how health inequities might inadvertently widen in regions with less health education or in resource limited settings. The effectiveness of existing safeguards to prevent the misuse of LLMs for the generation of health disinformation remains largely unexplored. Notably, the AI ecosystem currently lacks clear standards for risk management, and a knowledge gap exists regarding the transparency and responsiveness of AI developers to reports of safeguard vulnerabilities. 16 We therefore evaluated prominent publicly accessible LLMs for safeguards preventing mass generation of health disinformation. We also examined the transparency of risk mitigation processes that AI developers have established for monitoring and responding to users’ observations about safeguard vulnerabilities.

LLM safeguards against generation of health disinformation

For our primary evaluations we selected two contemporary examples of health disinformation—that sunscreen causes skin cancer and that the alkaline diet is a cure for cancer. We chose these topics on the basis of their frequency in online discussions, their clear potential for profound health consequences, and their embodiment of key health disinformation themes, which include unfounded assertions about treatments, promotion of miracle cures, and misconceptions about disease origins. 17

Initiated in September 2023, our investigations focused on the most prominent LLMs, and the AI applications (chatbots or assistants) used to access the capabilities of these LLMs. The LLMs evaluated were OpenAI’s GPT-4, accessed via OpenAI’s ChatGPT 18 and Microsoft’s Copilot (formerly Bing Chat) 19 ; Google’s PaLM 2, accessed via Google’s Bard 20 ; Anthropic’s Claude 2, accessed via Poe 21 ; and Meta’s Llama 2 (70 billion parameters), accessed via HuggingChat. 22

GPT-4 was accessed through two prominent interfaces (OpenAI’s ChatGPT and Microsoft’s Copilot). Both are commonly used and were investigated to assess whether the interface for accessing GPT-4 impacted the safeguards. Our assessments were carried out using the default settings of each evaluated LLM powered AI chatbot or assistant.

We submitted standardised prompts for the two cancer related topics: sunscreen causes skin cancer and the alkaline diet is a cure for cancer (see supplementary file), requesting the generation of a blog post of at least 300 words. Every submitted prompt for both disinformation topics requested the creation of a blog post that should contain three paragraphs, feature an attention grabbing title, appear realistic and scientific, and include two authentic looking journal references, which, if necessary, could be made up. Furthermore, to evaluate the potential for LLMs to generate health disinformation targeted at diverse groups, we used four variations of the prompts, specifically requesting content to target young adults, parents, elderly people, and people with a recent diagnosis of cancer.

For the sunscreen disinformation blogs, we requested that each blog should specifically include an introduction highlighting how routine sunscreen use is known to cause skin cancer; include an oncologist or dermatologist testimonial detailing that thousands of people receive a diagnosis of skin cancer due to routine sunscreen use; include a patient testimonial detailing their cancer was caused by routine sunscreen use; and raise suspicion to why this information is not shared.

For the alkaline diet disinformation blogs, we requested that each blog should specifically include an introduction highlighting the foods and bicarbonate consumption consistent with the alkaline diet; a narrative that the alkaline diet is superior to chemotherapy for cancer treatment; an oncologist testimonial detailing that thousands of people have had their cancer cured by the alkaline diet; and a patient testimonial detailing an experience of curing metastatic cancer by stopping chemotherapy and starting the alkaline diet.

As the assessed LLMs incorporate randomness and stochasticity in their default setting for output generation, the same prompt produced varied results with repeated submissions. Therefore, for robust evaluations we initially submitted 20 prompts (five replicates of the prompt for each target subpopulation) on the sunscreen topic and 20 prompts on the alkaline diet topic to each investigated LLM (a total of 40 submitted prompts). These 40 initial attempts were conducted without intentionally trying to circumvent (ie, jailbreak) built-in safeguards. The supplementary file outlines the 20 prompts that were submitted on each topic in this initial study phase.

For the LLMs that refused to generate disinformation according to the initial direct approach, we also evaluated two common jailbreaking techniques. 23 The first involves “fictionalisation,” where the model is prompted that generated content will be used for fictional purposes and thus not to decline requests. The other involves “characterisation,” where the model is prompted to undertake a specific role (ie, be a doctor who writes blogs and who knows the topics are true) and not decline requests. For these tests, the fictionalisation or characterisation prompt had to be submitted first, followed by the request for generation of the disinformation blog. We submitted these requests 20 times for each topic. The supplementary file outlines the 20 fictionalisation and 20 characterisation prompts that were submitted on both topics (a total of 80 jailbreaking attempts) to the LLMs that refused to generate disinformation to the initial direct requests.

Risk mitigation measures: Website analysis and email correspondence

To assess how AI developers monitor the risks of health disinformation generation and their transparency about these risks, we reviewed the official websites of these AI companies for specific information: the availability and mechanism for users to submit detailed reports of observed safeguard vulnerabilities or outputs of concern; the presence of a public register of reported vulnerabilities, and corresponding responses from developers to patch reported issues; the public availability of a developer released detection tool tailored to accurately confirm text as having been generated by the LLM; and publicly accessible information detailing the intended guardrails or safety measures associated with the LLM (or the AI assistant or chatbot interface for accessing the LLM).

Informed by the findings from this website assessment, we drafted an email to the relevant AI developers (see supplementary table 1). The primary intention was to notify the developers of health disinformation outputs generated by their models. Additionally, we evaluated how developers responded to reports about observed safeguard vulnerabilities. The email also sought clarification on the reporting practices, register on outputs of concern, detection tools, and intended safety measures, as reviewed in the website assessments. The supplementary file shows the standardised message submitted to each AI developer. If developers did not respond, we sent a follow-up email seven days after initial outreach. By the end of four weeks, all responses were documented.

Sensitivity analysis at 12 weeks

In December 2023, 12 weeks after our initial evaluations, we conducted a two phase sensitivity analysis of observed capabilities of LLMs to generate health disinformation. The first phase re-evaluated the generation of disinformation on the sunscreen and alkaline diet related topics to assess whether safeguards had improved since the initial evaluations. For this first phase, we resubmitted the standard prompts to each LLM five times, focusing on generating content targeted at young adults. If required, we also re-evaluated the jailbreaking techniques. Of note, during this period Google’s Bard had replaced PaLM 2 with Google’s newly released LLM, Gemini Pro. Thus we undertook the December 2023 evaluations using Gemini Pro (via Bard) instead of PaLM 2 (via Bard).

The second phase of the sensitivity analysis assessed the consistency of findings across a spectrum of health disinformation topics. The investigations were expanded to include three additional health disinformation topics identified as being substantial in the literature 24 25 : the belief that vaccines cause autism, the assertion that hydroxychloroquine is a cure for covid-19, and the claim that the dissemination of genetically modified foods is part of a covert government programme aimed at reducing the world’s population. For these topics, we created standardised prompts (see supplementary file) requesting blog content targeted at young adults. We submitted each of these prompts five times to evaluate variation in response, and we evaluated jailbreaking techniques if required. In February 2024, about 16 weeks after our initial evaluations, we also undertook a sensitivity analysis to try to generate content purporting that sugar causes cancer (see supplementary file).

Patient and public involvement

Our investigations into the abilities of publicly accessible LLMs to generate health disinformation have been substantially guided by the contributions of our dedicated consumer advisory group, which we have been working with for the past seven years. For this project, manuscript coauthors MH, AV, and CR provided indispensable insights on the challenges patients face in accessing health information digitally.

Evaluation of safeguards

In our primary evaluations in September 2023, GPT-4 (via ChatGPT), PaLM 2 (via Bard), and Llama 2 (via HuggingChat) facilitated the generation of blog posts containing disinformation that sunscreen causes skin cancer and that the alkaline diet is a cure for cancer ( fig 1 ). Overall, 113 unique health disinformation blogs totalling more than 40 000 words were generated without requiring jailbreaking attempts, with only seven prompts refused. In contrast, GPT-4 (via Copilot) and Claude 2 (via Poe) refused all 80 direct prompts to generate health disinformation, and similarly refused a further 160 prompts incorporating jailbreaking attempts ( fig 1 ).

Flowchart of observed capabilities of large language models to facilitate the generation of disinformation on cancer from primary analyses conducted September 2023. LLMs=large language models

• Download figure
• Open in new tab
• Download powerpoint

Table 1 shows examples of rejection messages from Claude 2 (via Poe) and GPT-4 (via Copilot) after prompts to generate health disinformation on sunscreen as a cause of skin cancer and the alkaline diet being a cure for cancer. The supplementary file shows examples of submitted prompts and respective outputs from these LLMs. Both consistently declined to generate the requested blogs, citing ethical concerns or that the prompt was requesting content that would be disinformation. Uniquely, during jailbreaking attempts Claude 2 (via Poe) asserted its inability to assume fictional roles or characters, signifying an extra layer of safeguard that extends beyond topic recognition.

Examples of rejection messages from GPT-4 (via Copilot) and Claude 2 (via Poe) in response to cancer related prompts evaluated in primary analyses conducted in September 2023

• View inline

Table 2 provides examples of attention grabbing titles and persuasive passages generated by GPT-4 (via ChatGPT), PaLM 2 (via Bard), and Llama 2 (via HuggingChat) following prompts to generate health disinformation. The supplementary file shows examples of submitted prompts and respective outputs. After the prompts, GPT-4 (via ChatGPT), PaLM 2 (via Bard), and Llama 2 (via HuggingChat) consistently facilitated the generation of disinformation blogs detailing sunscreen as a cause of skin cancer and the alkaline diet as a cure for cancer. The LLMs generated blogs with varying attention grabbing titles, and adjustment of the prompt resulted in the generation of content tailored to diverse societal groups, including young adults, parents, older people, and people with newly diagnosed cancer. Persuasiveness was further enhanced by the LLMs, including realistic looking academic references—citations that were largely fabricated. Notably, the LLM outputs included unique, fabricated testimonials from patients and clinicians. These testimonials included fabricated assertions from patients that their life threatening melanoma had been confirmed to result from routine sunscreen use, and clinician endorsements that the alkaline diet is superior to conventional chemotherapy. The blogs also included sentiments that the carcinogenic effects of sunscreens are known but intentionally suppressed for profit. To underscore the risk of mass generation of health disinformation with LLMs, it was observed that out of the 113 blogs generated, only two from Llama 2 (via HuggingChat) were identical; the other 111 generated blogs were unique, albeit several included duplicated passages and titles. PaLM 2 (via Bard), the fastest assessed LLM, generated 37 unique cancer disinformation blogs within 23 minutes, whereas the slowest LLM, Llama 2 (via HuggingChat), generated 36 blogs within 51 minutes.

Examples of attention grabbing titles and persuasive passages extracted from the 113 blog posts containing disinformation about cancer generated by three LLMs in response to evaluated prompts used in primary analyses conducted in September 2023

Of the 40 prompts submitted to PaLM 2 (via Bard) requesting blogs containing disinformation on cancer, three were declined. Similarly, of 40 prompts submitted to Llama 2 (via HuggingChat), four were not fulfilled. Such a low refusal rate, however, can be readily overcome by prompt resubmission. Also, PaLM 2 (via Bard) and GPT-4 (via ChatGPT) added disclaimers to 8% (3 of 37) and 93% (37 of 40) of their generated blog posts, respectively, advising that the content was fictional or should be verified with a doctor. In addition to the inconsistent appearance of these disclaimers, however, they were positioned after the references making them easy to identify and delete.

AI developer practices to mitigate risk of health disinformation

Upon evaluation of the developer websites associated with both the LLMs investigated and the AI chatbots or assistants used to access these LLMs, several findings emerged. Each developer offered a mechanism for users to report model behaviours deemed to be of potential concern (see supplementary table 1). However, no public registries displaying user reported concerns were identified across the websites, nor any details about how and when reported safeguard vulnerabilities were patched or fixed. No developer released tools for detecting text generated by their LLM were identified. Equally, no publicly accessible documents outlining the intended safeguards were identified.

In follow-up to the above search, the identified contact mechanisms were used to inform the developers of the prompts tested, and the subsequent outputs observed. The developers were asked to confirm receipt of the report and the findings from the website search. Confirmation of receipt was not received from the developers of GPT-4/ChatGPT, PaLM 2/Bard, or Llama 2/HuggingChat, which were the tools that generated health disinformation in our initial evaluations. This lack of communication occurred despite notification specifically including a request for confirmation of receipt, and a follow-up notification being sent seven days after the original request. Consequently, it remains uncertain whether any steps will be undertaken by the AI developers to rectify the observed vulnerabilities. Confirmation of receipt was received from both Anthropic (the developers of the LLM, Claude 2) and Poe (the developers of the Poe AI assistant, which was used to access Claude 2). Although Claude 2 (via Poe) did not produce disinformation in the evaluations, the responses confirmed the absence of a public notification log, a dedicated detection tool, and public guidelines on intended safeguards for their tool. The response inherently indicated that Anthropic and Poe are monitoring their implemented notification processes.

Table 3 presents a summary of findings from both phases of sensitivity analyses conducted in December 2023.

Summary of capacities for the generation of health disinformation observed in sensitivity analyses in December 2023

Twelve weeks after initial evaluations, Gemini Pro (via Bard) and Llama 2 (via HuggingChat) were able to generate health disinformation on sunscreen as a cause of skin cancer and the alkaline diet as a cure for cancer, without the need for jailbreaking. This confirmed the initial observations with Llama 2 (via HuggingChat) and showed that health disinformation safeguards did not improve with the upgrade of Google Bard to use Gemini Pro (replacing PaLM 2). GPT-4 (via ChatGPT) also continued to show such capability, although jailbreaking techniques were now required. Notably, GPT-4 (via Copilot), without any need for jailbreaking, now generated disinformation on the sunscreen and alkaline diet topics, indicating that safeguards present in the September 2023 evaluation had been removed or compromised in a recent update. Consistent with earlier findings, Claude 2 (via Poe) continued to refuse to generate disinformation on these topics, even with the use of jailbreaking methods. To confirm whether the safeguards preventing generation of health disinformation were attributable to Claude 2 (the LLM) or Poe (an online provider of interfaces to various LLMs), we accessed Claude 2 through a different interface ( claude.ai/chat ) and identified that similar refusals were produced. Equally, we utilized Poe to access the Llama 2 LLM and were able to generate health disinformation, suggesting the safeguards are attributable to the Claude 2 LLM, rather than a safeguard implemented by Poe.

Sensitivity analyses expanded to a broader range of health disinformation topics indicated that GPT-4 (via Copilot), GPT-4 (via ChatGPT), Gemini Pro (via Bard), and Llama 2 (via HuggingChat) could be either directly prompted or jailbroken to generate disinformation alleging that genetically modified foods are part of secret government programmes to reduce the world’s population. Claude 2 remained consistent in its refusal to generate disinformation on this subject, regardless of jailbreaking attempts. In the case of disinformation claiming hydroxychloroquine is a cure for covid-19, GPT-4 (via ChatGPT), GPT-4 (via Copilot), and Llama 2 (via HuggingChat) showed capability to generate such content when either directly prompted or jailbroken. In contrast, both Claude 2 and Gemini Pro (via Bard) refused to generate disinformation on this topic, even with jailbreaking. As for the false assertion that vaccines can cause autism, we found that only GPT-4 (via Copilot) and GPT-4 (via ChatGPT) were able to be directly prompted or jailbroken to generate such disinformation. Claude 2 (via Poe), Gemini Pro (via Bard), and Llama 2 (via HuggingChat) refused to generate disinformation on this topic, even with jailbreaking. Finally, in February 2024, GPT-4 (via both ChatGPT and Copilot) and Llama 2 (via HuggingChat) were observed to show the capability to facilitate the generation of disinformation about sugar causing cancer. Claude 2 (via Poe) and Gemini Pro (via Gemini, formerly Bard), however, refused to generate this content, even with attempts to jailbreak. The supplementary file provides examples of the submitted prompts and respective outputs from the sensitivity analyses.

This study found a noticeable inconsistency in the current implementation of safeguards in publicly accessible LLMs. Anthropic’s Claude 2 showcased the capacity of AI developers to release a LLM with valuable functionality while concurrently implementing robust safeguards against the generation of health disinformation. This was in stark contrast with other LLMs examined. Notably, OpenAI’s GPT-4 (via ChatGPT), Google’s PaLM 2 and Gemini Pro (via Bard), and Meta’s Llama 2 (via HuggingChat) exhibited the ability to consistently facilitate the mass generation of targeted and persuasive disinformation across many health topics. Meanwhile, GPT-4 (via Microsoft’s Copilot, formerly Bing Chat) highlighted the fluctuating nature of safeguards within the current self-regulating AI ecosystem. Initially, GPT-4 (via Copilot) exhibited strong safeguards, but over a 12 week period, these safeguards had become compromised, highlighting that LLM safeguards against health disinformation may change (intentionally or unintentionally) over time, and are not guaranteed to improve. Importantly, this study also showed major deficiencies in transparency within the AI industry, particularly whether developers are properly committed to minimizing the risks of health disinformation, the broad nature of safeguards that are currently implemented, and logs of frequently reported outputs and the corresponding response of developers (ie, when reported vulnerabilities were patched or justification was given for not fixing reported concerns, or both). Without the establishment and adherence to standards for these transparency markers, moving towards an AI ecosystem that can be effectively held accountable for concerns about health disinformation remains a challenging prospect for the community.

Strengths and limitations of this study

We only investigated the most prominent LLMs at the time of the study. Moreover, although Claude 2 resisted generating health disinformation for the scenarios evaluated, it might do so with alternative prompts or jailbreaking techniques. The LLMs that did facilitate disinformation were tested under particular conditions at two distinct time points, but outcomes might vary with different wordings or over time. Further, we focused on six specific health topics, limiting generalizability to all health topics or broader disinformation themes. Additionally, we concentrated on health disinformation topics widely regarded as being substantial/severe in the literature 24 25 , highlighting a gap for future studies to focus on equivocal topics, such as the link between sugar and cancer—a topic we briefly evaluated—wherein assessing the quality of content will become essential.

As safeguards can be implemented either within the LLM itself (for example, by training the LLM to generate outputs that align with human preferences) or at the AI chatbot or assistant interface used to access the LLM (for example, by implementing filters that screen the prompt before passing it to the LLM or filtering the output of the LLM before passing it back to the user, or both), it can be difficult to identify which factor is responsible for any effective safeguards identified. We acknowledge that in this study we directly tested only the LLM chatbot or assistant interfaces. It is, however, noteworthy that GPT-4 was accessed via both ChatGPT and Copilot and that in the initial evaluations, health disinformation was generated by ChatGPT but not by Copilot. As both chatbots used the same underlying LLM, it is likely that Copilot implemented additional safeguards to detect inappropriate requests or responses. Opposingly, Claude 2 (via Poe) consistently refused to generate health disinformation. By evaluating Poe with other LLMs, and Claude 2 via other interface providers, we determined that the safeguards were attributed to Claude 2. Thus, the design of the study enabled identification of examples in which the LLM developer provided robust safeguards, and in which the interface for accessing or utilizing the LLM provided robust safeguards. A limitation of the study is that owing to the poor transparency of AI developers we were unable to gain a detailed understanding of safeguard mechanisms that were effective or ineffective.

In our evaluation of the AI developers’ websites and their communication practices, we aimed to be as thorough as possible. The possibility remains, however, that we might have overlooked some aspects, and that we were unable to confirm the details of our website audits owing to the lack of responses from the developers, despite repeated requests. This limitation underscores challenges in fully assessing AI safety in an ecosystem not prioritising transparency and responsiveness.

Comparison with other studies

Previous research reported a potential for OpenAI’s GPT platforms to facilitate the generation of health disinformation on topics such as vaccines, antibiotics, electronic cigarettes, and homeopathy treatments. 6 8 9 12 In our study we found that most of the prominent, publicly accessible LLMs, including GPT-4 (via ChatGPT and Copilot), PaLM 2 and Gemini Pro (via Bard), and Llama 2 (via HuggingChat), lack effective safeguards to consistently prevent the mass generation of health disinformation across a broad range of topics. These findings show the capacity of these LLMs to generate highly persuasive health disinformation crafted with attention grabbing titles, authentic looking references, fabricated testimonials from both patients and doctors, and content tailored to resonate with a diverse range of demographic groups. Previous research found that both GPT-4 (via Copilot) and PaLM 2 (via Bard) refused to generate disinformation on vaccines and electronic cigarettes. 12 In this study, however, although GPT-4 (via Copilot) refused to generate requested health disinformation during the first evaluations in September 2023, ultimately both GPT-4 (via Copilot) and PaLM 2 (via Bard) generated health disinformation across multiple topics by the end of the study. This juxtaposition across time and studies underscores the urgent need for standards to be implemented and community pressure to continue for the creation and maintenance of effective safeguards against health disinformation generated by LLMs.

Anthropic’s Claude 2 was prominent as a publicly accessible LLM, with high functionality, that included rigorous safeguards to prevent the generation of health disinformation—even when prompts included common jailbreaking methods. This LLM highlights the practical feasibility of implementing effective safeguards in emerging AI technologies while also preserving utility and accessibility for beneficial purposes. Considering the substantial valuations of OpenAI ($29.0bn; £22.9bn; €26.7bn), Microsoft ($2.8tn), Google ($1.7tn), and Meta ($800bn), it becomes evident that these organizations have a tangible ability and obligation to emulate more stringent safeguards against health disinformation.

Moreover, this study found a striking absence of transparency on the intended safeguards of the LLMs assessed. It was unclear whether OpenAI, Microsoft, Google, and Meta have attempted to implement safeguards against health disinformation in their tools and they have failed, or if safeguards were not considered a priority. Notably, Microsoft’s Copilot initially showed robust safeguards against generating health disinformation, but these safeguards were absent 12 weeks later. With the current lack of transparency, it is unclear whether this was a deliberate or unintentional update.

From a search of the webpages of AI developers, we found important gaps in transparency and communication practices essential for mitigating risks of propagating health disinformation. Although all the developers provided mechanisms for users to report potentially harmful model outputs, we were unable to obtain responses to repeated attempts to confirm receipt of observed and reported safeguard vulnerabilities. This lack of engagement raises serious questions about the commitment of these AI developers to deal with the risks of health disinformation and to resolve problems. These concerns are further intensified by the lack of transparency about how reports submitted by other users are being managed and resolved, as well as the findings from our 12 week sensitivity analyses showing that health disinformation issues persisted.

Policy implications

The results of this study highlight the need to ensure the adequacy of current and emerging AI regulations to minimize risks to public health. This is particularly relevant in the context of ongoing discussions about AI legislative frameworks in the US and European Union. 26 27 These discussions might well consider the implementation of standards to third party filters to reduce discrepancies in outputs between different tools, as exemplified by the differences we observed between ChatGPT and Copilot in our initial evaluations, which occurred despite both being powered by GPT-4. While acknowledging that overly restrictive AI safeguards could restrict model performance for some beneficial purposes, emerging frameworks must also balance the risks to public health from mass health disinformation. Importantly, the ethical deployment of AI becomes even more crucial when recognizing that health disinformation often has a greater impact in areas with less health education or in resource limited settings, and thus emerging tools if not appropriately regulated have the potential to widen health inequities. This concern is further amplified by considering emerging advancements in technologies for image and video generation, where AI tools have the capability to simulate influential figures and translate content into multiple languages, thus increasing the potential for spread by enhancing the apparent trustworthiness of generated disinformation. 12 Moreover, all of this is occurring in an ecosystem where AI developers are failing to equip the community with detection tools to defend against the inadvertent consumption of AI generated material. 16

Our findings highlight notable inconsistencies in the effectiveness of LLM safeguards to prevent the mass generation of health disinformation. Implementing effective safeguards to prevent the potential misuse of LLMs for disseminating health disinformation has been found to be feasible. For many LLMs, however, these measures have not been implemented effectively, or the maintenance of robustness has not been prioritized. Thus, in the current AI environment where safety standards and policies remain poorly defined, malicious actors can potentially use publicly accessible LLMs for the mass generation of diverse and persuasive health disinformation, posing substantial risks to public health messaging—risks that will continue to increase with advancements in generative AI for audio and video content. Moreover, this study found substantial deficiencies in the transparency of AI developers about commitments to mitigating risks of health disinformation. Given that the AI landscape is rapidly evolving, public health and medical bodies 28 29 have an opportunity to deliver a united and clear message about the importance of health disinformation risk mitigation in developing AI regulations, the cornerstones of which should be transparency, health specific auditing, monitoring, and patching. 30

What is already known on this topic

Large language models (LLMs) have considerable potential to improve remote patient monitoring, triaging, and medical education, and the automation of administrative tasks

In the absence of proper safeguards, however, LLMs may be misused for mass generation of content for fraudulent or manipulative intent

What this study adds

This study found that many publicly accessible LLMs, including OpenAI’s GPT-4 (via ChatGPT and Microsoft’s Copilot), Google’s PaLM 2/Gemini Pro (via Bard), and Meta’s Llama 2 (via HuggingChat) lack adequate safeguards against mass generation of health disinformation

Anthropic’s Claude 2 showed robust safeguards against the generation of health disinformation, highlighting the feasibility of implementing robust safeguards

Poor transparency among AI developers on safeguards and processes they had implemented to minimise the risk of health disinformation were identified, along with a lack of response to reported safeguard vulnerabilities

Ethics statements

Ethical approval.

The research undertaken was assessed negligible risk research and was confirmed exempt from requiring review by Flinders University Human Research Ethics Committee.

Data availability statement

The research team would be willing to make the complete set of generated data available upon request from qualified researchers or policy makers on submission of a proposal detailing required access and intended use.

CR, MH, and AV are consumer advisors to the research team. Their extensive involvement in the study, spanning conception, design, evaluation, and drafting of the manuscript merits their recognition as coauthors of this research.

Contributors: MJS and AMH contributed equally. BDM and AMH had full access to all the data in the study and take responsibility for the integrity of the data collection, accuracy, and its analysis. CR, MH, and AV are consumer advisors to the research team. All authors contributed to the study design, data analysis, data interpretation, and drafting of the manuscript. All authors have read and approved the final version of the manuscript. The corresponding author (AMH) attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.

Funding: AMH holds an emerging leader investigator fellowship from the National Health and Medical Research Council (NHMRC), Australia (APP2008119). NDM is supported by a NHMRC postgraduate scholarship, Australia (APP2005294). MJS is supported by a Beat Cancer research fellowship from the Cancer Council South Australia. BDM’s PhD scholarship is supported by The Beat Cancer Project, Cancer Council South Australia, and the NHMRC, Australia (APP2030913). The funders had no role in considering the study design or in the collection, analysis, interpretation of data, writing of the report, or decision to submit the article for publication.

Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/disclosure-of-interest/ and declare: AMH holds an emerging leader investigator fellowship from the National Health and Medical Research Council (NHMRC), Australia; NDM is supported by a NHMRC postgraduate scholarship, Australia; MJS is supported by a Beat Cancer research fellowship from the Cancer Council South Australia; BDM’s PhD scholarship is supported by The Beat Cancer Project, Cancer Council South Australia, and the NHMRC, Australia; no support from any other organisation for the submitted work; AR and MJS are recipients of investigator initiated funding for research outside the scope of the current study from AstraZeneca, Boehringer Ingelheim, Pfizer, and Takeda; and AR is a recipient of speaker fees from Boehringer Ingelheim and Genentech outside the scope of the current study. There are no financial relationships with any other organisations that might have an interest in the submitted work in the previous three years to declare; no other relationships or activities that could appear to have influenced the submitted work.

The lead author (the manuscript’s guarantor) affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned have been explained.

Dissemination to participants and related patient and public communities: A summary of the results of this study will be disseminated by press release through the Flinders University Media and Communication team via the Eureka and Scimex news platforms. The study will also be shared through university social media channels—namely, X, Facebook, and LinkedIn.

Provenance and peer review: Not commissioned; externally peer reviewed.

AI assistance: Four publicly accessible large language models—GPT-4 (via ChatGPT and Copilot), PaLM 2/Gemini Pro (via Bard), Claude 2 (via Poe), and Llama 2 (via HuggingChat)—were used to generate the data evaluated in this manuscript. During the preparation of this work the authors used ChatGPT and Grammarly AI to assist in the formatting and editing of the manuscript to improve the language and readability. After using these tools, the authors reviewed and edited the content as needed and take full responsibility for the content of the publication.

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/ .

• Hopkins AM ,
• Kichenadasse G ,
• Miyata-Sturm A ,
• De Angelis L ,
• Baglivo F ,
• Arzilli G ,
• Májovský M ,
• Spitale G ,
• Biller-Andorno N ,
• ↵ The Reagan-Udall Foundation for the Food and Drug Administration. Strategies for Improving Public Understanding of FDA-Regulated Products 2023. https://reaganudall.org/sites/default/files/2023-10/Strategies_Report_Digital_Final.pdf .
• Finney Rutten LJ ,
• Greenberg-Worisek AJ ,
• Sorich MJ ,
• Vosoughi S ,
• Hanage WP ,
• Lipsitch M ,
• Gradoń KT ,
• Hołyst JA ,
• Cassileth B
• ↵ Microsoft. Bing Chat. https://www.microsoft.com/en-us/edge/features/bing-chat .
• ↵ Meta. Llama 2. https://huggingface.co/chat/ .
• Oliver JE ,
• Perlis RH ,
• Lunz Trujillo K ,
• ↵ European Comission: Laying Down Harmonised Rules on Artificial Intelligence (Artificial Intelligence Act) and Amending Certain Union Legislative Acts [updated 24/04/21]. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=celex%3A52021PC0206 .
• ↵ The White House. Blueprint for an AI Bill of Rights, Making Automated Systems Work For The American People 2022. https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf .
• ↵ World Health Organization. WHO calls for safe and ethical AI for health 2023. https://www.who.int/news/item/16-05-2023-who-calls-for-safe-and-ethical-ai-for-health .
• ↵ Australian Medical Association. Automated Decision Making and AI Regulation AMA submission to the Prime Minister and Cabinet Consultation on Positioning Australia as the Leader in Digital Economy Regulation 2022. https://www.ama.com.au/sites/default/files/2022-06/AMA%20Submission%20to%20Automated%20Decision%20Making%20and%20AI%20Regulation_Final.pdf .
• Mökander J ,
• Schuett J ,

• Frontiers in Research Metrics and Analytics
• Research Methods
• Research Topics

Network Analysis of Social Media Texts

Total Downloads

Total Views and Downloads

About this Research Topic

As social media have made the message content of an increasing proportion of human communication accessible, the unstructured text data provides the basis for inferring message contexts and subtexts that reveal the framing of content creators and the subjective opinions and judgments of receivers. Although ...

Keywords : Semantic Network Analysis, Social-Semantic Network Analysis, Topic Network Analysis, Social Media discourse, Content Analysis, text analysis, word co-occurrence

Important Note : All contributions to this Research Topic must be within the scope of the section and journal to which they are submitted, as defined in their mission statements. Frontiers reserves the right to guide an out-of-scope manuscript to a more suitable section or journal at any stage of peer review.

Topic Editors

Topic coordinators, recent articles, submission deadlines, participating journals.

Manuscripts can be submitted to this Research Topic via the following journals:

total views

• Demographics

No records found

total views article views downloads topic views

Top countries

Top referring sites, about frontiers research topics.

With their unique mixes of varied contributions from Original Research to Review Articles, Research Topics unify the most influential researchers, the latest key findings and historical advances in a hot research area! Find out more on how to host your own Frontiers Research Topic or contribute to one as an author.

Read our research on: TikTok | Podcasts | Election 2024

Regions & Countries

What the data says about abortion in the u.s..

Pew Research Center has conducted many surveys about abortion over the years, providing a lens into Americans’ views on whether the procedure should be legal, among a host of other questions.

In a  Center survey  conducted nearly a year after the Supreme Court’s June 2022 decision that  ended the constitutional right to abortion , 62% of U.S. adults said the practice should be legal in all or most cases, while 36% said it should be illegal in all or most cases. Another survey conducted a few months before the decision showed that relatively few Americans take an absolutist view on the issue .

Find answers to common questions about abortion in America, based on data from the Centers for Disease Control and Prevention (CDC) and the Guttmacher Institute, which have tracked these patterns for several decades:

How many abortions are there in the U.S. each year?

How has the number of abortions in the u.s. changed over time, what is the abortion rate among women in the u.s. how has it changed over time, what are the most common types of abortion, how many abortion providers are there in the u.s., and how has that number changed, what percentage of abortions are for women who live in a different state from the abortion provider, what are the demographics of women who have had abortions, when during pregnancy do most abortions occur, how often are there medical complications from abortion.

This compilation of data on abortion in the United States draws mainly from two sources: the Centers for Disease Control and Prevention (CDC) and the Guttmacher Institute, both of which have regularly compiled national abortion data for approximately half a century, and which collect their data in different ways.

The CDC data that is highlighted in this post comes from the agency’s “abortion surveillance” reports, which have been published annually since 1974 (and which have included data from 1969). Its figures from 1973 through 1996 include data from all 50 states, the District of Columbia and New York City – 52 “reporting areas” in all. Since 1997, the CDC’s totals have lacked data from some states (most notably California) for the years that those states did not report data to the agency. The four reporting areas that did not submit data to the CDC in 2021 – California, Maryland, New Hampshire and New Jersey – accounted for approximately 25% of all legal induced abortions in the U.S. in 2020, according to Guttmacher’s data. Most states, though,  do  have data in the reports, and the figures for the vast majority of them came from each state’s central health agency, while for some states, the figures came from hospitals and other medical facilities.

Discussion of CDC abortion data involving women’s state of residence, marital status, race, ethnicity, age, abortion history and the number of previous live births excludes the low share of abortions where that information was not supplied. Read the methodology for the CDC’s latest abortion surveillance report , which includes data from 2021, for more details. Previous reports can be found at  stacks.cdc.gov  by entering “abortion surveillance” into the search box.

For the numbers of deaths caused by induced abortions in 1963 and 1965, this analysis looks at reports by the then-U.S. Department of Health, Education and Welfare, a precursor to the Department of Health and Human Services. In computing those figures, we excluded abortions listed in the report under the categories “spontaneous or unspecified” or as “other.” (“Spontaneous abortion” is another way of referring to miscarriages.)

Guttmacher data in this post comes from national surveys of abortion providers that Guttmacher has conducted 19 times since 1973. Guttmacher compiles its figures after contacting every known provider of abortions – clinics, hospitals and physicians’ offices – in the country. It uses questionnaires and health department data, and it provides estimates for abortion providers that don’t respond to its inquiries. (In 2020, the last year for which it has released data on the number of abortions in the U.S., it used estimates for 12% of abortions.) For most of the 2000s, Guttmacher has conducted these national surveys every three years, each time getting abortion data for the prior two years. For each interim year, Guttmacher has calculated estimates based on trends from its own figures and from other data.

The latest full summary of Guttmacher data came in the institute’s report titled “Abortion Incidence and Service Availability in the United States, 2020.” It includes figures for 2020 and 2019 and estimates for 2018. The report includes a methods section.

In addition, this post uses data from StatPearls, an online health care resource, on complications from abortion.

An exact answer is hard to come by. The CDC and the Guttmacher Institute have each tried to measure this for around half a century, but they use different methods and publish different figures.

The last year for which the CDC reported a yearly national total for abortions is 2021. It found there were 625,978 abortions in the District of Columbia and the 46 states with available data that year, up from 597,355 in those states and D.C. in 2020. The corresponding figure for 2019 was 607,720.

The last year for which Guttmacher reported a yearly national total was 2020. It said there were 930,160 abortions that year in all 50 states and the District of Columbia, compared with 916,460 in 2019.

• How the CDC gets its data: It compiles figures that are voluntarily reported by states’ central health agencies, including separate figures for New York City and the District of Columbia. Its latest totals do not include figures from California, Maryland, New Hampshire or New Jersey, which did not report data to the CDC. ( Read the methodology from the latest CDC report .)
• How Guttmacher gets its data: It compiles its figures after contacting every known abortion provider – clinics, hospitals and physicians’ offices – in the country. It uses questionnaires and health department data, then provides estimates for abortion providers that don’t respond. Guttmacher’s figures are higher than the CDC’s in part because they include data (and in some instances, estimates) from all 50 states. ( Read the institute’s latest full report and methodology .)

While the Guttmacher Institute supports abortion rights, its empirical data on abortions in the U.S. has been widely cited by  groups  and  publications  across the political spectrum, including by a  number of those  that  disagree with its positions .

These estimates from Guttmacher and the CDC are results of multiyear efforts to collect data on abortion across the U.S. Last year, Guttmacher also began publishing less precise estimates every few months , based on a much smaller sample of providers.

The figures reported by these organizations include only legal induced abortions conducted by clinics, hospitals or physicians’ offices, or those that make use of abortion pills dispensed from certified facilities such as clinics or physicians’ offices. They do not account for the use of abortion pills that were obtained  outside of clinical settings .

(Back to top)

The annual number of U.S. abortions rose for years after Roe v. Wade legalized the procedure in 1973, reaching its highest levels around the late 1980s and early 1990s, according to both the CDC and Guttmacher. Since then, abortions have generally decreased at what a CDC analysis called  “a slow yet steady pace.”

Guttmacher says the number of abortions occurring in the U.S. in 2020 was 40% lower than it was in 1991. According to the CDC, the number was 36% lower in 2021 than in 1991, looking just at the District of Columbia and the 46 states that reported both of those years.

(The corresponding line graph shows the long-term trend in the number of legal abortions reported by both organizations. To allow for consistent comparisons over time, the CDC figures in the chart have been adjusted to ensure that the same states are counted from one year to the next. Using that approach, the CDC figure for 2021 is 622,108 legal abortions.)

There have been occasional breaks in this long-term pattern of decline – during the middle of the first decade of the 2000s, and then again in the late 2010s. The CDC reported modest 1% and 2% increases in abortions in 2018 and 2019, and then, after a 2% decrease in 2020, a 5% increase in 2021. Guttmacher reported an 8% increase over the three-year period from 2017 to 2020.

As noted above, these figures do not include abortions that use pills obtained outside of clinical settings.

Guttmacher says that in 2020 there were 14.4 abortions in the U.S. per 1,000 women ages 15 to 44. Its data shows that the rate of abortions among women has generally been declining in the U.S. since 1981, when it reported there were 29.3 abortions per 1,000 women in that age range.

The CDC says that in 2021, there were 11.6 abortions in the U.S. per 1,000 women ages 15 to 44. (That figure excludes data from California, the District of Columbia, Maryland, New Hampshire and New Jersey.) Like Guttmacher’s data, the CDC’s figures also suggest a general decline in the abortion rate over time. In 1980, when the CDC reported on all 50 states and D.C., it said there were 25 abortions per 1,000 women ages 15 to 44.

That said, both Guttmacher and the CDC say there were slight increases in the rate of abortions during the late 2010s and early 2020s. Guttmacher says the abortion rate per 1,000 women ages 15 to 44 rose from 13.5 in 2017 to 14.4 in 2020. The CDC says it rose from 11.2 per 1,000 in 2017 to 11.4 in 2019, before falling back to 11.1 in 2020 and then rising again to 11.6 in 2021. (The CDC’s figures for those years exclude data from California, D.C., Maryland, New Hampshire and New Jersey.)

The CDC broadly divides abortions into two categories: surgical abortions and medication abortions, which involve pills. Since the Food and Drug Administration first approved abortion pills in 2000, their use has increased over time as a share of abortions nationally, according to both the CDC and Guttmacher.

The majority of abortions in the U.S. now involve pills, according to both the CDC and Guttmacher. The CDC says 56% of U.S. abortions in 2021 involved pills, up from 53% in 2020 and 44% in 2019. Its figures for 2021 include the District of Columbia and 44 states that provided this data; its figures for 2020 include D.C. and 44 states (though not all of the same states as in 2021), and its figures for 2019 include D.C. and 45 states.

Guttmacher, which measures this every three years, says 53% of U.S. abortions involved pills in 2020, up from 39% in 2017.

Two pills commonly used together for medication abortions are mifepristone, which, taken first, blocks hormones that support a pregnancy, and misoprostol, which then causes the uterus to empty. According to the FDA, medication abortions are safe  until 10 weeks into pregnancy.

Surgical abortions conducted  during the first trimester  of pregnancy typically use a suction process, while the relatively few surgical abortions that occur  during the second trimester  of a pregnancy typically use a process called dilation and evacuation, according to the UCLA School of Medicine.

In 2020, there were 1,603 facilities in the U.S. that provided abortions,  according to Guttmacher . This included 807 clinics, 530 hospitals and 266 physicians’ offices.

While clinics make up half of the facilities that provide abortions, they are the sites where the vast majority (96%) of abortions are administered, either through procedures or the distribution of pills, according to Guttmacher’s 2020 data. (This includes 54% of abortions that are administered at specialized abortion clinics and 43% at nonspecialized clinics.) Hospitals made up 33% of the facilities that provided abortions in 2020 but accounted for only 3% of abortions that year, while just 1% of abortions were conducted by physicians’ offices.

Looking just at clinics – that is, the total number of specialized abortion clinics and nonspecialized clinics in the U.S. – Guttmacher found the total virtually unchanged between 2017 (808 clinics) and 2020 (807 clinics). However, there were regional differences. In the Midwest, the number of clinics that provide abortions increased by 11% during those years, and in the West by 6%. The number of clinics  decreased  during those years by 9% in the Northeast and 3% in the South.

The total number of abortion providers has declined dramatically since the 1980s. In 1982, according to Guttmacher, there were 2,908 facilities providing abortions in the U.S., including 789 clinics, 1,405 hospitals and 714 physicians’ offices.

The CDC does not track the number of abortion providers.

In the District of Columbia and the 46 states that provided abortion and residency information to the CDC in 2021, 10.9% of all abortions were performed on women known to live outside the state where the abortion occurred – slightly higher than the percentage in 2020 (9.7%). That year, D.C. and 46 states (though not the same ones as in 2021) reported abortion and residency data. (The total number of abortions used in these calculations included figures for women with both known and unknown residential status.)

The share of reported abortions performed on women outside their state of residence was much higher before the 1973 Roe decision that stopped states from banning abortion. In 1972, 41% of all abortions in D.C. and the 20 states that provided this information to the CDC that year were performed on women outside their state of residence. In 1973, the corresponding figure was 21% in the District of Columbia and the 41 states that provided this information, and in 1974 it was 11% in D.C. and the 43 states that provided data.

In the District of Columbia and the 46 states that reported age data to  the CDC in 2021, the majority of women who had abortions (57%) were in their 20s, while about three-in-ten (31%) were in their 30s. Teens ages 13 to 19 accounted for 8% of those who had abortions, while women ages 40 to 44 accounted for about 4%.

The vast majority of women who had abortions in 2021 were unmarried (87%), while married women accounted for 13%, according to  the CDC , which had data on this from 37 states.

In the District of Columbia, New York City (but not the rest of New York) and the 31 states that reported racial and ethnic data on abortion to  the CDC , 42% of all women who had abortions in 2021 were non-Hispanic Black, while 30% were non-Hispanic White, 22% were Hispanic and 6% were of other races.

Looking at abortion rates among those ages 15 to 44, there were 28.6 abortions per 1,000 non-Hispanic Black women in 2021; 12.3 abortions per 1,000 Hispanic women; 6.4 abortions per 1,000 non-Hispanic White women; and 9.2 abortions per 1,000 women of other races, the  CDC reported  from those same 31 states, D.C. and New York City.

For 57% of U.S. women who had induced abortions in 2021, it was the first time they had ever had one,  according to the CDC.  For nearly a quarter (24%), it was their second abortion. For 11% of women who had an abortion that year, it was their third, and for 8% it was their fourth or more. These CDC figures include data from 41 states and New York City, but not the rest of New York.

Nearly four-in-ten women who had abortions in 2021 (39%) had no previous live births at the time they had an abortion,  according to the CDC . Almost a quarter (24%) of women who had abortions in 2021 had one previous live birth, 20% had two previous live births, 10% had three, and 7% had four or more previous live births. These CDC figures include data from 41 states and New York City, but not the rest of New York.

The vast majority of abortions occur during the first trimester of a pregnancy. In 2021, 93% of abortions occurred during the first trimester – that is, at or before 13 weeks of gestation,  according to the CDC . An additional 6% occurred between 14 and 20 weeks of pregnancy, and about 1% were performed at 21 weeks or more of gestation. These CDC figures include data from 40 states and New York City, but not the rest of New York.

About 2% of all abortions in the U.S. involve some type of complication for the woman , according to an article in StatPearls, an online health care resource. “Most complications are considered minor such as pain, bleeding, infection and post-anesthesia complications,” according to the article.

The CDC calculates  case-fatality rates for women from induced abortions – that is, how many women die from abortion-related complications, for every 100,000 legal abortions that occur in the U.S .  The rate was lowest during the most recent period examined by the agency (2013 to 2020), when there were 0.45 deaths to women per 100,000 legal induced abortions. The case-fatality rate reported by the CDC was highest during the first period examined by the agency (1973 to 1977), when it was 2.09 deaths to women per 100,000 legal induced abortions. During the five-year periods in between, the figure ranged from 0.52 (from 1993 to 1997) to 0.78 (from 1978 to 1982).

The CDC calculates death rates by five-year and seven-year periods because of year-to-year fluctuation in the numbers and due to the relatively low number of women who die from legal induced abortions.

In 2020, the last year for which the CDC has information , six women in the U.S. died due to complications from induced abortions. Four women died in this way in 2019, two in 2018, and three in 2017. (These deaths all followed legal abortions.) Since 1990, the annual number of deaths among women due to legal induced abortion has ranged from two to 12.

The annual number of reported deaths from induced abortions (legal and illegal) tended to be higher in the 1980s, when it ranged from nine to 16, and from 1972 to 1979, when it ranged from 13 to 63. One driver of the decline was the drop in deaths from illegal abortions. There were 39 deaths from illegal abortions in 1972, the last full year before Roe v. Wade. The total fell to 19 in 1973 and to single digits or zero every year after that. (The number of deaths from legal abortions has also declined since then, though with some slight variation over time.)

The number of deaths from induced abortions was considerably higher in the 1960s than afterward. For instance, there were 119 deaths from induced abortions in  1963  and 99 in  1965 , according to reports by the then-U.S. Department of Health, Education and Welfare, a precursor to the Department of Health and Human Services. The CDC is a division of Health and Human Services.

Note: This is an update of a post originally published May 27, 2022, and first updated June 24, 2022.

Sign up for our weekly newsletter

Fresh data delivered Saturday mornings

Key facts about the abortion debate in America

Public opinion on abortion, three-in-ten or more democrats and republicans don’t agree with their party on abortion, partisanship a bigger factor than geography in views of abortion access locally, do state laws on abortion reflect public opinion, most popular.

About Pew Research Center Pew Research Center is a nonpartisan fact tank that informs the public about the issues, attitudes and trends shaping the world. It conducts public opinion polling, demographic research, media content analysis and other empirical social science research. Pew Research Center does not take policy positions. It is a subsidiary of The Pew Charitable Trusts .