literature review on machine learning

• Research article
• Open access
• Published: 15 February 2021

Systematic literature review of machine learning methods used in the analysis of real-world data for patient-provider decision making

• Alan Brnabic 1 &
• Lisa M. Hess   ORCID: orcid.org/0000-0003-3631-3941 2  

BMC Medical Informatics and Decision Making volume  21 , Article number:  54 ( 2021 ) Cite this article

29k Accesses

51 Citations

3 Altmetric

Metrics details

Machine learning is a broad term encompassing a number of methods that allow the investigator to learn from the data. These methods may permit large real-world databases to be more rapidly translated to applications to inform patient-provider decision making.

This systematic literature review was conducted to identify published observational research of employed machine learning to inform decision making at the patient-provider level. The search strategy was implemented and studies meeting eligibility criteria were evaluated by two independent reviewers. Relevant data related to study design, statistical methods and strengths and limitations were identified; study quality was assessed using a modified version of the Luo checklist.

A total of 34 publications from January 2014 to September 2020 were identified and evaluated for this review. There were diverse methods, statistical packages and approaches used across identified studies. The most common methods included decision tree and random forest approaches. Most studies applied internal validation but only two conducted external validation. Most studies utilized one algorithm, and only eight studies applied multiple machine learning algorithms to the data. Seven items on the Luo checklist failed to be met by more than 50% of published studies.

Conclusions

A wide variety of approaches, algorithms, statistical software, and validation strategies were employed in the application of machine learning methods to inform patient-provider decision making. There is a need to ensure that multiple machine learning approaches are used, the model selection strategy is clearly defined, and both internal and external validation are necessary to be sure that decisions for patient care are being made with the highest quality evidence. Future work should routinely employ ensemble methods incorporating multiple machine learning algorithms.

Peer Review reports

Traditional methods of analyzing large real-world databases (big data) and other observational studies are focused on the outcomes that can inform at the population-based level. The findings from real-world studies are relevant to populations as a whole, but the ability to predict or provide meaningful evidence at the patient level is much less well established due to the complexity with which clinical decision making is made and the variety of factors taken into account by the health care provider [ 1 , 2 ]. Using traditional methods that produce population estimates and measures of variability, it is very challenging to accurately predict how any one patient will perform, even when applying findings from subgroup analyses. The care of patients is nuanced, and multiple non-linear, interconnected factors must be taken into account in decision making. When data are available that are only relevant at the population level, health care decision making is less informed as to the optimal course of care for a given patient.

Clinical prediction models are an approach to utilizing patient-level evidence to help inform healthcare decision makers about patient care. These models are also known as prediction rules or prognostic models and have been used for decades by health care professionals [ 3 ]. Traditionally, these models combine patient demographic, clinical and treatment characteristics in the form of a statistical or mathematical model, usually regression, classification or neural networks, but deal with a limited number of predictor variables (usually below 25). The Framingham Heart Study is a classic example of the use of longitudinal data to build a traditional decision-making model. Multiple risk calculators and estimators have been built to predict a patient’s risk of a variety of cardiovascular outcomes, such as atrial fibrillation and coronary heart disease [ 4 , 5 , 6 ]. In general, these studies use multivariable regression evaluating risk factors identified in the literature. Based on these findings, a scoring system is derived for each factor to predict the likelihood of an adverse outcome based on a patient’s score across all risk factors evaluated.

With the advent of more complex data collection and readily available data sets for patients in routine clinical care, both sample sizes and potential predictor variables (such as genomic data) can exceed the tens of thousands, thus establishing the need for alternative approaches to rapidly process a large amount of information. Artificial intelligence (AI), particularly machine learning methods (a subset of AI), are increasingly being utilized in clinical research for prediction models, pattern recognition and deep-learning techniques used to combine complex information for example genomic and clinical data [ 7 , 8 , 9 ]. In the health care sciences, these methods are applied to replace a human expert to perform tasks that would otherwise take considerable time and expertise, and likely result in potential error. The underlying concept is that a machine will learn by trial and error from the data itself, to make predictions without having a pre-defined set of rules for decision making. Simply, machine learning can simply be better understood as “learning from data.” [ 8 ].

There are two types of learning from the data, unsupervised and supervised. Unsupervised learning is a type of machine learning algorithm used to draw inferences from datasets consisting of input data without labelled responses. The most common unsupervised learning method is cluster analysis, which is used for exploratory data analysis to find hidden patterns or grouping in data. Supervised learning involves making a prediction based on a set of pre-specified input and output variables. There are a number of statistical tools used for supervised learning. Some examples include traditional statistical prediction methods like regression models (e.g. regression splines, projection pursuit regression, penalized regression) that involve fitting a model to data, evaluating the fit and estimating parameters that are later used in a predictive equation. Other tools include tree-based methods (e.g. classification and regression trees [CART] and random forests), which successively partition a data set based on the relationships between predictor variables and a target (outcome) variable. Other examples include neural networks, discriminant functions and linear classifiers, support vector classifiers and machines. Often, predictive tools are built using various forms of model aggregation (or ensemble learning) that may combine models based on resampled or re-weighted data sets. These different types of models can be fitted to the same data using model averaging.

Classical statistical regression methods used for prediction modeling are well understood in the statistical sciences and the scientific community that employs them. These methods tend to be transparent and are usually hypothesis driven but can overlook complex associations with limited flexibility when a high number of variables are investigated. In addition, when using classic regression modeling, choosing the ‘right’ model is not straightforward. Non-traditional machine learning algorithms, and machine learning approaches, may overcome some of these limitations of classical regression models in this new era of big data, but are not a complete solution as they must be considered in the context of the limitations of data used in the analysis [ 2 ].

While machine learning methods can be used for both population-based models as well as for informed patient-provider decision making, it is important to note that the data, model, and outputs used to inform the care of an individual patient must meet the highest standards of research quality, as the choice made will likely have an impact on both the long- and short-term patient outcomes. While a range of uncertainty can be expected for population-based estimates, the risk of error for patient level models must be minimized to ensure quality patient care. The risks and concerns of utilizing machine learning for individual patient decision making have been raised by ethicists [ 10 ]. The risks are not limited to the lack of transparency, limited data regarding the confidence of the findings, and the risk of reducing patient autonomy in choice by relying on data that may foster a more paternalistic model of healthcare. These are all important and valid concerns, and therefore the role of machine learning for patient care must meet the highest standards to ensure that shared, not simply informed, evidence-based decision making be supported by these methods.

A systematic literature review was published in 2018 that evaluated the statistical methods that have been used to enable large, real-world databases to be used at the patient-provider level [ 11 ]. Briefly, this study identified a total of 115 articles that evaluated the use of logistic regression (n = 52, 45.2%), Cox regression (n = 24, 20.9%), and linear regression (n = 17, 14.8%). However, an interesting observation noted several studies utilizing novel statistical approaches such as machine learning, recursive partitioning, and development of mathematical algorithms to predict patient outcomes. More recently, publications are emerging describing the use of Individualized Treatment Recommendation algorithms and Outcome Weighted Learning for personalized medicine using large observational databases [ 12 , 13 ]. Therefore, this systematic literature review was designed to further pursue this observation to more comprehensively evaluate the use of machine learning methods to support patient-provider decision making, and to critically evaluate the strengths and weaknesses of these methods. For the purposes of this work, data supporting patient-provider decision making was defined as that which provided information specifically on a treatment or intervention choice; while both population-based and risk estimator data are certainly valuable for patient care and decision making, this study was designed to evaluate data that would specifically inform a choice for the patient with the provider. The overarching goal is to provide evidence of how large datasets can be used to inform decisions at the patient level using machine learning-based methods, and to evaluate the quality of such work to support informed decision making.

This study originated from a systematic literature review that was conducted in MEDLINE and PsychInfo; a refreshed search was conducted in September 2020 to obtain newer publications (Table 1 ). Eligible studies were those that analyzed prospective or retrospective observational data, reported quantitative results, and described statistical methods specifically applicable to patient-level decision making. Specifically, patient-level decision making referred to studies that provided data for or against a particular intervention at the patient level, so that the data could be used to inform decision making at the patient-provider level. Studies did not meet this criterion if only a population-based estimates, mortality risk predictors, or satisfaction with care were evaluated. Additionally, studies designed to improve diagnostic tools and those evaluating health care system quality indicators did not meet the patient-provider decision-making criterion. Eligible statistical methods for this study were limited to machine learning-based approaches. Eligibility was assessed by two reviewers and any discrepancies were discussed; a third reviewer was available to serve as a tie breaker in case of different opinions. The final set of eligible publications were then abstracted into a Microsoft Excel document. Study quality was evaluated using a modified Luo scale, which was developed specifically as a tool to standardize high-quality publication of machine learning models [ 14 ]. A modified version of this tool was utilized for this study; specifically, the optional item were removed, and three terms were clarified: item 6 (define the prediction problem) was redefined as “define the model,” item 7 (prepare data for model building) was renamed “model building and validation,” and item 8 (build the predictive model) was renamed “model selection” to more succinctly state what was being evaluated under each criterion. Data were abstracted and both extracted data and the Luo checklist items were reviewed and verified by a second reviewer to ensure data comprehensiveness and quality. In all cases of differences in eligibility assessment or data entry, the reviewers met and ensured agreement with the final set of data to be included in the database for data synthesis, with a third reviewer utilized as a tie breaker in case of discrepancies. Data were summarized descriptively and qualitatively, based on the following categories: publication and study characteristics; patient characteristics; statistical methodologies used, including statistical software packages; strengths and weaknesses; and interpretation of findings.

The search strategy was run on September 1, 2020 and identified a total of 34 publications that utilized machine learning methods for individual patient-level decision making (Fig.  1 ). The most common reason for study exclusion, as expected, was due to the study not meeting the patient-level decision making criterion. A summary of the characteristics of eligible studies and the patient data are included in Table 2 . Most of the real-world data sources included retrospective databases or designs (n = 27, 79.4%), primarily utilizing electronic health records. Six analyses utilized prospective cohort studies and one utilized data from a cross sectional study.

Prisma diagram of screening and study identification

General approaches to machine learning

The types of classification or prediction machine learning algorithms are reported in Table 2 . These included decision tree/random forest analyses (19 studies) [ 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 ] and neural networks (19 studies) [ 24 , 25 , 26 , 27 , 28 , 29 , 30 , 32 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 , 43 , 44 ]. Other approaches included latent growth mixture modeling [ 45 ], support vector machine classifiers [ 46 ], LASSO regression [ 47 ], boosting methods [ 23 ], and a novel Bayesian approach [ 26 , 40 , 48 ]. Within the analytical approaches to support machine learning, a variety of methods were used to evaluate model fit, such as Akaike Information Criterion, Bayesian Information Criterion, and the Lo-Mendel-Rubin likelihood ratio test [ 22 , 45 , 47 ], and while most studies included the area under the curve (AUC) of receiver-operator characteristic (ROC) curves (Table 3 ), analyses also included sensitivity/specificity [ 16 , 19 , 24 , 30 , 41 , 42 , 43 ], positive predictive value [ 21 , 26 , 32 , 38 , 40 , 41 , 42 , 43 ], and a variety of less common approaches such as the geometric mean [ 16 ], use of the Matthews correlation coefficient (ranges from -1.0, completely erroneous information, to + 1.0, perfect prediction) [ 46 ], defining true/false negatives/positives by means of a confusion matrix [ 17 ], calculating the root mean square error of the predicted versus original outcome profiles [ 37 ], or identifying the model with the best average performance training and performance cross validation [ 36 ].

Statistical software packages

The statistical programs used to perform machine learning varied widely across these studies, no consistencies were observed (Table 2 ). As noted above, one study using decision tree analysis used Quinlan’s C5.0 decision tree algorithm [ 15 ] while a second used an earlier version of this program (C4.5) [ 20 ]. Other decision tree analyses utilized various versions of R [ 18 , 19 , 22 , 24 , 27 , 47 ], International Business Machines (IBM) Statistical Package for the Social Sciences (SPSS) [ 16 , 17 , 33 , 47 ], the Azure Machine Learning Platform [ 30 ], or programmed the model using Python [ 23 , 25 , 46 ]. Artificial neural network analyses used Neural Designer [ 34 ] or Statistica V10 [ 35 ]. Six studies did not report the software used for analysis [ 21 , 31 , 32 , 37 , 41 , 42 ].

Families of machine learning algorithms

Also as summarized in Table 2 , more than one third of all publications (n = 13, 38.2%) applied only one family of machine learning algorithm to model development [ 16 , 17 , 18 , 19 , 20 , 34 , 37 , 41 , 42 , 43 , 46 , 48 ]; and only four studies utilized five or more methods [ 23 , 25 , 28 , 45 ]. One applied an ensemble of six different algorithms and the software was set to run 200 iterations [ 23 ], and another ran seven algorithms [ 45 ].

Internal and external validation

Evaluation of study publication quality identified the most common gap in publications as the lack of external validation, which was conducted by only two studies [ 15 , 20 ]. Seven studies predefined the success criteria for model performance [ 20 , 21 , 23 , 35 , 36 , 46 , 47 ], and five studies discussed the generalizability of the model [ 20 , 23 , 34 , 45 , 48 ]. Six studies [ 17 , 18 , 21 , 22 , 35 , 36 ] discussed the balance between model accuracy and model simplicity or interpretability, which was also a criterion of quality publication in the Luo scale [ 14 ]. The items on the checklist that were least frequently met are presented in Fig.  2 . The complete quality assessment evaluation for each item in the checklist is included in Additional file 1 : Table S1.

Least frequently met study quality items, modified Luo Scale [ 14 ]

There were a variety of approaches taken to validate the models developed (Table 3 ). Internal validation with splitting into a testing and validation dataset was performed in all studies. The cohort splitting approach was conducted in multiple ways, using a 2:1 split [ 26 ], 60/40 split [ 21 , 36 ], a 70/30 split [ 16 , 17 , 22 , 30 , 33 , 35 ], 75/25 split [ 27 , 40 ], 80/20 split [ 46 ], 90/10 split [ 25 , 29 ], splitting the data based on site of care [ 48 ], a 2/1/1 split for training, testing and validation [ 38 ], and splitting 60/20/20, where the third group was selected for model selection purposes prior to validation [ 34 ]. Nine studies did not specifically mention the form of splitting approach used [ 15 , 18 , 19 , 20 , 24 , 29 , 39 , 45 , 47 ], but most of those noted the use of k fold cross validation. One training set corresponded to 90% of the sample [ 23 ], whereas a second study was less clear, as input data were at the observation level with multiple observations per patient, and 3 of the 15 patients were included in the training set [ 37 ]. The remaining studies did not specifically state splitting the data into testing and validation samples, but most specified they performed five-fold cross validation (including one that generally mentioned cohort splitting) [ 18 , 45 ] or ten-fold cross validation strategies [ 15 , 19 , 20 , 28 ].

External validation was conducted by only two studies (5.9%). Hische and colleagues conducted a decision tree analysis, which was designed to identify patients with impaired fasting glucose [ 20 ]. Their model was developed in a cohort study of patients from the Berlin Potsdam Cohort Study (n = 1527) and was found to have a positive predictive value of 56.2% and a negative predictive value of 89.1%. The model was then tested on an independent from the Dresden Cohort (n = 1998) with a family history of type II diabetes. In external validation, positive predictive value was 43.9% and negative predictive value was 90.4% [ 20 ]. Toussi and colleagues conducted both internal and external validation in their decision tree analysis to evaluate individual physician prescribing behaviors using a database of 463 patient electronic medical records [ 15 ]. For the internal validation step, the cross-validation option was used from Quinlan’s C5.0 decision tree learning algorithm as their study sample was too small to split into a testing and validation sample, and external validation was conducted by comparing outcomes to published treatment guidelines. Unfortunately, they found little concordance between physician behavior and guidelines potentially due to the timing of the data not matching the time period in which guidelines were implemented, emphasizing the need for a contemporaneous external control [ 15 ].

Handling of missing values

Missing values were addressed in most studies (n = 21, 61.8%) in this review, but there were thirteen remaining studies that did not mention if there were missing data or how they were handled (Table 3 ). For those that reported methods related to missing data, there were a wide variety of approaches used in real-world datasets. The full information maximum likelihood method was used for estimating model parameters in the presence of missing data for the development of the model by Hertroijs and colleagues, but patients with missing covariate values at baseline were excluded from the validation of the model [ 45 ]. Missing covariate values were included in models as a discrete category [ 48 ]. Four studies removed patients from the model with missing data [ 46 ], resulting in the loss of 16%-41% of samples in three studies [ 17 , 36 , 47 ]. Missing data from primary outcome variables were reported among with 59% (men) and 70% (women) within a study of diabetes [ 16 ]. In this study, single imputation was used; for continuous variables CART (IBM SPSS modeler V14.2.03) and for categorical variables the authors used the weighted K-Nearest Neighbor approach using RapidMiner (V.5) [ 16 ]. Other studies reported exclusion but not specifically the impact on sample size [ 29 , 31 , 38 , 44 ]. Imputation was conducted in a variety of ways for studies with missing data [ 22 , 25 , 28 , 33 ]. Single imputation was used in the study by Bannister and colleagues, but followed by multiple imputation in the final model to evaluate differences in model parameters [ 22 ]. One study imputed with a standard last-imputation-forward approach [ 26 ]. Spline techniques were used to impute missing data in the training set of one study [ 37 ]. Missingness was largely retained as an informative variable, and only variables missing for 85% or more of participants were excluded by Alaa et al. [ 23 ] while Hearn et al. used a combination of imputation and exclusion strategies [ 40 ]. Lastly, missing or incomplete data were imputed using a model-based approach by Toussi et al. [ 15 ] and using an optimal-impute algorithm by Bertsimas et al. [ 21 ].

Strengths and weaknesses noted by authors

Publications summarized the strengths and weaknesses of the machine learning methods employed. Low complexity and simplicity of machine-based learning models were noted as strengths of this approach [ 15 , 20 ]. Machine learning approaches were both powerful and efficient methods to apply to large datasets [ 19 ]. It was noted that parameters in this study that were significant at the patient level were included, even if at the broader population-based level using traditional regression analysis model development they would have not been significant and therefore would have been otherwise excluded using traditional approaches [ 34 ]. One publication noted the value of machine learning being highly dependent on the model selection strategy and parameter optimization, and that machine learning in and of itself will not provide better estimates unless these steps are conducted properly [ 23 ].

Even when properly planned, machine learning approaches are not without issues that deserve attention in future studies that employ these techniques. Within the eligible publications, weaknesses included overfitting the model with the inclusion of too much detail [ 15 ]. Additional limitations are based on the data sources used for machine learning, such as the lack of availability of all desired variables and missing data that can affect the development and performance of these models [ 16 , 34 , 36 , 48 ]. The lack of all relevant variables was noted as a particular concern for retrospective database studies, where the investigator is limited to what has been recorded [ 26 , 28 , 29 , 38 , 40 ]. Importantly and as observed in the studies included in this review, the lack of external validation was stated as a limitation of studies included in this review [ 28 , 30 , 38 , 42 ].

Limitations can also be on the part of the research team, as the need for both clinical and statistical expertise in the development and execution of studies using machine learning-based methodology, and users are warned against applying these methods blindly [ 22 ]. The importance of the role of clinical and statistical experts in the research team was noted in one study and highlighted as a strength of their work [ 21 ].

This study systematically reviewed and summarized the methods and approaches used for machine learning as applied to observational datasets that can inform patient-provider decision making. Machine learning methods have been applied much more broadly across observational studies than in the context of individual decision making, so the summary of this work does not necessarily apply to all machine learning-based studies. The focus of this work is on an area that remains largely unexplored, which is how to use large datasets in a manner that can inform and improve patient care in a way that supports shared decision making with reliable evidence that is applicable to the individual patient. Multiple publications cite the limitations of using population-based estimates for individual decisions [ 49 , 50 , 51 ]. Specifically, a summary statistic at the population level does not apply to each person in that cohort. Population estimates represent a point on a potentially wide distribution, and any one patient could fall anywhere within that distribution and be far from the point estimate value. On the other extreme, case reports or case series provide very specific individual-level data, but are not generalizable to other patients [ 52 ]. This review and summary provides guidance and suggestions of best practices to improve and hopefully increase the use of these methods to provide data and models to inform patient-provider decision making.

It was common for single modeling strategies to be employed within the identified publications. It has long been known that single algorithms to estimation can produce a fair amount of uncertainty and variability [ 53 ]. To overcome this limitation, there is a need for multiple algorithms and multiple iterations of the models to be performed. This, combined with more powerful analytics in recent years, provides a new standard for machine learning algorithm choice and development. While in some cases, a single model may fit the data well and provide an accurate answer, the certainty of the model can be supported through novel approaches, such as model averaging [ 54 ]. Few studies in this review combined multiple families of modeling strategies along with multiple iterations of the models. This should become a best practice in the future and is recommended as an additional criterion to assess study quality among machine learning-based modeling [ 54 ].

External validation is critical to ensure model accuracy, but was rarely conducted in the publications included in this review. The reasons for this could be many, such as lack of appropriate datasets or due to the lack of awareness of the importance of external validation [ 55 ]. As model development using machine learning increases, there is a need for external validation prior to application of models in any patient-provider setting. The generalizability of models is largely unknown without these data. Publications that did not conduct external validation also did not note the need for this to be completed, as generalizability was discussed in only five studies, one of which had also conducted the external validation. Of the remaining four studies, the role of generalizability was noted in terms of the need for future external validation in only one study [ 48 ]. Other reviews that were more broadly conducted to evaluate machine learning methods similarly found a low rate of external validation (6.6% versus 5.9% in this study) [ 56 ]. It was shown that there was lower prediction accuracy by external validation than simply by cross validation alone. The current review, with a focus on machine learning to support decision making at a practical level, suggests external validation is an important gap that should be filled prior to using these models for patient-provider decision making.

Luo and others suggest that k -fold validation may be used with proper stratification of the response variable as part of the model selection strategy [ 14 , 55 ]. The studies identified in this review generally conducted 5- or tenfold validation. There is no formal rule for the selection for the value of k , which is typically based on the size of the dataset; as k increases, bias will be reduced, but in turn variance will increase. While the tradeoff has to be accounted for, k  = 5–10 has been found to be reasonable for most study purposes [ 57 ].

The evidence from identified publications suggests that the ethical concerns of lack of transparency and failure to report confidence in the findings are largely warranted. These limitations can be addressed through the use of multiple modeling approaches (to clarify the ‘black box’ nature of these approaches) and by including both external and high k-fold validation (to demonstrate the confidence in findings). To ensure these methods are used in a manner that improves patient care, the expectations of population-based risk prediction models of the past are no longer sufficient. It is essential that the right data, the right set of models, and appropriate validation are employed to ensure that the resulting data meet standards for high quality patient care.

This study did not evaluate the quality of the underlying real-world data used to develop, test or validate the algorithms. While not directly part of the evaluation in this review, researchers should be aware that all limitations of real-world data sources apply regardless of the methodology employed. However, when observational datasets are used for machine learning-based research, the investigator should be aware of the extent to which the methods they are using depend on the data structure and availability, and should evaluate a proposed data source to ensure it is appropriate for the machine learning project [ 45 ]. Importantly, databases should be evaluated to fully understand the variables included, as well as those variables that may have prognostic or predictive value, but may not be included in the dataset. The lack of important variables remains a concern with the use of retrospective databases for machine learning. The concerns with confounding (particularly unmeasured confounding), bias (including immortal time bias), and patient selection criteria to be in the database must also be evaluated [ 58 , 59 ]. These are factors that should be considered prior to implementing these methods, and not always at the forefront of consideration when applying machine learning approaches. The Luo checklist is a valuable tool to ensure that any machine-learning study meets high research standards for patient care, and importantly includes the evaluation of missing or potentially incorrect data (i.e. outliers) and generalizability [ 14 ]. This should be supplemented by a thorough evaluation of the potential data to inform the modeling work prior to its implementation, and ensuring that multiple modeling methods are applied.

This review found a wide variety of approaches, methods, statistical software and validation strategies that were employed in the application of machine learning methods to inform patient-provider decision making. Based on these findings, there is a need to ensure that multiple modeling approaches are employed in the development of machine learning-based models for patient care, which requires the highest research standards to reliably support shared evidence-based decision making. Models should be evaluated with clear criteria for model selection, and both internal and external validation are needed prior to applying these models to inform patient care. Few studies have yet to reach that bar of evidence to inform patient-provider decision making.

Availability of data and materials

All data generated or analyzed during this study are included in this published article and its supplementary information files.

Abbreviations

Artificial intelligence

Area under the curve

Classification and regression trees

Logistic least absolute shrinkage and selector operator

Steyerberg EW, Claggett B. Towards personalized therapy for multiple sclerosis: limitations of observational data. Brain. 2018;141(5):e38-e.

Fröhlich H, Balling R, Beerenwinkel N, Kohlbacher O, Kumar S, Lengauer T, et al. From hype to reality: data science enabling personalized medicine. BMC Med. 2018;16(1):150.

Article   PubMed   PubMed Central   Google Scholar  

Steyerberg EW. Clinical prediction models. Berlin: Springer; 2019.

Book   Google Scholar  

Schnabel RB, Sullivan LM, Levy D, Pencina MJ, Massaro JM, D’Agostino RB Sr, et al. Development of a risk score for atrial fibrillation (Framingham Heart Study): a community-based cohort study. Lancet. 2009;373(9665):739–45.

D’Agostino RB, Wolf PA, Belanger AJ, Kannel WB. Stroke risk profile: adjustment for antihypertensive medication. Framingham Study Stroke. 1994;25(1):40–3.

Article   CAS   PubMed   Google Scholar  

Framingham Heart Study: Risk Functions 2020. https://www.framinghamheartstudy.org/ .

Gawehn E, Hiss JA, Schneider G. Deep learning in drug discovery. Mol Inf. 2016;35:3–14.

Article   CAS   Google Scholar  

Vamathevan J, Clark D, Czodrowski P, Dunham I, Ferran E, Lee G, et al. Applications of machine learning in drug discovery and development. Nat Rev Drug Discov. 2019;18(6):463–77.

Article   CAS   PubMed   PubMed Central   Google Scholar  

Marcus G. Deep learning: A critical appraisal. arXiv preprint arXiv:180100631. 2018.

Grote T, Berens P. On the ethics of algorithmic decision-making in healthcare. J Med Ethics. 2020;46(3):205–11.

Article   PubMed   Google Scholar  

Brnabic A, Hess L, Carter GC, Robinson R, Araujo A, Swindle R. Methods used for the applicability of real-world data sources to individual patient decision making. Value Health. 2018;21:S102.

Article   Google Scholar  

Fu H, Zhou J, Faries DE. Estimating optimal treatment regimes via subgroup identification in randomized control trials and observational studies. Stat Med. 2016;35(19):3285–302.

Liang M, Ye T, Fu H. Estimating individualized optimal combination therapies through outcome weighted deep learning algorithms. Stat Med. 2018;37(27):3869–86.

Luo W, Phung D, Tran T, Gupta S, Rana S, Karmakar C, et al. Guidelines for developing and reporting machine learning predictive models in biomedical research: a multidisciplinary view. J Med Internet Res. 2016;18(12):e323.

Toussi M, Lamy J-B, Le Toumelin P, Venot A. Using data mining techniques to explore physicians’ therapeutic decisions when clinical guidelines do not provide recommendations: methods and example for type 2 diabetes. BMC Med Inform Decis Mak. 2009;9(1):28.

Ramezankhani A, Hadavandi E, Pournik O, Shahrabi J, Azizi F, Hadaegh F. Decision tree-based modelling for identification of potential interactions between type 2 diabetes risk factors: a decade follow-up in a Middle East prospective cohort study. BMJ Open. 2016;6(12):e013336.

Pei D, Zhang C, Quan Y, Guo Q. Identification of potential type II diabetes in a Chinese population with a sensitive decision tree approach. J Diabetes Res. 2019;2019:4248218.

Neefjes EC, van der Vorst MJ, Verdegaal BA, Beekman AT, Berkhof J, Verheul HM. Identification of patients with cancer with a high risk to develop delirium. Cancer Med. 2017;6(8):1861–70.

Mubeen AM, Asaei A, Bachman AH, Sidtis JJ, Ardekani BA, Initiative AsDN. A six-month longitudinal evaluation significantly improves accuracy of predicting incipient Alzheimer’s disease in mild cognitive impairment. J Neuroradiol. 2017;44(6):381–7.

Hische M, Luis-Dominguez O, Pfeiffer AF, Schwarz PE, Selbig J, Spranger J. Decision trees as a simple-to-use and reliable tool to identify individuals with impaired glucose metabolism or type 2 diabetes mellitus. Eur J Endocrinol. 2010;163(4):565.

Bertsimas D, Dunn J, Pawlowski C, Silberholz J, Weinstein A, Zhuo YD, et al. Applied informatics decision support tool for mortality predictions in patients with cancer. JCO Clin Cancer Inform. 2018;2:1–11.

Bannister CA, Halcox JP, Currie CJ, Preece A, Spasic I. A genetic programming approach to development of clinical prediction models: a case study in symptomatic cardiovascular disease. PLoS ONE. 2018;13(9):e0202685.

Article   PubMed   PubMed Central   CAS   Google Scholar  

Alaa AM, Bolton T, Di Angelantonio E, Rudd JHF, van der Schaar M. Cardiovascular disease risk prediction using automated machine learning: a prospective study of 423,604 UK Biobank participants. PLoS ONE. 2019;14(5):e0213653.

Baxter SL, Marks C, Kuo TT, Ohno-Machado L, Weinreb RN. Machine learning-based predictive modeling of surgical intervention in glaucoma using systemic data from electronic health records. Am J Ophthalmol. 2019;208:30–40.

Dong Y, Xu L, Fan Y, Xiang P, Gao X, Chen Y, et al. A novel surgical predictive model for Chinese Crohn’s disease patients. Medicine (Baltimore). 2019;98(46):e17510.

Hill NR, Ayoubkhani D, McEwan P, Sugrue DM, Farooqui U, Lister S, et al. Predicting atrial fibrillation in primary care using machine learning. PLoS ONE. 2019;14(11):e0224582.

Kang AR, Lee J, Jung W, Lee M, Park SY, Woo J, et al. Development of a prediction model for hypotension after induction of anesthesia using machine learning. PLoS ONE. 2020;15(4):e0231172.

Karhade AV, Ogink PT, Thio Q, Cha TD, Gormley WB, Hershman SH, et al. Development of machine learning algorithms for prediction of prolonged opioid prescription after surgery for lumbar disc herniation. Spine J. 2019;19(11):1764–71.

Kebede M, Zegeye DT, Zeleke BM. Predicting CD4 count changes among patients on antiretroviral treatment: Application of data mining techniques. Comput Methods Programs Biomed. 2017;152:149–57.

Kim I, Choi HJ, Ryu JM, Lee SK, Yu JH, Kim SW, et al. A predictive model for high/low risk group according to oncotype DX recurrence score using machine learning. Eur J Surg Oncol. 2019;45(2):134–40.

Kwon JM, Jeon KH, Kim HM, Kim MJ, Lim S, Kim KH, et al. Deep-learning-based out-of-hospital cardiac arrest prognostic system to predict clinical outcomes. Resuscitation. 2019;139:84–91.

Kwon JM, Lee Y, Lee Y, Lee S, Park J. An algorithm based on deep learning for predicting in-hospital cardiac arrest. J Am Heart Assoc. 2018;7(13):26.

Scheer JK, Smith JS, Schwab F, Lafage V, Shaffrey CI, Bess S, et al. Development of a preoperative predictive model for major complications following adult spinal deformity surgery. J Neurosurg Spine. 2017;26(6):736–43.

Lopez-de-Andres A, Hernandez-Barrera V, Lopez R, Martin-Junco P, Jimenez-Trujillo I, Alvaro-Meca A, et al. Predictors of in-hospital mortality following major lower extremity amputations in type 2 diabetic patients using artificial neural networks. BMC Med Res Methodol. 2016;16(1):160.

Rau H-H, Hsu C-Y, Lin Y-A, Atique S, Fuad A, Wei L-M, et al. Development of a web-based liver cancer prediction model for type II diabetes patients by using an artificial neural network. Comput Methods Programs Biomed. 2016;125:58–65.

Ng T, Chew L, Yap CW. A clinical decision support tool to predict survival in cancer patients beyond 120 days after palliative chemotherapy. J Palliat Med. 2012;15(8):863–9.

Pérez-Gandía C, Facchinetti A, Sparacino G, Cobelli C, Gómez E, Rigla M, et al. Artificial neural network algorithm for online glucose prediction from continuous glucose monitoring. Diabetes Technol Therapeut. 2010;12(1):81–8.

Azimi P, Mohammadi HR, Benzel EC, Shahzadi S, Azhari S. Use of artificial neural networks to decision making in patients with lumbar spinal canal stenosis. J Neurosurg Sci. 2017;61(6):603–11.

Bowman A, Rudolfer S, Weller P, Bland JDP. A prognostic model for the patient-reported outcome of surgical treatment of carpal tunnel syndrome. Muscle Nerve. 2018;58(6):784–9.

Hearn J, Ross HJ, Mueller B, Fan CP, Crowdy E, Duhamel J, et al. Neural networks for prognostication of patients with heart failure. Circ. 2018;11(8):e005193.

Google Scholar  

Isma’eel HA, Cremer PC, Khalaf S, Almedawar MM, Elhajj IH, Sakr GE, et al. Artificial neural network modeling enhances risk stratification and can reduce downstream testing for patients with suspected acute coronary syndromes, negative cardiac biomarkers, and normal ECGs. Int J Cardiovasc Imaging. 2016;32(4):687–96.

Isma’eel HA, Sakr GE, Serhan M, Lamaa N, Hakim A, Cremer PC, et al. Artificial neural network-based model enhances risk stratification and reduces non-invasive cardiac stress imaging compared to Diamond-Forrester and Morise risk assessment models: a prospective study. J Nucl Cardiol. 2018;25(5):1601–9.

Jovanovic P, Salkic NN, Zerem E. Artificial neural network predicts the need for therapeutic ERCP in patients with suspected choledocholithiasis. Gastrointest Endosc. 2014;80(2):260–8.

Zhou HF, Huang M, Ji JS, Zhu HD, Lu J, Guo JH, et al. Risk prediction for early biliary infection after percutaneous transhepatic biliary stent placement in malignant biliary obstruction. J Vasc Interv Radiol. 2019;30(8):1233-41.e1.

Hertroijs DF, Elissen AM, Brouwers MC, Schaper NC, Köhler S, Popa MC, et al. A risk score including body mass index, glycated haemoglobin and triglycerides predicts future glycaemic control in people with type 2 diabetes. Diabetes Obes Metab. 2018;20(3):681–8.

Oviedo S, Contreras I, Quiros C, Gimenez M, Conget I, Vehi J. Risk-based postprandial hypoglycemia forecasting using supervised learning. Int J Med Inf. 2019;126:1–8.

Khanji C, Lalonde L, Bareil C, Lussier MT, Perreault S, Schnitzer ME. Lasso regression for the prediction of intermediate outcomes related to cardiovascular disease prevention using the TRANSIT quality indicators. Med Care. 2019;57(1):63–72.

Anderson JP, Parikh JR, Shenfeld DK, Ivanov V, Marks C, Church BW, et al. Reverse engineering and evaluation of prediction models for progression to type 2 diabetes: an application of machine learning using electronic health records. J Diabetes Sci Technol. 2016;10(1):6–18.

Patsopoulos NA. A pragmatic view on pragmatic trials. Dialogues Clin Neurosci. 2011;13(2):217–24.

Lu CY. Observational studies: a review of study designs, challenges and strategies to reduce confounding. Int J Clin Pract. 2009;63(5):691–7.

Morgenstern H. Ecologic studies in epidemiology: concepts, principles, and methods. Annu Rev Public Health. 1995;16(1):61–81.

Vandenbroucke JP. In defense of case reports and case series. Ann Intern Med. 2001;134(4):330–4.

Buckland ST, Burnham KP, Augustin NH. Model selection: an integral part of inference. Biometrics. 1997;53:603–18.

Zagar A, Kadziola Z, Lipkovich I, Madigan D, Faries D. Evaluating bias control strategies in observational studies using frequentist model averaging 2020 (submitted).

Kang J, Schwartz R, Flickinger J, Beriwal S. Machine learning approaches for predicting radiation therapy outcomes: a clinician’s perspective. Int J Radiat Oncol Biol Phys. 2015;93(5):1127–35.

Scott IM, Lin W, Liakata M, Wood J, Vermeer CP, Allaway D, et al. Merits of random forests emerge in evaluation of chemometric classifiers by external validation. Anal Chim Acta. 2013;801:22–33.

Kuhn M, Johnson K. Applied predictive modeling. Berlin: Springer; 2013.

Hess L, Winfree K, Muehlenbein C, Zhu Y, Oton A, Princic N. Debunking Myths While Understanding Limitations. Am J Public Health. 2020;110(5):E2-E.

Thesmar D, Sraer D, Pinheiro L, Dadson N, Veliche R, Greenberg P. Combining the power of artificial intelligence with the richness of healthcare claims data: Opportunities and challenges. PharmacoEconomics. 2019;37(6):745–52.

Download references

Acknowledgements

Not applicable.

No funding was received for the conduct of this study.

Author information

Authors and affiliations.

Eli Lilly and Company, Sydney, NSW, Australia

Alan Brnabic

Eli Lilly and Company, Indianapolis, IN, USA

Lisa M. Hess

You can also search for this author in PubMed   Google Scholar

Contributions

AB and LMH contributed to the design, implementation, analysis and interpretation of the data included in this study. AB and LMH wrote, revised and finalized the manuscript for submission. AB and LMH have both read and approved the final manuscript.

Corresponding author

Correspondence to Lisa M. Hess .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

Authors are employees of Eli Lilly and Company and receive salary support in that role.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1..

Table S1. Study quality of eligible publications, modified Luo scale [14].

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Brnabic, A., Hess, L.M. Systematic literature review of machine learning methods used in the analysis of real-world data for patient-provider decision making. BMC Med Inform Decis Mak 21 , 54 (2021). https://doi.org/10.1186/s12911-021-01403-2

Download citation

Received : 07 July 2020

Accepted : 20 January 2021

Published : 15 February 2021

DOI : https://doi.org/10.1186/s12911-021-01403-2

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Machine learning
• Decision making
• Decision tree
• Random forest
• Automated neural network

BMC Medical Informatics and Decision Making

ISSN: 1472-6947

• General enquiries: [email protected]

A Literature Review of Using Machine Learning in Software Development Life Cycle Stages

Ieee account.

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

Methodology

• Open access
• Published: 28 April 2022

An intelligent literature review: adopting inductive approach to define machine learning applications in the clinical domain

• Renu Sabharwal   ORCID: orcid.org/0000-0001-9728-8001 1 &
• Shah J. Miah 1  

Journal of Big Data volume  9 , Article number:  53 ( 2022 ) Cite this article

6685 Accesses

9 Citations

Metrics details

Big data analytics utilizes different techniques to transform large volumes of big datasets. The analytics techniques utilize various computational methods such as Machine Learning (ML) for converting raw data into valuable insights. The ML assists individuals in performing work activities intelligently, which empowers decision-makers. Since academics and industry practitioners have growing interests in ML, various existing review studies have explored different applications of ML for enhancing knowledge about specific problem domains. However, in most of the cases existing studies suffer from the limitations of employing a holistic, automated approach. While several researchers developed various techniques to automate the systematic literature review process, they also seemed to lack transparency and guidance for future researchers. This research aims to promote the utilization of intelligent literature reviews for researchers by introducing a step-by-step automated framework. We offer an intelligent literature review to obtain in-depth analytical insight of ML applications in the clinical domain to (a) develop the intelligent literature framework using traditional literature and Latent Dirichlet Allocation (LDA) topic modeling, (b) analyze research documents using traditional systematic literature review revealing ML applications, and (c) identify topics from documents using LDA topic modeling. We used a PRISMA framework for the review to harness samples sourced from four major databases (e.g., IEEE, PubMed, Scopus, and Google Scholar) published between 2016 and 2021 (September). The framework comprises two stages—(a) traditional systematic literature review consisting of three stages (planning, conducting, and reporting) and (b) LDA topic modeling that consists of three steps (pre-processing, topic modeling, and post-processing). The intelligent literature review framework transparently and reliably reviewed 305 sample documents.

Introduction

Organizations are continuously harnessing the power of various big data adopting different ML techniques. Captured insights from big data may create a greater impact to reshape their business operations and processes. As a vital technique, big data analytics methods are used to transform complicated and huge amounts of data, known as ‘Big Data, in order to uncover hidden patterns, new learning, untold facts or associations, anomalies, and other perceptions [ 41 ]. Big Data alludes to the enormous amount of data that a traditional database management system cannot handle. In most of the cases, traditional software functions would be inadequate to analyze or process them. Big data are characterized by the 5 V’s, which refers to volume, variety, velocity, veracity, and value [ 22 ]. ML is a vital approach to design useful big data analytics techniques, which is a rapidly growing sub-field in information sciences that deals with all these characteristics. ML employs numerous methods for machines to learn from past experiences (e.g., past datasets) reducing the extra burden of writing codes in traditional programming [ 7 , 26 ]. Clinical care enterprises face a huge challenge due to the increasing demand of big data processing to improve clinical care outcomes. For example, an electronic health record contains a huge amount of patient information, drug administration, imaging data using various modalities. The variety and quantity of the huge data provide in the clinical domain as an ideal topic to appraise the value of ML in research.

Existing ML approaches, such as Oala et al. [ 35 ] proposed an algorithmic framework that give a path towards the effective and reliable application of ML in the healthcare domain. In conjunction with their systematic review, our research offers a smart literature review that consolidates a traditional literature review followed the PRISMA framework guidelines and topic modeling using LDA, focusing on the clinical domain. Most of the existing literature focused on the healthcare domain [ 14 , 42 , 49 ] are more inclusive and of a broader scope with a requisite of medical activities, whereas our research is primarily focused is clinical, which assist in diagnosing and treating patients as well as includes clinical aspects of medicine.

Since clinical research has developed, the area has become increasingly attractive to clinical researchers, in particular for learning insights of ML applications in clinical practices . This is because of its practical pertinence to clinical patients, professionals, clinical application designers, and other specialists supported by the omnipresence of clinical disease management techniques. Although the advantage is presumed for the target audience, such as self-management abilities (self-efficacy and investment behavior) and physical or mental condition of life amid long-term ill patients, clinical care specialists (such as further developing independent direction and providing care support to patients), their clinical care have not been previously assessed and conceptualized as a well-defined and essential sub-field of health care research. It is important to portray similar studies utilizing different types of review approaches in the aspect of the utilization of ML/DL and its value. Table 1 represents some examples of existing studies with various points and review approaches in the domain.

Although the existing studies included in Table 1 give an understanding of designated aspects of ML/DL utilization in clinical care, they show a lack of focus on how key points addressed in existing ML/DL research are developing. Further to this, they indicate a clear need towards an understanding of multidisciplinary affiliations and profiles of ML/DL that could provide significant knowledge to new specialists or professionals in this space. For instance, Brnabic and Hess [ 8 ] recommended a direction for future research by stating that “ Future work should routinely employ ensemble methods incorporating various applications of machine learning algorithms” (p. 1).

ML tools have become the central focus of modern biomedical research, because of better admittance to large datasets, exponential processing power, and key algorithmic developments allowing ML models to handle increasingly challenging data [ 19 ]. Different ML approaches can analyze a huge amount of data, including difficult and abnormal patterns. Most studies have focused on ML and its impacts on clinical practices [ 2 , 9 , 10 , 24 , 26 , 34 , 43 ]. Fewer studies have examined the utilization of ML algorithms [ 11 , 20 , 45 , 48 ] for more holistic benefits for clinical researchers.

ML becomes an interdisciplinary science that integrates computer science, mathematics, and statistics. It is also a methodology that builds smart machines for artificial intelligence. Its applications comprise algorithms, an assortment of instructions to perform specific tasks, crafted to independently learn from data without human intercession. Over time, ML algorithms improve their prediction accuracy without a need for programming. Based on this, we offer an intelligent literature review using traditional literature review and Latent Dirichlet Allocation (LDA Footnote 1 ) topic modeling in order to meet knowledge demands in the clinical domain. Theoretical measures direct the current study results because previous literature provides a strong foundation for future IS researchers to investigate ML in the clinical sector. The main aim of this study is to develop an intelligent literature framework using traditional literature. For this purpose, we employed four digital databases -IEEE, Google Scholar, PubMed, and Scopus then performed LDA topic modeling, which may assist healthcare or clinical researchers in analyzing many documents intelligently with little effort and a small amount of time.

Traditional systematic literature is destined to be obsolete, time-consuming with restricted processing power, resulting in fewer sample documents investigated. Academic and practitioner-researchers are frequently required to discover, organize, and comprehend new and unexplored research areas. As a part of a traditional literature review that involves an enormous number of papers, the choice for a researcher is either to restrict the number of documents to review a priori or analyze the study using some other methods.

The proposed intelligent literature review approach consists of Part A and Part B, a combination of traditional systematic literature review and topic modeling that may assist future researchers in using appropriate technology, producing accurate results, and saving time. We present the framework below in Fig.  1 .

Proposed intelligent literature review framework

The traditional literature review identified 534,327 articles embraces Scopus (24,498), IEEE (2558), PubMed (11,271), and Google Scholar (496,000) articles, which went through three stages–Planning the review, conducting the review, and reporting the review and analyzed 305 articles, where we performed topic modeling using LDA.

We follow traditional systematic literature review methodologies [ 25 , 39 , 40 ] including a PRISMA framework [ 37 ]. We review four digital databases and deliberately develop three stages entailing planning, conducting, and reporting the review (Fig.  2 ).

Traditional literature review three stages

Planning the review

Research articles : the research articles are classified using some keywords mentioned below in Tables 2 , 3 .

Digital database : Four databases (IEEE, PubMed, Scopus, and Google Scholar) were used to collect details for reviewing research articles.

Review protocol development : We first used Scopus to search the information and found many studies regarding this review. We then searched PubMed, IEEE, and Google scholar for articles and extracted only relevant papers matching our keywords and review context based on their full-text availability.

Review protocol evaluation : To support the selection of research articles and inclusion and exclusion criteria, the quality of articles was explored and assessed to appraise their suitability and impartiality [ 44 ]. Only articles with keywords “machine learning” and “clinical” in document titles and abstracts were selected.

Conducting the review

The second step is conducting the review, which includes a description of Search Syntax and data synthesis.

Search syntax Table 4 details the syntax used to select research articles.

Data synthesis

We used a qualitative meta-synthesis technique to understand the methodology, algorithms, applications, qualities, results, and current research impediments. Qualitative meta-synthesis is a coherent approach for analyzing data across qualitative studies [ 4 ]. Our first search identified 534,327 papers, comprising Scopus (24,498), IEEE (2,558), PubMed (11,271), and Google Scholar (496,000) articles with the selected keywords. After subjecting this dataset to our inclusion and exclusion criteria, articles were reduced to Scopus (181), IEEE (62), PubMed (37), and Google Scholar (46) (Fig.  3 ).

PRISMA framework of traditional literature review

Reporting the review

This section displays the result of the traditional literature review.

Demonstration of findings

A search including linear literature and citation chaining was acted in digital databases, and the resulted papers were thoroughly analyzed to choose only the most pertinent articles, at last, 305 articles were included for the Part B review. Information of such articles were classified, organized, and demonstrated to show the finding.

Report the findings

The word cloud is displayed on the selected 305 research articles which give an overview of the frequency of the word within those 305 research articles. The chosen articles are moved to the next step to perform the conversion of PDF files to text documents for performing LDA topic modeling (Fig. 4 ).

Word cloud on 305 articles

Conversion of pdf files to a text document

The Python coding is used to convert pdf files shared on GitHub https://github.com/MachineLearning-UON/Topic-modeling-using-LDA.git . The one text document is prepared with 305 research papers collected from a traditional literature review.

Topic modelling for intelligent literature review

Our intelligent literature review is developed using a combination of traditional literature review and topic modeling [ 22 ]. We use topic modeling—probability generating, a text-mining technique widely used in computer science for text mining and data recovery. Topic modeling is used in numerous papers to analyze [ 1 , 5 , 17 , 36 ] and use various ML algorithms [ 38 ] such as Latent Semantic Indexing (LSI), Latent Semantic Analysis (LSA), Latent Dirichlet Allocation (LDA), Non-Negative Matrix Factorization (NMF), Parallel Latent Dirichlet Allocation (PLDA), and Pachinko Allocation Model (PAM). We developed the LDA-based methodological framework so it would be most widely and easily used [ 13 , 17 , 21 ] as a very elementary [ 6 ] approach. LDA is an unsupervised and probabilistic ML algorithm that discovers topics by calculating patterns of word co-occurrence across many documents or corpus [ 16 ]. Each LDA topic is distributed across each document as a probability.

While there are numerous ways of conducting a systematic literature review, most strategies require a high expense of time and prior knowledge of the area in advance. This study examined the expense of various text categorization strategies, where the assumptions and cost of the strategy are analyzed [ 5 ]. Interestingly, except manually reading the articles and topic modeling, all the strategies require prior knowledge of the articles' categories and high pre-examination costs. However, topic modeling can be automated, alternate the utilization of researchers' time, demonstrating a perfect match for the utilization of topic modeling as a part of an Intelligent literature review. Topic modeling has been used in a few papers to categorize research papers presented in Table 5 .

The articles/papers in the above table analyzed are speeches, web documents, web posts, press releases, and newspapers. However, none of those have developed the framework to perform traditional literature reviews from digital databases then use topic modeling to save time. However, this research points out the utilization of LDA in academics and explores four parameters—text pre-processing, model parameters selection, reliability, and validity [ 5 ]. Topic modeling identifies patterns of the repetitive word across a corpus of documents. Patterns of word co-occurrence are conceived as hidden ‘topics’ available in the corpus. First, documents must be modified to be machine-readable, with only their most informative features used for topic modeling. We modify documents in a three-stage process entailing pre-processing, topic modeling, and post-processing, as defined in Fig.  1 earlier.

The utilization of topic modeling presents an opportunity for researchers to use advanced technology for the literature review process. Topic modeling has been utilized online and requires many statistical skills, which not all researchers have. Therefore, we have shared the codes in GitHub with the default parameter for future researchers.

Pre-processing

Székely and Brocke [ 46 ] explained that pre-processing is a seven-step process which explored below and mentioned in Fig.  1 as part B:

Load data—The text data file is imported using the python command.

Optical character recognition—using word cloud, characters are recognized.

Filtering non-English words—non-English words are removed.

Document tokenization—Split the text into sentences and the sentences into words. Lowercase the words and remove punctuation.

Text cleaning—the text has been cleaned using portstemmer.

Word lemmatization—words in the third person are changed to the first person, and past and future verb tenses are changed into the present.

Stop word removal—All stop words are removed.

Topic modelling using LDA

Several research articles have been selected to run LDA topic modeling, explained in Table 5 . LDA model results present the coherence score for all the selected topics and a list of the most frequently used words for each.

Post-processing

The goal of the post-processing stage is to identify and label topics and topics relevant for use in the literature review. The result of the LDA model is presented as a list of topics and probabilities of each document (paper). The list is utilized to assign a paper to a topic by arranging the list by the highest probability for each paper for each topic. All the topics contain documents that are like each other. To reduce the risk of error in topic identification, a combination of inspecting the most frequent words for each topic and a paper view is used. After the topic review, it will present in the literature review.

Following the intelligent literature review, results of the LDA model should be approved or validated by statistical, semantic, or predictive means. Statistical validation defines the mutual information tests of result fit to model assumptions; semantics validation requires hand-coding to decide if the importance of specific words varies significantly and as expected with tasks to different topics which is used in the current study to validate LDA model result; and predictive validation refers to checking if events that ought to have expanded the prevalence of particular topic if out interpretations are right, did so [ 6 , 21 ].

LDA defines that each word in each document comes from a topic, and the topic is selected from a set of keywords. So we have two matrices:

ϴtd = P(t|d) which is the probability distribution of topics in documents

Фwt = P(w|t), which is the probability distribution of words in topics

And, we can say that the probability of a word given document, i.e., P(w|d), is equal to:

where T is the total number of topics; likewise, let’s assume there are W keywords for all the documents.

If we assume conditional independence, we can say that

And hence P(w|d) is equal to

that is the dot product of ϴtd and Фwt for each topic t.

Our systematic literature review identified 305 research papers after performing a traditional literature review. After executing LDA topic modeling, only 115 articles show the relevancy with our topic "machine learning application in clinical domain'. The following stages present LDA topic modeling process.

The 305 research papers were stacked into a Python environment then converted into a single text file. The seven steps have been carried out, described earlier in Pre-processing .

• Topic modeling

The two main parameters of the LDA topic model are the dictionary (id2word)-dictionary and the corpus—doc_term_matrix. The LDA model is created by running the command:

# Creating the object for LDA model using gensim library

LDA = gensim.models.ldamodel.LdaModel

# Build LDA model

lda_model = LDA(corpus=doc_term_matrix, id2word = dictionary, num_topics=20, random_state=100,

chunksize = 1000, passes=50,iterations=100)

In this model, ‘num_topics’ = 20, ‘chunksize’ is the number of documents used in each training chunk, and ‘passes’ is the total number of training passes.

Firstly, the LDA model is built with 20 topics; each topic is represented by a combination of 20 keywords, with each keyword contributing a certain weight to a topic. Topics are viewed and interpreted in the LDA model, such as Topic 0, represented as below:

(0, '0.005*"analysis" + 0.005*"study" + 0.005*"models" + 0.004*"prediction" + 0.003*"disease" + 0.003*"performance" + 0.003*"different" + 0.003*"results" + 0.003*"patient" + 0.002*"feature" + 0.002*"system" + 0.002*"accuracy" + 0.002*"diagnosis" + 0.002*"classification" + 0.002*"studies" + 0.002*"medicine" + 0.002*"value" + 0.002*"approach" + 0.002*"variables" + 0.002*"review"'),

Our approach to finding the ideal number of topics is to construct LDA models with different numbers of topics as K and select the model with the highest coherence value. Selecting the ‘K' value that denotes the end of the rapid growth of topic coherence ordinarily offers significant and interpretable topics. Picking a considerably higher value can provide more granular sub-topics if the ‘K’ selection is too large, which can cause the repetition of keywords in multiple topics.

Model perplexity and topic coherence values are − 8.855378536321144 and 0.3724024189689453, respectively. To measure the efficiency of the LDA model is lower the perplexity, the better the model is. Topics and associated keywords were then examined in an interactive chart using the pyLDAvis package, which presents the topics are 20 and most salient terms in those 20 topics, but these 20 topics overlap each other as shown in Fig.  5 , which means the keywords are repeated in these 20 topics and topics are overlapped, which means so decided to use num_topics = 9 and presented PyLDAvis Figure below. Each bubble on the left-hand side plot represents a topic. The bigger the bubble is, the more predominant that topic is. A decent topic will have a genuinely big, non-overlapping bubble dispersed throughout the graph instead of grouped in one quadrant. A topic model with many topics will typically have many overlaps, small-sized bubbles clustered in one locale of the graph, as shown in Fig.  6 .

PyLDAvis graph with 20 topics in the clinical domain

PyLDAvis graph with nine vital topics in the clinical domain

Each bubble addresses a generated topic. The larger the bubble, the higher percentage of the number of keywords in the corpus is about that topic which can be seen on the GitHub file. Blue bars address the general occurrence of each word in the corpus. If no topic is selected, the blue bars of the most frequently used words are displayed, as depicted in Fig.  6 .

The further the bubbles are away from each other, the more various they are. For example, we can tell that topic 1 is about patient information and studies utilized deep learning to analyze the disease, which can be seen in GitHub file codes ( https://github.com/MachineLearning-UON/Topic-modeling-using-LDA.git ) and presented in Fig.  7 .

PyLDAvis graph with topic 1

Red bars give the assessed number of times a given topic produced a given term. As you can see from Fig.  7 , there are around 4000 of the word 'analysis', and this term is utilized 1000 times inside topic 1. The word with the longest red bar is the most used by the keywords having a place with that topic.

A good topic model will have big and non-overlapping bubbles dispersed throughout the chart. As we can see from Fig.  6 , the bubbles are clustered within one place. One of the practical applications of topic modeling is discovering the topic in a provided document. We find out the topic number with the highest percentage contribution in that document, as shown in Fig.  8 .

Dominant topics with topic percentage contribution

The next stage is to process the discoveries and find a satisfactory depiction of the topics. A combination of evaluating the most continuous words utilized to distinguish the topic. For example, the most frequent words for the papers in topic 2 are "study" and "analysis", which indicate frequent words for ML usage in the clinical domain.

The topic name is displayed with the topic number from 0 to 8, which represents in the Table 6 , which includes the Topic number and Topic words.

The result represents the percentage of the topics in all documents, which presents that topic 0 and topic 6 have the highest percentage and used in 58 and 57 documents, respectively, with 115 papers. The result of this research was an overview of the exploration areas inside the paper corpus, addressed by 9 topics.

This paper presented a new methodology that is uncommon in scholarly publications. The methodology utilizes ML to investigate sample articles/papers to distinguish research directions. Even though the structure of the ML-based methodology has its restrictions, the outcomes and its ease of use leave a promising future for topic modeling-based systematic literature reviews.

The principal benefit of the methodological framework is that it gives information about an enormous number of papers, with little effort on the researcher's part, before time-exorbitant manual work is to be finished. By utilizing the framework, it is conceivable to rapidly explore a wide range of paper corpora and assess where the researcher's time and concentration should be spent. This is particularly significant for a junior researcher with minimal earlier information on a research field. If default boundaries and cleaning settings can be found for the steps in the framework, a completely programmed gathering of papers could be empowered, where limited works have been introduced to accomplish an overview of research directions.

From a literature review viewpoint, the advantage of utilizing the proposed framework is that the inclusion and exclusion selection of papers for a literature review will be delayed to a later stage where more information is given, resulting in a more educated dynamic interaction. The framework empowers reproducibility, as every step can be reproduced in the systematic review process that ultimately empowers with transparency. The whole process has been demonstrated as a case concept on GitHub by future researchers.

The study has introduced an intelligent literature review framework that uses ML to analyze existing research documents or articles. We demonstrate how topic modeling can assist literature review by reducing the manual screening of huge quantities of literature for more efficient use of researcher time. An LDA algorithm provides default parameters and data cleaning steps, reducing the effort required to review literature. An additional advantage of our framework is that the intelligent literature review offers accurate results with little time, and it comprises traditional ways to analyze literature and LDA topic modeling.

This framework is constructed in a step-by-step manner. Researchers can use it efficiently because it requires less technical knowledge than other ML algorithms. There is no restriction on the quantity of the research papers it can measure. This research extends knowledge to similar studies in this field [ 12 , 22 , 23 , 26 , 30 , 46 ] which present topic modeling. The study acknowledges the inspiring concept of smart literature defined by Asmussen and Møller [ 3 ]. The researchers previously provided a brief description of how LDA is utilized in topic modeling. Our research followed the basic idea but enhanced its significance to broaden its scale and focus on a specific domain such as the clinical domain to produce insights from existing research articles. For instance, Székely and Vom [ 46 ] utilized natural language processing to analyze 9514 sustainability reports published between 1999 and 2015. They identified 42 topics but did not develop any framework for future researchers. This was considered a significant gap in the research. Similarly, Kushwaha et al. [ 22 ] used a network analysis approach to analyze 10-year papers without providing any clear transparent outcome (e.g., how the research step-by-step produces an outcome). Likewise, Asmussen and Møller [ 3 ] developed a smart literature review framework that was limited to analyzing 650 sample articles through a single method. However, in our research, we developed an intelligent literature review that combines traditional and LDA topic modeling, so that future researchers can get assistance to gain effective knowledge regarding literature review when it becomes a state-of-the-art in research domains.

Our research developed a more effective intelligent framework, which combines traditional literature review and topic modeling using LDA, which provides more accurate and transparent results. The results are shared via public access on GitHub using this link https://github.com/MachineLearning-UON/Topic-modeling-using-LDA.git .

This paper focused on creating a methodological framework to empower researchers, diminishing the requirement for manually scanning documents and assigning the possibility to examine practically limitless. It would assist in capturing insights of an enormous number of papers quicker, more transparently, with more reliability. The proposed framework utilizes the LDA's topic model, which gathers related documents into topics.

A framework employed topic modeling for rapidly and reliably investigating a limitless number of papers, reducing their need to read individually, is developed. Topic modeling using the LDA algorithm can assist future researchers as they often need an outline of various research fields with minimal pre-existing knowledge. The proposed framework can empower researchers to review more papers in less time with more accuracy. Our intelligent literature review framework includes a holistic literature review process (conducting, planning, and reporting the review) and an LDA topic modeling (pre-processing, topic modeling, and post-processing stages), which conclude the results of 115 research articles are relevant to the search.

The automation of topic modeling with default parameters could also be explored to benefit non-technical researchers to explore topics or related keywords in any problem domain. For future directions, the principal points should be addressed. Future researchers in other research fields should apply the proposed framework to acquire information about the practical usage and gain ideas for additional advancement of the framework. Furthermore, research in how to consequently specify model parameters could extraordinarily enhance the ease of use for the utilization of topic modeling for non-specialized researchers, as the determination of model parameters enormously affects the outcome of the framework.

Future research may be utilized more ML analytics tools as complete solution artifacts to analyze different forms of big data. This could be adopting design science research methodologies for benefiting design researchers who are interested in building ML-based artifacts [ 15 , 28 , 29 , 31 , 32 , 33 ].

Availability of data and materials

Data will be supplied upon request.

LDA is a probabilistic method for topic modeling in text analysis, providing both a predictive and latent topic representation.

Abbreviations

The Institute of Electrical and Electronics Engineers

• Machine learning
• Latent Dirichlet Allocation

Organizational Capacity

Latent Semantic Indexing

Latent Semantic Analysis

Non-Negative Matrix Factorization

Parallel Latent Dirichlet Allocation

Pachinko Allocation Model

Abuhay TM, Kovalchuk SV, Bochenina K, Mbogo G-K, Visheratin AA, Kampis G, et al. Analysis of publication activity of computational science society in 2001–2017 using topic modelling and graph theory. J Comput Sci. 2018;26:193–204.

Article   Google Scholar  

Adlung L, Cohen Y, Mor U, Elinav E. Machine learning in clinical decision making. Med. 2021;2(6):642–65.

Asmussen CB, Møller C. Smart literature review: a practical topic modeling approach to exploratory literature review. J Big Data. 2019;6(1):1–18.

Beck CT. A meta-synthesis of qualitative research. MCN Am J Mater Child Nurs. 2002;27(4):214–21.

Behera RK, Bala PK, Dhir A. The emerging role of cognitive computing in healthcare: a systematic literature review. Int J Med Informatics. 2019;129:154–66.

Blei DM. Probabilistic topic models. Commun ACM. 2012;55(4):77–84.

Blei DM, Ng AY, Jordan MI. Latent Dirichlet allocation. J Mach Learn Res. 2003;3:993–1022.

MATH   Google Scholar  

Brnabic A, Hess LM. Systematic literature review of machine learning methods used in the analysis of real-world data for patient-provider decision making. BMC Med Inform Decis Mak. 2021;21(1):1–19.

Cabitza F, Locoro A, Banfi G. Machine learning in orthopedics: a literature review. Front Bioeng Biotechnol. 2018;6:75.

Chang C-H, Lin C-H, Lane H-Y. Machine learning and novel biomarkers for the diagnosis of Alzheimer’s disease. Int J Mol Sci. 2021;22(5):2761.

Connor KL, O’Sullivan ED, Marson LP, Wigmore SJ, Harrison EM. The future role of machine learning in clinical transplantation. Transplantation. 2021;105(4):723–35.

Dias R, Torkamani A. Artificial intelligence in clinical and genomic diagnostics. Genome Med. 2019;11(1):1–12.

DiMaggio P, Nag M, Blei D. Exploiting affinities between topic modeling and the sociological perspective on culture: application to newspaper coverage of US government arts funding. Poetics. 2013;41(6):570–606.

Forest P-G, Martin D. Fit for Purpose: Findings and recommendations of the external review of the Pan-Canadian Health Organizations: Summary Report: Health Canada Ottawa, ON; 2018.

Genemo H, Miah SJ, McAndrew A. A design science research methodology for developing a computer-aided assessment approach using method marking concept. Educ Inf Technol. 2016;21(6):1769–84.

Greene D, Cross JP. Exploring the political agenda of the european parliament using a dynamic topic modeling approach. Polit Anal. 2017;25(1):77–94.

Grimmer J. A Bayesian hierarchical topic model for political texts: measuring expressed agendas in Senate press releases. Polit Anal. 2010;18(1):1–35.

Grimmer J, Stewart BM. Text as data: the promise and pitfalls of automatic content analysis methods for political texts. Polit Anal. 2013;21(3):267–97.

Hassan N, Slight R, Weiand D, Vellinga A, Morgan G, Aboushareb F, et al. Preventing sepsis; how can artificial intelligence inform the clinical decision-making process? A systematic review. Int J Med Inform. 2021;150:104457.

Hirt R, Koehl NJ, Satzger G, editors. An end-to-end process model for supervised machine learning classification: from problem to deployment in information systems. Designing the Digital Transformation: DESRIST 2017 Research in Progress Proceedings of the 12th International Conference on Design Science Research in Information Systems and Technology Karlsruhe, Germany 30 May-1 Jun; 2017: Karlsruher Institut für Technologie (KIT).

Koltsova O, Koltcov S. Mapping the public agenda with topic modeling: the case of the Russian live journal. Policy Internet. 2013;5(2):207–27.

Kushwaha AK, Kar AK, Dwivedi YK. Applications of big data in emerging management disciplines: a literature review using text mining. Int J Inf Manag Data Insights. 2021;1(2):100017.

Google Scholar  

Li S, Wang H. Traditional literature review and research synthesis. The Palgrave handbook of applied linguistics research methodology. 2018:123–44.

Magrabi F, Ammenwerth E, McNair JB, De Keizer NF, Hyppönen H, Nykänen P, et al. Artificial intelligence in clinical decision support: challenges for evaluating AI and practical implications. Yearb Med Inform. 2019;28(01):128–34.

Maier D, Waldherr A, Miltner P, Wiedemann G, Niekler A, Keinert A, et al. Applying LDA topic modeling in communication research: toward a valid and reliable methodology. Commun Methods Meas. 2018;12(2–3):93–118.

Mårtensson G, Ferreira D, Granberg T, Cavallin L, Oppedal K, Padovani A, et al. The reliability of a deep learning model in clinical out-of-distribution MRI data: a multicohort study. Med Image Anal. 2020;66:101714.

Mendo IR, Marques G, de la Torre DI, López-Coronado M, Martín-Rodríguez F. Machine learning in medical emergencies: a systematic review and analysis. J Med Syst. 2021;45(10):1–16.

Miah SJ. An ontology based design environment for rural business decision support. Nathan: Griffith University Nathan; 2008.

Miah SJ, A new semantic knowledge sharing approach for e-government systems. 4th IEEE International Conference on Digital Ecosystems and Technologies; 2010: IEEE.

Miah SJ, Camilleri E, Vu HQ. Big Data in healthcare research: a survey study. J Comput Inf Syst. 2021. https://doi.org/10.1080/08874417.2020.1858727 .

Miah SJ, Gammack J, Kerr D, Ontology development for context-sensitive decision support. Third International Conference on Semantics, Knowledge and Grid (SKG 2007); 2007: IEEE.

Miah SJ, Gammack JG. Ensemble artifact design for context sensitive decision support. Australas J Inf Syst. 2014. https://doi.org/10.3127/ajis.v18i2.898 .

Miah SJ, Gammack JG, McKay J. A metadesign theory for tailorable decision support. J Assoc Inf Syst. 2019;20(5):4.

Mimno D, Blei D, editors. Bayesian checking for topic models. Proceedings of the 2011 conference on empirical methods in natural language processing; 2011.

Oala L, Murchison AG, Balachandran P, Choudhary S, Fehr J, Leite AW, et al. Machine learning for health: algorithm auditing & quality control. J Med Syst. 2021;45(12):1–8.

Ouhbi S, Idri A, Fernández-Alemán JL, Toval A. Requirements engineering education: a systematic mapping study. Requir Eng. 2015;20(2):119–38.

Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ. 2020;372:n71.

Quinn KM, Monroe BL, Colaresi M, Crespin MH, Radev DR. How to analyze political attention with minimal assumptions and costs. Am J Polit Sci. 2010;54(1):209–28.

Rowley J, Slack F. Conducting a literature review. Management research news. 2004.

Rozas LW, Klein WC. The value and purpose of the traditional qualitative literature review. J Evid Based Soc Work. 2010;7(5):387–99.

Sabharwal R, Miah SJ. A new theoretical understanding of big data analytics capabilities in organizations: a thematic analysis. J Big Data. 2021;8(1):1–17.

Salazar-Reyna R, Gonzalez-Aleu F, Granda-Gutierrez EM, Diaz-Ramirez J, Garza-Reyes JA, Kumar A. A systematic literature review of data science, data analytics and machine learning applied to healthcare engineering systems. Management Decision. 2020.

Shah P, Kendall F, Khozin S, Goosen R, Hu J, Laramie J, et al. Artificial intelligence and machine learning in clinical development: a translational perspective. NPJ Digit Med. 2019;2(1):1–5.

Sone D, Beheshti I. Clinical application of machine learning models for brain imaging in epilepsy: a review. Front Neurosci. 2021;15:761.

Spasic I, Nenadic G. Clinical text data in machine learning: systematic review. JMIR Med Inform. 2020;8(3):e17984.

Székely N, Vom Brocke J. What can we learn from corporate sustainability reporting? Deriving propositions for research and practice from over 9,500 corporate sustainability reports published between 1999 and 2015 using topic modelling technique. PLoS ONE. 2017;12(4):e0174807.

Verma D, Bach K, Mork PJ, editors. Application of machine learning methods on patient reported outcome measurements for predicting outcomes: a literature review. Informatics; 2021: Multidisciplinary Digital Publishing Institute.

Weng W-H. Machine learning for clinical predictive analytics. Leveraging data science for global health. Cham: Springer; 2020. p. 199–217.

Book   Google Scholar  

Yin Z, Sulieman LM, Malin BA. A systematic literature review of machine learning in online personal health data. J Am Med Inform Assoc. 2019;26(6):561–76.

Download references

Acknowledgements

Not applicable.

Author information

Authors and affiliations.

Newcastle Business School, The University of Newcastle, Newcastle, NSW, Australia

Renu Sabharwal & Shah J. Miah

You can also search for this author in PubMed   Google Scholar

Contributions

The first author conducted the research, while the second author has ensured quality standards and rewritten the entire findings linking to underlying theories. Both authors read and approved the final manuscript.

Corresponding author

Correspondence to Renu Sabharwal .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests, additional information, publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Sabharwal, R., Miah, S.J. An intelligent literature review: adopting inductive approach to define machine learning applications in the clinical domain. J Big Data 9 , 53 (2022). https://doi.org/10.1186/s40537-022-00605-3

Download citation

Received : 18 November 2021

Accepted : 06 April 2022

Published : 28 April 2022

DOI : https://doi.org/10.1186/s40537-022-00605-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Clinical research
• Systematic literature review

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Published: 15 May 2024

A large-scale machine learning analysis of inorganic nanoparticles in preclinical cancer research

• Bárbara B. Mendes   ORCID: orcid.org/0000-0001-8630-1119 1   na1 ,
• Zilu Zhang   ORCID: orcid.org/0009-0000-2180-5957 2   na1 ,
• João Conniot 1 ,
• Diana P. Sousa   ORCID: orcid.org/0000-0003-3474-5417 1 ,
• João M. J. M. Ravasco 1 ,
• Lauren A. Onweller   ORCID: orcid.org/0009-0004-0865-4495 2 ,
• Andżelika Lorenc   ORCID: orcid.org/0000-0002-1474-7864 3 , 4 ,
• Tiago Rodrigues   ORCID: orcid.org/0000-0002-1581-5654 3 ,
• Daniel Reker   ORCID: orcid.org/0000-0003-4789-7380 2 , 5 &
• João Conde   ORCID: orcid.org/0000-0001-8422-6792 1  

Nature Nanotechnology ( 2024 ) Cite this article

Metrics details

• Nanoparticles
• Nanotechnology in cancer

Owing to their distinct physical and chemical properties, inorganic nanoparticles (NPs) have shown promising results in preclinical cancer therapy, but designing and engineering them for effective therapeutic purposes remains a challenge. Although a comprehensive database of inorganic NP research is not currently available, it is crucial for developing effective cancer therapies. In this context, machine learning (ML) has emerged as a transformative tool, but its adaptation to nanomedicine is hindered by inexistent or small datasets. Here we assembled a large database of inorganic NPs, comprising experimental datasets from 745 preclinical studies in cancer nanomedicine. Using descriptive statistics and explainable ML models we mined this database to gain knowledge of inorganic NP design patterns and inform future NP research for cancer treatment. Our analyses suggest that NP shape and therapy type are prominent features in determining in vivo efficacy, measured as a percentage of tumour reduction. Moreover, our database provides a large-scale open-access resource for discriminative ML that the broader nanotechnology community can utilize. Our work blueprints data mining for translational cancer research and offers evidence for standardizing NP reporting to accelerate and de-risk inorganic NP-based drug delivery, which may help to improve patient outcomes in clinical settings.

This is a preview of subscription content, access via your institution

Access options

Access Nature and 54 other Nature Portfolio journals

Get Nature+, our best-value online-access subscription

24,99 € / 30 days

cancel any time

Subscribe to this journal

Receive 12 print issues and online access

251,40 € per year

only 20,95 € per issue

Buy this article

• Purchase on Springer Link
• Instant access to full article PDF

Prices may be subject to local taxes which are calculated during checkout

Data availability

The full database is publicly available via GitHub at https://github.com/RekerLab/NanoAnalysis . Source data are provided with this paper.

Code availability

The code to create our machine learning models and perform the analysis described here is publicly available via GitHub at https://github.com/RekerLab/NanoAnalysis .

Mendes, B. B., Sousa, D. P., Conniot, J. & Conde, J. Nanomedicine-based strategies to target and modulate the tumor microenvironment. Trends Cancer 7 , 847–862 (2021).

Article   CAS   PubMed   Google Scholar  

Bobo, D., Robinson, K. J., Islam, J., Thurecht, K. J. & Corrie, S. R. Nanoparticle-based medicines: a review of FDA-approved materials and clinical trials to date. Pharm. Res. 33 , 2373–2387 (2016).

Anselmo, A. C. & Mitragotri, S. Nanoparticles in the clinic: an update. Bioeng. Transl. Med. 4 , e10143 (2019).

Article   PubMed   PubMed Central   Google Scholar  

Anselmo, A. C. & Mitragotri, S. Nanoparticles in the clinic: an update post COVID-19 vaccines. Bioeng. Transl. Med. 6 , e10246 (2021).

Article   CAS   PubMed   PubMed Central   Google Scholar  

Mendes, B. B. et al. Nanodelivery of nucleic acids. Nat. Rev. Methods Primers 2 , 24 (2022).

van der Meel, R. et al. Smart cancer nanomedicine. Nat. Nanotechnol. 14 , 1007–1017 (2019).

Janjua, T. I., Cao, Y., Yu, C. & Popat, A. Clinical translation of silica nanoparticles. Nat. Rev. Mater. 6 , 1072–1074 (2021).

Das, C. G. A., Kumar, V. G., Dhas, T. S., Karthick, V. & Kumar, C. M. V. Nanomaterials in anticancer applications and their mechanism of action - a review. Nanomedicine 47 , 102613 (2023).

Gavas, S., Quazi, S. & Karpiński, T. M. Nanoparticles for cancer therapy: current progress and challenges. Nanoscale Res. Lett. 16 , 173 (2021).

Faria, M., Björnmalm, M., Crampin, E. J. & Caruso, F. A few clarifications on MIRIBEL. Nat. Nanotechnol. 15 , 2–3 (2020).

Faria, M. et al. Minimum information reporting in bio–nano experimental literature. Nat. Nanotechnol. 13 , 777–785 (2018).

Lorenc, A. et al. Machine learning for next-generation nanotechnology in healthcare. Matter 4 , 3078–3080 (2021).

Article   CAS   Google Scholar  

Mitchell, M. J. et al. Engineering precision nanoparticles for drug delivery. Nat. Rev. Drug Discov. 20 , 101–124 (2021).

Boehnke, N. et al. Massively parallel pooled screening reveals genomic determinants of nanoparticle delivery. Science 377 , eabm5551 (2023).

Article   Google Scholar  

Brockow, K. et al. Experience with polyethylene glycol allergy-guided risk management for COVID-19 vaccine anaphylaxis. Allergy 77 , 2200–2210 (2022).

Sellaturay, P., Nasser, S., Islam, S., Gurugama, P. & Ewan, P. W. Polyethylene glycol (PEG) is a cause of anaphylaxis to the Pfizer/BioNTech mRNA COVID-19 vaccine. Clin. Exp. Allergy 51 , 861–863 (2021).

Stone, C. A. Jr. et al. Immediate hypersensitivity to polyethylene glycols and polysorbates: more common than we have recognized. J. Allergy Clin. Immunol. Pract. 7 , 1533–1540.e8 (2019).

Article   PubMed   Google Scholar  

Chenthamara, D. et al. Therapeutic efficacy of nanoparticles and routes of administration. Biomater. Res. 23 , 20 (2019).

Sung, H. et al. Global cancer statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J. Clin. 71 , 209–249 (2021).

Global Burden of Disease 2019 Cancer Collaboration. Cancer incidence, mortality, years of life lost, years lived with disability, and disability-adjusted life years for 29 cancer groups from 2010 to 2019: a systematic analysis for the Global Burden of Disease Study 2019. JAMA Oncol. 8 , 420–444 (2022).

Alvarez, E. M. et al. The global burden of adolescent and young adult cancer in 2019: a systematic analysis for the Global Burden of Disease Study 2019. Lancet Oncol. 23 , 27–52 (2022).

Chen, Y., Chen, H. & Shi, J. In vivo bio-safety evaluations and diagnostic/therapeutic applications of chemically designed mesoporous silica nanoparticles. Adv. Mater. 25 , 3144–3176 (2013).

Iscaro, A., Howard, F. N. & Muthana, M. Nanoparticles: properties and applications in cancer immunotherapy. Curr. Pharm. Des. 25 , 1962–1979 (2019).

Zhou, H. et al. Biodegradable inorganic nanoparticles for cancer theranostics: insights into the degradation behavior. Bioconjug. Chem. 31 , 315–331 (2020).

Zhang, Y. et al. Prolonged local in vivo delivery of stimuli-responsive nanogels that rapidly release doxorubicin in triple-negative breast cancer cells. Adv. Healthc. Mater. 9 , 1901101 (2020).

Conde, J., Oliva, N., Zhang, Y. & Artzi, N. Local triple-combination therapy results in tumour regression and prevents recurrence in a colon cancer model. Nat. Mater. 15 , 1128–1138 (2016).

Kwong, B., Gai, S. A., Elkhader, J., Wittrup, K. D. & Irvine, D. J. Localized immunotherapy via liposome-anchored anti-CD137 + IL-2 prevents lethal toxicity and elicits local and systemic antitumor immunity. Cancer Res. 73 , 1547–1558 (2013).

Li, W. et al. Hyaluronic acid ion-pairing nanoparticles for targeted tumor therapy. J. Control. Release 225 , 170–182 (2016).

Lei, C. et al. Local release of highly loaded antibodies from functionalized nanoporous support for cancer immunotherapy. J. Am. Chem. Soc. 132 , 6906–6907 (2010).

Fransen, M. F., van der Sluis, T. C., Ossendorp, F., Arens, R. & Melief, C. J. M. Controlled local delivery of CTLA-4 blocking antibody induces CD8 + T-cell-dependent tumor eradication and decreases risk of toxic side effects. Clin. Cancer Res. 19 , 5381–5389 (2013).

Ishihara, J. et al. Matrix-binding checkpoint immunotherapies enhance antitumor efficacy and reduce adverse events. Sci. Transl. Med. 9 , eaan0401 (2017).

Errington, T. M., Denis, A., Perfito, N., Iorns, E. & Nosek, B. A. Challenges for assessing replicability in preclinical cancer biology. eLife 10 , e67995 (2021).

Wilhelm, S. et al. Analysis of nanoparticle delivery to tumours. Nat. Rev. Mater. 1 , 16014 (2016).

Cheng, Y.-H., He, C., Riviere, J. E., Monteiro-Riviere, N. A. & Lin, Z. Meta-analysis of nanoparticle delivery to tumors using a physiologically based pharmacokinetic modeling and simulation approach. ACS Nano 14 , 3075–3095 (2020).

Zhong, R. et al. Hydrogels for RNA delivery. Nat. Mater. https://doi.org/10.1038/s41563-023-01472-w (2023).

Lasagna-Reeves, C. et al. Bioaccumulation and toxicity of gold nanoparticles after repeated administration in mice. Biochem. Biophys. Res. Commun. 393 , 649–655 (2010).

Hatakeyama, H., Akita, H. & Harashima, H. A multifunctional envelope type nano device (MEND) for gene delivery to tumours based on the EPR effect: a strategy for overcoming the PEG dilemma. Adv. Drug Deliv. Rev. 63 , 152–160 (2011).

Harris, J. M., Martin, N. E. & Modi, M. Pegylation. Clin. Pharmacokinet. 40 , 539–551 (2001).

Suk, J. S., Xu, Q., Kim, N., Hanes, J. & Ensign, L. M. PEGylation as a strategy for improving nanoparticle-based drug and gene delivery. Adv. Drug Deliv. Rev. 99 , 28–51 (2016).

Zhang, M. et al. Influencing factors and strategies of enhancing nanoparticles into tumors in vivo. Acta Pharm. Sin. B 11 , 2265–2285 (2021).

Nguyen, L. N. M. et al. The exit of nanoparticles from solid tumours. Nat. Mater. 22 , 1261–1272 (2023).

Setyawati, M. I. et al. Titanium dioxide nanomaterials cause endothelial cell leakiness by disrupting the homophilic interaction of VE–cadherin. Nat. Commun. 4 , 1673 (2013).

Shamay, Y. et al. Quantitative self-assembly prediction yields targeted nanomedicines. Nat. Mater. 17 , 361–368 (2018).

Reker, D. et al. Computationally guided high-throughput design of self-assembling drug nanoparticles. Nat. Nanotechnol. 16 , 725–733 (2021).

Bannigan, P. et al. Machine learning models to accelerate the design of polymeric long-acting injectables. Nat. Commun. 14 , 35 (2023).

Lundberg, S. M. et al. From local explanations to global understanding with explainable AI for trees. Nat. Mach. Intell. 2 , 56–67 (2020).

Caballero, D. et al. Precision biomaterials in cancer theranostics and modelling. Biomaterials 280 , 121299 (2022).

Zhao, Y. et al. A comparison between sphere and rod nanoparticles regarding their in vivo biological behavior and pharmacokinetics. Sci. Rep. 7 , 4131 (2017).

Kolhar, P. et al. Using shape effects to target antibody-coated nanoparticles to lung and brain endothelium. Proc. Natl Acad. Sci. USA 110 , 10753–10758 (2013).

Zhang, M., Kim, H. S., Jin, T. & Moon, W. K. Near-infrared photothermal therapy using EGFR-targeted gold nanoparticles increases autophagic cell death in breast cancer. J. Photochem. Photobiol. B 170 , 58–64 (2017).

Jo, Y. et al. Chemoresistance of cancer cells: requirements of tumor microenvironment-mimicking in vitro models in anti-cancer drug development. Theranostics 8 , 5259–5275 (2018).

Guo, B. et al. Molecular engineering of conjugated polymers for biocompatible organic nanoparticles with highly efficient photoacoustic and photothermal performance in cancer theranostics. ACS Nano 11 , 10124–10134 (2017).

Li, Z. et al. Small gold nanorods laden macrophages for enhanced tumor coverage in photothermal therapy. Biomaterials 74 , 144–154 (2016).

Das, P., Delost, M. D., Qureshi, M. H., Smith, D. T. & Njardarson, J. T. A survey of the structures of US FDA approved combination drugs. J. Med. Chem. 62 , 4265–4311 (2019).

Fernandes Neto, J. M. et al. Multiple low dose therapy as an effective strategy to treat EGFR inhibitor-resistant NSCLC tumours. Nat. Commun. 11 , 3157 (2020).

Kim, M. H. et al. The effect of VEGF on the myogenic differentiation of adipose tissue derived stem cells within thermosensitive hydrogel matrices. Biomaterials 31 , 1213–1218 (2010).

Pedregosa, F. et al. Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12 , 2825–2830 (2011).

Google Scholar  

Chen, T. & Guestrin, C. XGBoost: a scalable tree boosting system. in Proc. of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining 785–794 (Association for Computing Machinery, 2016).

Lundberg, S. M. & Lee, S.-I. A unified approach to interpreting model predictions. in Advances in Neural Information Processing Systems (eds Guyon, I. et al.) Vol. 30 (Curran Associates, Inc., 2017).

Download references

Acknowledgements

This project received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (ERC-StG-2019-848325), the Duke Science and Technology Initiative and the National Institutes of Health NIGMS grant R35GM151255. We acknowledge Fundação para a Ciência e a Tecnologia (FCT) for financial support in the framework of the PhD grant 2020.06638.BD (D.P.S.), the Duke Department of Biomedical Engineering for support through a BME Fellowship (Z.Z.), the National Science Foundation (NSF) for support through the Graduate Research Fellowship DGE2129754 (L.A.O.) and the ERASMUS+ programme (A.L.).

Author information

These authors contributed equally: Bárbara B. Mendes, Zilu Zhang.

Authors and Affiliations

ToxOmics, NOVA Medical School, Faculdade de Ciências Médicas (NMS|FCM), Universidade NOVA de Lisboa, Lisbon, Portugal

Bárbara B. Mendes, João Conniot, Diana P. Sousa, João M. J. M. Ravasco & João Conde

Department of Biomedical Engineering, Duke University, Durham, NC, USA

Zilu Zhang, Lauren A. Onweller & Daniel Reker

Instituto de Investigação do Medicamento (iMed), Faculdade de Farmácia, Universidade de Lisboa, Lisbon, Portugal

Andżelika Lorenc & Tiago Rodrigues

Department of Biopharmacy, Ludwik Rydygier Collegium Medicum in Bydgoszcz, Nicolaus Copernicus University in Toruń, Bydgoszcz, Poland

Andżelika Lorenc

Duke Cancer Institute, Duke University School of Medicine, Durham, NC, USA

Daniel Reker

You can also search for this author in PubMed   Google Scholar

Contributions

J. Conde conceived the idea and concept of the study. D.R. conceived the ML platform. T.R. conceived the data curation. B.B.M., J. Conniot, D.P.S., J.M.J.M.R. and J. Conde collected all of the data from the published manuscripts, organized the dataset and calculated the correlations. Z.Z., L.A.O. and A.L. conducted the data analysis, text mining and designed, implemented and evaluated the ML models. J. Conde, D.R. and T.R provided guidance and supervised the work. All authors contributed to the writing and editing of the paper, and all authors approved the final version of the paper.

Corresponding authors

Correspondence to Tiago Rodrigues , Daniel Reker or João Conde .

Ethics declarations

Competing interests.

J. Conde and T.R. are co-founders and shareholders of TargTex SA Targeted Therapeutics for Glioblastoma Multiforme. J. Conde is a member of the Global Burden of Disease (GBD) consortium from the Institute for Health Metrics and Evaluation (IHME), University of Washington, USA, and member of the Scientific Advisory Board of Vector Bioscience, Cambridge. T.R. acts as a consultant to the pharmaceutical, biotechnology and technology industry and is a full member of the Acceleration Consortium, University of Toronto. D.R. acts as a consultant to the pharmaceutical and biotechnology industry, as a scientific mentor for Start2 and serves on the scientific advisory board of Areteia Therapeutics. The other authors declare no competing interests.

Peer review

Peer review information.

Nature Nanotechnology thanks Natalie Boehnke, Karolina Jagiełło and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplementary information.

Supplementary Results and Discussion, Figs. 1– 9, Tables 1–8 and Refs. 1–11.

Source data

Source data fig. 2.

Alluvial plot source data for Fig. 2c.

Source Data Fig. 4

Raw data for Fig. 4d, alluvial plot source data for Fig. 4e and statistical source data for Fig. 4f.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Cite this article.

Mendes, B.B., Zhang, Z., Conniot, J. et al. A large-scale machine learning analysis of inorganic nanoparticles in preclinical cancer research. Nat. Nanotechnol. (2024). https://doi.org/10.1038/s41565-024-01673-7

Download citation

Received : 03 May 2023

Accepted : 10 April 2024

Published : 15 May 2024

DOI : https://doi.org/10.1038/s41565-024-01673-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

A systematic review and meta-analysis of artificial neural network, machine learning, deep learning, and ensemble learning approaches in field of geotechnical engineering

• Open access
• Published: 13 May 2024

Cite this article

You have full access to this open access article

• Elaheh Yaghoubi 1 ,
• Elnaz Yaghoubi 1 ,
• Ahmed Khamees 2 &
• Amir Hossein Vakili   ORCID: orcid.org/0000-0001-8920-172X 3 , 4  

45 Accesses

Explore all metrics

Artificial neural networks (ANN), machine learning (ML), deep learning (DL), and ensemble learning (EL) are four outstanding approaches that enable algorithms to extract information from data and make predictions or decisions autonomously without the need for direct instructions. ANN, ML, DL, and EL models have found extensive application in predicting geotechnical and geoenvironmental parameters. This research aims to provide a comprehensive assessment of the applications of ANN, ML, DL, and EL in addressing forecasting within the field related to geotechnical engineering, including soil mechanics, foundation engineering, rock mechanics, environmental geotechnics, and transportation geotechnics. Previous studies have not collectively examined all four algorithms—ANN, ML, DL, and EL—and have not explored their advantages and disadvantages in the field of geotechnical engineering. This research aims to categorize and address this gap in the existing literature systematically. An extensive dataset of relevant research studies was gathered from the Web of Science and subjected to an analysis based on their approach, primary focus and objectives, year of publication, geographical distribution, and results. Additionally, this study included a co-occurrence keyword analysis that covered ANN, ML, DL, and EL techniques, systematic reviews, geotechnical engineering, and review articles that the data, sourced from the Scopus database through the Elsevier Journal, were then visualized using VOS Viewer for further examination. The results demonstrated that ANN is widely utilized despite the proven potential of ML, DL, and EL methods in geotechnical engineering due to the need for real-world laboratory data that civil and geotechnical engineers often encounter. However, when it comes to predicting behavior in geotechnical scenarios, EL techniques outperform all three other methods. Additionally, the techniques discussed here assist geotechnical engineering in understanding the benefits and disadvantages of ANN, ML, DL, and EL within the geo techniques area. This understanding enables geotechnical practitioners to select the most suitable techniques for creating a certainty and resilient ecosystem.

Avoid common mistakes on your manuscript.

1 Introduction

Geotechnical engineering involves investigating and utilizing naturally occurring materials, including soil, rock, and intermediate geomaterials, such as coal [ 1 , 2 ]. Among these materials, soil is distinguished due to its complex physical, mechanical, and chemical properties in engineering materials [ 3 , 4 ]. These materials exhibit inherent anisotropic and heterogeneous characteristics resulting from various origins and formation mechanisms, presenting difficulties in understanding and forecasting [ 5 , 6 ]. Traditionally, geotechnical engineers employ two primary approaches for investigating material behaviors: (1) laboratory and field tests and (2) numerical and analytical methods [ 7 , 8 ]. While laboratory and field tests offer descriptive insights, they often entail substantial costs and time commitments [ 7 , 9 ]. Conversely, numerical methods, like finite elements [ 10 , 11 , 12 ] or discrete analyses [ 12 , 13 ], provide cost-effective virtual assessments of geotechnical material behavior [ 7 , 14 ].

Computational intelligence and soft computing analyses have gained recognition due to the complex challenges encountered in various engineering applications. These approaches have gradually replaced the need for complex calculations [ 15 , 16 , 17 ]. There are numerous advantages to employing AI techniques in geotechnical engineering [ 18 ], including:

AI can model intricate and nonlinear processes without presuming initial input–output relationships [ 19 , 20 ].

AI demonstrates its effectiveness in forecasting, surveillance, choice-making, recognition, and classification in various situations [ 21 ].

AI has the capability to provide precise predictions even when there are no established physical parameter relationships available [ 22 ].

AI has the ability to process extensive datasets, identify patterns, and occasionally generate missing data [ 23 , 24 ].

Artificial neural networks (ANNs) can evaluate all feasible alternatives for a given project outcome using complex mathematical models and advanced software tools [ 25 , 26 ]. The integration of ANNs with optimization algorithms is essential to mitigate error rates, particularly in complex scenarios like compressed sensing [ 27 , 28 ]. ANN provides essential tools for geotechnical engineers in prominent consulting firms, enabling them to make quick and informed decisions, thereby improving performance and mitigating risks [ 29 ].

Geotechnical challenges are full of uncertainties and include different factors that avoid direct determination by engineers, leading to the quick adoption of machine learning (ML) techniques [ 30 , 31 , 32 ]. ML techniques can recognize potential correlations in data without any prior presumptions [ 33 , 34 , 35 , 36 ]. Additionally, deep learning (DL), a subfield of ML, aims to enhance the learning algorithms' capability to comprehend complex data. This is achieved using ANNs with multiple layers of interconnected nodes [ 37 ]. While DL has exhibited success in tackling learning challenges, its performance is influenced by various factors, and optimizing DL remains an ongoing focus of research in the field of AI [ 38 , 39 ]. Furthermore, as computational efficiency advances, ongoing investigations into AI and DL are taking place [ 40 , 41 ].

The primary objectives of this research are to comprehensively assess the applications of ANN, ML, DL, and EL in geotechnology forecasting and to establish a systematic categorization framework. Through the analysis of an extensive dataset, this study aims to provide insights into utilizing these techniques in addressing geotechnical challenges, enabling informed decision-making in this field. Table 1 , which serves as an abbreviation table, provides crucial references to assist readers in understanding the fundamental ideas presented in the paper. Figure  1 illustrates various sections covered in this review paper.

Outline of various sections covered in the current review paper

2 Literature review

Geotechnical engineering is a multidisciplinary field that encompasses various sub-disciplines within engineering and geology [ 1 , 8 ]. It involves the study of soil and rock behavior to ensure the stability, safety, and longevity of infrastructure and construction projects [ 8 , 9 , 10 ]. In structural engineering, it addresses foundation design and soil–structure interaction [ 42 , 43 ]. Construction engineering involves ground structures, excavation, soil improvement, and earthwork [ 8 , 44 ]. Environmental engineering focuses on geoenvironmental concerns, while earthquake engineering deals with seismic geotechnics and ground motions [ 8 , 9 , 45 ].

Mechanical engineering aspects include rock mechanics, soil mechanics, and ice mechanics [ 9 , 46 ]. Geology plays a role in geological engineering, geomaterials analysis, and geohazard assessment [ 8 , 9 ]. Hydraulic engineering covers earth dams, scouring, groundwater drainage, and marine geotechnics [ 8 , 47 ], while transportation engineering includes tunneling and road engineering [ 8 , 9 ]. Figure  2 illustrates these diverse sub-disciplines within the field of geotechnical engineering. Geotechnical engineers apply their expertise across these domains, ensuring the proper utilization of soil and rock properties in diverse construction and environmental contexts.

Overview of geotechnical engineering sub-disciplines

AI consists of a sophisticated collection of programming techniques [ 48 , 49 ]. Many of these techniques are founded on the idea that knowledge gaining, organization, access, and modification, in both humans and machines, form the basis for 'intelligent' decision-making [ 50 , 51 , 52 , 53 , 54 ]. AI techniques find application in a wide array of geographical issues, including modeling individual and collective decision-making and developing expert and 'intelligent' geographical information systems [ 55 ]. Geotechnical engineers employ various AI techniques to solve diverse challenges [ 56 ]. Adopting AI applications in geotechnical engineering has revolutionized the resources available to industry experts, providing them with advanced tools for in-depth data analysis and intricate modeling [ 57 , 58 ] decision-making [ 59 ]. Recent instances of GeoAI endeavors involve the identification of terrain features [ 60 , 61 ], the detection of densely distributed building footprints [ 62 , 63 , 64 ], the extraction of information from scanned historical maps [ 65 , 66 , 67 ], and semantic classification, such as with LiDAR point clouds [ 68 , 69 , 70 ], novel methods for spatial interpolation [ 71 ], and advances in traffic forecasting [ 72 , 73 , 74 ]. Integrating AI applications in this field enhances the analytical capabilities of industry professionals and fundamentally alters their decision-making processes [ 75 , 76 , 77 , 78 , 79 ]. Through precise data analysis and the application of dynamic modeling, AI enables professionals to optimize site selection, fine-tune design specifications, and adeptly anticipate and manage risks, ultimately leading to the successful and sustainable execution of geotechnical projects [ 80 ]. AI plays a pivotal role in advancing sustainable construction and infrastructure projects by efficiently allocating resources, reducing environmental impacts, and optimizing material usage, energy consumption, and waste management techniques in geotechnical engineering [ 81 , 82 , 83 ]. It is a powerful tool in sustainable construction, effectively managing resources to minimize environmental impacts [ 84 , 85 , 86 , 87 ]. By optimizing material distribution and utilization, reducing energy consumption, and limiting waste, AI not only results in cost savings but also significantly diminishes the ecological footprint in construction [ 88 , 89 , 90 ]. Furthermore, AI can also be utilized to enhance other critical factors, such as mechanical strength and bearing capacity [ 91 , 92 ]. This comprehensive approach utilizes AI’s capabilities to address a broader spectrum of considerations in construction, resulting in improved sustainability and performance. Nevertheless, the integration of AI in geotechnical engineering faces challenges, mainly due to the necessity for comprehensive and reliable data, particularly in specialized or remote projects [ 18 , 93 , 94 , 95 ]. Ensuring data quality is essential, emphasizing the significance of a balanced approach in developing and validating AI models [ 96 , 97 , 98 ]. Table 2 discusses some review articles by researchers using AI in geotechnical engineering from 2017 onwards. Also, these reviews have concentrated on one or two AI techniques. In contrast, this review article offers a comprehensive exploration of all four AI techniques (ANN, DL, ML, and EL) within the field of geotechnical engineering. The study showed a co-occurrence keyword analysis encompassing AI techniques (ANN, DL, ML, and EL), systematic review, geotechnical engineering, and review; the data were gathered from the Scopus database and then visualized utilizing VOS Viewer. The dimensions and annotations of each circle represent the importance of the corresponding keyword. Lines connecting them represent connections between these keywords. Various colors signify separate clusters, each associated with its own specialized domain of knowledge. Figure  3 visually represents the research trend observed from 2020 to 2023.

Application of AI methods in review papers within the field of geotechnical engineering, supported by total publications retrieved from the Scopus database

3 AI techniques and algorithms overview

The advent of big data, cloud computing, artificial neural networks, and machine learning has empowered engineers to develop machines capable of emulating human intelligence [ 107 , 108 ]. Expanding on these advancements, this research designates machines capable of perceiving, recognizing, learning, reacting, and problem-solving as AI [ 109 , 110 , 111 ]. This inevitably signifies a transformative influence on future workplaces, as AI has the potential to enhance human performance to higher standards [ 112 , 113 , 114 ]. Consequently, it is poised to emerge as the next groundbreaking innovation [ 115 ]. AI is classified into four distinct approaches, including artificial neural networks (ANN), machine learning (ML), deep learning (DL), and ensemble learning (EL). The categorization of these methods is depicted in Fig.  4 . Furthermore, for clarity, Table  3 offers a comprehensive comparison of these techniques across various dimensions. As demonstrated in Table  3 , the assessment of complexity, data requirements, and interpretability can vary depending on the specific architecture and algorithm. These characteristics can be influenced by factors such as data quality and domain expertise.

Various categorizations of AI

3.1 Artificial neural network (ANN) and its application in geotechnical engineering field

The development of the ANN appeared as a solution for tackling challenges involving complex patterns and predictions [ 120 , 121 ]. Inspired by the information processing mechanisms of the human brain, studies have defined the complex, multi-layered structure of ANN [ 122 , 123 ]. These neural networks consist of three essential layers: the input, hidden, and output [ 124 ]. Neurons are distributed across these layers in a multilayer ANN, each neuron serving as a crucial processing unit. The initial level, represented by the input layer, acquires information to reduce errors and enhance computations [ 125 , 126 ]. Consequently, the logical determination of the number of neurons is crucial. The input signal can move to subsequent layers due to the interconnectivity among neurons. Neuron weight signifies their capacity to communicate with one another; moreover, the weight and neuron count in preceding layers determine the number of neurons in each layer [ 127 , 128 ]. It is worth noting that the discretion of the number of hidden layers and neurons is possible. Like other networks, ANNs serve as an exceptional modeling tool for analysis. They excel in defining nonlinear network function evaluation, pattern recognition, data classification, simulation, clustering, and optimization, all essential features of AI [ 129 ]. ANN can be categorized into six distinct network types, which include:

Feed Forward Neural Network (FFNN) FFNN is a foundational framework within supervised ANNs, demonstrating notable proficiency in recognizing patterns [ 130 ]. It systematically handles information through input, hidden, and output layers linearly, devoid of feedback connections [ 131 ]. It is skilled at complex pattern learning, although it requires precise adjustment of hyperparameters to achieve the best results.

Back Propagation Neural Network (BPNN) The Backpropagation Neural Network (BPNN) is a widespread ANN used in supervised learning tasks. It demonstrates proficiency in comprehending complex relationships [ 132 ]. Functioning through interlinked layers, it refines weights using backpropagation to reduce output differences [ 133 ].

Radial Basis Function Neural Network (RBFNN) The RBFNN is a neural network model designed for various ANN applications [ 134 , 135 , 136 ]. It utilizes radial basis functions. This architecture enables it to excel in both pattern recognition and regression tasks [ 137 ]. By employing radial basis functions, it efficiently processes data and adjusts parameters dynamically, ensuring accurate and reliable results [ 138 , 139 ].

Bayesian Regression Neural Network (BRNN) The BRNN combines neural networks with Bayesian regression to represent complex models [ 140 , 141 ]. It utilizes neural networks to manage nonlinear patterns and employs Bayesian methods to measure uncertainty, making it advantageous for various applications [ 142 ].

Generalized Regression Neural Network (GRNN) The GRNN is a sophisticated model proficient in making predictions by estimating functions through a radial basis function strategy [ 143 , 144 ]. This feature makes it especially apt for efficient training and approximating smooth functions [ 145 , 146 , 147 ]. The GRNN, based on in radial basis functions (RBFs), is recognized for its effectiveness in regression assignments.

Differentiated Evolution Neural Network (DENN) DENN is a type of ANN that uses the differential evolution algorithm to optimize its network structure and parameters. The DENN integrates advanced evolution strategies in neural network training to accelerate convergence speed, improve solution quality, and enhance generalization capabilities for complex optimization tasks [ 148 , 149 ].

ANN can be employed for a range of tasks in geoengineering, including Soil Classification [ 93 , 150 , 151 ] and Property Estimation [ 93 , 152 , 153 ], Settlement and Settlement Prediction [ 93 , 154 , 155 , 156 ], Slope Stability Analysis [ 157 , 158 , 159 ], Seismic Hazard Assessment [ 160 , 161 ], Groundwater Flow Modeling [ 162 , 163 ], Tunneling and Excavation [ 164 , 165 , 166 ], Site Characterization [ 95 , 167 ], Risk Assessment [ 168 , 169 ], Material Behavior Modeling [ 170 , 171 ], and Optimization [ 172 , 173 ].

Different types of ANNs, including GRNN, DENN, BRNN, RBFNN, and FFNN, may be chosen based on the specific problem, data availability, and the desired level of complexity. For example, RBFNNs may be used for data interpolation and function approximation, while FFNNs are suitable for general regression and classification tasks. DENN, if applicable to geotechnical problems, may offer specific advantages in terms of optimization and adaptation [ 174 ].

Table 4 displays the employment of ANN techniques in geotechnical engineering.

From the information provided, it is clear that a diverse array of advanced AI techniques, including various types of neural networks and hybrid models, have been successfully utilized in research within the field of geotechnical engineering [ 104 , 192 ]. These approaches have addressed various geotechnical challenges, from soil property prediction to estimating material strengths and evaluating geotechnical structure performance. The outcomes substantiate the efficacy of AI-based models in providing accurate and dependable forecasts across different facets of geotechnical engineering. Furthermore, these models offer the potential to improve computational efficiency and make valuable contributions to advancing more sustainable practices in soil stabilization and subgrade construction. [ 8 , 100 , 104 , 176 ].

The study performed a keyword analysis, giving particular attention to the application of ANN techniques in the field of Geotechnical Engineering. The data were gathered from the Scopus database and then visualized utilizing VOS Viewer. Over the period from 2016 to 2023, a total of 1254 manuscripts were cumulatively published. The size and label of each circle correspond to the significance of the respective keyword. Connecting lines indicate relationships between the keywords. Different colors denote distinct clusters based on their specific areas of expertise, which is presented in Fig.  5 . Furthermore, based on data from the WOS database, a geographic analysis demonstrates the utilization of ANN techniques in geotechnical engineering between 2016 and 2023, as depicted in Fig.  6 .

Keywords related to ANN in the field of geotechnical engineering, extracted from the Scopus database

Utilization of ANN techniques in the analysis of geotechnical engineering, evidenced by total publications categorized by country in the WOS database

3.2 Machine learning (ML) and its application in geotechnical engineering field

ML represents a vital advancement in AI [ 121 ]. ML is achieved through iterative algorithms that learn from relevant data specific to a particular training task. This enables computers to recognize complex patterns and bring to light insights without the need for direct programming [ 193 ]. ML aims to automate analytical modeling, especially for tasks involving high-dimensional data, such as classification, regression, and clustering [ 121 ]. Different varieties of ML models contain:

Reinforcement learning: reinforcement learning involves training an agent to interact with its environment using feedback signals, aiming to develop a strategy that maximizes anticipated rewards. As indicated by [ 194 , 195 , 196 , 197 , 198 , 199 , 200 , 201 , 202 , 203 , 204 , 205 , 206 , 207 , 208 ], this type of ML can be classified into four methods.

Value-based methods focus on acquiring value functions (e.g., Q-values or state values) to guide action selection based on these estimates.

Policy-based methods directly learn policies to choose actions that lead to maximum expected rewards.

Actor-critic methods combine estimating value functions with policy optimization.

Model-based methods entail learning a model of the environment to plan and make decisions.

Unsupervised learning: unsupervised learning involves identifying patterns or structures in data without prior knowledge of the desired outcome. It is trained on data that lack labels, aiming to learn a representation that captures the inherent structure of the dataset [ 194 , 195 , 196 , 197 , 198 ]. This learning includes a variety of techniques, such as clustering, dimensionality reduction, density estimation, and anomaly detection [ 209 , 210 , 211 ]. Clustering groups similar data points based on specific features or similarities [ 196 , 212 , 213 ]. Dimensionality reduction methods aim to reduce the number of features while retaining important information [ 214 , 215 , 216 , 217 ]. Density estimation focuses on estimating the probability density function of a dataset [ 218 , 219 , 220 ]. Anomaly detection identifies data points that deviate significantly from expected or normal behavior [ 210 , 221 , 222 ]. Some well-known and commonly used techniques in Unsupervised Learning, such as Principal Component Analysis (PCA) [ 223 , 224 , 225 ], K-Means Clustering [ 226 , 227 , 228 ], Hierarchical Clustering [ 229 , 230 ], Gaussian Mixture Models (GMM) [ 231 , 232 , 233 ], are mentioned here.

Supervised learning: supervised learning trains an algorithm using labeled data, where each input corresponds to a known output. The algorithm learns to link input features with desired outcomes through these labeled examples. This learning process is divided into two main types: classification, which categorizes data into predefined classes, and regression, which predicts continuous numerical values. Classification yields distinct class labels, while regression deals with a range of continuous outputs [ 194 , 195 , 196 , 197 , 198 , 199 , 234 , 235 ]. Some widely recognized and commonly used techniques in supervised learning are mentioned here, including Linear Regression [ 236 ], Logistic Regression (LR) [ 237 , 238 ], Bayesian Linear regression (BLR) [ 239 , 240 ], Random Forest [ 241 , 242 ], Support Vector Machines (SVM) [ 243 , 244 ].

ML techniques, including supervised, unsupervised, and reinforcement learning, have a wide range of applications in geoengineering [ 245 , 246 ]. These techniques can enhance decision-making, optimize processes, and gain geotechnical and geographical data insights [ 247 ]. Supervised learning can be applied in slope stability analysis, foundation design, and material classification [ 248 , 249 , 250 ], while clustering for site characterization and dimensionality reduction uses unsupervised learning [ 251 , 252 ]. Additionally, reinforcement learning is applied for optimal excavation tunneling and resource management [ 253 , 254 ]. The successful application of ML in geoengineering depends on the availability of high-quality data, domain expertise, and careful model selection and validation. Table 5 is dedicated to ML methodologies designed to tackle specific challenges in geotechnical engineering.

Table 4 offers a summary of recent studies exploring the utilization of ML in geotechnical engineering. These investigations contain various subjects, ranging from soil classification and spatial interpolation to slope stability and rock mass categorization. Also it contains predictions for unconfined compressive strength (UCS), evaluations of soil layering, projections for shear strength of fiber-reinforced soil (FRS), estimations of cation exchange capacity (CEC), and assessments of gully erosion susceptibility. The results of these inquiries demonstrate the effectiveness of ML algorithms in dealing with various challenges within geotechnical engineering. ML models have demonstrated notable accuracy in tasks like soil classification, spatial property variability prediction, slope stability assessment, rock mass categorization, UCS prediction, identification of soil layers, FRS shear strength prediction, CEC estimation, and gully erosion susceptibility mapping [ 106 , 192 , 269 ]. Furthermore, the research emphasizes the significance of factors such as the quality and representativeness of the training dataset, model complexity, and the specific application context when deploying ML algorithms in geotechnical engineering. Additionally, further validation of ML models using new databases is often necessary to evaluate their broader applicability. According to a search of the Scopus database for Elsevier journal papers, researchers published 1,401 research papers on ML in geotechnical engineering between 2016 and 2023. Figure  7 shows these data, along with keywords related to Machine learning techniques in geotechnical engineering, extracted from the most relevant articles. Moreover, as indicated by the WOS database, a geographical data analysis demonstrates the application of ML techniques in geotechnical engineering between 2016 and 2023, as shown in Fig.  8 .

ML keywords in geotechnical engineering from the Scopus database

Utilization of ML techniques in the analysis of geotechnical engineering, evidenced by total publications categorized by country in the WOS database

3.3 Deep learning (DL) and its application in geotechnical engineering field

Recently, DL has demonstrated remarkable advancements and achievements across a wide range of fields [ 270 ]. DL, a subset of ML, aims to develop algorithms that can gradually comprehend complex data representations. This is accomplished by employing neural networks consisting of interconnected layers of nodes. DL algorithms typically use extensive datasets during training, enabling them to identify complex patterns and attain highly accurate predictions [ 271 ]. DL algorithms possess the unique capability to autonomously identify features, circumventing the need for ML algorithms, which accelerates data classification processes [ 121 ]. Additionally, DL demonstrates exceptional efficiency in handling substantial volumes of information within tight timeframes. Notably, one of the most noteworthy characteristics of DL is its capacity to enhance its intelligence over time continually [ 270 ]. Table 5 provides an overview of DL methods. Other DL approaches often integrate and complement the methods outlined in Table  6 to enhance overall efficiency and effectiveness. Also, Table  7 is specifically dedicated to the application of DL techniques in the field of geotechnical engineering.

DL techniques have found applications in various aspects of geoengineering due to their ability to process and analyze complex data patterns, including geological feature detection [ 93 , 105 , 286 ], landslide prediction[ 169 , 287 , 288 ], seismic data analysis [ 105 , 289 ], groundwater modeling [ 290 ], infrastructure monitoring [ 93 , 291 ], soil classification[ 292 , 293 ], geospatial data analysis [ 105 , 290 ], mining and resource management [ 294 , 295 ], environmental impact assessment [ 105 , 296 ]. To implement DL in geoengineering, access to relevant datasets, machine learning and deep learning expertise, and computing resources for model training will be required [ 297 , 298 ].

The research conducted a keyword analysis with a specific emphasis on the utilization of Deep Learning techniques in Geotechnical Engineering. It was found that researchers published 1,040 research papers on deep learning in this field between 2016 and 2023. The data were collected from the Scopus database and visualized using VOS Viewer, as illustrated in Fig.  9 ; this graphical representation captures the evolving research trends spanning from 2019 to 2023. Moreover, as indicated by the WOS database, a geographical data analysis demonstrates the application of DL techniques in geotechnical engineering between 2016 and 2023, as shown in Fig.  10 .

DL keywords in geotechnical engineering from the Scopus database

Utilization of DL techniques in the analysis of geotechnical engineering, evidenced by total publications categorized by country in the WOS database

3.4 Ensemble learning (EL) its application in geotechnical engineering field

EL, a method in ML, combines the forecasts of multiple models to enhance overall performance [ 299 , 300 ]. EL aims to improve predictive performance, accuracy, and generalization on various tasks [ 301 , 302 ]. Ensemble methods work best when the base models are diverse, meaning they make errors on different subsets of the data or have different approaches to solving the problem [ 303 , 304 ]. This diversity helps in reducing the overall error [ 305 , 306 ]. Several widely recognized EL techniques include:

Bagging (Bootstrap Aggregating): This ensemble method involves training multiple models on different subsets of the data, and their predictions are aggregated [ 307 , 308 , 309 ]. Random Forest [ 310 , 311 ] and Bagged Decision Trees [ 312 , 313 , 314 ] are the well-known methods in this category.

Boosting: Boosting enhances predictive performance by training weak models sequentially. Each model corrects the mistakes made by its predecessor, resulting in a strong learner within the ensemble [ 315 , 316 ]. AdaBoost (Adaptive Boosting) [ 317 , 318 , 319 ], Gradient Boosting Machines (GBM) [ 320 , 321 , 322 ], XGBoost (Extreme Gradient Boosting) [ 323 , 324 , 325 ], LightGBM (Light Gradient Boosting Machine) [ 326 , 327 ], and CatBoost (Categorical Boosting) [ 328 , 329 ] are widely recognized techniques within this classification.

Stacking Ensembles (SE): In this approach, a meta-model is trained to learn how to best combine predictions from the base models [ 330 , 331 ]. Stacking Classifier [ 332 , 333 , 334 ] and Stacking Regressor [ 335 , 336 ] are recognized techniques within the SE category.

Voting Ensembles (VE): Models in this ensemble provide predictions, and a majority vote determines the final output [ 337 , 338 ]. Hard Voting [ 339 , 340 , 341 ] and Soft Voting [ 342 , 343 , 344 ] are established methods within the VE classification.

In geotechnical engineering, EL technique is frequently employed to heighten the precision of soil and rock behavior predictions [ 345 ]. In geotechnical engineering, stacking is a frequently used EL method [ 346 , 347 ]. This involves training multiple ML models on the same dataset and combining their predictions to generate a conclusive forecast, which can be executed through techniques like weighted averaging or voting [ 348 ]. Another common technique utilized in geotechnical engineering is Bagging [ 349 , 350 , 351 ]. Multiple ML approaches are trained on distinct subsets of the dataset, and their predictions are aggregated to form a final prediction. This helps mitigate the overfitting of the models to the training data [ 352 , 353 , 354 ].

It is important to note that the choice of EL method and the specific application will depend on the nature of the geoengineering problem, the available data, and the goals of the analysis [ 93 , 286 , 355 , 356 ]. EL can significantly enhance the predictive capabilities and robustness of models in geoengineering, ultimately leading to safer and more effective engineering solutions [ 357 , 358 , 359 ]. EL is a developing field ready to have a revolutionary impact on geotechnical engineering. Table 8 provides a collection of recent studies that have successfully utilized EL to address a range of challenges in the field of geotechnical engineering. These include forecasting soil liquefaction susceptibility, categorizing rock mass quality, approximating lateral wall deflection in braced excavations, projecting soil properties through raw soil spectra data, and anticipating landslide susceptibility. These efforts emphasize the potential of EL in increasing the accuracy, efficiency, and reliability of geotechnical analyses and designs [ 357 , 363 ].

The data, sourced from the Scopus database, were subsequently visualized using VOS Viewer. Over the period from 2016 to 2023, researchers published 609 research papers on ensemble learning in geotechnical engineering. The size and label of each circle in the visualization indicate the significance of the respective keyword, while connecting lines signify relationships between them. Figure  11 presents these data along with keywords associated with Ensemble Learning (EL) approaches in geotechnical engineering, extracted from the most pertinent articles. Furthermore, according to the WOS database, the application of EL techniques in geotechnical engineering is demonstrated through geographical data analysis, as depicted in Fig.  12 , which visually depicts the research pattern observed from 2020 to 2023.

Keywords related to EL in the field of geotechnical engineering, extracted from the Scopus database

To gain insight into the performance of various EL-based models in the geotechnical field, Kardani et al. [ 293 ] examined the effectiveness of different EL techniques in predicting the resilient modulus of subgrade soils. They found that the bagging ensemble model outperformed other models tested, including the voting ensemble, voting ensemble with random forest, and stacking ensemble. Their conclusion was that the bagging ensemble outperformed other methods, making it suitable for estimating the resilient modulus with superior performance and an acceptable degree of accuracy. This model not only demonstrated higher prediction accuracy and generalization ability, but also exhibited several advantages such as stability, reduced noise, and ease of use. On the other hand, learning the art of ensemble modeling can be challenging, and making incorrect selections may lead to reduced prediction precision. Additionally, ensemble modeling can be costly in terms of both time and space. However, additional research using various datasets should be conducted to predict different geotechnical parameters, ensuring the performance of the bagging ensemble methods and other EL-based methods. Therefore, it is strongly recommended to utilize EL-based methods in the geotechnical field for predicting mechanical, physical, and chemical properties of soils. Further research is necessary to make reliable decisions about their performance in the geotechnical area.

4 Discussions and challenges linked with AI in geotechnical engineering

ANN models are adaptable and capable of capturing complex patterns in data. However, these models require precise adjustment of hyperparameters to achieve peak performance [ 116 ]. The performance of ANN depends on factors such as architecture, data quality, and data quantity [ 124 , 365 , 366 ]. Therefore, ANN is able to be the best choice for small datasets or when interpretability is crucial.

Various techniques are included in ML models, such as supervised learning, unsupervised learning, or reinforcement learning, which are well suited for different tasks. The performance of ML models varies according to the algorithm and the data being utilized. In comparison to DL models, ML models are frequently found to be more interpretable [ 367 , 368 ]. They are considered a favorable option when there are limited data or the need for transparent models [ 117 ].

DL models, such as CNNs and RNNs, manage extensive, high-dimensional datasets proficiently. They can automatically acquire hierarchical features [ 369 , 370 ]. DL demands significant computational power, sizable datasets, and precise parameter optimization. Additionally, DL models may not always offer interpretability, which can present limitations in certain use cases [ 118 ].

EL combines multiple models to enhance predictive performance, often surpassing the performance of individual models. It achieves this by reducing overfitting and increasing robustness, making it suitable for diverse datasets and applications [ 119 ]. Furthermore, EL demonstrates a reduced susceptibility to noise and outliers [ 371 ].

Assessing ANN, ML, DL, and EL for geoengineering in terms of accuracy and performance can be a complex task, given that the efficacy of each method relies on diverse variables, such as the particular problem, dataset characteristics, and model setup. These approaches rely on geoengineering problems, the data, computational resources, and interpretability needs. Both ML and ANN demonstrate a moderate level of complexity and are mainly applied in the field of geotechnics. Notably, ML has attracted substantial attention from researchers due to its high interpretability and optimal performance, even with small data. This interest is substantiated by data from WOS covering the period from 2019 to 2023, which reveals that a significant number of articles published in Springer Nature, Elsevier, and IEEE journals within the geotechnical domain underscore the prevalent preference for employing ML among researchers in this field as shown in Fig.  13 .

Comparative analysis of esteemed journals (Springer, Elsevier, and IEEE) in the domains of ANN, ML, DL, and EL in geoengineering, 2019–2023, using the WOS database

Figure  13 illustrates the number of research papers published in reputable journals, such as those from Springer, Elsevier, and IEEE, focusing on the areas of ANN, ML, DL, and in the field of Geotechnical Engineering. These data have been sourced from WOS.

Based on data from the WOS database, ANN is frequently employed in geotechnical engineering, even though ML, DL, and EL methods have demonstrated substantial potential as illustrated in Fig.  14 . This preference for using ANN in geotechnical engineering may be attributed to the common requirement for real-world laboratory data frequently encountered by civil and geotechnical engineers or the potential limitation in expertise for effectively employing ML, DL, and EL methods in data-driven prediction. However, EL techniques consistently outperform the other three methods in the context of predicting geotechnical behaviors.

Comparison of various ANN, ML, DL, and EL methods used in geotechnical engineering based on the total number of publications from 2019 to 2023, using the WOS database

According to the data obtained from the WOS database, Fig.  14 provides an overview of the utilization of ANN, ML, DL, and EL approaches in geotechnical engineering from 2019 to 2023. The data clearly show that ANN has maintained its status as a consistently preferred technique within this field. Additionally, it is noteworthy that ML has exhibited a steady and upward trend over the years. In 2022 and 2023, researchers demonstrated a nearly equal preference for both ANN and ML techniques within the field of geotechnical engineering.

As depicted in Fig.  14 , it is evident that the EL methods have been consistently popular over the years. Notably, the utilization of EL in the field of geotechnical engineering experienced a substantial increase from 2021 to 2022, reaching its peak adoption rate during this period. DL methods have not been widely adopted in recent years, but they started gaining recognition in geotechnical engineering in 2020. However, their popularity among geotechnical engineers remains limited due to the substantial amount of data required for accurate forecasting using this learning approach.

A widespread trend toward the utilization of artificial intelligence techniques, including ANN, ML, DL, and EL, in the field of geotechnical engineering is observed globally. This analysis, spanning from 2016 to 2023, involves the classification of data using WOS enabling thorough examination of transformations on a continental scale (refer to Fig.  15 ). This comparison reveals that this subject matter is actively embraced across all continents.

Continental comparison of ANN, ML, DL, and EL in geotechnical engineering (2016–2023) using WOS database

5 Future research directions and opportunities linked with AI application in geotechnical engineering

Future research in the field of geotechnical engineering and artificial intelligence (AI) should prioritize interdisciplinary collaboration, bringing together geotechnical engineering expertise and AI proficiency. This synergy has the potential to yield innovative solutions and provide a deeper understanding of how AI can effectively address the multifaceted challenges within geotechnical engineering. Furthermore, researchers should investigate geographical variations in the utilization of AI techniques in geotechnical engineering, examining how these methods are applied differently in various regions and identifying the factors influencing these variations. Additionally, the integration of AI for real-time monitoring and decision-making during geotechnical construction and operations should be explored, focusing on the development of adaptive AI-driven systems that can enhance safety and operational efficiency. Finally, researchers should delve into the concept of human–machine collaboration, examining how AI can assist geotechnical practitioners in decision-making, risk assessment, and project design. These research directions, aligned with the standards of scholarly articles, aim to foster innovation and provide practical solutions for the geotechnical engineering community. Figure  16 offers a visual depiction of the critical future research direction in the application of AI within the realm of geotechnical engineering.

Visual overview of key future research directions in the application of AI within the realm of geotechnical engineering

From a geotechnical engineering perspective, there are numerous topics that can still be studied and addressed in future research. One potential area of research is the application of various AI methods to predict the dynamic response of different soils, contingent on the availability of adequate datasets. In addition, a simple review of Tables 4 , 5 , 7 , and 8 and available papers in the field of geotechnical engineering confirms that soil improvement, as a hot topic in general, has received less attention from the AI approach. It is well known that soil properties, including soil gradation, consistency, compaction parameters, consolidation, dispersivity, collapsibility, swelling potential, durability, strength, elasticity, stress–strain curves, peck strain energy, resilient modulus, dynamic response, erodibility, chemical compositions, hydraulic conductivity, electrical conductivity, and liquefaction potential, can be altered through stabilization with traditional materials like lime and cement, or through the use of waste by-products such as lignosulfonate, travertine waste, red mud, sewage sludge, water treatment sludge, fly ash, various types of slags, as well as soil reinforcement using different materials like fibers and geosynthetic materials, or alternative soil improvement techniques such as electroosmosis [ 46 , 372 , 373 , 374 , 375 , 376 , 377 , 378 , 379 , 380 , 381 , 382 , 383 , 384 , 385 ]. However, it is evident that AI-based prediction of soil parameters after stabilization or reinforcement with various techniques and materials deserves more attention, especially considering the substantial number of experimental papers in this field and the availability of sufficient datasets. Therefore, future research studies can focus on the prediction of stabilized and reinforced soil parameters.

6 Conclusions

ANN, ML, DL, and EL are pivotal approaches for extracting valuable insights and making autonomous predictions in various fields, including geotechnology. This study aimed to comprehensively assess the applications of these techniques in geoengineering, filling a critical gap in the existing literature.

Evaluation of a vast dataset extracted from the Web of Science and Scopus databases revealed significant insights. ANN remains a widely used technique in geotechnical engineering, often due to the necessity for real-world laboratory data frequently encountered by civil and geotechnical engineers. Additionally, the expertise gap in effectively applying ML, DL, and EL methods for data-driven predictions may influence the preference for ANN. However, when it comes to predicting geotechnical behaviors, EL techniques consistently outperform the other three methods, showcasing their effectiveness in this domain.

Each of these techniques possesses its unique strengths and limitations. ANN models are adaptable and excel at capturing complex data patterns, but they require meticulous hyperparameter tuning and are suitable for scenarios with limited data or where interpretability is crucial. ML models encompass various techniques suitable for diverse tasks, offering interpretable solutions and being favored when data are limited. DL models handle high-dimensional data effectively but demand substantial computational resources and careful parameter optimization. Conversely, EL combines multiple models to enhance predictive performance, exhibiting robustness and reduced sensitivity to noise and outliers. The integration of ANN, ML, DL, and EL techniques has significantly contributed to advancing the field of geotechnology. Researchers and practitioners in this domain should continue to explore and harness the potential of these methodologies to address the evolving challenges in geotechnical engineering effectively.

Data availability

The datasets generated during and analyzed during the current study are available from the corresponding author on reasonable request.

Sun J, Huang Y (2022) Modeling the simultaneous effects of particle size and porosity in simulating geo-materials. Materials 15(4):1576

Article   Google Scholar  

Abuel-Naga HM, Bouazza A (2014) Numerical experiment-artificial intelligence approach to develop empirical equations for predicting leakage rates through GM/GCL composite liners. Geotext Geomembr 42(3):236–245

Xiong Z, Zhong L, Wang H, Li X (2021) Structural defects, mechanical behaviors, and properties of two-dimensional materials. Materials 14(5):1192

Razeghi HR, Ghadir P, Javadi AA (2022) Mechanical strength of saline sandy soils stabilized with alkali-activated cements. Sustainability 14(20):13669

Yu M, Hu Z, Zhou J, Lu Y, Guo W, Zhang Z (2023) Retrieving grain boundaries in 2D materials. Small 19(7):2205593

Kang DW, Choi KH, Lee SJ, Park BJ (2019) Mapping anisotropic and heterogeneous colloidal interactions via optical laser tweezers. J Phys Chem Lett 10(8):1691–1697

Wallace M, Ng K (2016) Development and application of underground space use in Hong Kong. Tunn Undergr Space Technol 55:257–279

Baghbani A, Choudhury T, Costa S, Reiner J (2022) Application of artificial intelligence in geotechnical engineering: a state-of-the-art review. Earth Sci Rev 228:103991

Beiranvand B, Rajaee T (2022) Application of artificial intelligence-based single and hybrid models in predicting seepage and pore water pressure of dams: a state-of-the-art review. Adv Eng Softw 173:103268

Assadi-Langroudi A et al (2022) Recent advances in nature-inspired solutions for ground engineering (NiSE). Int J Geosynth Ground Eng 8(1):3

Liu H, Maghoul P, Shalaby A, Thomson D (2023) Ultrasonic characterization of frozen soils using a multiphase poromechanical approach. Comput Geotech 153:105068

Liu H, Maghoul P, Mantelet G, Shalaby A (2022) GeoNDT: a fast general-purpose computational tool for geotechnical non-destructive testing applications. Acta Geotech 17(8):3515–3534

Liu C, Phan DT (2023) Analytical modeling of elastic moduli dispersion and poromechanical responses of a dual-porosity dual-permeability porous cylinder under dynamic forced deformation test. Rock Mech Rock Eng 56(3):2249–2269

Liu H, Maghoul P, Shalaby A (2020) Laboratory-scale characterization of saturated soil samples through ultrasonic techniques. Sci Rep 10(1):3216

Lee S, Lee SR, Kim Y (2003) An approach to estimate unsaturated shear strength using artificial neural network and hyperbolic formulation. Comput Geotech 30(6):489–503

Pujitha AK, Sivaswamy J (2018) Solution to overcome the sparsity issue of annotated data in medical domain. CAAI Trans Intell Technol 3(3):153–160

Xie H-B, Guo T, Bai S, Dokos S (2014) Hybrid soft computing systems for electromyographic signals analysis: a review. Biomed Eng Online 13(1):1–19

Sharma S, Ahmed S, Naseem M, Alnumay WS, Singh S, Cho GH (2021) A survey on applications of artificial intelligence for pre-parametric project cost and soil shear-strength estimation in construction and geotechnical engineering. Sensors 21(2):463

Hu EY, Bouteiller J-MC, Song D, Baudry M, Berger TW (2015) Volterra representation enables modeling of complex synaptic nonlinear dynamics in large-scale simulations. Front Comput Neurosci 9:112

Kawamura S, Deng M (2020) Recent developments on modeling for a 3-DOF micro-hand based on AI methods. Micromachines 11(9):792

Lozada DN, Carter AH (2020) Genomic selection in winter wheat breeding using a recommender approach. Genes 11(7):779

Raman DV, Anderson J, Papachristodoulou A (2017) Delineating parameter unidentifiabilities in complex models. Phys Rev E 95(3):032314

Raissi M, Yazdani A, Karniadakis GE (2020) Hidden fluid mechanics: learning velocity and pressure fields from flow visualizations. Science 367(6481):1026–1030

Article   MathSciNet   Google Scholar  

Hsu W et al (2015) An integrated, ontology-driven approach to constructing observational databases for research. J Biomed Inform 55:132–142

Shahin MA, Jaksa MB, Maier HR (2001) Artificial neural network applications in geotechnical engineering. Aust Geomech 36(1):49–62

Google Scholar  

Johnson JL (2018) Design of experiments and progressively sequenced regression are combined to achieve minimum data sample size. Int J Hydromechatron 1(3):308–331

Zhou Y, Sun Q, Liu J (2018) Robust optimisation algorithm for the measurement matrix in compressed sensing. CAAI Trans Intell Technol 3(3):133–139

Chen S, Du H, Wu L, Jin J, Qiu B (2017) Compressed sensing MRI via fast linearized preconditioned alternating direction method of multipliers. Biomed Eng Online 16:1–18

Kostić S, Vasović N, Todorović K, Samčović A (2016) Application of artificial neural networks for slope stability analysis in geotechnical practice. In: 2016 13th Symposium on neural networks and applications (NEUREL). IEEE, pp 1–6

Goh AT, Zhang W (2014) An improvement to MLR model for predicting liquefaction-induced lateral spread using multivariate adaptive regression splines. Eng Geol 170:1–10

Wang L, Wu C, Gu X, Liu H, Mei G, Zhang W (2020) Probabilistic stability analysis of earth dam slope under transient seepage using multivariate adaptive regression splines. Bull Eng Geol Environ 79:2763–2775

Zhang W, Zhang Y, Goh AT (2017) Multivariate adaptive regression splines for inverse analysis of soil and wall properties in braced excavation. Tunn Undergr Space Technol 64:24–33

Goh ATC, Zhang W, Zhang Y, Xiao Y, Xiang Y (2018) Determination of earth pressure balance tunnel-related maximum surface settlement: a multivariate adaptive regression splines approach. Bull Eng Geol Environ 77:489–500

van Natijne AL, Lindenbergh RC, Bogaard TA (2020) Machine learning: new potential for local and regional deep-seated landslide nowcasting. Sensors 20(5):1425

Zhang W, Goh AT, Zhang Y, Chen Y, Xiao Y (2015) Assessment of soil liquefaction based on capacity energy concept and multivariate adaptive regression splines. Eng Geol 188:29–37

Zhang W, Wu C, Li Y, Wang L, Samui P (2021) Assessment of pile drivability using random forest regression and multivariate adaptive regression splines. Georisk Assess Manag Risk Eng Syst Geohazards 15(1):27–40

Jin S, Zeng X, Xia F, Huang W, Liu X (2021) Application of deep learning methods in biological networks. Brief Bioinform 22(2):1902–1917

Zhan Z-H, Li J-Y, Zhang J (2022) Evolutionary deep learning: a survey. Neurocomputing 483:42–58

Mavaie P, Holder L, Beck D, Skinner MK (2021) Predicting environmentally responsive transgenerational differential DNA methylated regions (epimutations) in the genome using a hybrid deep-machine learning approach. BMC Bioinform 22(1):1–25

Nguyen G et al (2019) Machine learning and deep learning frameworks and libraries for large-scale data mining: a survey. Artif Intell Rev 52:77–124

Zhang W, Zhang R, Wang W, Zhang F, Goh ATC (2019) A multivariate adaptive regression splines model for determining horizontal wall deflection envelope for braced excavations in clays. Tunn Undergr Space Technol 84:461–471

Sigdel LD, Al-Qarawi A, Leo CJ, Liyanapathirana S, Hu P (2021) Geotechnical design practices and soil–structure interaction effects of an integral bridge system: a review. Appl Sci 11(15):7131

Tavakoli R, Kamgar R, Rahgozar R (2020) Optimal location of energy dissipation outrigger in high-rise building considering nonlinear soil–structure interaction effects. Period Polytech Civ Eng 64(3):887–903

Kim J, Lee S, Seo J, Lee D-E, Choi HS (2021) The integration of earthwork design review and planning using UAV-based point cloud and BIM. Appl Sci 11(8):3435

Onyelowe KC, Fazel Mojtahedi F, Golaghaei Darzi A, Kontoni D-PN (2023) Solving large deformation problems in geotechnical and geo-environmental engineering with the smoothed particle hydrodynamics: a state-of-the-art review of constitutive solutions. Environ Earth Sci 82(17):394

Vakili AH, Salimi M, Lu Y, Shamsi M, Nazari Z (2022) Strength and post-freeze–thaw behavior of a marl soil modified by lignosulfonate and polypropylene fiber: an environmentally friendly approach. Constr Build Mater 332:127364

Xu L, Peng X, Jiang H, An X, Xi X (2022) Distributive hydraulic engineering, cross-scale landscape planning, and climate change resilience: on the water-adaptive strategy in the Huai’an–Yangzhou Section of China’s Grand Canal. River Res Appl 39:1224

Sarker IH (2022) AI-based modeling: techniques, applications and research issues towards automation, intelligent and smart systems. SN Comput Sci 3(2):158

Kitsios F, Kamariotou M (2021) Artificial intelligence and business strategy towards digital transformation: a research agenda. Sustainability 13(4):2025

Bag S, Gupta S, Kumar A, Sivarajah U (2021) An integrated artificial intelligence framework for knowledge creation and B2B marketing rational decision making for improving firm performance. Ind Market Manag 92:178–189

Trunk A, Birkel H, Hartmann E (2020) On the current state of combining human and artificial intelligence for strategic organizational decision making. Bus Res 13(3):875–919

Harfouche A, Quinio B, Saba M, Saba PB (2023) The recursive theory of knowledge augmentation: integrating human intuition and knowledge in artificial intelligence to augment organizational knowledge. Inf Syst Front 25(1):55–70

Rajagopal NK et al (2022) Future of business culture: an artificial intelligence-driven digital framework for organization decision-making process. Complexity 2022:1–14

Grant R, Phene A (2022) The knowledge based view and global strategy: past impact and future potential. Glob Strategy J 12(1):3–30

Nishant R, Kennedy M, Corbett J (2020) Artificial intelligence for sustainability: challenges, opportunities, and a research agenda. Int J Inf Manag 53:102104

Samui P (2020) Application of artificial intelligence in geo-engineering. In: Information technology in geo-engineering: proceedings of the 3rd international conference (ICITG), Guimarães, Portugal 3. Springer, pp 30–44

Almajed A, Lemboye K, Moghal AAB (2022) A critical review on the feasibility of synthetic polymers inclusion in enhancing the geotechnical behavior of soils. Polymers 14(22):5004

Phoon K-K (2023) What geotechnical engineers want to know about reliability. ASCE-ASME J Risk Uncertain Eng Syst Part A Civ Eng 9(2):03123001

Sircar A, Yadav K, Rayavarapu K, Bist N, Oza H (2021) Application of machine learning and artificial intelligence in oil and gas industry. Pet Res 6(4):379–391

Li W, Hsu C-Y (2020) Automated terrain feature identification from remote sensing imagery: a deep learning approach. Int J Geogr Inf Sci 34(4):637–660

Xiong L, Li S, Tang G, Strobl J (2022) Geomorphometry and terrain analysis: Data, methods, platforms and applications. Earth-Sci Rev 233:104191

Feng L, Xu P, Tang H, Liu Z, Hou P (2023) National-scale mapping of building footprints using feature super-resolution semantic segmentation of Sentinel-2 images. GISci Remote Sens 60(1):2196154

Li S, Bao T, Liu H, Deng R, Zhang H (2023) Multilevel feature aggregated network with instance contrastive learning constraint for building extraction. Remote Sens 15(10):2585

Xie Y, Cai J, Bhojwani R, Shekhar S, Knight J (2020) A locally-constrained YOLO framework for detecting small and densely-distributed building footprints. Int J Geogr Inf Sci 34(4):777–801

Uhl JH, Leyk S, Chiang Y-Y, Knoblock CA (2022) Towards the automated large-scale reconstruction of past road networks from historical maps. Comput Environ Urban Syst 94:101794

Avcı C, Sertel E, Kabadayı ME (2022) Deep learning-based road extraction from historical maps. IEEE Geosci Remote Sens Lett 19:1–5

Duan W, Chiang Y-Y, Leyk S, Uhl JH, Knoblock CA (2020) Automatic alignment of contemporary vector data and georeferenced historical maps using reinforcement learning. Int J Geogr Inf Sci 34(4):824–849

Wen S, Wang T, Tao S (2022) Hybrid CNN-LSTM architecture for LiDAR point clouds semantic segmentation. IEEE Robot Autom Lett 7(3):5811–5818

Cai Y, Fan L, Atkinson PM, Zhang C (2022) Semantic segmentation of terrestrial laser scanning point clouds using locally enhanced image-based geometric representations. IEEE Trans Geosci Remote Sens 60:1–15

Diab A, Kashef R, Shaker A (2022) Deep learning for LiDAR point cloud classification in remote sensing. Sensors 22(20):7868

Zhu D, Cheng X, Zhang F, Yao X, Gao Y, Liu Y (2020) Spatial interpolation using conditional generative adversarial neural networks. Int J Geogr Inf Sci 34(4):735–758

Ren Y, Chen H, Han Y, Cheng T, Zhang Y, Chen G (2020) A hybrid integrated deep learning model for the prediction of citywide spatio-temporal flow volumes. Int J Geogr Inf Sci 34(4):802–823

Yin G et al (2022) A novel interpolation method to predict soil heavy metals based on a genetic algorithm and neural network model. Sci Total Environ 825:153948

Gao S, He D, Zhang Z, Tang X, Zhao Z (2023) A novel dynamic interpolation method based on both temporal and spatial correlations. Appl Intell 53(5):5100–5125

Jiang L, Qin X, Yam KC, Dong X, Liao W, Chen C (2023) Who should be first? How and when AI-human order influences procedural justice in a multistage decision-making process. PLoS ONE 18(7):e0284840

Monkul MM, Özhan HO (2021) Microplastic contamination in soils: a review from geotechnical engineering view. Polymers 13(23):4129

Javaid M, Haleem A, Singh RP, Suman R (2022) Artificial intelligence applications for industry 4.0: a literature-based study. J Ind Integr Manag 7(01):83–111

Mohiuddin Babu M, Akter S, Rahman M, Billah MM, Hack-Polay D (2022) The role of artificial intelligence in shaping the future of Agile fashion industry. Prod Plan Control. https://doi.org/10.1080/09537287.2022.2060858

Di Vaio A, Hassan R, Alavoine C (2022) Data intelligence and analytics: A bibliometric analysis of human–artificial intelligence in public sector decision-making effectiveness. Technol Forecast Soc Chang 174:121201

Eslami A, Nabizadeh A, Akbarzadeh Kasani H (2022) Geotechnical and geophysical characterisations of construction waste-infilled quarry for housing and commercial developments: case study of Tehran, Iran. Waste Manag Res 40(3):349–359

Barišić I, Netinger Grubeša I, Hackenberger DK, Palijan G, Glavić S, Trkmić M (2022) Multidisciplinary approach to agricultural biomass ash usage for earthworks in road construction. Materials 15(13):4529

Wang Y (2022) The impacts of improvements in the unified economic and environmental efficiency of transportation infrastructure on industrial structure transformation and upgrade from the perspective of resource factors. PLoS ONE 17(12):e0278722

Andeobu L, Wibowo S, Grandhi S (2022) Artificial intelligence applications for sustainable solid waste management practices in Australia: a systematic review. Sci Total Environ 834:155389

Hou Y, Dong Q, Wang D, Liu J (2023) Introduction to ‘Artificial intelligence in failure analysis of transportation infrastructure and materials.’ Philos Trans R Soc 381:20220177

Najjar MK, Figueiredo K, Evangelista ACJ, Hammad AW, Tam VW, Haddad A (2022) Life cycle assessment methodology integrated with BIM as a decision-making tool at early-stages of building design. Int J Constr Manag 22(4):541–555

Ogunmakinde OE, Egbelakin T, Sher W (2022) Contributions of the circular economy to the UN sustainable development goals through sustainable construction. Resour Conserv Recycl 178:106023

Xie Y, Zhao Y, Chen Y, Allen C (2022) Green construction supply chain management: Integrating governmental intervention and public–private partnerships through ecological modernisation. J Clean Prod 331:129986

Hammond GP, Li B (2016) Environmental and resource burdens associated with world biofuel production out to 2050: footprint components from carbon emissions and land use to waste arisings and water consumption. GCB Bioenergy 8(5):894–908

Yaro NSA et al (2023) A comprehensive overview of the utilization of recycled waste materials and technologies in asphalt pavements: towards environmental and sustainable low-carbon roads. Processes 11(7):2095

Srivastava PR, Mangla SK, Eachempati P, Tiwari AK (2023) An explainable artificial intelligence approach to understanding drivers of economic energy consumption and sustainability. Energy Econ 125:106868

Sikder A, Saha P, Singha PS (2023) Sugar industry waste produced geopolymer concrete and its compressive strength prediction via statistical analysis and artificial intelligence approach. Innov Infrastruct Solut 8(7):201

Ahmad M, Rashid K, Tariq Z, Ju M (2021) Utilization of a novel artificial intelligence technique (ANFIS) to predict the compressive strength of fly ash-based geopolymer. Constr Build Mater 301:124251

Zhang W, Gu X, Tang L, Yin Y, Liu D, Zhang Y (2022) Application of machine learning, deep learning and optimization algorithms in geoengineering and geoscience: comprehensive review and future challenge. Gondwana Res 109:1–17

Zhang W, Pradhan B, Stuyts B, Xu C (2023) Application of artificial intelligence in geotechnical and geohazard investigations. Geol J 58(6):2187–2194

Phoon K-K, Zhang W (2023) Future of machine learning in geotechnics. Georisk Assess Manag Risk Eng Syst Geohazards 17(1):7–22

McAlpine ED, Pantanowitz L, Michelow PM (2021) Challenges developing deep learning algorithms in cytology. Acta Cytol 65(4):301–309

Ruamviboonsuk P, Chantra S, Seresirikachorn K, Ruamviboonsuk V, Sangroongruangsri S (2021) Economic evaluations of artificial intelligence in ophthalmology. Asia-Pacific J Ophthalmol 10(3):307–316

Schoenherr JR, Abbas R, Michael K, Rivas P, Anderson TD (2023) Designing AI using a human-centered approach: explainability and accuracy toward trustworthiness. IEEE Trans Technol Soc 4(1):9–23

Jeremiah JJ, Abbey SJ, Booth CA, Kashyap A (2021) Results of application of artificial neural networks in predicting geo-mechanical properties of stabilised clays—a review. Geotechnics 1(1):147–171

Jong S, Ong D, Oh E (2021) State-of-the-art review of geotechnical-driven artificial intelligence techniques in underground soil–structure interaction. Tunn Undergr Space Technol 113:103946

Zhang W, Gu X, Hong L, Han L, Wang L (2023) Comprehensive review of machine learning in geotechnical reliability analysis: algorithms, applications and further challenges. Appl Soft Comput 136:110066

Agwu OE, Akpabio JU, Alabi SB, Dosunmu A (2018) Artificial intelligence techniques and their applications in drilling fluid engineering: a review. J Pet Sci Eng 167:300–315

Gao W (2018) A comprehensive review on identification of the geomaterial constitutive model using the computational intelligence method. Adv Eng Inform 38:420–440

Moayedi H, Mosallanezhad M, Rashid ASA, Jusoh WAW, Muazu MA (2020) A systematic review and meta-analysis of artificial neural network application in geotechnical engineering: theory and applications. Neural Comput Appl 32:495–518

Zhang W, Li H, Li Y, Liu H, Chen Y, Ding X (2021) Application of deep learning algorithms in geotechnical engineering: a short critical review. Artif Intell Rev 54:1–41

Reddy YR (2017) Applications of artificial intelligence and machine learning in geotechnical engineering. Int J Emerg Technol Innov Res 2349–5162

Kuang L et al (2021) Application and development trend of artificial intelligence in petroleum exploration and development. Pet Explor Dev 48(1):1–14

Hussain AA, Al-Turjman F (2021) Artificial intelligence and blockchain: a review. Trans Emerg Telecommun Technol 32(9):e4268

Pham ST, Sampson PM (2022) The development of artificial intelligence in education: a review in context. J Comput Assist Learn 38(5):1408–1421

Khan MA, Khojah M, Vivek V (2022) Artificial intelligence and big data: the advent of new pedagogy in the adaptive e-learning system in the higher educational institutions of Saudi Arabia. Educ Res Int 2022:1–10

Kumar K, Thakur GSM (2012) Advanced applications of neural networks and artificial intelligence: a review. Int J Inf Technol Comput Sci 4(6):57

Dwivedi YK et al (2021) Artificial intelligence (AI): multidisciplinary perspectives on emerging challenges, opportunities, and agenda for research, practice and policy. Int J Inf Manag 57:101994

Khogali HO, Mekid S (2023) The blended future of automation and AI: examining some long-term societal and ethical impact features. Technol Soc 73:102232

Horáková T, Houška M, Dömeová L (2017) Classification of the educational texts styles with the methods of artificial intelligence. J Balt Sci Educ 16(3):324

Lawler RW, Rushby N (2013) An interview with Robert Lawler. Br J Edu Technol 44(1):20–30

Zurada J (1992) Introduction to artificial neural systems. West Publishing Co., Eagan

Zhou Z-H (2021) Machine learning. Springer, Singapore

Book   Google Scholar  

Kelleher JD (2019) Deep learning. MIT Press, Cambridge

Zhou Z-H, Zhou Z-H (2021) Ensemble learning. Springer, Singapore

Kurani A, Doshi P, Vakharia A, Shah M (2023) A comprehensive comparative study of artificial neural network (ANN) and support vector machines (SVM) on stock forecasting. Ann Data Sci 10(1):183–208

Wazirali R, Yaghoubi E, Abujazar MSS, Ahmad R, Vakili AH (2023) State-of-the-art review on energy and load forecasting in microgrids using artificial neural networks, machine learning, and deep learning techniques. Electr Power Syst Res 225:109792

Fitz S, Romero P (2021) Neural networks and deep learning: a paradigm shift in information processing, machine learning, and artificial intelligence. In: Rau R, Wardrop R, Zingales L (eds) The Palgrave handbook of technological finance. Springer, Cham, pp 589–654

Chapter   Google Scholar  

Wang S, Cheng TH, Lim MH (2022) A hierarchical taxonomic survey of spiking neural networks. Memet Comput 14(3):335–354

Kalinić Z, Marinković V, Kalinić L, Liébana-Cabanillas F (2021) Neural network modeling of consumer satisfaction in mobile commerce: an empirical analysis. Expert Syst Appl 175:114803

Uzair M, Jamil N (2020) Effects of hidden layers on the efficiency of neural networks. In: 2020 IEEE 23rd international multitopic conference (INMIC). IEEE, pp 1–6

Dey P (2022) Artificial neural network in diagnostic cytology. CytoJournal 19:27

Shah A et al (2023) A comprehensive study on skin cancer detection using artificial neural network (ANN) and convolutional neural network (CNN). Clin eHealth 6:76

Gholami V, Sahour H (2022) Simulation of rainfall-runoff process using an artificial neural network (ANN) and field plots data. Theor Appl Climatol 147:1–12

Kim D, Hur J (2018) Short-term probabilistic forecasting of wind energy resources using the enhanced ensemble method. Energy 157:211–226

Li H, Zhang L (2020) A bilevel learning model and algorithm for self-organizing feed-forward neural networks for pattern classification. IEEE Trans Neural Netw Learn Syst 32(11):4901–4915

Abdolrasol MG et al (2021) Artificial neural networks based optimization techniques: a review. Electronics 10(21):2689

Abba SI et al (2020) Implementation of data intelligence models coupled with ensemble machine learning for prediction of water quality index. Environ Sci Pollut Res 27:41524–41539

Pozzi I, Bohte S, Roelfsema P (2020) Attention-gated brain propagation: how the brain can implement reward-based error backpropagation. Adv Neural Inf Process Syst 33:2516–2526

Ojo S, Imoize A, Alienyi D (2021) Radial basis function neural network path loss prediction model for LTE networks in multitransmitter signal propagation environments. Int J Commun Syst 34(3):e4680

Heidari A, Navimipour NJ, Unal M (2023) A secure intrusion detection platform using blockchain and radial basis function neural networks for internet of drones. IEEE Internet Things J 10:8445

Liu G, Hou Z (2020) Cooperative adaptive iterative learning fault-tolerant control scheme for multiple subway trains. IEEE Trans Cybernet 52(2):1098–1111

Fath AH, Madanifar F, Abbasi M (2020) Implementation of multilayer perceptron (MLP) and radial basis function (RBF) neural networks to predict solution gas–oil ratio of crude oil systems. Petroleum 6(1):80–91

Shen Y, Pan X, Zheng Z, Gerstoft P (2020) Matched-field geoacoustic inversion based on radial basis function neural network. J Acoust Soc Am 148(5):3279–3290

Quan H, Dong S, Zhao D, Li H, Geng J, Liu H (2023) Generic AI models for mass transfer coefficient prediction in amine-based CO 2 absorber, Part II: RBFNN and RF model. AIChE J 69(1):e17904

Pérez-Rodríguez P, Flores-Galarza S, Vaquera-Huerta H, del Valle-Paniagua DH, Montesinos-López OA, Crossa J (2020) Genome-based prediction of Bayesian linear and non-linear regression models for ordinal data. Plant Genome 13(2):e20021

Li J et al (2022) Probability prediction approach of fatigue failure for the subsea wellhead using bayesian regularization artificial neural network. J Mar Sci Eng 10(11):1627

Sun W, Paiva AR, Xu P, Sundaram A, Braatz RD (2020) Fault detection and identification using Bayesian recurrent neural networks. Comput Chem Eng 141:106991

Emayavaramban G et al (2021) SEMG based classification of hand gestures using artificial neural network. Mater Today Proc 37:2591–2598

Dou M, Qin C, Li G, Wang C (2020) Research on calculation method of free flow discharge based on artificial neural network and regression analysis. Flow Meas Instrum 72:101707

Jafari M, Shahsavar A (2020) The application of artificial neural networks in modeling and predicting the effects of melatonin on morphological responses of citrus to drought stress. PLoS ONE 15(10):e0240427

Zheng X et al (2022) Comparison of GRNN and RF algorithms for predicting heat transfer coefficient in heat exchange channels with bulges. Appl Therm Eng 217:119263

Salgado C, Dam R, Salgado W, Werneck R, Pereira C, Schirru R (2020) The comparison of different multilayer perceptron and general regression neural networks for volume fraction prediction using MCNPX code. Appl Radiat Isot 162:109170

Baioletti M, Di Bari G, Milani A, Poggioni V (2020) Differential evolution for neural networks optimization. Mathematics 8(1):69

Haritha K, Shailesh S, Judy M, Ravichandran K, Krishankumar R, Gandomi AH (2023) A novel neural network model with distributed evolutionary approach for big data classification. Sci Rep 13(1):11052

Khan MS, Ivoke J, Nobahar M, Amini F (2022) Artificial neural network (ANN) based soil temperature model of highly plastic clay. Geomech Geoeng 17(4):1230–1246

Pawar A, Jolly A, Pandey V, Chaurasiya PK, Verma TN, Meshram K (2023) Artificial intelligence algorithms for prediction of cyclic stress ratio of soil for environment conservation. Environ Chall 12:100730

Mohammadi M, Fatemi Aghda SM, Talkhablou M, Cheshomi A (2022) Prediction of the shear strength parameters from easily-available soil properties by means of multivariate regression and artificial neural network methods. Geomech Geoeng 17(2):442–454

Özdemir E (2022) A new predictive model for uniaxial compressive strength of rock using machine learning method: artificial intelligence-based age-layered population structure genetic programming (ALPS-GP). Arab J Sci Eng 47(1):629–639

Aouadj A, Bouafia A (2022) CPT-based method using hybrid artificial neural network and mathematical model to predict the load-settlement behaviour of shallow foundations. Geomech Geoeng 17(1):321–333

Sasmal SK, Behera RN (2022) Transient settlement estimation of shallow foundation under eccentrically inclined static and cyclic load on granular soil using artificial intelligence techniques. Geomech Geoeng 18:1–17

Zhang N, Zhou A, Pan Y, Shen S-L (2021) Measurement and prediction of tunnelling-induced ground settlement in karst region by using expanding deep learning method. Measurement 183:109700

Bardhan A, Samui P (2022) Probabilistic slope stability analysis of Heavy-haul freight corridor using a hybrid machine learning paradigm. Transp Geotech 37:100815

Gao W, Raftari M, Rashid ASA, Mu’azu MA, Jusoh WAW (2020) A predictive model based on an optimized ANN combined with ICA for predicting the stability of slopes. Eng Comput 36:325–344

Ahangari Nanehkaran Y et al (2022) Application of machine learning techniques for the estimation of the safety factor in slope stability analysis. Water 14(22):3743

Broccardo M et al (2020) Induced seismicity risk analysis of the hydraulic stimulation of a geothermal well on Geldinganes, Iceland. Nat Hazard 20(6):1573–1593

Convertito V, Ebrahimian H, Amoroso O, Jalayer F, De Matteis R, Capuano P (2021) Time-dependent seismic hazard analysis for induced seismicity: the case of St Gallen (Switzerland), geothermal field. Energies 14(10):2747

Cahyadi TA, Syihab Z, Widodo LE, Notosiswoyo S, Widijanto E (2021) Analysis of hydraulic conductivity of fractured groundwater flow media using artificial neural network back propagation. Neural Comput Appl 33:159–179

Di Salvo C (2022) Improving results of existing groundwater numerical models using machine learning techniques: a review. Water 14(15):2307

Erharter GH, Marcher T, Reinhold C (2020) Artificial neural network based online rockmass behavior classification of TBM data. In: Information technology in geo-engineering: proceedings of the 3rd international conference (ICITG), Guimarães, Portugal 3. Springer, pp 178–188

Ling J, Li X, Li H, Shen Y, Rui Y, Zhu H (2022) Data acquisition-interpretation-aggregation for dynamic design of rock tunnel support. Autom Constr 143:104577

Liu J, Jiang Y, Han W, Sakaguchi O (2021) Optimized ANN model for predicting rock mass quality ahead of tunnel face using measure-while-drilling data. Bull Eng Geol Environ 80:2283–2305

Shahri AA, Shan C, Zäll E, Larsson S (2021) Spatial distribution modeling of subsurface bedrock using a developed automated intelligence deep learning procedure: a case study in Sweden. J Rock Mech Geotech Eng 13(6):1300–1310

Reddy YR (2022) Reducing the risks in geotechnical engineering using artificial intelligence techniques. Int J Emerg Technol Innov Res 2349–5162

Nanehkaran YA et al (2023) Riverside landslide susceptibility overview: leveraging artificial neural networks and machine learning in accordance with the United Nations (UN) sustainable development goals. Water 15(15):2707

Onyelowe KC, Aneke FI, Onyia ME, Ebid AM, Usungedo T (2022) AI (ANN, GP, and EPR)-based predictive models of bulk density, linear-volumetric shrinkage & desiccation cracking of HSDA-treated black cotton soil for sustainable subgrade. Geomech Geoeng 18:1–20

Baghbani A, Costa S, Faradonbeh RS, Soltani A, Baghbani H (2023) Modeling the effects of particle shape on damping ratio of dry sand by simple shear testing and artificial intelligence. Appl Sci 13(7):4363

Liu L, Moayedi H, Rashid ASA, Rahman SSA, Nguyen H (2020) Optimizing an ANN model with genetic algorithm (GA) predicting load-settlement behaviours of eco-friendly raft-pile foundation (ERP) system. Eng Comput 36:421–433

Li J et al (2022) Facilitate geoengineering and completions with machine learning methods: case study from ordos tight oil field. In: Abu Dhabi international petroleum exhibition and conference. SPE, p D021S062R003

Liu S, Chang R, Zuo J, Webber RJ, Xiong F, Dong N (2021) Application of artificial neural networks in construction management: current status and future directions. Appl Sci 11(20):9616

Armaghani DJ, Mirzaei F, Shariati M, Trung NT, Shariati M, Trnavac D (2020) Hybrid ANN-based techniques in predicting cohesion of sandy-soil combined with fiber. Geomech Eng 20(3):191–205

Koopialipoor M, Fahimifar A, Ghaleini EN, Momenzadeh M, Armaghani DJ (2020) Development of a new hybrid ANN for solving a geotechnical problem related to tunnel boring machine performance. Eng Comput 36:345–357

Pham V-N, Do H-D, Oh E, Ong DE (2021) Prediction of unconfined compressive strength of cement-stabilized sandy soil in Vietnam using artificial neural networks (ANNs) model. Int J Geotech Eng 15(9):1177–1187

Zhang P, Yin Z-Y, Jin Y-F (2022) Bayesian neural network-based uncertainty modelling: application to soil compressibility and undrained shear strength prediction. Can Geotech J 59(4):546–557

Asare EN, Affam M, Ziggah YY (2023) A hybrid intelligent prediction model of autoencoder neural network and multivariate adaptive regression spline for uniaxial compressive strength of rocks. Model Earth Syst Environ 9:1–17

Narmandakh D, Butscher C, Ardejani FD, Yang H, Nagel T, Taherdangkoo R (2023) The use of feed-forward and cascade-forward neural networks to determine swelling potential of clayey soils. Comput Geotech 157:105319

Liu Y, Yang Z, Li X (2022) Adaptive ensemble learning of radial basis functions for efficient geotechnical reliability analysis. Comput Geotech 146:104753

Zhang L, Du Y-H, Yang X-J, Fan H-H (2022) Application of artificial neural network in predicting the dispersibility of soil. Iran J Sci Technol Trans Civ Eng 46(3):2315–2324

Williams CG, Ojuri OO (2021) Predictive modelling of soils’ hydraulic conductivity using artificial neural network and multiple linear regression. SN Appl Sci 3:1–13

Vakili AH, Davoodi S, Arab A, Selamat MB (2015) Use of artificial neural network in predicting permeability of dispersive clay treated with lime and pozzolan. IJSRES 3(1):23–37

Shaik S, Krishna KSR, Abbas M, Ahmed M, Mavaluru D (2019) Applying several soft computing techniques for prediction of bearing capacity of driven piles. Eng Comput 35:1463–1474

Hong C, Luo G, Chen W (2022) Safety analysis of a deep foundation ditch using deep learning methods. Gondwana Res 123:16

Bunawan AR, Momeni E, Armaghani DJ, Rashid ASA (2018) Experimental and intelligent techniques to estimate bearing capacity of cohesive soft soils reinforced with soil–cement columns. Measurement 124:529–538

Moayedi H, Gör M, Khari M, Foong LK, Bahiraei M, Bui DT (2020) Hybridizing four wise neural-metaheuristic paradigms in predicting soil shear strength. Measurement 156:107576

Jaafari A et al (2019) Meta optimization of an adaptive neuro-fuzzy inference system with grey wolf optimizer and biogeography-based optimization algorithms for spatial prediction of landslide susceptibility. CATENA 175:430–445

Goudjil K, Arabet L (2021) Assessment of deflection of pile implanted on slope by artificial neural network. Neural Comput Appl 33(4):1091–1101

Arabet L, Hidjeb M, Belaabed F (2022) A comparative study of reinforced soil shear strength prediction by the analytical approach and artificial neural networks. Eng Technol Appl Sci Res 12(6):9795–9801

Ebid AM (2021) 35 Years of (AI) in geotechnical engineering: state of the art. Geotech Geol Eng 39(2):637–690

Hamerly G, Elkan C (2002) Alternatives to the k-means algorithm that find better clusterings. In: Proceedings of the eleventh international conference on Information and knowledge management, pp 600–607

Mahesh B (2020) Machine learning algorithms—a review. Int J Sci Res 9(1):381–386

An Q, Rahman S, Zhou J, Kang JJ (2023) A comprehensive review on machine learning in healthcare industry: classification, restrictions, opportunities and challenges. Sensors 23(9):4178

Sarker IH (2021) Machine learning: algorithms, real-world applications and research directions. SN Comput Sci 2(3):160

Coronnello C, Francipane MG (2022) Moving towards induced pluripotent stem cell-based therapies with artificial intelligence and machine learning. Stem Cell Rev Rep 18:1–11

Hsu B-M (2020) Comparison of supervised classification models on textual data. Mathematics 8(5):851

Zhang S, May D, Gül M, Musilek P (2022) Reinforcement learning-driven local transactive energy market for distributed energy resources. Energy and AI 8:100150

Lu T, Schuurmans D, Boutilier C (2018) Non-delusional Q-learning and value-iteration. In: Advances in neural information processing systems, vol 31

Rashid T, Samvelyan M, De Witt CS, Farquhar G, Foerster J, Whiteson S (2020) Monotonic value function factorisation for deep multi-agent reinforcement learning. J Mach Learn Res 21(1):7234–7284

MathSciNet   Google Scholar  

Marchesini E, Farinelli A (2021) Centralizing state-values in dueling networks for multi-robot reinforcement learning mapless navigation. In: 2021 IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE, pp 4583–4588

Pope AP et al (2021) Hierarchical reinforcement learning for air-to-air combat. In: 2021 international conference on unmanned aircraft systems (ICUAS). IEEE, pp 275–284

Sharma K, Singh B, Herman E, Regine R, Rajest SS, Mishra VP (2021) Maximum information measure policies in reinforcement learning with deep energy-based model. In: 2021 International conference on computational intelligence and knowledge economy (ICCIKE). IEEE, pp 19–24

Wang L, Zhang W, He X, Zha H (2018) Supervised reinforcement learning with recurrent neural network for dynamic treatment recommendation. In: Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp 2447–2456

Russek EM, Momennejad I, Botvinick MM, Gershman SJ, Daw ND (2017) Predictive representations can link model-based reinforcement learning to model-free mechanisms. PLoS Comput Biol 13(9):e1005768

Polydoros AS, Nalpantidis L (2017) Survey of model-based reinforcement learning: applications on robotics. J Intell Rob Syst 86(2):153–173

Ayoub A, Jia Z, Szepesvari C, Wang M, Yang L (2020) Model-based reinforcement learning with value-targeted regression. In: International conference on machine learning. PMLR, pp 463–474

Alghanmi N, Alotaibi R, Buhari SM (2022) Machine learning approaches for anomaly detection in IoT: an overview and future research directions. Wireless Pers Commun 122(3):2309–2324

Usmani UA, Happonen A, Watada J (2022) A review of unsupervised machine learning frameworks for anomaly detection in industrial applications. In: Science and information conference. Springer, pp 158–189

Nassif AB, Talib MA, Nasir Q, Dakalbab FM (2021) Machine learning for anomaly detection: a systematic review. IEEE Access 9:78658–78700

Yuan G, Sun P, Zhao J, Li D, Wang C (2017) A review of moving object trajectory clustering algorithms. Artif Intell Rev 47:123–144

Kassambara A (2017) Practical guide to cluster analysis in R: unsupervised machine learning. Sthda

Zebari R, Abdulazeez A, Zeebaree D, Zebari D, Saeed J (2020) A comprehensive review of dimensionality reduction techniques for feature selection and feature extraction. J Appl Sci Technol Trends 1(2):56–70

Anowar F, Sadaoui S, Selim B (2021) Conceptual and empirical comparison of dimensionality reduction algorithms (PCA, KPCA, LDA, MDS, SVD, LLE, ISOMAP, LE, ICA, t-SNE). Comput Sci Rev 40:100378

Sharma N, Saroha K (2015) Study of dimension reduction methodologies in data mining. In: International conference on computing, communication & automation. IEEE, pp 133–137

Gisbrecht A, Hammer B (2015) Data visualization by nonlinear dimensionality reduction. Wiley Interdiscip Rev Data Min Knowl Discov 5(2):51–73

Nachman B, Shih D (2020) Anomaly detection with density estimation. Phys Rev D 101(7):075042

Carleo G et al (2019) Machine learning and the physical sciences. Rev Mod Phys 91(4):045002

Wang Z, Scott DW (2019) Nonparametric density estimation for high-dimensional data—algorithms and applications. Wiley Interdiscip Rev Comput Stat 11(4):e1461

Lavin A, Ahmad S (2015) Evaluating real-time anomaly detection algorithms—the Numenta anomaly benchmark. In 2015 IEEE 14th international conference on machine learning and applications (ICMLA). IEEE, pp 38–44

Beggel L, Pfeiffer M, Bischl B (2020) Robust anomaly detection in images using adversarial autoencoders. In: Machine learning and knowledge discovery in databases: European conference, ECML PKDD 2019, Würzburg, Germany, September 16–20, 2019, Proceedings, Part I. Springer, pp 206–222

Chowdhury A, Bose A, Zhou S, Woodruff DP, Drineas P (2022) A Fast, provably accurate approximation algorithm for sparse principal component analysis reveals human genetic variation across the world. In: International conference on research in computational molecular biology. Springer, pp 86–106

Weaving D, Beggs C, Dalton-Barron N, Jones B, Abt G (2019) Visualizing the complexity of the athlete-monitoring cycle through principal-component analysis. Int J Sports Physiol Perform 14(9):1304–1310

Jolliffe IT, Cadima J (2016) Principal component analysis: a review and recent developments. Philos Trans R Soc A Math Phys Eng Sci 374(2065):20150202

Cohn R, Holm E (2021) Unsupervised machine learning via transfer learning and k-means clustering to classify materials image data. Integr Mater Manuf Innov 10(2):231–244

Sinaga KP, Yang M-S (2020) Unsupervised K-means clustering algorithm. IEEE Access 8:80716–80727

Aytaç E (2020) Unsupervised learning approach in defining the similarity of catchments: Hydrological response unit based k-means clustering, a demonstration on Western Black Sea Region of Turkey. Int Soil Water Conserv Res 8(3):321–331

Malik A, Tuckfield B (2019) Applied unsupervised learning with R: uncover hidden relationships and patterns with k-means clustering, hierarchical clustering, and PCA. Packt Publishing Ltd, Birmingham

Nouraei H, Nouraei H, Rabkin SW (2022) Comparison of unsupervised machine learning approaches for cluster analysis to define subgroups of heart failure with preserved ejection fraction with different outcomes. Bioengineering 9(4):175

Vakeel A, Vantari NR, Reddy SN, Muthyapu R, Chavan A (2022) Machine learning models for predicting and clustering customer churn based on boosting algorithms and Gaussian mixture model. In: 2022 International conference for advancement in technology (ICONAT). IEEE, pp 1–5

Ma Y, Hao Y (2020) Antenna classification using Gaussian mixture models (GMM) and machine learning. IEEE Open J Antennas Propag 1:320–328

Wang Z, Ritou M, Da Cunha C, Furet B (2020) Contextual classification for smart machining based on unsupervised machine learning by Gaussian mixture model. Int J Comput Integr Manuf 33(10–11):1042–1054

Goldstein A, Fink L, Meitin A, Bohadana S, Lutenberg O, Ravid G (2018) Applying machine learning on sensor data for irrigation recommendations: revealing the agronomist’s tacit knowledge. Precision Agric 19:421–444

Greener JG, Kandathil SM, Moffat L, Jones DT (2022) A guide to machine learning for biologists. Nat Rev Mol Cell Biol 23(1):40–55

Mühlbacher T, Piringer H (2013) A partition-based framework for building and validating regression models. IEEE Trans Visual Comput Gr 19(12):1962–1971

Ghavamipour AR, Turkmen F, Jiang X (2022) Privacy-preserving logistic regression with secret sharing. BMC Med Inform Decis Mak 22(1):1–11

Lydersen S (2022) Logistic regression with more than two categories. Tidsskrift for Den Norske Legeforening

Mongwe WT, Mbuvha R, Marwala T (2021) Bayesian inference of local government audit outcomes. PLoS ONE 16(12):e0261245

Ribeiro M, Nunes I, Castro L, Costa-Santos C, Henriques TS (2023) Machine learning models based on clinical indices and cardiotocographic features for discriminating asphyxia fetuses—Porto retrospective intrapartum study. Front Public Health 11:1099263

Pellegrino E et al (2021) Machine learning random forest for predicting oncosomatic variant NGS analysis. Sci Rep 11(1):21820

Mucesh S et al (2021) A machine learning approach to galaxy properties: joint redshift–stellar mass probability distributions with Random Forest. Mon Not R Astron Soc 502(2):2770–2786

Bansal M, Goyal A, Choudhary A (2022) A comparative analysis of K-nearest neighbor, genetic, support vector machine, decision tree, and long short term memory algorithms in machine learning. Decis Anal J 3:100071

Bennett-Lenane H, Griffin BT, O’Shea JP (2022) Machine learning methods for prediction of food effects on bioavailability: a comparison of support vector machines and artificial neural networks. Eur J Pharm Sci 168:106018

Tehrani FS, Santinelli G, Herrera Herrera M (2021) Multi-regional landslide detection using combined unsupervised and supervised machine learning. Geomat Natl Hazards Risk 12(1):1015–1038

Tehrani FS, Calvello M, Liu Z, Zhang L, Lacasse S (2022) Machine learning and landslide studies: recent advances and applications. Nat Hazards 114(2):1197–1245

Egbueri JC (2023) Use of joint supervised machine learning algorithms in assessing the geotechnical peculiarities of erodible tropical soils from southeastern Nigeria. Geomech Geoeng 18(1):16–33

Ma J et al (2022) Machine learning models for slope stability classification of circular mode failure: an updated database and automated machine learning (AutoML) approach. Sensors 22(23):9166

Lin Y, Zhou K, Li J (2018) Prediction of slope stability using four supervised learning methods. IEEE Access 6:31169–31179

Nanehkaran YA et al (2023) Comparative analysis for slope stability by using machine learning methods. Appl Sci 13(3):1555

Cannistraci CV, Ravasi T, Montevecchi FM, Ideker T, Alessio M (2010) Nonlinear dimension reduction and clustering by Minimum Curvilinearity unfold neuropathic pain and tissue embryological classes. Bioinformatics 26(18):i531–i539

Wang L (2016) Discovering phase transitions with unsupervised learning. Phys Rev B 94(19):195105

Soranzo E, Guardiani C, Wu W (2023) Reinforcement learning for the face support pressure of tunnel boring machines. Geosciences 13(3):82

Erharter GH, Hansen TF, Liu Z, Marcher T (2021) Reinforcement learning based process optimization and strategy development in conventional tunneling. Autom Constr 127:103701

Eyo E, Abbey S (2022) Multiclass stand-alone and ensemble machine learning algorithms utilised to classify soils based on their physico-chemical characteristics. J Rock Mech Geotech Eng 14(2):603–615

Shi C, Wang Y (2021) Non-parametric machine learning methods for interpolation of spatially varying non-stationary and non-Gaussian geotechnical properties. Geosci Front 12(1):339–350

Mitelman A, Yang B, Urlainis A, Elmo D (2023) Coupling geotechnical numerical analysis with machine learning for observational method projects. Geosciences 13(7):196

Mali N, Dutt V, Uday K (2021) Determining the geotechnical slope failure factors via ensemble and individual machine learning techniques: a case study in Mandi, India. Front Earth Sci 9:701837

Tse KC, Chan AC, Yau KK (2017) Machine learning study on man-made features in hong kong—a data driven approach to feature classification.

Santos AEM, Lana MS, Pereira TM (2022) Evaluation of machine learning methods for rock mass classification. Neural Comput Appl 34(6):4633–4642

Rahman T, Sarkar K (2021) Lithological control on the estimation of uniaxial compressive strength by the P-wave velocity using supervised and unsupervised learning. Rock Mech Rock Eng 54:3175–3191

Hudson KS, Ulmer KJ, Zimmaro P, Kramer SL, Stewart JP, Brandenberg SJ (2023) Unsupervised machine learning for detecting soil layer boundaries from cone penetration test data. Earthq Eng Struct Dyn 52:3201

Chou J-S, Truong D-N, Le T-L, Truong TTH (2021) Bio-inspired optimization of weighted-feature machine learning for strength property prediction of fiber-reinforced soil. Expert Syst Appl 180:115042

Jafarzadeh A, Pal M, Servati M, FazeliFard M, Ghorbani M (2016) Comparative analysis of support vector machine and artificial neural network models for soil cation exchange capacity prediction. Int J Environ Sci Technol 13:87–96

Pal SC et al (2020) Ensemble of machine-learning methods for predicting gully erosion susceptibility. Remote Sens 12(22):3675

Eyo E, Abbey S (2021) Machine learning regression and classification algorithms utilised for strength prediction of OPC/by-product materials improved soils. Constr Build Mater 284:122817

Onyelowe KC, Mahesh CB, Srikanth B, Nwa-David C, Obimba-Wogu J, Shakeri J (2021) Support vector machine (SVM) prediction of coefficients of curvature and uniformity of hybrid cement modified unsaturated soil with NQF inclusion. Clean Eng Technol 5:100290

Manzouri F, Zare M, Shojaei S (2022) Exploring the potential of spatial artificial neural network in estimating topsoil salinity changes of in arid lands. Spat Inf Res 30(4):551–562

Tahmasebi P, Kamrava S, Bai T, Sahimi M (2020) Machine learning in geo-and environmental sciences: from small to large scale. Adv Water Resour 142:103619

Shen W, Li Y, Liu Y, Han J, Wang J, Yuan X (2021) Entity linking meets deep learning: techniques and solutions. IEEE Trans Knowl Data Eng 35:2556

Mathew A, Amudha P, Sivakumari S (2021) Deep learning techniques: an overview. In: Advanced machine learning technologies and applications: proceedings of AMLTA 2020, pp 599–608

Xu W, He J, Shu Y, Zheng H (2020) Advances in convolutional neural networks. IntechOpen, London

Vamosi S, Reutterer T, Platzer M (2022) A deep recurrent neural network approach to learn sequence similarities for user-identification. Decis Support Syst 155:113718

Chai R et al (2017) Improving EEG-based driver fatigue classification using sparse-deep belief networks. Front Neurosci 11:103

Comşa I-M, Versari L, Fischbacher T, Alakuijala J (2021) Spiking autoencoders with temporal coding. Front Neurosci 15:712667

Creswell A, Bharath AA (2018) Denoising adversarial autoencoders. IEEE Trans Neural Netw Learn Syst 30(4):968–984

Li H, Parikh NA, He L (2018) A novel transfer learning approach to enhance deep neural network classification of brain functional connectomes. Front Neurosci 12:491

Azarafza M, Hajialilue Bonab M, Derakhshani R (2022) A deep learning method for the prediction of the index mechanical properties and strength parameters of marlstone. Materials 15(19):6899

Guan Q, Yang Z, Guo N, Hu Z (2023) Finite element geotechnical analysis incorporating deep learning-based soil model. Comput Geotech 154:105120

Xu Z, Ma W, Lin P, Hua Y (2022) Deep learning of rock microscopic images for intelligent lithology identification: neural network comparison and selection. J Rock Mech Geotech Eng 14(4):1140–1152

Bekele YW (2021) Physics-informed deep learning for one-dimensional consolidation. J Rock Mech Geotech Eng 13(2):420–430

Liu M, Liao S, Yang Y, Men Y, He J, Huang Y (2021) Tunnel boring machine vibration-based deep learning for the ground identification of working faces. J Rock Mech Geotech Eng 13(6):1340–1357

Liu Z, Hu S, Sun Y, Azmoon B (2022) An exploratory investigation into image-data-driven deep learning for stability analysis of geosystems. Geotech Geol Eng 40(2):735–750

Zhang Z, Pan Q, Yang Z, Yang X (2023) Physics-informed deep learning method for predicting tunnelling-induced ground deformations. Acta Geotech 18:1–16

Campos Montero F (2023) Deep learning for geotechnical engineering: the effectiveness of generative adversarial networks in subsoil schematization.

Abbaszadeh Shahri A, Shan C, Larsson S (2022) A novel approach to uncertainty quantification in groundwater table modeling by automated predictive deep learning. Natl Resour Res 31(3):1351–1373

Nanehkaran Y, Licai Z, Chen J, Azarafza M, Yimin M (2022) Application of artificial neural networks and geographic information system to provide hazard susceptibility maps for rockfall failures. Environ Earth Sci 81(19):475

Kikuchi T, Sakita K, Nishiyama S, Takahashi K (2023) Landslide susceptibility mapping using automatically constructed CNN architectures with pre-slide topographic DEM of deep-seated catastrophic landslides caused by Typhoon Talas. Nat Hazards 117(1):339–364

Zhong Z, Sun AY, Wu X (2020) Inversion of time-lapse seismic reservoir monitoring data using CycleGAN: a deep learning-based approach for estimating dynamic reservoir property changes. J Geophys Res Solid Earth 125(3):e2019JB018408

Abbaszadeh Shahri A, Chunling S, Larsson S (2023) A hybrid ensemble-based automated deep learning approach to generate 3D geo-models and uncertainty analysis. Eng Comput. https://doi.org/10.1007/s00366-023-01852-5

Soga K, Schooling J (2016) Infrastructure sensing. Interface focus 6(4):20160023

Shahin MA (2016) State-of-the-art review of some artificial intelligence applications in pile foundations. Geosci Front 7(1):33–44

Kardani N, Aminpour M, Raja MNA, Kumar G, Bardhan A, Nazem M (2022) Prediction of the resilient modulus of compacted subgrade soils using ensemble machine learning methods. Transp Geotech 36:100827

Rasol M et al (2022) GPR monitoring for road transport infrastructure: a systematic review and machine learning insights. Constr Build Mater 324:126686

Yang W, Xia K, Fan S (2023) Oil logging reservoir recognition based on TCN and SA-BiLSTM deep learning method. Eng Appl Artif Intell 121:105950

Yao P, Yu Z, Zhang Y, Xu T (2023) Application of machine learning in carbon capture and storage: An in-depth insight from the perspective of geoscience. Fuel 333:126296

Kasravi J, Safarzadeh MA, Hashemi A (2017) A population-feedback control based algorithm for well trajectory optimization using proxy model. J Rock Mech Geotech Eng 9(2):281–290

Zhu D et al (2023) Deep learning approach of drilling decision for subhorizontal drain geosteering based on APC-LSTM model. SPE Drill Complet 38(01):1–17

Sagi O, Rokach L (2018) Ensemble learning: a survey. Wiley Interdiscip Rev Data Min Knowl Discov 8(4):e1249

Ngo G, Beard R, Chandra R (2022) Evolutionary bagging for ensemble learning. Neurocomputing 510:1–14

Kshatri SS, Singh D, Narain B, Bhatia S, Quasim MT, Sinha GR (2021) An empirical analysis of machine learning algorithms for crime prediction using stacked generalization: an ensemble approach. IEEE Access 9:67488–67500

Ullah I, Liu K, Yamamoto T, Zahid M, Jamal A (2021) Electric vehicle energy consumption prediction using stacked generalization: an ensemble learning approach. Int J Green Energy 18(9):896–909

Kumari P, Toshniwal D (2021) Extreme gradient boosting and deep neural network based ensemble learning approach to forecast hourly solar irradiance. J Clean Prod 279:123285

Kazmaier J, Van Vuuren JH (2022) The power of ensemble learning in sentiment analysis. Expert Syst Appl 187:115819

Wang G, Song Q, Zhu X (2021) Ensemble learning based classification algorithm recommendation. arXiv preprint arXiv:2101.05993

Bose S et al (2021) An ensemble machine learning model based on multiple filtering and supervised attribute clustering algorithm for classifying cancer samples. PeerJ Comput Sci 7:e671

Clark RD, Liang W, Lee AC, Lawless MS, Fraczkiewicz R, Waldman M (2014) Using beta binomials to estimate classification uncertainty for ensemble models. J Cheminform 6(1):1–19

Inamullah, Hassan S, Alrajeh NA, Mohammed EA, Khan S (2023) Data diversity in convolutional neural network based ensemble model for diabetic retinopathy. Biomimetics 8(2):187

Knafou J et al (2023) Ensemble of deep learning language models to support the creation of living systematic reviews for the COVID-19 literature. Syst Rev 12(1):94

Juraev F, El-Sappagh S, Abdukhamidov E, Ali F, Abuhmed T (2022) Multilayer dynamic ensemble model for intensive care unit mortality prediction of neonate patients. J Biomed Inform 135:104216

Perales-Gonzalez C, Fernandez-Navarro F, Carbonero-Ruz M, Perez-Rodriguez J (2021) Global negative correlation learning: a unified framework for global optimization of ensemble models. IEEE Trans Neural Netw Learn Syst 33(8):4031–4042

Harrou F, Saidi A, Sun Y (2019) Wind power prediction using bootstrap aggregating trees approach to enabling sustainable wind power integration in a smart grid. Energy Convers Manag 201:112077

Xie J, Zhang J, Xie X, Bi Z, Li Z (2019) Ensemble of bagged regression trees for concrete dam deformation predicting. IOP Conf Ser Earth Environ Sci 376(1):012040

Modi S, Bhattacharya J, Basak P (2021) Convolutional neural network–bagged decision tree: a hybrid approach to reduce electric vehicle’s driver’s range anxiety by estimating energy consumption in real-time. Soft Comput 25:2399–2416

Simidjievski N, Todorovski L, Džeroski S (2016) Modeling dynamic systems with efficient ensembles of process-based models. PLoS ONE 11(4):e0153507

Bian X et al (2018) Robust boosting neural networks with random weights for multivariate calibration of complex samples. Anal Chim Acta 1009:20–26

Miao Q, Cao Y, Xia G, Gong M, Liu J, Song J (2015) RBoost: label noise-robust boosting algorithm based on a nonconvex loss function and the numerically stable base learners. IEEE Trans Neural Netw Learn Syst 27(11):2216–2228

Walker KW, Jiang Z (2019) Application of adaptive boosting (AdaBoost) in demand-driven acquisition (DDA) prediction: a machine-learning approach. J Acad Librariansh 45(3):203–212

Sevinç E (2022) An empowered AdaBoost algorithm implementation: a COVID-19 dataset study. Comput Ind Eng 165:107912

Kim C, Park T (2022) Predicting determinants of lifelong learning intention using Gradient Boosting Machine (GBM) with grid search. Sustainability 14(9):5256

Sunaryono D, Sarno R, Siswantoro J (2022) Gradient boosting machines fusion for automatic epilepsy detection from EEG signals based on wavelet features. J King Saud Univ Comput Inf Sci 34(10):9591–9607

Chen H, Shen Z, Wang L, Qi C, Tian Y (2022) Prediction of undrained failure envelopes of skirted circular foundations using gradient boosting machine algorithm. Ocean Eng 258:111767

Kavzoglu T, Teke A (2022) Predictive performances of ensemble machine learning algorithms in landslide susceptibility mapping using random forest, extreme gradient boosting (XGBoost) and natural gradient boosting (NGBoost). Arab J Sci Eng 47(6):7367–7385

Ching PML, Zou X, Wu D, So RHY, Chen G (2022) Development of a wide-range soft sensor for predicting wastewater BOD5 using an eXtreme gradient boosting (XGBoost) machine. Environ Res 210:112953

He W et al (2022) Rapid and uninvasive characterization of bananas by hyperspectral imaging with extreme gradient boosting (XGBoost). Anal Lett 55(4):620–633

Rufo DD, Debelee TG, Ibenthal A, Negera WG (2021) Diagnosis of diabetes mellitus using gradient boosting machine (LightGBM). Diagnostics 11(9):1714

Zhang J, Mucs D, Norinder U, Svensson F (2019) LightGBM: AN effective and scalable algorithm for prediction of chemical toxicity—application to the Tox21 and mutagenicity data sets. J Chem Inf Model 59(10):4150–4158

Demir S, Sahin EK (2023) Predicting occurrence of liquefaction-induced lateral spreading using gradient boosting algorithms integrated with particle swarm optimization: PSO-XGBoost, PSO-LightGBM, and PSO-CatBoost. Acta Geotech 18(6):3403–3419

Hancock JT, Khoshgoftaar TM (2020) CatBoost for big data: an interdisciplinary review. J Big Data 7(1):1–45

Chan A, Peck R, Gibbs M, van der Schaar M (2023) Synthetic model combination: a new machine learning method for pharmacometric model ensembling. CPT Pharmacomet Syst Pharmacol 12(7):953

Wang N, Zhou W, Li H (2021) Learning diverse models for end-to-end ensemble tracking. IEEE Trans Image Process 30:2220–2231

Waqas Khan P, Byun Y-C (2022) Multi-fault detection and classification of wind turbines using stacking classifier. Sensors 22(18):6955

Zhao R, Mu Y, Zou L, Wen X (2022) A hybrid intrusion detection system based on feature selection and weighted stacking classifier. IEEE Access 10:71414–71426

Chatterjee S, Byun Y-C (2022) EEG-based emotion classification using stacking ensemble approach. Sensors 22(21):8550

Djarum DH, Ahmad Z, Zhang J (2023) Reduced Bayesian optimized stacked regressor (RBOSR): a highly efficient stacked approach for improved air pollution prediction. Appl Soft Comput 144:110466

Cai Y et al (2023) An adaptive stacking regressor with a self-iterative optimization module for improving fractional woody cover mapping. IEEE Geosci Remote Sens Lett. https://doi.org/10.1109/LGRS.2023.3281646

Peng Y, Rios A, Kavuluru R, Lu Z (2018) Extracting chemical–protein relations with ensembles of SVM and deep learning models. Database 2018:bay073

Manganelli S et al (2019) Development, validation and integration of in silico models to identify androgen active chemicals. Chemosphere 220:204–215

Morgan-Benita JA et al (2022) Hard voting ensemble approach for the detection of type 2 diabetes in mexican population with non-glucose related features. Healthcare 10(8):1362

Stephen O, Madanian S, Nguyen M (2022) A hard voting policy-driven deep learning architectural ensemble strategy for industrial products defect recognition and classification. Sensors 22(20):7846

Shareef AQ, Kurnaz S (2023) Deep learning based COVID-19 detection via hard voting ensemble method. Wirel Pers Commun. https://doi.org/10.1007/s11277-023-10485-2

Verma R, Chandra S (2023) RepuTE: a soft voting ensemble learning framework for reputation-based attack detection in fog-IoT milieu. Eng Appl Artif Intell 118:105670

Khan MA, Iqbal N, Jamil H, Kim D-H (2023) An optimized ensemble prediction model using AutoML based on soft voting classifier for network intrusion detection. J Netw Comput Appl 212:103560

Sherazi SWA, Bae J-W, Lee JY (2021) A soft voting ensemble classifier for early prediction and diagnosis of occurrences of major adverse cardiovascular events for STEMI and NSTEMI during 2-year follow-up in patients with acute coronary syndrome. PLoS ONE 16(6):e0249338

Wudil YS, Al-Najjar OA, Al-Osta MA, Baghabra Al-Amoudi OS, Gondal MA (2023) Investigating the soil unconfined compressive strength based on laser-induced breakdown spectroscopy emission intensities and machine learning techniques. ACS Omega 8(29):26391–26404

Chen Y, Xu Y, Jamhiri B, Wang L, Li T (2022) Predicting uniaxial tensile strength of expansive soil with ensemble learning methods. Comput Geotech 150:104904

Kardani N, Zhou A, Nazem M, Shen S-L (2021) Improved prediction of slope stability using a hybrid stacking ensemble method based on finite element analysis and field data. J Rock Mech Geotech Eng 13(1):188–201

Indrasiri PL, Halgamuge MN, Mohammad A (2021) Robust ensemble machine learning model for filtering phishing URLs: expandable random gradient stacked voting classifier (ERG-SVC). IEEE Access 9:150142–150161

Jibanchand N, Devi KR (2023) Application of ensemble learning in predicting shallow foundation settlement in cohesionless soil. Int J Geotech Eng 17:1–12

Li L, Iskander M (2021) Evaluation of roundness parameters in use for sand. J Geotech Geoenviron Eng 147(9):04021081

Zhang W, Wu C, Zhong H, Li Y, Wang L (2021) Prediction of undrained shear strength using extreme gradient boosting and random forest based on Bayesian optimization. Geosci Front 12(1):469–477

Dong X, Yu Z, Cao W, Shi Y, Ma Q (2020) A survey on ensemble learning. Front Comput Sci 14:241–258

Liu Y, Zhao Q (2022) Ensemble learning. In: Handbook on computer learning and intelligence: volume 2: Deep Learning, Intelligent Control and Evolutionary Computation. World Scientific, pp 635–660

Mahajan P, Uddin S, Hajati F, Moni MA (2023) Ensemble learning for disease prediction: a review. Healthcare 11(12):1808

Corner A, Pidgeon N (2020) Like artificial trees? The effect of framing by natural analogy on public perceptions of geoengineering. In: The ethics of nanotechnology, geoengineering, and clean energy. Routledge, pp 361–374

Horton JB et al (2023) Solar geoengineering research programs on national agendas: a comparative analysis of Germany, China, Australia, and the United States. Clim Change 176(4):37

Rabbani A, Samui P, Kumari S (2023) Implementing ensemble learning models for the prediction of shear strength of soil. Asian J Civ Eng 24:1–17

Zhang R, Wu C, Goh AT, Böhlke T, Zhang W (2021) Estimation of diaphragm wall deflections for deep braced excavation in anisotropic clays using ensemble learning. Geosci Front 12(1):365–373

Zhang W, Li H, Han L, Chen L, Wang L (2022) Slope stability prediction using ensemble learning techniques: a case study in Yunyang County, Chongqing, China. J Rock Mech Geotech Eng 14(4):1089–1099

Wang ZZ, Goh SH (2022) A maximum entropy method using fractional moments and deep learning for geotechnical reliability analysis. Acta Geotech 17(4):1147–1166

Sahu A, Samui P, Determination of liquefaction susceptibility of soil: a deep learning approach.

Sheng D, Yu J, Tan F, Tong D, Yan T, Lv J (2023) Rock mass quality classification based on deep learning: a feasibility study for stacked autoencoders. J Rock Mech Geotech Eng 15(7):1749–1758

Huang F, Zhang J, Zhou C, Wang Y, Huang J, Zhu L (2020) A deep learning algorithm using a fully connected sparse autoencoder neural network for landslide susceptibility prediction. Landslides 17:217–229

Padarian J, Minasny B, McBratney A (2019) Using deep learning to predict soil properties from regional spectral data. Geoderma Reg 16:e00198

Norouzi M, Ayoubi S, Jalalian A, Khademi H, Dehghani A (2010) Predicting rainfed wheat quality and quantity by artificial neural network using terrain and soil characteristics. Acta Agric Scand Sect B Soil Plant Sci 60(4):341–352

Salehzadeh H, Gholipoor M, Abbasdokht H, Baradaran M (2016) Optimizing plant traits to increase yield quality and quantity in tobacco using artificial neural network. Int J Plant Prod 10(1):97

Alaskar H, Saba T (2021) Machine learning and deep learning: a comparative review. In: Proceedings of integrated intelligence enable networks and computing: IIENC 2020, pp 143–150

Chauhan NK, Singh K (2018) A review on conventional machine learning vs deep learning. In: 2018 International conference on computing, power and communication technologies (GUCON). IEEE, pp 347–352

Shrestha A, Mahmood A (2019) Review of deep learning algorithms and architectures. IEEE Access 7:53040–53065

Pouyanfar S et al (2018) A survey on deep learning: algorithms, techniques, and applications. ACM Comput Surv 51(5):1–36

Krawczyk B, Minku LL, Gama J, Stefanowski J, Woźniak M (2017) Ensemble learning for data stream analysis: a survey. Inf Fusion 37:132–156

Vakili AH, Ghasemi J, Bin Selamat MR, Salimi M, Farhadi MS (2018) Internal erosional behaviour of dispersive clay stabilized with lignosulfonate and reinforced with polypropylene fiber. Constr Build Mater 193:405–415

Vakili AH, Kaedi M, Mokhberi M, Bin Selamat MR, Salimi M (2018) Treatment of highly dispersive clay by lignosulfonate addition and electroosmosis application. Appl Clay Sci 152:1–8

Shahsavani S, Vakili AH, Mokhberi M (2020) The effect of wetting and drying cycles on the swelling-shrinkage behavior of the expansive soils improved by nanosilica and industrial waste. Bull Eng Geol Environ 79(9):4765–4781

Vakili AH, Shojaei SI, Salimi M, Bin Selamat MR, Farhadi MS (2020) Contact erosional behaviour of foundation of pavement embankment constructed with nanosilica-treated dispersive soils. Soils Found 60(1):167–178

Vakili A, Selamat M, Moayedi H (2013) An assessment of physical and mechanical properties of dispersive clay treated with lime. Casp J Appl Sci Res 2:197–204

Khoshbakht EB, Vakili AH, Farhadi MS, Salimi M (2019) Reducing the negative impact of freezing and thawing cycles on marl by means of the electrokinetical injection of calcium chloride. Cold Reg Sci Technol 157:196–205

Vakili AH, Salimi M, Shamsi M (2021) Application of the dynamic cone penetrometer test for determining the geotechnical characteristics of marl soils treated by lime. Heliyon 7(9):e08062

Parsaei M, Vakili AH, Salimi M, Farhadi MS, Falamaki A (2021) Effect of electric arc and ladle furnace slags on the strength and swelling behavior of cement-stabilized expansive clay. Bull Eng Geol Environ 80(8):6303–6320

Vakili AH, Selamat MRB, Salimi M, Gararei SG (2021) Evaluation of pozzolanic Portland cement as geotechnical stabilizer of a dispersive clay. Int J Geotech Eng 15(4):504–511

Vakili AH, Rastegar S, Golkarfard H, Salimi M, Izadneshan Z (2023) Effect of polypropylene fibers on internal erosional behavior of poorly graded sandy soil stabilized with the binary mixtures of clay and polyvinyl acetate. Environ Earth Sci 82(12):1–18

Keskin I, Arslan O, Vakili AH (2023) Investigating the impact of travertine powder on strength and permeability of swelling clay. Phys Chem Earth Parts A/B/C 132:103494

Falamaki A et al (2023) Experimental investigation of the effect of landfill leachate on the mechanical and cracking behavior of polypropylene fiber-reinforced compacted clay liner. Environ Sci Pollut Res 30:1–18

Vakili AH, Salimi M, Keskin I, Abujazar MSS, Shamsi M (2023) Effects of polyvinyl acetate content on contact erosion parameters of pavement embankment constructed by dispersive soils. Bull Eng Geol Environ 82(10):398

Jamshidi M, Mokhberi M, Vakili AH, Nasehi A (2023) Effect of chitosan bio-polymer stabilization on the mechanical and dynamic characteristics of marl soils. Transp Geotech 42:101110

Download references

Open access funding provided by the Scientific and Technological Research Council of Türkiye (TÜBİTAK).

Author information

Authors and affiliations.

Department of Electrical and Electronics Engineering, Faculty of Engineering, Karabuk University, Karabuk, Turkey

Elaheh Yaghoubi & Elnaz Yaghoubi

College of Science and Technology, Umm Al-Aranib, Libya

Ahmed Khamees

Department of Environmental Engineering, Faculty of Engineering, Karabuk University, Karabuk, Turkey

Amir Hossein Vakili

Department of Civil Engineering, Faculty of Engineering, Zand Institute of Higher Education, Shiraz, Iran

You can also search for this author in PubMed   Google Scholar

Contributions

Elaheh Yaghoubi involved in supervision, conceptualization, methodology, writing—reviewing and editing. Elnaz Yaghoubi took part in software, data curation, writing—original draft. Ahmed Khamees involved in visualization, investigation. Amir Hossein Vakili took part in supervision, conceptualization, methodology, writing—reviewing and editing.

Corresponding author

Correspondence to Amir Hossein Vakili .

Ethics declarations

Conflict of interest.

The authors declare no conflict of interest.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Yaghoubi, E., Yaghoubi, E., Khamees, A. et al. A systematic review and meta-analysis of artificial neural network, machine learning, deep learning, and ensemble learning approaches in field of geotechnical engineering. Neural Comput & Applic (2024). https://doi.org/10.1007/s00521-024-09893-7

Download citation

Received : 20 November 2023

Accepted : 23 April 2024

Published : 13 May 2024

DOI : https://doi.org/10.1007/s00521-024-09893-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Artificial neural networks
• Machine learning
• Deep learning
• Ensemble learning
• Geotechnical engineering
• Find a journal
• Publish with us
• Track your research

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Healthc Eng
• v.2021; 2021

This article has been retracted.

Literature review on the applications of machine learning and blockchain technology in smart healthcare industry: a bibliometric analysis.

School of Management, Jilin University, Changchun 130022, China

Biaoan Shan

Associated data.

The data used to support the study are available from the corresponding author upon request.

The emergence of machine learning (ML) and blockchain (BC) technology has greatly enriched the functions and services of healthcare, giving birth to the new field of “smart healthcare.” This study aims to review the application of ML and BC technology in the smart medical industry by Web of Science (WOS) using bibliometric visualization. Through our research, we identify the countries with the greatest output, the major research subjects, funding funds, and the research hotspots in this field. We also find out the key themes and future research areas in application of ML and BC technology in healthcare area. We reveal the different aspects of research under the two technologies and how they relate to each other around five themes.

1. Introduction

With the revolution of the medical infrastructure in the recent years, the smart healthcare system has been paid more considerable attention [ 1 ]. Smart healthcare is a novel concept that refers to a set of rules that integrate prevention, diagnosis, treatment, and management. Different from traditional medical systems, smart medical systems can connect and exchange information at any time and place [ 2 ].

Compared with traditional medical treatment, smart healthcare has the characteristics of preventability, immediacy, and interconnection of information. Through wireless network, using portable mobile devices, medical staff can constantly perceive, process, and analyze major medical events (preventability). Doctors can grasp the case information of each patient at any time and quickly develop a diagnosis and treatment plan (immediacy). Medical personnel can log in the medical system anywhere to inquire about medical images and medical advice and patient's referral information can be accessed at any hospital through the medical network (interconnection of information). These functions are supported by new digital technologies. BC follows absolute privacy rules to identify users related to transactions. It is mainly used for the management of information systems to help achieve secure storage, transactions, process automation, and other applications [ 3 ]. ML is the leading technology for performing complex analysis, intelligent judgment, and creative problem solving in healthcare [ 4 ].

Generally, previous studies related to application of digital technologies in smart healthcare domain were limited to study in one field or one country. No studies have mapped the current status of these two technologies in the medical field. Also, there is no relative study that specifically addresses the relationship between authors, affiliations, keywords, and the hotpots of the research. In the past five years, the study of smart healthcare has attracted extensive attention from scholars of a series of disciplines, which requires us to integrate the viewpoints of scholars of different disciplines and study the status to seek deeper discoveries.

Therefore, this research proposed portraying the status of application of two types of digital technologies, ML and BC, in smart healthcare studies by bibliometric visualization. In this paper, we have presented a comprehensive review on the application of ML and BL techniques in the healthcare sector. We analyze the research status in terms of countries, institutions, publication volume, authors, journals, sponsors, and subject areas. In addition, this paper subdivides the main application scenarios of the prior art in the medical field. Our research will provide healthcare practitioners with an insight to keep ML and BC technologies fully utilized. Finally, we analyze the latest research trends based on ML and BC technology in order to provide a research direction for future research.

2. Methodology

2.1. research method.

This study mainly applied bibliometrics to conduct a comprehensive analysis of the articles published on WOS related to the application of machine learning and blockchain technology in the field of smart healthcare. Bibliometric analysis makes a visual analysis of the research field by analyzing the information obtained from the database, such as titles, abstracts, keywords, and references. Bibliometric analysis helps researchers review a research area more scientifically and systematically and discover future research trends [ 5 ]. Since WOS contains scientific literature of all disciplines, our literature collection process is carried out in Web of Science [ 6 ].

2.2. Research Process

Our research process is divided into four steps: problem formulation, literature search, basic analysis, and VOSviewer analysis.

• Problem formulation: Our study aims to provide a comprehensive presentation of the application of ML and BC in the field of smart medicine by visualizing existing publications. To achieve this goal, we developed and solved the following research questions: How much attention have scholars paid to emerging technologies and smart medicine in recent years? What are the representative literatures? What are the current research topics and how are they evolving?
• Literature search: Literature search is divided into two steps: subject word search and manual screening. The first step is subject word search in the Web of Science core data set using the retrieval formula TS = (“machine learning” OR “blockchain”) AND TS = (“smart healthcare”). The first round of screening finally obtained 118 documents (data gathering took place in May 2021). The second step is manual literature screening. Our purpose is to eliminate the document that is not relevant to this field and to obtain the research direction and research progress in this field more accurately. The team members carefully read the titles, keywords, and abstracts of 118 documents and scanned the full text when necessary to ensure the relevance of the document. Finally, 112 documents were obtained as our research objects.
• Basic analysis: Basic analysis refers to the analysis results that can be directly exported from the WOS database or can be obtained through statistics. We conducted a statistical analysis of the year in which the literature was published, the research field, the fund sponsor, and so on. Through basic analysis, researchers can gauge changes in scholars' and institutions' interest in research in the field and understand research trends.
• VOSviewer analysis: VOSviewer analysis is to use VOSviewer to build a bibliometric network. We listed the most influential authors and literature in the field and showed heat maps of keyword research. We also used a clustering technique based on keywords cooccurrence to study the conceptual structure and distinctive clusters of a research field [ 7 ].

3.1. Number of Documents per Year

Research on this topic first appeared in the WOS database in 2015 ( Figure 1 ). In 2015, Tucker et al. [ 8 ] published an article titled “ Machine learning classification of medication adherence in patients with movement disorders using non-wearable sensors ” in Computers in Biology and Medicine . This paper demonstrates that nonwearable hardware and data mining models can be used to monitor medication compliance outside of traditional healthcare settings. It has initiated the research of machine learning in the field of intelligent medicine and laid a foundation for later research.

Number of documents per year from the studies.

The number of international documents on smart healthcare study is increasing year by year ( Figure 1 ). The peak number of articles was 54 in 2020, double the number of articles published in 2019. This study finds that there are three reasons for the rapid growth of publications. First, the demand for applications in the medical industry has been growing with the data-intensive development of smart applications, and the market for smart medicine is gradually expanding [ 9 ]. Second, more and more international researchers (such as Venkatesan et al. [ 10 ]) began to study the application of blockchain and machine learning in the field of intelligent medicine. Finally, more and more journals such as Sensors are more willing to accept research related to this field.

3.2. Major Research Subject Areas

Figure 2 shows the subject area with the widest application of machine learning and blockchain in smart healthcare research is Computer Science, with 76 documents (33.33%). The second major research area is Engineering with 66 documents (28.95%), and Telecommunications ranked third with 47 documents (20.61%). An article may cover multiple subject classifications. As for the application of machine learning and blockchain technologies in the field of intelligent medicine, the vast majority of researches start from the technical perspective, such as Computer Science, Engineering, and Telecommunications. However, there were only three articles related to biochemistry, such as Health Care Sciences & Services, accounting for 1% of the total literature. This indicates that the application of blockchain technology and machine learning technology in the healthcare field is still in the experimental stage of technology research and development, and the application of technology is not mature and widespread.

Number of documents based on subject areas of the smart healthcare studies.

3.3. Sponsoring Funding

Figure 3 shows the main funders to help publish research on the application of ML and BC technologies in smart healthcare are the National Natural Science Foundation with 8 papers, the European Commission with 6 papers, and King Saud University with 5 papers. This result is directly related to the number of articles published in China. According to the funding of various institutions, China attaches great importance to the application of machine learning and blockchain in the field of smart health.

Documents based on sponsoring funding of the studies.

3.4. Countries

The countries with the highest number of publications of smart healthcare research are India and China, with 23 documents each ( Figure 4 ). It was followed by Saudi Arabia with 22 documents, South Korea with 17 documents, the United States with 15 documents, and Pakistan with 12 documents ( Figure 4 shows the number of documents issued by all the issuing countries). It is reasonable considering the population size of India and China. Asian countries publish more articles than European regions.

Number of documents by country of smart healthcare studies.

India, China, Saudi Arabia, and other countries have published more documents in the field of smart medicine, which can be explained from three aspects. First of all, the medical level of developing countries such as India, China, and Saudi Arabia needs to be improved and they need to rely on new technologies to further improve the efficiency and quality of medical treatment. Secondly, machine learning, blockchain, and other new technologies have wide application markets in China, India, and Saudi Arabia. There is a great demand for new technologies in the medical industry, which provides conditions for technology commercialization. Finally, the large number of documents in countries such as China and Saudi Arabia is also influenced by the funding they received.

3.5. The Most Cited Papers

The publication of smart healthcare studies with the most 10 citations is listed in Table 1 . Referring to the largest number of articles in the first ten articles that appeared in 2005, Tucker et al. have developed a noninvasive sensor to monitor adherence of Parkinson's patients to drugs, by reading data of patients, to distinguish between patients taking their medicine diagnosis and treatment at home and it can realize the early warning and effectiveness of clinical trials as high as 97%.

The most cited papers in smart healthcare research.

Through the analysis of the most cited literatures, it is found that the main application of machine learning in the medical field is the diagnosis of diseases in these highly cited articles such as remote healthcare, disease diagnosis, healthcare monitoring, medication adherence, and body sensor. The main application of blockchain in the medical field is data encryption. The article related to blockchain focuses on its role in Multilevel Privacy-Preserving, monitoring patient vital sign, privacy health data, and Drug Supply Chain Integrity Management.

3.6. Research Hotspots

Figure 5 is the keyword heat map made according to the keywords in 112 articles. The darker the color, the higher the frequency of the keyword. Through the analysis of the heat map ( Figure 5 ), we found that, in addition to machine learning, blockchain, and smart healthcare, Internet and the Internet of Things are also core keywords of this area.

Map of hotspots.

IoT appears in the keywords because the outbreak of chronic diseases such as Novel Coronavirus has once again aroused the attention to intelligent medical treatment. IoT technology is used to monitor patient's vital sign. Smart healthcare can reply to such complex and urgent medical emergencies and make up for the shortcomings of the original medical system. The Internet of Things enables wearable devices to collect large amounts of data. It is considered as the communication between objects due to being embedded in sensors [ 11 , 12 ]. The Internet of Things in healthcare connects millions of sensors to a patient's body to continuously monitor his or her physical condition.

4. Theme Clustering Analysis

Through the author keywords and keywords plus, we used VOSviewer to construct the keyword network diagram. VOSviewer is a software tool for building and visualizing bibliometric networks that generate clusters that represent the underlying structure of a document. Specifically, we used bibliometric tools to analyze the keywords in articles, extract the most commonly used terms, and map them according to their interrelationships. From this analysis, we can determine the topics that are most commonly used in this field. In 112 articles, a keyword that appears at least three times is required to appear in the keyword network. Therefore, from 582 keywords, there were only 55 keywords that meet the thresholds. Figure 6 shows that there were five groups of study themes based on research keywords related to the study. Large circles indicate terms that are used more frequently, and color-coded lines indicate cooccurrence of terms. We found that the circles of the same color might come from a single document, so we read the full text of the documents mapping circles of the same color to see how they fit together. We name the clustering results according to the technical similarity.

Map of study field.

4.1. Cluster 1: Machine Learning

Cluster 1 is what we call Machine Learning (red). The keywords for this topic were smart healthcare, machine learning, biomedical monitoring, data analytics, data mining, deep learning, fusion, human activity recognition, model, medical diagnostic imaging, neural network, and smart home.

Machine learning, a field of artificial intelligence, has been widely used in the medical industry. With the development of ICT technology and the arrival of the era of big data, information such as patient information, medical treatment records, and medication status has been digitized, and a large amount of data has been generated in the field of medicine and healthcare [ 13 ]. The medical industry uses machine learning to deeply analyze complex medical data, which has become the main direction of machine learning research.

Cluster 1 centers around the application of machine learning in the medical field. Specifically, the research focuses on how the medical industry uses machine learning methods such as deep learning, neural network learning, and feature fusion to realize data analysis and mining. The goal is to realize the identification of human activities, health monitoring, disease prediction, and diagnosis and promote the development of the field of smart medicine.

Gumaei et al. [ 14 ] applied machine learning to human activity recognition. They proposed a human activity recognition framework based on multisensor hybrid deep learning model. This method can process multisensor data more intelligently and help medical institutions better care for the elderly and patients. Souri et al. [ 15 ] applied machine learning to the student health monitoring system, collected data through the Internet of Things, and analyzed data through machine learning to continuously monitor the physical condition of students. Ali et al. [ 16 ] built a disease prediction system based on deep learning feature fusion and information gain technique to predict the incidence of heart disease and create conditions for the effective treatment of patients with heart disease. Chui et al. [ 17 ] reviewed previous research on disease diagnosis in the field of intelligent medicine, summarized emerging machine learning algorithms, and discussed the challenges of deploying disease diagnosis in the future.

4.2. Cluster 2: Artificial Intelligence

When exploring the second group, Artificial Intelligence cluster (green), there were items like architecture, cloud computing, edge, edge computing, fog computing, medical services, pathology, and smart city.

Artificial intelligence refers to the intelligence shown by machines made by people. In general, artificial intelligence refers to the technology that represents human intelligence through ordinary computer programs. In the field of computer, artificial intelligence has been paid more and more attention; and, in the robot, economic and political decision-making, control system, and simulation system have been applied. Artificial intelligence is also getting more and more attention in the medical field.

Cluster 2 focuses on the application of artificial intelligence in the field of intelligent medicine. Specifically, the role of artificial intelligence is mainly reflected in the analysis and processing of data generated by intelligent devices such as the Internet of Things and sensors.

Artificial intelligence is attracting more and more attention in the medical field [ 18 , 19 ], and it is mostly used to assist decision-making in the medical field. Kamruzzaman [ 18 ] found that AI systems could be used for ancillary medical care by processing data from patients who admitted to hospital for emergency treatment or providing early detection of major diseases. With the assistance of artificial intelligence, the patient's human body data and genetic data can be automatically analyzed and generated medical reports to provide decision support to doctors. The reason why artificial intelligence can assist decision-making is the huge growth of medical data analysis and research. On the one hand, it can increase the accuracy and timeliness of decision-making, and, on the other hand, it can greatly alleviate the shortage of medical resources (medical staff and equipment) and save costs. Chui et al. [ 19 ] applied artificial intelligence to a particular disease for three kinds of detection of speech disorders.

4.3. Cluster 3: Blockchain

The third group of keywords, Blockchain cluster (yellow), includes access control, cloud, electronic medical records, Ethereum, privacy, security, and smart contracts.

Blockchain technology refers to a distributed, decentralized, and immutable digital ledger that records transactions through a global computer network, in which the information is highly secure [ 20 ]. It has grown into a very rich and efficient technology to serve and instruct the healthcare patient's point of view, to maintain patient data privacy, and to provide physicians with real-time, accurate, and trusted data of the process [ 21 ].

Cluster 3 revolves around the application of blockchain in the field of smart medicine. Specifically, the role of blockchain in the medical field is divided into two categories, the first category revolves around the encryption of medical data generated by individual patients, and the other category is applied to the supply chain system management of drugs. As the scalability of wearable medical devices increases, the smart medical field faces challenges in maintaining privacy and security. Scholars are building different frameworks for blockchain to improve the security of medical data.

In addition, blockchain technology can also assist in decision-making. Gul et al. [ 22 ] proposed an intelligent medical business model based on blockchain, which is customer-centered business. On the other hand, blockchain technology is applied in the drug supply chain management, supply chain involves a certain degree of stakeholders, and one of the biggest challenges associated with the supply chain is temperature monitoring and preventing fake drugs. Singh et al. [ 23 ] proposed a blockchain framework based on IoT sensors to track the movement of drugs in the supply chain. The safety of the pharmaceutical supply chain has become a major public health concern, and this is a collective process [ 24 ].

4.4. Cluster 4: Sensor Cluster

Sensor cluster (purple) was dominated by the keywords sensor, feature extraction, activity recognition, classification, prediction, smart healthcare system, care, and healthcare.

Sensor technology, communication technology, and computer technology together form the three pillars of modern information technology. With the development of sensor technology, sensors are becoming more and more intelligent and widely used. Sensors can collect data in real time. By analyzing the collected data, targets such as tracking patients' physical conditions, taking medicines, and identifying activities can be realized [ 25 ]. The application of sensors has greatly improved the quality and efficiency of the medical industry and reduced the cost of treatment, so it is widely used in the medical field.

Cluster 4 focuses on the application of sensors in the field of smart healthcare. Specifically, Cluster 4 mainly studies the method of using sensors to collect data in the medical industry, using feature extraction technology to extract data features, and realizing activity recognition and state prediction through classification algorithm. Since data collection in the medical industry mainly relies on sensors and big data is the basis of realizing smart medicine, sensors play a vital role in the process of smart medical industry.

By analyzing data collected from nonwearable multimodal sensors, Tucker et al. [ 8 ] modeled and predicted patients' adherence to medication after leaving the clinic. The method can give early warning to the patient's safety status in time and prompt the patient to take the medicine on time. Syed et al. [ 26 ] placed multiple wearable sensors on the left ankle, right arm, and chest of the tester, using related technologies to identify the movement of different body parts and determine the type of movement. This method is a good way to remotely monitor the activity status and health status of the elderly. Alo et al. [ 27 ] argued that current activity identification methods are not good at analyzing complex and dynamic activities. Therefore, a deep stacked autoencoder algorithm is proposed for the smartphone-embedded accelerometer sensor data, which can automatically extract the data features and improve the accuracy of activity recognition. Khan et al. [ 28 ] believed that wearable devices with embedded sensors can record data but have limitations such as battery life and hardware cost. Therefore, the authors further propose that channel state information can be used instead of wearable sensor devices to collect data to predict the health status of patients.

4.5. Cluster 5: IoT Cluster

IoT cluster (blue) was related to the keywords authentication, COVID-19, efficient, big data, industries, Internet, and network.

Industry 5.0 is the Fifth Industrial Revolution consisting of intelligent digital information and manufacturing technology [ 29 ]. The Fifth Industrial Revolution has promoted the wide application of technologies such as the Internet of Things (IoT). The IoT combines information sensing devices with the Internet, enabling people, machines, and things to get rid of the limitations of time and space and realize interconnection. The rapid development of the IoT has created opportunities for providing personalized services in the medical field [ 30 ].

Cluster 5 centers on the application of the IoT in the field of smart healthcare. Research topics can be specifically divided into two categories. The first category focuses on how the medical industry applies IoT technology to epidemic prevention and control and patient treatment during the epidemic period. The IoT has the advantage of usability and accessibility, which makes data security a challenge while providing convenience. The second category of research focuses on the security of open data in the IoT and how to protect the data with technologies such as blockchain.

At the beginning of COVID-19, the most effective way to prevent the spread of the virus was to keep a physical distance and wear a mask. In order to effectively monitor the physical distance, Vedaei et al. [ 31 ] proposed a system composed of an IoT node, a smartphone application, and machine learning tools. The system records the user's health index, displays his or her health status, and prompts the user about a safe physical distance. In the environment of IoT, data security and information privacy are also concerned. Wazid et al. [ 32 ] proposed a blockchain-based secure authentication key management scheme to ensure the security of communication data. Wang et al. [ 33 ] invented the GuardHealth system, which can not only realize the sharing of medical data but also guarantee the privacy of medical data. Tahir et al. [ 3 ] proposed a blockchain-based network authentication framework for the IoT using a probabilistic model to enhance access control. It can be seen that the security management authentication and access control of the IoT are closely related to blockchain.

The study found that the emergence of these five types of research is based on modern technology and the environment on the existing healthcare services put forward new requirements. The first is the accuracy of data collection, which makes the medical system reliable and effective (sensors), and the second is that patient treatment requires continuous testing and analysis of medical data under different monitoring data, resulting in a large amount of data. How to analyze this data has become a huge challenge (machine learning). The third is the effectiveness of treatment (AI); the need for patient numbers, types of analysis, and response times has become more urgent, new treatments need to be simulated in a robust, and scalable framework that delivers the highest quality results in the shortest possible time. The fourth is the confidentiality of data (blockchain), which is sensitive because it can be maliciously used by terrorists or pharmaceutical monopolists against specific sectors. This requires that such a framework be tamper-proof and nonhackable, preventing any fraudulent manipulation of sensitive medical data. The Internet of Things is the interconnection between medical devices.

5. Future Research of Machine Learning and Blockchain Technology

After that, we continued to use the authors' keywords to conduct analysis with the help of the software VOSviewer. The lighter the color, the newer the research, and we tried to find and explain this research trend through the color. Figure 7 shows the trend of the research topic over time; the lighter the color, the newer the research.

Map of research trends.

5.1. Biomedical Monitor

In the research on machine learning, scholars currently monitor and diagnose hormone indicators of chronic diseases (diabetes, Alzheimer's disease, cognitive impairment, etc.) through different algorithms and try to improve the accuracy of monitoring and diagnosis through methodology. Future research will focus on biological monitoring.

Tchito Tchapga et al. [ 34 ] have begun to discuss the impact of the size of the data set on machine learning algorithms and have proposed the Spark algorithm to perform the classification of biomedical images. Zhao et al. [ 35 ] designed an intelligent disease recognition algorithm based on deep learning algorithm, which has good international adaptability and science for monitoring the adaptive behavior of children with ADHD. Wu et al. [ 36 ] have been able to evaluate videos through in-depth learning.

5.2. Fog Computing

The Internet of Things (IoT) is facing challenges due to the explosive growth of devices and the amount, variety, accuracy, and speed of the data these devices generate. This requires ultralow latency, reliable service, and security and privacy. Fog computing is a promising solution to overcome these challenges. Fog computing is a highly virtualized technology that provides computing, storage, and networking services in multiple application domains.

The combination of IoT devices and cloud services became the basis of fog computing. Smart medicine promises a health system that connects people and medical institutions in an intelligent way by using technologies such as wearables and IoT devices to access information in real time to deliver healthcare services. A computational distributed fog computing approach processes data from dementia and COPD patients' perceptions in the home environment and sends it to physicians to help them make quick decisions in the comfort of their homes in order to better treat patients with Parkinson's disease [ 37 ]. Fog computing is emerging as a key enabler of smart healthcare to ensure adequate patient safety. Fog computing nodes form a distributed network. Blockchain technology is a trusted shared distributed decentralized database archiving technology that caches transaction records across the network and can be easily integrated with fog computing. Tuli et al. proposed another blockchain-based security framework that integrates sensor FOG nodes and cloud servers. Their framework ensures reliability security and QoS [ 9 ].

5.3. Activity Cognition

Sensors can realize the recognition of activities and state prediction through feature extraction, with the assistance of classification algorithm. At present, the application of smart medical systems is still in the initial stage, in which data collection and feature extraction are the key to deploy smart medical infrastructure. Sensors are the foundation of all medical data collection, and one direction for future research is to improve the performance of complex human activity recognition frameworks. The scholars further proposed that channel state information could be used instead of wearable sensor devices to collect data, and feature selection algorithm and classification algorithm could be used to select and classify features, so as to predict the health status of patients.

5.4. Security and Privacy

The Internet of Things can provide a new turning point by facilitating communication with nearby entities or objects. Through the Internet of Things, existing research allows doctors and other healthcare personnel to know the status of patients. It can also be stored in a database for further access and can be analyzed so that any serious problems can be prevented or cured if they occur. However, the front-end devices of the Internet of Things, such as temperature sensors, card readers, wearable devices, mobile phones, and other embedded computer systems, are in an open environment, so there will be access control problems. The Internet of Things has the advantages of accessibility and convenience, which at the same time lays hidden dangers for the security of medical data. Therefore, security and privacy are an important trend of future research. [ 3 , 32 , 33 ].

6. Conclusion

This paper conducts a bibliometric analysis of research related to machine learning and blockchain technologies' application in smart healthcare. Based on the quantitative analysis of 112 intelligent medical research literatures in the Web of Science (WOS) database, the relevant conclusions were drawn. Research hotspots in this area include machine learning in telemedicine, disease diagnosis and monitoring, blockchain technology in medical data privacy, drug supply chain management, and the role of the Internet of Things in monitoring patient's vital signs. The most researched subjects illustrate the contributions of blockchain and machine learning technologies to the world of computing, engineering, and even medicine, both now and in the future. Future research will focus on fog computing, edge computing, and machine learning methods that lead to more accurate disease prevention and treatment. As implications for practice, identifying blockchain and machine learning as key themes for the application of two emerging technologies in the field of intelligent medicine helps to understand the development of research in this field, the common themes, and background, as well as the gaps and trends in research.

Our study has two main contributions. First, in theory, we used the method of bibliometric visualization to analyze the application of ML and BC technologies in medical field. The purpose of this study was to guide the researchers in different fields to further study the ML and BC technologies in the field of smart healthcare, so that they can know which journals, authors, and articles they can refer to when studying this phenomenon. We analyzed the latest research trends based on ML and BC technology in order to provide a research direction for future research. Second, we divided the study into five clusters by clustering and revealed the substantive relationship between the clusters. Our research provided healthcare practitioners with an insight to keep ML and BC technologies fully utilized. With this study as a foundation, new research can address the lack of research and advanced knowledge in the field. We hope that future studies will provide a deeper contribution by measuring references to data obtained through Web of Science and explain the impact of the study.

Data Availability

Conflicts of interest.

The authors declare that they have no conflicts of interest.