This paper is in the following e-collection/theme issue:

Non-Randomized Studies (funded, eHealth) (126) Clinical Information and Decision Making (1931) Advanced Data Analytics in eHealth (212) Clinical Informatics (1335) Machine Learning (2089) Decision Support for Health Professionals (1552) E-Health / Health Services Research and New Models of Care (726)

Published on in Vol 14 (2025)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/70303, first published .
Improving the Predictive Accuracy of the National Early Warning Score 2: Protocol for Algorithm Refinement

Improving the Predictive Accuracy of the National Early Warning Score 2: Protocol for Algorithm Refinement

Improving the Predictive Accuracy of the National Early Warning Score 2: Protocol for Algorithm Refinement

Authors of this article:

Chris Plummer1, 2 Author Orcid Image ;   Cen Cong3 Author Orcid Image ;   Madison Milne-Ives3, 4 Author Orcid Image ;   Lynsey Threlfall2, 5 Author Orcid Image ;   Peta Le Roux6 Author Orcid Image ;   Edward Meinert3, 7 Author Orcid Image
Article Authors Cited by (1) Tweetations Metrics

    Protocol

    1Department of Cardiology, Newcastle upon Tyne Hospitals NHS Foundation Trust, Newcastle upon Tyne, United Kingdom

    2NIHR Newcastle Biomedical Research Centre, Newcastle University, Newcastle upon Tyne, United Kingdom

    3Translational and Clinical Research Institute, Newcastle University, Newcastle upon Tyne, United Kingdom

    4Centre for Health Technology, School of Nursing and Midwifery, University of Plymouth, Plymouth, United Kingdom

    5Acute Medicine, Newcastle upon Tyne Hospitals NHS Foundation Trust, Newcastle upon Tyne, United Kingdom

    6Information Services, Newcastle upon Tyne Hospitals NHS Foundation Trust, Newcastle upon Tyne, United Kingdom

    7Department of Primary Care and Public Health, Imperial College London, London, United Kingdom

    Corresponding Author:

    Chris Plummer, BSc, BM, BCh, PhD

    Department of Cardiology

    Newcastle upon Tyne Hospitals NHS Foundation Trust

    Freeman Hospital

    Newcastle upon Tyne, NE7 7DN

    United Kingdom

    Phone: 44 01912336161

    Email: chris.plummer@newcastle.ac.uk


    Background: The National Early Warning Score 2 (NEWS2) has been widely adopted for predicting patient deterioration in health care settings using routinely collected physiological observations. The use of NEWS2 has been shown to reduce in-hospital mortality, but it has limited accuracy in the prediction of clinically important outcomes, especially over longer time periods.

    Objective: This project aims to improve the predictive accuracy of the NEWS2 scoring system, particularly its accuracy over more than 24 hours and its predictive value in older patients and children. It will investigate whether using the currently collected data differently and the inclusion of additional data would result in an improved algorithm.

    Methods: The study will use historical patient data from the Newcastle upon Tyne Hospitals NHS Foundation Trust, including observational data (eg, vital signs), BMI- related data, and other outcome-related variables (eg, mortality rates) to train and test an algorithm to predict the risk of key clinical outcomes, including mortality, intensive therapy unit admission, sepsis, and cardiac arrest, to demonstrate a proof of concept for a modified scoring system. The algorithm’s performance will be assessed based on its accuracy, precision, F1-score, area under the curve, and receiver operating characteristic curve.

    Results: The study is expected to start in April 2025. The findings are expected to be produced by the end of 2026 and will be disseminated at symposia, conferences, and in journal publications.

    Conclusions: The refined NEWS2 algorithm will address limited accuracy in predicting clinical deterioration beyond 24 hours in the original system by incorporating additional variables. Improved accuracy in the early detection of deterioration can lead to timely interventions, potentially reducing mortality and adverse clinical events. The enhanced algorithm also has the potential to be integrated into existing clinical decision support systems to facilitate health care professionals’ decision-making.

    International Registered Report Identifier (IRRID): PRR1-10.2196/70303

    JMIR Res Protoc 2025;14:e70303

    doi:10.2196/70303

    Keywords

    clinical deterioration; National Early Warning Score; performance; proof-of-concept 


    The National Early Warning Score 2 (NEWS2) system is a widely used tool for the systematic documentation and identification of clinical deterioration. It has been demonstrated to be effective at reducing in-hospital mortality [1-4], it facilitates effective communication between clinicians and enables timely interventions to improve patient outcomes, but NEWS2 has limited positive and negative predictive accuracy [5-7], particularly in predicting adverse events beyond 24 hours [8-10].

    The use of digital technologies in health care presents an opportunity to evaluate whether the inclusion of additional routinely collected variables would improve the predictive accuracy of an early warning score and thus reduce patient risk. The same technologies could then be used to provide clinical decision support using artificial intelligence–derived algorithms that would not be practical using paper observation charts and health care records [1,11].

    This further personalization of risk prediction is increasingly important in our aging population. Older adults are particularly vulnerable to sudden changes in physiological status [12], but this group was not included in the development of NEWS2 [13]. Improved predictive accuracy in this growing frail and vulnerable group could significantly improve patient outcomes [13-15].

    In an evaluation of the performance of the original National Early Warning Score (NEWS), it demonstrated consistent predictive accuracy across multiple patient groups [5]. However, the performance of NEWS2 varies between care settings and between patients with different characteristics [11,16,17]. NEWS2 has been found to have significantly lower specificity, sensitivity, and positive predictive value for those who were at risk of type II respiratory failure, but higher specificity and positive predictive value for those with documented type II respiratory failure status, than the original NEWS [16]. The accuracy of NEWS has also been found to vary between care settings [17]. These observations suggest that the inclusion of individualized patient and care setting characteristics may improve the predictive accuracy, sensitivity, and specificity of the system and thus improve outcomes in a range of health care settings and in diverse patient groups.

    To determine which variables may improve the risk prediction of NEWS2, we have reviewed 6 databases (CINAHL, PubMed, Embase, ScienceDirect, Cochrane Library, and Web of Science) [18]. Our preliminary findings show that demographic variables including age and ethnicity [6,19], trend data [20-22], and adjustments to component weighting [23,24] may improve the performance of NEWS2 in predicting clinical deterioration.

    The principal aim of this work is to develop an algorithm that can improve the predictive accuracy and thus clinical value of the NEWS2 system. The objectives of this study are as follows:

    1. To determine which additional variables have been previously used to enhance the accuracy of clinical early warning scores, including NEWS2.
    2. To collect a large set of patient data that links observations, demographic, and admissions data with key deterioration-related outcomes.
    3. To use these data to train an artificial intelligence model to determine which variables, and what weightings, provide optimal predictive accuracy.
    4. To test the accuracy, sensitivity, and specificity of the algorithm.
    5. To consider how the new model could be applied across a wide range of health care settings, including different physiological observation equipment, different electronic patient record systems, or paper-based clinical record systems.

    Overview

    The study is a retrospective cohort study using historical, anonymized patient data to explore ways in which a new alerting system could improve NEWS2. The Newcastle Hospitals National Health Service (NHS) Foundation Trust has an established information governance system for the use of anonymized patient data for research. This study will use historical, anonymized hospital inpatient and emergency care data from the Newcastle upon Tyne Hospitals NHS Foundation Trust, collected routinely over 6 years (December 2018 to June 2024).

    Patient and public involvement (PPI) refers to research being carried out “with” or “by” members of the public rather than “to,” “about” or “for” them [25]. To support PPI in research, we will recruit PPI members through the Newcastle BRC Informatics and Precision Care for an Ageing Population theme and Newcastle University’s Clinical Trials Unit. They will be included in all project management group meetings and will provide their lived experience with hospital monitoring to highlight any concerns or barriers they foresee in the adoption and implementation of a modified NEWS system. At least 1 older adult will be included to ensure that we are including a perspective that has been missing in the development of previous NEWS systems. A logic diagram of the study design is presented in Multimedia Appendix 1.

    The model will be developed following the recommended process of building a predictive model with machine learning [26,27]. The research question and a clear goal of building a model will be first defined. The following types of data (Table 1) will then be obtained from the Trust.

    Table 1. Input variables.
    TypeInput variables
    General
    • Patient ID
    Demographic data
    • Age
    • Biological sex
    • Ethnicity
    Observational data (vital signs and other measures)
    • Observation date, time, and location
    • Encounter: ID and type
    • Heart rate
    • Temperature
    • Respiratory rate
    • Need of oxygen therapy
    • Level of consciousness or new confusion (Alert, Verbal, Pain, or Unresponsive score)
    • Blood pressure (systolic and diastolic)
    • Paired value One (lying blood pressure)
    • Paired value Two (standing blood pressure)
    • Oxygen therapy percentage—inspired oxygen fraction (FiO2)
    • Oxygen saturation (SpO2)
    • Nursing concern
    • Urine output
    • Pain score
    • Mask code (oxygen therapy device used)
    • Partial Obs set (NEWS2a variables)
    • Partial Obs set (additional variables) 
    BMI data (in separate data file)
    • Height (numeric, units, date, and time)
    • Weight (numeric, units, date, and time)
    • BMI (score date and time) 
    Outcome-related data
    • Mortality
    • Cardiac arrest
    • Resuscitation
    • Unplanned admission to critical care: include either intensive therapy unit or high-dependency unit
    • Sepsis
    • NEWS2 score
    • Northumbria Healthcare NHSb Foundation Trust risk rating (NuTH)- overwrite thresholds (NuTH specific)

    aNEWS2: National Early Warning Score.

    bNHS: National Health Service.

    The data will then be deidentified before the research team can access it within a secure project environment in the Trust’s IT network for training and testing the algorithm. Data will also be prepared by handling missing values and any inconsistencies or imbalances. Deidentified data from the Trust’s clinical data marts, including patient observations, admissions information, demographic data, and outcomes data, will be divided into training and testing subsets. A decision will then be made on whether to test the performance of multiple existing models with additional data or improve the performance of 1 model (eg, eXtreme Gradient Boosting). Using these datasets, we will train and test an algorithm that optimizes the variables and their weightings to predict the risk of key clinical outcomes, including mortality, intensive therapy unit admission, sepsis, and cardiac arrest, to demonstrate a proof of concept for a modified scoring system.

    The following metrics will be used to evaluate the model’s performance. Definitions and equations are obtained from the literature [28,29].

    1. Accuracy: the number of correct predictions divided by the total number of predictions.
    2. Precision: the ratio of true positives and total positives predicted.
    3. F1-score: the harmonic mean of precision and recall.
    4. Area under the curve (AUC): the probability that the model, if given a randomly chosen positive and negative example, will rank the positive higher than the negative. The closer the AUC is to 1.0, the better the model’s ability to separate classes from each other.
    5. Receiver-operating characteristic curve: this curve is typically plotted in terms of true positive rate and false positive rate. It is drawn by calculating the true positive rate and the false positive rate at every possible threshold (in practice, at selected intervals), then graphing the true positive rate over the false positive rate.

    The performance of the model will be optimized through hyperparameter tuning and configuration refining. The final model will be tested with the test set against predetermined metrics. Depending on funding availability, the model may be validated using external collaborator’s datasets.

    Ethical Considerations

    This study (Integrated Research Application System Project ID 344472) has received ethical approval from the UK Health Research Authority Research Ethical Committee (Rec reference 25/WA/0060). Informed consent and compensation do not apply to the study, as no human participants will be involved. A data scientist at the Trust will exclude all data from patients who have opted out of their data being used for research purposes through local or national processes. All patient data used in this project are fully deidentified by the NHS before being accessed by the research team, ensuring no personal information about participants is collected. Data sharing is restricted to authorized team members with NHS-approved credentials and access is managed through secure NHS systems with strict monitoring. During the project, data analysis will be conducted within password-protected environments. After the project, data will be securely archived or deleted following NHS policies. Patients will not be identified in any way in the findings to ensure participant confidentiality.


    Algorithm development has not yet commenced as of February 26, 2025. Work is expected to start in April 2025 and findings are expected to be produced by December 2026. By the end of this study, we will have trained and tested an algorithm based on historical patient data from the Newcastle upon Tyne Hospitals NHS Foundation Trust. This will provide a proof-of-concept of whether and which additional data can improve the predictive accuracy of the NEWS2 system.

    After refinement, the algorithm is expected to be able to predict clinical disease deterioration accurately. This will be assessed by comparing the algorithm’s predictions against observed outcomes, including mortality, cardiac arrest, unplanned admissions to critical care units (intensive therapy units or high-dependency units), resuscitation events, and the occurrence or progression of sepsis. A summary of outcome items and measures is listed in Table 2.

    Table 2. Outcome items and measures.
    ItemDefinitionMeasureExpected result
    MortalityWhether the patient survives within a specified time frameQuantitative: mortality rateHigh accuracy and precision against patient records
    Cardiac arrestSudden loss of all heart activityQuantitative: number of correctly predicted cardiac arrest casesHigh accuracy and precision against documented cardiac arrest cases
    Unplanned critical care admissionsWhether the patient requires admission to intensive careQuantitative: number of correctly predicted admissions

    		High accuracy and precision against observed admissions to critical care units
    Resuscitation eventsProcess of reviving acutely ill patientQuantitative: number of correctly predicted resuscitation eventsHigh accuracy and precision against the actual number of events
    SepsisLife-threatening reaction to an infectionQuantitative: number of correctly predicted sepsis eventsHigh accuracy and precision against documented sepsis cases

    Principal Findings

    The anticipated results of the study include the algorithm’s increased predictive accuracy. There are 2 main ways in which this could improve patient outcomes (1) improving the system’s ability to accurately predict patients who will deteriorate within and beyond 24 hours would allow health care professionals to optimize their care and prevent deterioration and (2) reducing false positive alerts would allow care to be concentrated where it will achieve the greatest benefit.

    Comparison to Prior Work

    Several studies have developed machine learning algorithms (eg, logistic regression, support vector machine, gradient boosting decision tree, and XGBoost) to predict clinical deterioration in emergency departments, such as in-hospital cardiac arrest, clinical deterioration, and mortality, using historical prehospital data or emergency department data in China and Korea [30-32]. While these algorithms achieved high accuracy (AUC over 0.8) in predicting clinical deterioration in emergency department patients, their relevance and potential generalizability to different population demographics and health care contexts remain unknown. This study uses historical patient data from a UK-based NHS Trust, making the algorithm more relevant to the UK health care system. In addition, it includes more demographic variables (ie, age and ethnicity) and BMI-related data to inform a more comprehensive risk assessment. However, this study still requires validation with external datasets to increase its generalizability.

    Future Directions

    Patient and Public Involvement and Engagement members will help guide the dissemination of the results of the study to the population through publicly accessible forums. Following this work, we will test the new system on data in other NHS Trusts across the United Kingdom to assess its generalizability. In 2026, we will conduct implementation studies in Newcastle and then in other NHS Trusts to assess the adoption of the system and gather initial data on patient outcomes to construct a learning health care system to evaluate the real-world performance of the algorithm and facilitate continual improvement. By 2029, we aim to implement the system in NHS Trusts across the United Kingdom and conduct large-scale trials to evaluate its effectiveness in improving patient outcomes.

    Strengths and Limitations

    This work highlights the importance of investigating the use of additional clinical variables to those used in NEWS2 in the development of a National Early Warning Score. The study design was informed by an evidence synthesis of the literature. Despite these strengths, this project also has limitations. Some retrospective datasets may be of low quality or incomplete. Furthermore, external validation will be needed to test algorithm generalizability.

    Conclusion

    This work has identified potential areas of improvement to the NEWS2 system and outlined a project for improving its ability to predict adverse outcomes through the use of additional patient data or changes to the scoring process. The project’s outputs will be shared widely to facilitate collaboration between health care professionals, researchers, and policymakers to support the implementation of improved clinical early warning systems in diverse populations.

    Acknowledgments

    We would like to thank the editors for their valuable feedback and suggestions during the review of this paper. This work was supported by the National Institute for Health and Care Research (NIHR) Newcastle Biomedical Research Centre (BRC) based at the Newcastle upon Tyne Hospitals NHS Foundation Trust, Newcastle University, and the Cumbria, Northumberland, and Tyne and Wear (CNTW) NHS Foundation Trust. The views expressed in this publication are those of the authors and not necessarily those of the NIHR BRC or any of the authors’ affiliated universities. The open access publication fee was paid by the Newcastle BRC. The funding body was not involved in the study design, data collection or analysis, or the writing and decision to submit the article for publication. EM is the cofounder and Chief Executive Officer of Gnosis Health Limited, a spin-out company from Newcastle University that develops software for the management of chronic diseases. This affiliation did not influence the design, conduct, or reporting of the research or views presented in this manuscript, and no conflict of interest arose in its development.

    Data Availability

    Data sharing is not applicable to this article as no data sets were generated or analyzed during this study.

    Authors' Contributions

    CP contributed to conceptualization, funding acquisition, writing—review and editing. CC contributed to writing—original draft, writing—review and editing. MM contributed to methodology, writing—original draft. LT contributed to writing—review and editing. PLR contributed to writing—review and editing. EM contributed to conceptualization, supervision, writing—review and editing.

    Conflicts of Interest

    EM is the Editor-in-Chief of JMIRx Med.

    Multimedia Appendix 1

    Study logic diagram.

    PNG File , 76 KB
    Multimedia Appendix 2

    Peer review report from the NIHR Newcastle Biomedical Research Centre (National Institute for Health and Care Research, UK).

    PDF File (Adobe PDF File), 144 KB
    1. Holland M, Kellett J. The United Kingdom's National Early Warning Score: should everyone use it? A narrative review. Internal and Emergency Medicine. Jan 05, 2023;18(2):573-583. [FREE Full text] [CrossRef] [Medline]
    2. Finnikin S, Wilke V. What's behind the NEWS? National Early Warning Scores in primary care. Br J Gen Pract. Jun 2020;70(695):272-273. [FREE Full text] [CrossRef] [Medline]
    3. Myrstad M, Ihle-Hansen H, Tveita AA, Andersen EL, Nygård S, Tveit A, et al. National Early Warning Score 2 (NEWS2) on admission predicts severe disease and in-hospital mortality from Covid-19 - a prospective cohort study. Scand J Trauma Resusc Emerg Med. Jul 13, 2020;28(1):66. [FREE Full text] [CrossRef] [Medline]
    4. Scott LJ, Redmond NM, Garrett J, Whiting P, Northstone K, Pullyblank A. Distributions of the National Early Warning Score (NEWS) across a healthcare system following a large-scale roll-out. Emerg Med J. May 2019;36(5):287-292. [FREE Full text] [CrossRef] [Medline]
    5. Price C, Prytherch D, Kostakis I, Briggs J. Evaluating the performance of the National Early Warning Score in different diagnostic groups. Resuscitation. Dec 2023;193:110032. [FREE Full text] [CrossRef] [Medline]
    6. Carr E, Bendayan R, Bean D, Stammers M, Wang W, Zhang H, et al. Shi T, et al. Evaluation and improvement of the National Early Warning Score (NEWS2) for COVID-19: a multi-hospital study. BMC Med. Jan 21, 2021;19(1):23. [FREE Full text] [CrossRef] [Medline]
    7. Williams B. The National Early Warning Score: from concept to NHS implementation. Clin Med (Lond). Nov 2022;22(6):499-505. [FREE Full text] [CrossRef] [Medline]
    8. Holland M, Kellett J. A systematic review of the discrimination and absolute mortality predicted by the National Early Warning Scores according to different cut-off values and prediction windows. Eur J Intern Med. 2022;98:15-26. [FREE Full text] [CrossRef] [Medline]
    9. Wei S, Xiong D, Wang J, Liang X, Wang J, Chen Y. The accuracy of the National Early Warning Score 2 in predicting early death in prehospital and emergency department settings: a systematic review and meta-analysis. Ann Transl Med. Jan 31, 2023;11(2):95. [FREE Full text] [CrossRef] [Medline]
    10. Mcneill H, Khairat S. Impact of intensive care unit readmissions on patient outcomes and the evaluation of the National Early Warning Score to prevent readmissions: literature review. JMIR Perioper Med. May 08, 2020;3(1):e13782. [FREE Full text] [CrossRef] [Medline]
    11. Zhang K, Zhang X, Ding W, Xuan N, Tian B, Huang T, et al. The prognostic accuracy of National Early Warning Score 2 on predicting clinical deterioration for patients with COVID-19: a systematic review and meta-analysis. Front Med (Lausanne). 2021;8:699880. [FREE Full text] [CrossRef] [Medline]
    12. Clegg A, Young J, Iliffe S, Rikkert MO, Rockwood K. Frailty in elderly people. The Lancet. 2013;381(9868):752-762. [CrossRef]
    13. Vardy ER, Lasserson D, Barker RO, Hanratty B. NEWS2 and the older person. Clin Med (Lond). 2022;22(6):522-524. [FREE Full text] [CrossRef] [Medline]
    14. Kemp K, Alakare J, Harjola VP, Strandberg T, Tolonen J, Lehtonen L, et al. National Early Warning Score 2 (NEWS2) and 3-level triage scale as risk predictors in frail older adults in the emergency department. BMC Emerg Med. Oct 28, 2020;20(1):83. [FREE Full text] [CrossRef] [Medline]
    15. Sze S, Pan D, Williams CML, Wong N, Sahota A, Bell D, et al. Letter to the editor: variability but not admission or trends in NEWS2 score predicts clinical outcome in elderly hospitalised patients with COVID-19. J Infect. 2021;82(1):159-198. [FREE Full text] [CrossRef] [Medline]
    16. Pimentel MAF, Redfern OC, Gerry S, Collins GS, Malycha J, Prytherch D, et al. A comparison of the ability of the national early warning score and the national early warning score 2 to identify patients at risk of in-hospital mortality: a multi-centre database study. Resuscitation. 2019;134:147-156. [FREE Full text] [CrossRef] [Medline]
    17. Zaidi H, Bader-El-Den M, McNicholas J. Using the National Early Warning Score (NEWS/NEWS 2) in different Intensive Care Units (ICUs) to predict the discharge location of patients. BMC Public Health. Sep 05, 2019;19(1):1231. [FREE Full text] [CrossRef] [Medline]
    18. Milne-Ives M, Riccalton V, Plummer C, Threlfall L, Cong C, Ananthakrishnan A, et al. Modifications to the National Early Warning Score: a scoping review protocol. BMJ Open. Oct 09, 2024;14(10):e089061. [FREE Full text] [CrossRef] [Medline]
    19. Nissen SK, Candel BGJ, Nickel CH, de Jonge E, Ryg J, Bogh SB, et al. The impact of age on predictive performance of National Early Warning Score at arrival to emergency departments: development and external validation. Ann Emerg Med. Apr 2022;79(4):354-363. [FREE Full text] [CrossRef] [Medline]
    20. Forster S, McKeever TM, Churpek M, Gonem S, Shaw D. Predicting outcome in acute respiratory admissions using patterns of National Early Warning Scores. Clin Med (Lond). Sep 2022;22(5):409-415. [FREE Full text] [CrossRef] [Medline]
    21. Gonem S, Taylor A, Figueredo G, Forster S, Quinlan P, Garibaldi JM, et al. Dynamic early warning scores for predicting clinical deterioration in patients with respiratory disease. Respir Res. 2022;23(1):203. [FREE Full text] [CrossRef] [Medline]
    22. Zhu Y, Chiu YD, Villar SS, Brand JW, Patteril MV, Morrice DJ, et al. Dynamic individual vital sign trajectory early warning score (DyniEWS) versus snapshot national early warning score (NEWS) for predicting postoperative deterioration. Resuscitation. Dec 2020;157:176-184. [FREE Full text] [CrossRef] [Medline]
    23. Clarke J, Gallifant J, Grant D, Desai N, Glover G. Predictive value of the National Early Warning Score 2 for hospitalised patients with viral respiratory illness is improved by the addition of inspired oxygen fraction as a weighted variable. BMJ Open Respir Res. 2023;10(1):e001657. [FREE Full text] [CrossRef] [Medline]
    24. Malycha J, Farajidavar N, Pimentel MAF, Redfern O, Clifton DA, Tarassenko L, et al. The effect of fractional inspired oxygen concentration on early warning score performance: a database analysis. Resuscitation. 2019;139:192-199. [FREE Full text] [CrossRef] [Medline]
    25. Definitions for patient and public involvement and engagement. NIHR Great Ormond Street Hospital Biomedical Research Centre. 2025. URL: https:/​/www.​gosh.nhs.uk/​our-research/​our-research-infrastructure/​nihr-great-ormond-street-hospital-brc/​nihr-gosh-brc-and-crf-joint-support-research-strategies/​our-joint-strategy-for-patient-and-public-involvement-and-engagement/​definitions-for-patient-and-public-involvement-and-engagement/​ [accessed 2025-02-26]
    26. Gupta R, Sharma A, Alam T. Building predictive models with machine learning. In: Studies in Big Data. Singapore. Springer Nature Singapore; 2024:39-59.
    27. Efthimiou O, Seo M, Chalkou K, Debray T, Egger M, Salanti G. Developing clinical prediction models: a step-by-step guide. BMJ. 2024;386:e078276. [FREE Full text] [CrossRef] [Medline]
    28. Joshi AV. Machine Learning and Artificial Intelligence. 2nd Edition. Cham, Switzerland. Springer International Publishing; Dec 16, 2022.
    29. Google Developers Site. Classification: accuracy, recall, precision, and related metrics. URL: https://developers.google.com/machine-learning/crash-course/classification/accuracy-precision-recall [accessed 2024-12-09]
    30. Zhai Q, Lin Z, Ge H, Liang Y, Li N, Ma Q, et al. Using machine learning tools to predict outcomes for emergency department intensive care unit patients. Sci Rep. 2020;10(1):20919. [FREE Full text] [CrossRef] [Medline]
    31. Kim JH, Choi A, Kim MJ, Hyun H, Kim S, Chang H. Development of a machine-learning algorithm to predict in-hospital cardiac arrest for emergency department patients using a nationwide database. Sci Rep. 2022;12(1):21797. [FREE Full text] [CrossRef] [Medline]
    32. Choi A, Choi SY, Chung K, Chung HS, Song T, Choi B, et al. Development of a machine learning-based clinical decision support system to predict clinical deterioration in patients visiting the emergency department. Sci Rep. 2023;13(1):8561. [FREE Full text] [CrossRef] [Medline]

    ‎
    AUC: area under the curve
    NEWS: National Early Warning Score
    NEWS2: National Early Warning Score 2
    NHS: National Health Service
    PPI: patient and public involvement
    XGBoost: extreme gradient boosting

    Edited by A Schwartz; The proposal for this study was peer-reviewed by: NIHR Newcastle Biomedical Research Centre (National Institute for Health and Care Research, UK). See the Multimedia Appendix for the peer-review report; submitted 19.12.24; accepted 08.04.25; published 21.07.25.

    Copyright

    ©Chris Plummer, Cen Cong, Madison Milne-Ives, Lynsey Threlfall, Peta Le Roux, Edward Meinert. Originally published in JMIR Research Protocols (https://www.researchprotocols.org), 21.07.2025.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Research Protocols, is properly cited. The complete bibliographic information, a link to the original publication on https://www.researchprotocols.org, as well as this copyright and license information must be included.