Case Report
Early warning scores for the detection of deteriation in specific patients groups
Inhoud:

    Auteur(s):

    A.P.M. Rongen (1), A. van Leent (1), S. E. Hoeks (1), Ruud van Dam (2), Lotte Snijders (2), Jim Smit (2), Jasper van Bommel (2) 

    1Department of Anesthesiology, 2Department of Intensive Care Erasmus Medical Center, Erasmus University, Rotterdam, the Netherlands

    Correspondentie:

    A.P.M. Rongen - a.rongen@erasmusmc.nl
    Case Report

    Early warning scores for the detection of deteriation in specific patients groups

    Abstract

    Background Our objective was to compare the prognostic accuracy of general and specific early warning scores (EWS) for specific patient groups for the prediction of serious adverse events. Our expectation is that patient-specific EWSs will be more accurate in detecting SAEs in these patient-groups compared with general EWSs.

    Methods The data sources EMBASE, MEDLINE, CINAHL, Web of Science Core Collection, Cochrane Central Register of Controlled Trials and Google Scholar were searched from inception until August 2021. Validation studies on both general and patient-specific EWSs in patients admitted to a cardiothoracic, neurologic/neurosurgical, pulmonary, hepatologic or hematologic ward, to predict SAEs (mortality, cardiac arrest and unplanned admission to a higher level of care) were eligible for inclusion.

    Results Twenty-two studies were included of which eighteen were rated as high risk of bias on at least one domain. These studies were performed in cardiothoracic, hematology, neurology and pulmonary patient subgroups and the EWSs used in these groups were mostly general. A meta-analysis could only be performed for the EWSs predicting mortality in pulmonary patients of which CURB65 showed the most favorable prognostic accuracy.

    Conclusions We conclude there is a lack of validated patient-specific EWSs. Therefore, we cannot make any conclusions on whether either patient-specific or general EWSs are more suitable in predicting SAEs in pulmonary, neurologic, hematologic and cardiothoracic patients. Patient-specific EWSs in specific disease groups require further validation of their performance in detecting adverse events in order to conclude superiority of patient-specific, or general EWSs.

    Study registration PROSPERO CRD42021284623.

    Introduction

    Multiple studies[1-3] provide evidence that hospitalised patients suffering from serious adverse events (SAEs), i.e. cardiopulmonary arrest or admission to an intensive care unit, show detectable signs of clinical deterioration in the hours preceding these events. Therefore, many of these SAE’s are considered preventable. Early warning scores (EWSs) have been introduced as a potential solution to this problem. These tools are prediction models that assess patient characteristics and physiological parameters to stratify the risk of clinical deterioration. Subsequently, they trigger more intensive care, such as increased nursing attention, alerting the care provider, or activating a rapid response team (RRT). At first, these EWSs seemed successful in preventing SAEs leading to general implementation in heterogeneous patient populations.[4] However, a recent Cochrane review[5] showed that EWSs and RRTs may lead to little or no difference in SAEs. This may be explained by heterogeneity in study populations, but also by the general nature of the EWSs. This raises the question whether the system would work better if patient-specific instead of general EWSs would be used for specific (homogeneous) patient populations.

    The prognostic accuracy of these EWSs (e.g. NEWS, NEWS2, qSOFA) has been evaluated in systemic reviews but these studies were either based on general patient populations,[6] or focused on a single or limited EWSs in specific patient groups.[7,8] A comprehensive review and comparison of the prognostic accuracy of all available general and patient-specific EWSs is essential in the search of the best available EWS instrument per patient group, but is still lacking.

    Therefore, the aim of this study was to 1) provide an overview of the different EWSs used in specific patient populations and 2) to outline their prognostic accuracy to predict SAEs. Based on our results we will make recommendations on which tool to use in these specific patient groups. Our expectation is that there are multiple patient-specific EWSs for different patient groups, which will be more accurate in detecting SAEs in these groups compared with general population EWSs.

    Methods

    This review was registered on the PROSPERO database (ID CRD42021284623). We followed the PRISMA-DTA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) guidelines for diagnostic test accuracy reviews when reporting results.[9] A protocol was not prepared.

    Study identification

    The databases EMBASE, MEDLINE, CINAHL, Web of Science Core Collection, Cochrane Central Register of Controlled Trials and Google Scholar were searched from inception until August 8th 2021. An experienced health sciences librarian assisted in the development of the search strategy. The complete search strategy is presented in Appendix A and was modified appropriately for each database. Searches were not limited by language, date or publication status. The PubMed function ‘Similar Articles’ and reference lists of all included articles were screened for additional studies.

    Study selection

    Studies were included if: (1) they included validation of any EWS, consisting of an aggregated weighted score, in hospitalised, nonpregnant adult patients on general hospital wards; (2) the EWS validation was performed specifically for patients admitted to a cardiothoracic, neurologic/neurosurgical, pulmonary, hepatologic or hematologic wards and [10] they reported EWS performance on at least one of the following SAEs; mortality, cardiac arrest and unplanned admission to a higher level of care.

    Studies were excluded if; (1) the EWS was not an aggregated weighted score but a complicated statistical model that would not be generally applicable; (2) the reported data was not sufficient to construct a 2x2 contingency table in order to check the validity of the published results and exclude studies that reported spurious results.

    Titles and abstracts from the search results and respectively full-texts of all potentially eligible studies were independently reviewed by two reviewers (AR/AvL) following the in- and exclusion criteria. Disagreements were resolved by referral to a third reviewer (JvB).

    Data extraction and quality assessment

    Data extraction was performed independently by two investigators (AR/AvL) and consisted of author and study information (study design, country, year of publication, period of enrollment, number of patients included, type, number and threshold of EWS under investigation), patient characteristics (age, gender, reason for hospital admission), outcome(s) assessed, sensitivity, specificity, positive and negative likelihood ratio’s and area under the curve (AUC). Contingency tables were extracted or reconstructed for expressing the diagnostic accuracy. Where studies reported more than one set of 2x2 contingency tables, e.g. for different thresholds, then each set of 2x2 data was extracted.

    Study quality and risk of bias was assessed independently by two reviewers (AR/AvL) using the revised tool for the quality assessment of diagnostic accuracy studies (QUADAS-2).[11] Again, disagreements were resolved by referral to a third reviewer (JvB). The QUADAS-2 assesses four potential areas for bias and applicability of the research question: patient selection, index test, reference standard, and flow and timing.

    Data synthesis

    We generated study-specific estimates of sensitivity, specificity, and corresponding 95% confidence intervals and graphically illustrated these using forest plots. We grouped studies by patient population and outcomes assessed.

    EWSs which were analyzed in at least 5 studies were considered appropriate for inclusion in the meta-analysis. Studies can report a single threshold or multiple threshold values for different EWS scores and/or outcomes. We applied the bivariate random-effects model to calculate summary estimates of sensitivity, specificity, and predictive values of studies based on single data points from each study. As only single data points can be included, we selected the optimal threshold per score by the method described by Steinhauser et al[12] (R package diagmeta).

    We constructed summary receiver operating characteristic (SROC) curves to represent the overall prognostic accuracy of the instruments. This is conceptually very similar to the ROC, however, each data point comes from a different study, not a different threshold giving a curve which is solely shaped by the results across the studies.[13] We calculated 95% confidence intervals and 95% prediction intervals around estimates where appropriate.

    We used R for Windows (Version 4.1.0; R Foundation for Statistical Computing, Vienna, Austria) in all analyses (mada and diagmeta package).

    Results

    Study Selection and Characteristics

    Figure 1 shows the detailed study selection process based on PRISMA-DTA reporting guidelines.[14] A total of 2300 published studies were initially identified. After removing duplicate articles and screening the abstracts, we identified 213 studies. Of these, 192 studies were excluded with reasons in the full-text assessments. In case of insufficient information, the corresponding authors were contacted through e-mail; only studies with adequate information were included. Finally, 21 articles met the eligibility criteria and were included in the review. We identified and included one more article through snowballing, which was also included, resulting in a total of 22 articles.[15-36]

    Figure 1. PRISMA flow chart

    The characteristics of the 22 included studies are described in table 1. These studies comprised only four of the five predefined patient subgroups (i.e. cardiothoracic, hematology, neurology and pulmonary) of which pulmonary diseases was the largest (n=17). Within this group, pathology consisted of COVID-19, pneumonia and acute exacerbation of chronic obstructive pulmonary disease (COPD). Eighteen different EWSs were analyzed throughout the studies regarding pulmonary diseases, however, 10 (56%) of them were analyzed just once. The other patient subgroups consisted of cardiothoracic (n=1), hematologic (n=1) and neurologic (n=3) patients. Of the outcome events mortality was the most often studied outcome (n=19) whereas cardiac arrest was the least studied (n=1). Included studies were predominantly retrospective (77%) and consisted of study sample sizes varying from 43 to 442893.

    Table 1 - Study characteristics

    Quality Assessment

    Table 2 shows the summarized results of quality assessments using QUADAS-2. Of the total of 22, 18 studies were scored on at least one risk of high bias. Eight studies scored high risk of bias on patient selection.[22-26,31-33] Inadequate exclusion criteria were the prime cause (e.g. no exclusion of patients with treatment limitations while the main outcome was ICU admission; patients excluded who lacked any element of the EWS, which could lead to confounding by indication because the sickest patients might be monitored more closely). Seven studies scored unclear risk on the index test because no information was available on whether the person interpreting the EWS was blinded to the outcome.[16,17,21,23,24,31,34] Studies (n=12) that mentioned no threshold and no information about blinding were given high risk of bias.[15,18,20,22,26-28,30,32,33,35,36] In addition, six studies were assigned high risk in flow and timing[15,16,18,23,27,31] as data were collected prospectively as part of routine care and so it was considered likely that any EWS would be acted on before outcomes were assessed. Four studies that revealed a significant percentage of missing data were assigned as high risk in flow and timing.[27-29,33] For applicability concerns, all of the studies for index test and reference standard were treated as low risk. Those studies (n=3)[26,28,29] with possible higher mortality in tertiary urban care settings were given high risk for applicability concerns for patient selection.

    Table 2 - Results of the QUADAS-2 assessment

    Pulmonary disease subgroup

    Mortality

    Of the 17 studies in the pulmonary patient subgroup 15 studies analyzed mortality using 18 different EWSs (Table 1).[15,16,18,20-22,24-26,29-31,33,34,36] The sensitivities and specificities of each instrument at the optimal thresholds are depicted in a forest plot in Figure 2 (& appendix B).

    Figure 2 - Paired Forest Plot - pulmonary & mortality

    EWSs which were analyzed in at least 5 studies were considered appropriate for inclusion in the meta-analysis and therefore a total of 14 articles were included in the meta-analysis in which 4 EWSs were studied (i.e. National Early Warning Score (NEWS), NEWS2, CURB65 and the quick sequential organ failure assessment (qSOFA) score).[15,16,18,20-22,24-26,29,31,33,34,36] Figure 3 shows the SROC plot for NEWS, NEWS2, CURB65 and qSOFA on predicting mortality in pulmonary patient subgroups with CURB65 showing the best prognostic accuracy based on the summary points. Regarding the other most frequently used instruments in this patient subgroup, based on the test accuracy studies, the sensitivity ranged considerably from 46% to 89% (NEWS); 28% to 90% (NEWS2) and 31% to 82% (qSOFA) (Appendix Table 1). A similar range was found for the specificity: 27% to 80% (NEWS); 30% to 93% (NEWS2) and 57% to 98% (qSOFA). However, these values are difficult to compare because different thresholds were used throughout the studies.

    Figure 3 - SROC

    Admission to a higher level of care

    A total of 4 studies[17,20,23,24] analyzed admission to a higher level of care in patients with pulmonary disease. The following EWSs were assessed in these studies: MEWS, NEWS, NEWS2, CURB-65 and Lac-CURB65. Sensitivity varied from 0,24 using CURB65[24] with a threshold of 3 to 0,93 using MEWS.[17] Specificity varied from 0,17 using Lac-CURB65[24] with a moderate threshold to 0,98 using NEWS2[23] with a threshold of 7. Moreover, MEWS[17] scored high on both sensitivity and specificity in this patient subgroup.

    Cardiothoracic disease

    There was only one study performed in cardiothoracic patients[19] assessing the prognostic accuracy of NEWS after major cardiac surgery. Regarding mortality sensitivity and specificity were 42% and 19%, and 90% and 98% respectively for thresholds 5 and 7. For cardiac arrest and admission to a higher level of care, sensitivities and specificities of 21% and 90%, and 41% and 91%% were found for a threshold of 5; and 8% and 98%, and 18% and 98% for a threshold of 7 respectively.

    Hematological disease

    The prognostic accuracy of LEWS, MEWS and PARS was evaluated for in-hospital mortality of patients undergoing allogeneic stem cell transplantation in one study.[35] The sensitivity and specificity of in-hospital mortality for threshold 7 were 100% and 95%, 67% and 92%, and 83% and 92%  for respectively LEWS, MEWS and PARS.

    Neurological disease

    Three studies investigated different EWSs in neurological patients of which two assessed the prognostic accuracy of MEWS and EWS on mortality in stroke patients,[27,28] and one admission to a higher level of care in patients with brain tumors.[32]

    Discussion

    This review provides an overview of the different EWSs used in specific patient populations and evaluates their ability to predict SAEs. We expected to find many different EWSs developed specifically for certain patient groups, however, this was not the case. We found that, despite the use of mostly general EWSs in very heterogeneous patient populations, validation of patient-specific EWSs in specific patient subgroups is limited. It is possible that these patient-specific EWSs were developed but never validated and therefore were not found. This raises the question of the degree of validity of these instruments and challenges the justness of using these for the general patient population. On the other hand, one could hypothesize about the necessity of these patient-specific EWSs rather than general EWSs; do we really need them?

    We were surprised to find so few validated patient-specific EWSs in this review. Our hypothesis was that these would perform better in comparison with general EWSs since our assumption was that certain patient groups would show unique signs of deterioration prior to developing a SAE. Possibly this hypothesis is correct and more patient-specific EWSs should be developed and validated. But another possibility is that patients of all categories deteriorate the same way, despite their diagnoses, and patient-specific EWSs are not superior to general EWSs to predict SAEs. In our opinion, future research should exist of validation of more EWSs of both patient-specific- and general nature in specific patient groups in order to make comparisons about prognostic accuracy. Another possible explanation for the lack of patient-specific EWSs in literature could be the hospital-wide approach in which most RRTs work in which they do not distinguish specific patient groups.

    Regarding the prognostic accuracy of EWSs in the pulmonary disease subgroup the most relevant finding was that the CURB65 was most accurate in predicting mortality. CURB65 is a disease specific EWS which was developed for predicting mortality in community-acquired pneumonia. The score is an acronym for the following physiological parameters; confusion of new onset, blood urea nitrogen, respiratory rate, blood pressure and age 65 or older.

    Only one study evaluated the prognostic accuracy of EWSs in specific patient populations.[37] Alhmoud and colleagues found a good performance of general EWS instruments in medical and surgical settings. However, they excluded all studies in which instruments specifically developed and validated for specified patient groups were analyzed.

    Another systematic review[38] compared the prognostic accuracy of EWSs using statistical modeling versus aggregate-weighted tools. Here, Linnen and collegues found superior prognostic performance and reduced work-load using statistical modeling based on six included studies. However, EWSs using statistical modeling appear to perform best when tailored to the targeted patient population (or are derived in-house). Therefore, hospitals considering adoption of an EWS using statistical modeling should consider that externally developed EWSs appear to experience a performance drop when applied to new patient populations. Our goal was to make recommendations for EWSs to be easily used widespread in specific patient-groups. For this reason, we solely included aggregate-weighted tools.

    Nowadays, more and more data-driven methods using hospitals own clinical data are used to gain information from admitted patients in order to determine the best treatment options. Some of these even provide real-time availability of data using dashboards to quickly reflect on treatment regiments and clinical outcomes.[39] Future developments in digitalization and artificial intelligence might enable the implementation of more accurate statistically advanced prediction models for specific patient-groups.

    There are several strengths of this review. First, a thorough search was done in all relevant electronic databases, irrespective of any filters. Second, we included EWSs specifically derived and validated for particular disease populations. We only included studies where patients admitted to hospital wards were assessed, and excluded other settings such as the emergency department, to optimize the homogeneity of the patient populations. Furthermore, excluding studies that reported insufficient data to construct 2x2 contingency tables enabled us to check the validity of the published results and exclude studies that reported spurious results.

    The most important limitation of our review is the inclusion of studies with low methodological quality which could have biased our findings. However, this was inevitable due to the paucity of available evidence. Future research should ideally include methodologically rigorous studies with low risk of bias. In accordance with McGaughey and colleagues,[5] we believe there is a need for development of a patient-informed core-outcome-set comprising clear and consistent definitions and recommendations for measurement as well as EWS and RRT interventions conforming to a standard to facilitate meaningful comparison and meta-analysis. We believe this should be done for homogeneous patient-specific populations, using robust outcomes such as in-hospital mortality, ICU admission and cardiac arrest. Standardization of EWSs, thresholds and clinical outcomes throughout published literature would help to improve the applicability of study findings. Second, since this was a review on prognostic accuracy, Cochrane’s Quality Assessment of Prognostic Accuracy Studies (QUAPAS) tool[40] would be even better suitable to assess the methodological quality of included studies. However, since this tool is not yet available for use we chose the best available alternative, QUADAS-2.

    Publication bias is another possible limitation of this review, i.e. the tendency on the parts of investigators, reviewers and editors to submit or accept manuscripts for publication based on the direction or strength of the study findings.[41] This form of bias usually affects smaller studies and those with lower methodological quality, which are more likely to be published when they show larger effects, thereby affecting the overall estimates of meta-analyses in which they are included.

    Different types of trigger systems are used to identify deteriorating ward patients. Single-parameter systems use a predefined set of physiological parameters to call a medical emergency team (e.g. respiratory rate > 36/min). Studies have found that aggregate-weighted tools are better at discriminating patients at risk of SAEs.[42,43] For that reason, we did include only the latter in this review. However, this may have led to bias since we might have missed studies reporting on; and validating single parameter systems in these specific patient populations.

    Unfortunately it was not feasible to construct another meta-analysis on other outcomes or patient subgroups, since not enough evidence was available.

    Conclusions

    Based on this systematic review, we can conclude that the number of patient subgroup specific EWSs is much smaller than we expected. However, in pulmonary patients adverse outcome is most accurately predicted with the CURB65, a subgroup specific EWS. The overall performance of general EWS in different subgroups is very variable and it remains to be investigated whether specific EWS can benefit these patients. Future developments in digitalization and artificial intelligence might enable the implementation of more accurate statistically advanced prediction models for specific patient groups.

    Disclosures

    All authors declare no conflict of interest. No funding or financial support was received. 

    The authors thank Maarten F.M. Engel for developing the search strategy and compiling the literature database for this report.

    Template data collection forms, data extracted from included studies, data used for all analyses, analytic code or any other materials used in the review can be obtained through e-mailing the authors.

    Vragen

    Referenties

    1. Alam N, Hobbelink EL, van Tienhoven AJ, van de Ven PM, Jansma EP, Nanayakkara PW. The impact of the use of the Early Warning Score (EWS) on patient outcomes: a systematic review. Resuscitation 2014;85(5):587-94.
    2. Buist M, Bernard S, Nguyen TV, Moore G, Anderson J. Association between clinically abnormal observations and subsequent in-hospital mortality: a prospective study. Resuscitation 2004;62(2):137-78.
    3. McGaughey J, Fergusson DA, Van Bogaert P, Rose L. Early warning systems and rapid response systems for the prevention of patient deterioration on acute adult hospital wards. Cochrane Database Syst Rev 2021;11:CD005529.
    4. Gerry S, Bonnici T, Birks J, et al. Early warning scores for detecting deterioration in adult hospital patients: systematic review and critical appraisal of methodology. BMJ 2020;369:m1501.
    5. Zhang K, Zhang X, Ding W, et al. National Early Warning Score Does Not Accurately Predict Mortality for Patients With Infection Outside the Intensive Care Unit: A Systematic Review and Meta-Analysis. Front Med (Lausanne) 2021;8:704358.
    6. McInnes MDF, Moher D, Thombs BD, et al. Preferred Reporting Items for a Systematic Review and Meta-analysis of Diagnostic Test Accuracy Studies: The PRISMA-DTA Statement. JAMA 2018;319(4):388-96.
    7. Whiting PF, Rutjes AW, Westwood ME, et al. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med 2011;155(8):529-36.
    8. Aliberti MJR, Covinsky KE, Garcez FB, et al. A fuller picture of COVID-19 prognosis: the added value of vulnerability measures to predict mortality in hospitalised older adults. Age Ageing 2021;50(1):32-39.
    9. Barlow G, Nathwani D, Davey P. The CURB65 pneumonia severity score outperforms generic sepsis and early warning scores in predicting mortality in community-acquired pneumonia. Thorax 2007;62(3):253-62.
    10. Barnett WR, Radhakrishnan M, Macko J, Hinch BT, Altorok N, Assaly R. Initial MEWS score to predict ICU admission or transfer of hospitalized patients with COVID-19: A retrospective study. J Infect 2021;82(2):282-327.
    11. Bradley P, Frost F, Tharmaratnam K, Wootton DG, Research NWCOfR. Utility of established prognostic scores in COVID-19 hospital admissions: multicentre prospective evaluation of CURB-65, NEWS2 and qSOFA. BMJ Open Respir Res 2020;7(1):1-9.
    12. Chiu YD, Villar SS, Brand JW, et al. Logistic early warning scores to predict death, cardiac arrest or unplanned intensive care unit re-admission after cardiac surgery. Anaesthesia 2020;75(2):162-70.
    13. De Socio GV, Gidari A, Sicari F, Palumbo M, Francisci D. National Early Warning Score 2 (NEWS2) better predicts critical Coronavirus Disease 2019 (COVID-19) illness than COVID-GRAM, a multi-centre study. Infection 2021;49(5):1033-38.
    14. Echevarria C, Steer J, Bourke SC. Comparison of early warning scores in patients with COPD exacerbation: DECAF and NEWS score. Thorax 2019;74(10):941-46.
    15. Fan G, Tu C, Zhou F, et al. Comparison of severity scores for COVID-19 patients with pneumonia: a retrospective study. Eur Respir J 2020;56(3)
    16. Gidari A, De Socio GV, Sabbatini S, Francisci D. Predictive value of National Early Warning Score 2 (NEWS2) for intensive care unit admission in patients with SARS-CoV-2 infection. Infect Dis (Lond) 2020;52(10):698-704.
    17. Grudzinska FS, Aldridge K, Hughes S, et al. Early identification of severe community-acquired pneumonia: a retrospective observational study. BMJ Open Respir Res 2019;6(1):e000438.
    18. Hodgson LE, Congleton J, Venn R, Forni LG, Roderick PJ. NEWS 2 – too little evidence to implement? Clin Med (Lond) 2018;18(5):371-73.
    19. Jo S, Jeong T, Lee JB, Jin Y, Yoon J, Park B. Validation of modified early warning score using serum lactate level in community-acquired pneumonia patients. The National Early Warning Score-Lactate score. Am J Emerg Med 2016;34(3):536-41.
    20. Knoery C, Barlas RS, Vart P, et al. Modified early warning score and risk of mortality after acute stroke. Clin Neurol Neurosurg 2021;202:106547.
    21. Liljehult J, Christensen T. Early warning score predicts acute mortality in stroke patients. Acta Neurol Scand 2016;133(4):261-7.
    22. Liu FY, Sun XL, Zhang Y, et al. Evaluation of the Risk Prediction Tools for Patients With Coronavirus Disease 2019 in Wuhan, China: A Single-Centered, Retrospective, Observational Study. Crit Care Med 2020;48(11):e1004-e11.
    23. Murphy SA, Haja Mydin H, Fatah S, Antunes G. Predicting end-of-life in patients with an exacerbation of COPD by routine clinical assessment. Respir Med 2010;104(11):1668-74.
    24. Myrstad M, Ihle-Hansen H, Tveita AA, et al. National Early Warning Score 2 (NEWS2) on admission predicts severe disease and in-hospital mortality from Covid-19 – a prospective cohort study. Scand J Trauma Resusc Emerg Med 2020;28(1):66.
    25. Peng L, Luo Z, Liang L, et al. Comparison of the Performance of 24 Early Warning Scores with the Updated National Early Warning Score (NEWS2) for Predicting Unplanned Intensive Care Unit (ICU) Admission in Postoperative Brain Tumor Patients: A Retrospective Study at a Single Center. Med Sci Monit 2021;27:e929168.
    26. Pokeerbux MR, Yelnik CM, Faure E, et al. National early warning score to predict intensive care unit transfer and mortality in COVID-19 in a French cohort. Int J Clin Pract 2021;75(6):e14121.
    27. Richardson D, Faisal M, Fiori M, Beatson K, Mohammed M. Use of the first National Early Warning Score recorded within 24 hours of admission to estimate the risk of in-hospital mortality in unplanned COVID-19 patients: a retrospective cohort study. BMJ Open 2021;11(2):e043721.
    28. von Lilienfeld-Toal M, Midgley K, Lieberbach S, et al. Observation-based early warning scores to detect impending critical illness predict in-hospital and overall survival in patients undergoing allogeneic stem cell transplantation. Biol Blood Marrow Transplant 2007;13(5):568-76.
    29. Wang L, Lv Q, Zhang X, et al. The utility of MEWS for predicting the mortality in the elderly adults with COVID-19: a retrospective cohort study with comparison to other predictive clinical scores. PeerJ 2020;8:e10018.
    30. Alhmoud B BT, Patel R, Melley D, Williams B, Banerjee A. Performance of universal early warning scores in different patient subgroups and clinical settings: a systematic review. BMJ Open 2021;11(4):1-16.
    31. Dickersin K. The existence of publication bias and risk factors for its occurrence. JAMA 1990;263(10):1385-94.