May 2022


July 2022

S.A. Wortel (1,2), F. Bakhshi-Raiez (1,2), W.G. Tjon Sjoe Sjoe (1,2), E.P.A. van der Zwan (1,2), N.F. de Keizer (1,2), D.A. Dongelmans (2,3) 1 Department of Medical Informatics, 2 Dutch National Intensive Care Evaluation (NICE), 3 Department of Intensive Care Medicine, Amsterdam UMC, University of Amsterdam, Amsterdam, the Netherlands


S.A. Wortel -
Original Article

The role of the Dutch National Intensive Care Evaluation registry during the COVID-19 pandemic


In this paper we describe how the data infrastructure of an existing national quality registry was adapted to meet public health, research and capacity needs for information on COVID-19 hospitalisations in the Netherlands.

From the start of the COVID-19 pandemic, there was a need for knowledge about the disease and its spread. Data were necessary on, for example, the number of COVID-19 hospitalisations, length of stay of COVID-19 patients and their outcomes. For instance, for the Dutch Centre for Disease Control (i.e., RIVM) and the government to monitor the pandemic and to prevent ICUs from overflowing, to help physicians make treatment and triage decisions and to inform the public.

Reliable data sources were essential. It seems imperative that if a national health crisis arises, data already captured and available in electronic health records (EHRs) can be extracted. However, this requires a FAIR data infrastructure in which the data are Findable, Accessible, and most importantly Interoperable to be Re-used.[1] In current practice, it was not possible to gather data from the EHRs at a national level since different EHR software, data models and coding systems are being used and much of the information is embedded in free text notes.

In the Netherlands, the National Intensive Care Evaluation (NICE) registry collects data on intensive care patients (demographics, comorbidities, reason for ICU admission, acute physiological disturbance) and their outcomes to monitor and improve the quality of care.[2] This quality registry receives monthly batches of data from intensive care units (ICUs) extracted from local EHRs. These are usually manually validated and supplemented by intensivists, nurses or data managers before being uploaded to the registry database. As the ICU capacity was of major concern during this pandemic and the need for (near) real-time information on the number of patient admissions and their outcomes was high, the infrastructure of the NICE registry was adapted to capture the data on COVID-19 admissions in a fast and accessible manner. This meant that intensivists were asked to enter a limited dataset on COVID-19 patients on a daily basis. In this paper, we describe how the set-up of an existing national quality registry was used to meet the need for information on COVID-19 hospitalisations in the Netherlands.

The NICE team, consisting of software engineers, medical informaticians, data scientists and intensivists used an agile approach to accommodate all stakeholder requirements. Stakeholders were data collectors, i.e., intensivists, nurses and support staff who entered and validated data, and the data endusers. End-users included intensivists who needed knowledge on COVID-19, its prognoses and risk factors; ICU managers who had to plan and scale up the number of available beds; national and regional coordination centres for patient distribution that harmonised fair share and transport of patients; and finally, the RIVM for pandemic control and the government. The main requirements included the following:

  • Users should be able to enter and update data on COVID-19 admission easily, with minimal effort and (near) real-time.
  • Users should be able to register ICU admissions as well as regular hospital admissions.
  • A unique patient identifier such as the citizen service number should be used to follow patients during consecutive admissions in different hospitals.
  • The safety measures to ensure the privacy of patients and end-users should meet the high standards as described by the General Data Protection Regulation (GDPR) and ISO27001/NEN7510.
  • COVID-19 data should be (legally and real-time) accessible to the RIVM to model the course of the pandemic and make predictions.
  • COVID-19 data and the patients’ clinical characteristics from the regular NICE registry should be available to answer research questions and provide insights into the disease outcomes and risk factors.
  • Aggregated and current overviews on the number of COVID-19 cases, patient characteristics and outcomes should be available to inform the public, media and Dutch regional care networks.

COVID-19 module
Based on the requirements, the NICE developed an online data entry application for COVID-19 admissions. The COVID-19 application was implemented as part of the NICE Online infrastructure, i.e., the secured part of the NICE website. Each hospitalised patient, either admitted to the ward or to the ICU, with COVID-19 or suspicion of COVID-19, could be entered in the application by hospital workers with access rights to the NICE Online infrastructure. The users could either add a new ICU admission or hospital admission and fill in a form for each admission. The forms consisted of a limited set of variables. These were: the citizen service number, date of birth, hospital admission date, ICU admission date (if applicable), whether the patient was transferred from another ICU (yes or no), COVID-19 status (lab-confirmed, CT confirmed, suspected or negative), ICU discharge date (if applicable), hospital discharge date, discharge destination, and death in hospital (yes or no).

The Ministry of Health, Welfare and Sport (VWS) gave permission for the use of the citizen service number for registering the COVID-19 patients and communicated this to all the hospitals. Collecting a unique patient identification variable was necessary in order to be able to follow COVID-19 patients transferred between hospitals. Otherwise COVID patients could be counted more than once, which would in turn affect the reproduction number and the modelling of the COVID-19 pandemic.

At a later stage, it was also possible for NICE Online users to upload the data in a single .csv file. The NICE website underwent and passed an extensive security check, commissioned by the Ministry of VWS. As the registry already worked with patient data, security safeguards were in place and privacy was well organised. Yet additional measures were recommended and adopted accordingly, for instance setting up geo-blocking to protect the website from cyber-attacks from countries outside the EU.

COVID-19 data usage
The COVID-19 data from the NICE registry were the main data source for the Dutch RIVM and government to model the pandemic, make predictions and analyse the effectiveness of the COVID-19 vaccinations. Results were used in the briefings of the House of Representatives to make decisions on COVID-19 measures and to inform the public. This way the Outbreak Management Team, an advisory body to the Dutch government, had timely and accurate information on patient flow and outcomes in terms of mortality and length of stay. The admissions with a positive COVID-19 status or suspicion of COVID-19 were used to create overviews on the NICE website on the number of admissions per day, length of stay and hospital mortality. These overviews were updated in real-time and were easily available to the general public and media. For the ICUs and the Dutch regional care networks, additional overviews were made available in the secured part of the NICE website. These overviews displayed the number of COVID-19 admissions per hospital and per Dutch regional care network. Furthermore, the COVID-19 status of ICU patients from the COVID-19 application admitted to the ICU was linked to the clinical characteristics of the ICU patients from the NICE registry. These data were used to create regularly updated reports on the clinical characteristics and outcomes of COVID-19 admissions and for different research purposes, for example to describe and compare the clinical characteristics and outcomes between COVID-19 patients and patients with other community-acquired viral pneumonias,[3] to describe differences in the planning of nursing staff, and the impact of those differences on nursing workload in the COVID-19 period versus a recent non-COVID-19 period,[4] to compare patient characteristics and outcomes between the first wave and subsequent waves of the pandemic,[5] and to assess the association between body mass index and mortality of COVID-19 patients.[6] Lastly, COVID-19 data were also linked to the national insurance claims database to determine the long-term outcomes of COVID-19 patients after hospitalisation. A compact overview of the COVID-19 data collection, storage and usage is shown in figure 1. 

Due to the efforts of all the Dutch ICUs to collect data, the NICE quality registry the NICE quality registry was pivotal during the COVID-19 pandemic in the Netherlands. The registry collected data on all COVID-19 admissions in the Netherlands and these data were used to aid the RIVM to monitor and control the pandemic, the government in making decisions on COVID-19 measures, to inform physicians, hospitals and the public, and to conduct scientific research. Intensivists in the Netherlands played an important role in data collection and critically reviewing the data representation on the NICE website.

Figure 1 V30-5

Because the NICE registry already had national coverage, the RIVM and the Ministry of VWS asked the registry coordinator to facilitate the collection of data on COVID-19 patient admissions. Initially, only the COVID-19 patients admitted to the ICU were included. However, when it became apparent how valuable these data were and more information was needed on COVID-19 patients on the general ward (e.g. mortality, length of stay), the RIVM and Ministry of VWS asked the registry coordinator to enable data to be collected on COVID-19 admissions from the general wards as well. The Ministry of VWS communicated the need for these data to the hospitals and asked them to also register the COVID-19 patients admitted to general wards. In some hospitals, the same people who registered the ICU patients also registered the general ward patients and in other hospitals others were designated to do so. The coordinator of the NICE registry facilitated this by creating the necessary accounts for the persons registering the admissions, answering questions via emails, sending reminders out, offering the option to upload admission records in batches (instead of manually entering each patient) and keeping the number of variables which needed to be collected low.

The existing NICE data infrastructure did have some limitations. In an effort to keep the COVID-19 registration process as simple as possible, the NICE started with a limited manual data entry process. Despite consistency checks at data entry point, manual data collection can be prone to errors. Furthermore, it was possible to alter the COVID-19 status from suspected to positive or negative but due to the database design, it was not possible to specify when the patient tested positive or negative. So, patients admitted for another reason than COVID-19 who were infected during hospital admission were considered COVID-19 admissions and might have an artificially longer length of stay for COVID-19. In retrospect, the better way to register the COVID-19 status would be to also register the dates to which the status applied. Furthermore, setting up geo-blocking resulted in data not being available for our foreign partners and end-users.

We have seen that the current approach used for registering COVID-19 patients was adequate; however, this approach is not sustainable. Ideally, data should be extracted directly from the source (EHRs) to prevent double registration and reduce the overall registration burden for healthcare professionals. In order to extract data from the source, we need to ensure that the source data are FAIR.[1] Further research is needed to develop and evaluate what this FAIR EHR infrastructure should look like to be directly available if another pandemic or disease outbreak arises. Although several projects have been launched in the Netherlands to improve registration at the source and work towards a FAIR data infrastructure, this process is not going smoothly. Collaboration between EHR suppliers, government and national initiatives such as the NICE registry must ensure that these kinds of projects accelerate so that we can act more efficiently in the future.

The infrastructure of a national quality registry has shown to be sufficient to aid the RIVM to monitor and control the pandemic, the government in making decisions on COVID-19 measures, to inform hospitals and the public and to conduct research. Intensivists, ICU nurses and data managers played an important role in facilitating this. Adjustments were needed to facilitate the information needs of the various stakeholders. So, for future pandemics or disease outbreaks, it is important to improve the current national data infrastructure by, for example, registering data according to international information and terminology standards at the source (EHR) and applying FAIR principles to improve the reuse and exchange of data.

The authors would like to thank the board members of the NICE foundation, Dylan de Lange, Sesmu Arbous, Roy van den Berg, Rob Bosman, Bas van Bussel, Michiel Erkamp, Mart de Graaff, Marga Hoogendoorn, David Moolenaar, Jan Jaap Spijkstra, and Ruud de Waal for all the fruitful discussions enabling the development of the data infrastructure. The authors would like to acknowledge the efforts of the intensivists, nurses and all other hospital workers in recording the data on COVID-19 patients. Without their efforts and contributions it would not have been possible to provide all the stakeholders (RIVM, researchers and general public) with the necessary information during the COVID-19 pandemic.

D.A. Dongelmans and N.F. de Keizer are members of the board of the NICE foundation. The NICE foundation pays the department of Medical Informatics, Amsterdam UMC, for processing data of all Dutch ICUs into audit and feedback information. F. Bakhshi-Raiez, N.F. de Keizer, W.G. Tjon Sjoe Sjoe, S.A. Wortel, and E.P.A. van der Zwan, are employees of the department of medical informatics and work on the NICE project.


  1. Wilkinson MD, Dumontier M, Aalbersberg IJ, et al. The FAIR guiding principles for scientific data management and stewardship. Sci Data. 2016;3:1-9.
  2. Van de Klundert N, Holman R, Dongelmans DA, de Keizer NF. Data resource profile: the Dutch National Intensive Care Evaluation (NICE) registry of admissions to adult intensive care units. Int J Epidemiol. 2015;44:1850.
  3. Hoogendoorn ME, Brinkman S, Bosman RJ, Haringman J, de Keizer NF, Spijkstra JJ. The impact of COVID-19 on nursing workload and planning of nursing staff on the Intensive Care; a prospective descriptive multicenter study. Int J Nurs Stud. 2021;121:104005.
  4. Dongelmans DA, Termorshuizen F, Brinkman S, et al. Characteristics and outcome of COVID-19 patients admitted to the ICU: a nationwide cohort study on the comparison between the first and the consecutive upsurges of the second wave of the COVID-19 pandemic in the Netherlands. Ann Intensive Care. 2022;12:5.
  5. Kooistra EJ, Brinkman S, van der Voort PH, et al. Body mass index and mortality in coronavirus disease 2019 and other diseases: a cohort study in 35,506 ICU patients. Crit Care Med.