ABSTRACT

CORRESPONDING AUTHOR:
Dr. Raghavendra Bhupal Y. J,
S/o Yashvanth Bhupal,
Door No. 35, Amar Krupa,
3rd Cross, Gandhi Nagar,
Bellary-583103.
Email: raghavendrabhupal@gmail.com

ABSTRACT: BACKGROUND AND OBJECTIVES: Various prognostic scores are used to evaluate the outcome in the PICU. Among these scores PRISM (Pediatric Risk of Mortality) and PIM (Pediatric Index of Mortality), are the most recent versions. The objectives of this study were to compare performance of PIM II and PRISM III scores at a general pediatric intensive care unit, relation between observed mortality and survival and predicted mortality and survival. METHODS: Present prospective cohort study was conducted at Pediatric Intensive Care Unit, KLES, Dr. Prabhakar Kore Hospital and Medical Research Centre, Belgaum during period of July-2009 to October-2009. Ethical clearance was obtained from Ethical and Research Committee of Jawaharlal Nehru Medical College, Belgaum. A total of 316 patients were enrolled out of 350 admissions during study period. Within first hour of admission PIM II was assessed and at 24 hours PRISM III score was assessed. Patients were followed up for outcome measured in form of survivors and non survivors. RESULTS: A total 316 patients were enrolled, 28 patients were excluded as their outcome was not known. Among the 288 patients 261 were survivors and 27 were non-survivors. Majority of survivors and non survivors were males but there was no statistical significant influence on outcome. The overall number of estimated mortality was 13.2 and 16.5 with PIM II and PRISM III respectively compared to observed mortality of 27. PIM II (x²=16.34; p<0.001) and PRISM III (x²=6.78; p=0.009) had poor calibration. PRISM III showed the better discrimination (ROC=0.892) followed by PIM II 0.871 (0.827 to 0.907). PIM II and PRISM III (0.850 to 0.925) revealed positive and significant correlation, with spearman's rank correlation (r= 0.318; p<0.0001). INTERPRETATION AND CONCLUSION: In this study PIM II and PRISM III under predicted mortality and also had poor calibration with good discrimination. Overall both scores exhibited good capacity to discriminate between survivors and non survivors and can be used as a tool with comparable performance for prognostic evaluation.

KEYWORDS: Pediatric Risk of Mortality (PRISM) III; Pediatric Index of Mortality (PIM) II.

INTRODUCTION: The advent of evidence based medicine in the past decade has made the Cartesian premise that ‘if something can't be quantified, its existence should be questioned', even more pertinent in the practice of modern medicine. Intensive care has developed over the past 30 years with little rigorous scientific evidence about what is, or is not, clinically effective.

Pediatric intensive care units (PICU) aim at promoting quality care with the object of achieving the best results and better prognosis for critically ill children.¹ These units are points of major technology transfer and constitute one of the main consumers of hospital critical care segment.

However, when patients with varying prognosis and degrees of clinical severity are being treated the final results of employing the resources available at such units is often uncertain.²  Scoring systems have been developed in response to an increasing emphasis on the evaluation and monitoring of health services.²

The evaluation of severity of illness in the critically ill patient is made through the use of severity scores and prognostic models.³ Severity scores are instruments that aim at stratifying patients based on the severity of illness, assigning to each patient an increasing score as their severity of illness increases, predict a certain outcome (usually the vital status at hospital discharge) based on a given set of prognostic variables and a certain modeling equation.⁴

There are five major purposes of severity-of-illness scoring systems. First, scoring systems have been used in randomized controlled trials (RCT) and other clinical investigations.⁵The second purpose of severity-of-illness scoring systems is to quantify severity of illness for hospital and health care system administrative decisions such as resource allocation. The third purpose of these scoring systems is to assess ICU performance and compare the quality of care of different ICUs and within the same ICU over time.

Severity-of illness scoring systems could be used to assess the impact on patient outcomes of planned changes in the ICU, such as changes in bed number, staffing ratios, and medical coverage.⁶ The fourth purpose of these scoring systems is to assess the prognosis of individual patients in order to assist families and caregivers in making decisions about ICU care. Finally, scoring systems are now being used to evaluate suitability of patients for novel therapy (e.g., the use of the APACHE II assessment for prescription of recombinant human activated protein C).

Numerous scoring systems have been proposed in an effort to increase the prognostic accuracy and predicting outcome. The APACHE system developed by William Knaus et al in 1981, demonstrated the ability to evaluate, in an accurate and reproducible form, the severity of disease in this population.⁷ Two years later, Jean-Roger Le Gall and co-workers published a simplified version of this model, the Simplified Acute Physiology Score (SAPS).⁸

Another simplification of the original APACHE system, the APACHE II, was published in 1985 by the same authors of the original model.⁹ This system introduced the possibility to predict mortality, needing for this purpose the selection of a major reason for intensive care unit (ICU) admission from a list comprising 50 operative and non-operative diagnoses. Additional contributions for the prediction of prognosis comprise the Mortality Probability Models (MPM),¹⁰ developed by Stanley Lemeshow using logistic regression techniques.

All existing models aim to predict an outcome (vital status at hospital discharge) based on a given set of variables. The outcome of a patient with a certain clinical condition (defined by the registered variables), is estimated by treating in a hypothetical reference ICU. Initially scoring systems were developed for trauma patients and were either specific anatomical methods (abbreviated injury scale 1969, burn score 1971, injury severity score 1974) or specific physiological methods (trauma index 1971, Glasgow coma scale 1974, trauma score 1981 and sepsis score 1983).¹¹

Pediatric ICUs compare components that are related with disease severity and the resources available with the outcomes of specific types of patients. Mortality and length of hospital stay are examples of the most used outcomes. In order to measure severity risk of mortality, scores are employed that establish a numerical scale and in this way they compare estimated mortality in percent with the observed mortality.

Known as prognostic scores, these can be used to evaluate the quality of medical care and to optimize the employment of resources, aiming at improving the cost-benefit relationship. Since they compare mortality adjusted by disease severity, these scores can also be used for comparisons between clinical trials and for planning technological resources.

The principal scores that have been developed for the pediatric population are the PRISM (Pediatric Risk of Mortality)¹² and PIM (Pediatric Index of Mortality), with their most recent versions being PRISM III¹³ and PIM-I1.¹⁴ These scores were developed by identifying variables relevant to mortality risk and scoring them after a multivariate statistical analysis by logistic regression.

The performance of the PRISM and PIM systems have been compared a number of times by the authors who developed the scores themselves, but have rarely been compared independently. To date, those studies that have been performed independently have not used heterogenic groups of patients from PICUs, but have investigated certain specific disease categories, new versions of the methods^15,16 or homogenous groups of high mortality patients.

In this independent study, our objective is to compare the performance of the PRISM III and the PIM II at a general PICU. In this study, we are investigating the relationship between observed mortality and survival and the predicted mortality and survival rates as estimated by the two scores.

OBJECTIVES: To compare the performance of the PIM II (Pediatric Index of Mortality) and PRISM III (Pediatric Risk of Mortality) scores at a general pediatric intensive care unit, the relation between observed mortality and survival and predicted mortality and survival.

METHODOLOGY: The present prospective cohort study was carried out at Pediatric Intensive Care Unit of KLES Dr. Prabhakar Kore Hospital and Medical Research Centre, Belgaum attached to Jawaharlal Nehru Medical College Belgaum during the period of July-2009 to November-2009 on 288 patients. The sample size was calculated based on patient data at hospital for the last three consecutive years.

Pediatric Intensive Care Unit of KLES Dr Prabhakar Kore Hospital and Medical Research Centre, Belgaum serves as a referral center for four districts of Karnataka (Belgaum, Bagalkot, Raichur and Dharwad), along with bordering areas of Goa and Maharashtra. It is a 14 bedded PICU which is well equipped with all the monitors and mechanical ventilators. It is also backed by 24 hours in hospital laboratory facilities, portable X ray, bedside ECHO and ultrasound facilities and ABG machine.

All cases admitted to the PICU as per the PICU admission guidelines of Indian academy of paediatrics¹⁷ were included in the study. Patients expired within first eight hours and patients discharged within 24 hours after admission were excluded from the study. The study was approved by the Ethical and Research Committee of Jawaharlal Nehru Medical College, Belgaum. All the patients fulfilling selection criteria were included in the study after obtaining informed consent.

At admission detailed history was taken and systemic examination was done by the on duty resident and recorded in a predesigned and pretested proforma (Annexure II). Within the first hour of admission PIM II was assessed and all the precautions required for the calculation of the PIM II score was followed. Further at 24 hours admission to the PICU PRISM III score was assessed. The study variables included age, sex, socio-economic status, length of hospital stay, number of organ failure on admission, primary affected system/illness and PIM II and PRISM III variables.

The calculated PIM II scores were converted to expected mortality rate by using software developed by SFAR. PRISM III scores which were calculated converted to estimated mortality rate. Patients were followed up for the outcome. The outcome was measured in the form of survivors and non survivors.

The association between the study variables was analyzed using Chi-square test, simple descriptive analysis like, mean, median, and standard deviation. Comparison of the general similarity between observed mortality and that expected mortality by standardized mortality ratio (SMR) was calculated. A SMR ratio of SMR <1 was considered as over estimation, equal to 1 as accurate estimation and >1 as underestimation of mortality.

For the aptness of the two models, the HOSMER-LEMESHOW goodness of fit test¹⁸ was employed to test the agreement between observed and expected mortality (calibration) and a p value of >0.05 it indicates good fit The capacity for discrimination between survivors and non survivors was made using the typical area under a receiver operating characteristic curve (ROC curve).¹⁹

An AUC of > 0.9 was considered excellent discrimination, between 0.80 to 0.89 good and 0.70-0.79 as fair. Quantitative correlation between the results of the scores was analyzed using the spearman rank test.

RESULTS: A total of 316 patients were enrolled out of 350 admissions during the study period, after applying the exclusion criteria (6 expired within 24 hrs after admission, rest were discharged within 24 hours after admission). A total of 288 patients were analyzed as 12 patients were discharged against medical advice and 16 discharged on request, they were excluded from the study as their outcome was not known. Majorities (279) were medical cases and rests (9) were surgical cases. Among the 288 patients analyzed, 261 were survivors and 27 were non-survivors, with the overall observed mortality rate of 9.3%.

Majority of the survivors and non survivors were male (53.7% and 55.6% respectively) and there was no statistical significance between the groups (p=0.849). Children between the age group of 3 to 8 years and 8 to 18 years(33.3% and 36.4% respectively) formed the majority among the survivors and the non survivors with no statistical significance (p=0.353) (Graph 1).

Majority of the survivors and non survivors (65.5% vs 44.4%) belonged to class I Socio economic status. 70.8% were directly admitted to PICU, 18% were transferred from the ward, 9% were referred from private practitioners and 2.06% were transferred from OT and had no significant influence on the outcome. (p=0.3159) (Graph 2).

When the primary organ affected/illness were correlated with the outcome, as shown in the graph 5, dengue cases constituted the majority (55.2% vs 29.6%) followed by the CNS cases (12.6% vs 25.9%) among survivors and non survivors and there were significant mortalities among the dengue cases (p=0.032) (Graph 3). Systolic blood pressure, Fi0₂ % / Pa02 ratio, base excess and need for mechanical ventilation < 1 hour after admission were statistically significant and affected the outcome (Table 1).

Majority of survivors (72%) were between the range of 0 to 5 and majority of non survivors (40.7%) were between range of >15 to 30 (Table 2). It was seen that systolic blood pressure, heart rate, body temperature, pupil reflex, mental status, serum potassium, Blood Urea Nitrogen, and prolonged activated plasma thromboplastin time were statistically significant and affected the outcome (Table 3).

PIM II underestimated mortalities compared to the observed mortalities which was statistically significant (p<0.001). PIM II underestimated the mortality so had poor calibration for various levels of probability of death (Table 4).

PRISM III underestimated mortalities compared to the observed mortalities which was statistically significant (p=0.009). PRISM III underestimated the mortality so had poor calibration for various level of probability of death (Table 5).

As the area under the curve was 0.871 (CI — 0.827 to 0.907) for PIM II and 0.892 (CI - 0.850 to 0.925) for PRISM III, both discriminated probability of death and survival well. But when pair wise comparison of ROC curve was done p value was 0.696, there was no statistical difference between these scores for discrimination (Figure 1).

DISCUSSION: The principal scores that have been developed for the pediatric population are the PRISM (Pediatric Risk of Mortality) and PIM (Pediatric Index of Mortality), with their most recent versions being PRISM III and PIM-II. The performance of the PRISM III and PIM II systems have been compared a number of times by the authors who developed the scores themselves, but have rarely been compared independently.

To date, those studies that have been performed independently have not used heterogenic groups of patients from PICUs, but have investigated certain specific disease categories, new versions of the methods or homogenous groups of high mortality patients. The PIM II and PRISM III scores were first validated in developed countries. PIM II was validated in Australia and New Zealand whereas PRISM III was validated in United States of America.

In the present study, demographic profile like age and gender did not show any significant influence on the outcome. Similar observations were noted in other Indian studies.^20,21 Majority of the patients were admitted directly to the paediatric intensive care unit without any significant influence on the outcome which was similar to the observation shown in studies conducted in developed countries.^22,23

In the present study, the observed mortality rate was 9.3%. PIM II and PRISM III underestimated the mortality with a predictive mortality rate of 4.6% (SMR 1.32) and 5.6% (SMR 1.64) respectively. Similar observations of under prediction of mortality by the scores were shown in studies from both developed and developing countries.^{20,24,25,26,27}

In an Indian study where the characteristics of the study population and ICU setting were similar to the present study PIM II and PRISM III under predicted the mortality with an predictive mortality rate of PRISM 29% (SMR 1.2), PIM II 22%(SMR 1.57) and PIM 22 % (SMR 1.57) respectively against the observed mortality rate of 35%.²⁰

Another study from developing country, showed the under prediction of mortality by using PRISM and PIM II but with good calibration.²⁴Leteurteet al²⁴ also demonstrated the under prediction of mortality by the scores. Contrary to these observations, studies from developed countries have shown accurate prediction of mortality using these scores,^{22,23,28,29,30,31} a Korean study²³ showed the predictive mortality rate of 13.9% (SMR 1) and with 14.1% (SMR 0.99) using PIM II and PRISM III respectively versus the observed mortality rate of 14%.²³ Martha et al³¹ also demonstrated similar observation.

The various parameters of PIM II and PRISM III affecting the outcome in the present study were systolic BP, Fi02%/Pa02 ratio, base excess, requirement of mechanical ventilation within first hour and no pupillary reaction to light and systolic BP, Heart Rate, Body temperature, Pupil reflex, mental status, serum potassium, Blood urea nitrogen and derangement of activated Partial Thromboplastic Time respectively and were similar to an Korean study.²³

PIM II and PRISM III overall under predicted mortality but had satisfactory prediction and discrimination under the age of two years, for high risk and low risk diagnostic groups, in patients with dengue fever, in non-ventilated patients. PIM II and PRISM III for no organ failure at admission over estimated (SMR=0.66) and accurately estimated (SMR=1.15) mortalities respectively. Isolated satisfactory prediction for ventilated patients was seen only in PRISM III.

In the Indian study²⁰ comparing the performance of the scores, similar observation of accurately prediction of mortality by the score in a less than one year of age was noted, whereas under prediction of mortality in infants was seen in a study by Ozer et al.²⁵ The performance of the scoring systems were compared to validate in terms of discrimination and calibration in the present study, both the scores had poor calibration.

The discriminatory power was evaluated using ROC, with PRISM III (AUC being 0.892) having better discriminatory power than PIM II (AUC being 0.871) with the positive and significant correlation using spearman's rank correlation (r=0.310; p<0.0001 value). Similar observations of positive correlation were seen in studies done by Qureshi et al²⁶ (r=0.74; p=0.001) and Martha et a1³¹ (r=0.65; p<0.001).

Various studies have also shown similar good discrimination and poor calibration both from developed and developing countries.^{20,25,26,27,29} The Indian study with the similar characteristics of study population and ICU setting as the present study, evaluating the performance of the scores, under predicted mortality with SMR for PRISM, PIM and PIM II been 1.2, 1.57 and 1.57 and documented AUC of 0.80, 0.82 and 0.81 respectively.²⁰

Similar observation of poor calibration with good discrimination of PIM II in a single PICU of Argentina was demonstrated by Eulmesekianet al²⁷ and of PRISM III by Leteurtre et al.²⁴ Wells et al³² demonstrated poor performance of PRISM score in terms of poor calibration and poor discrimination. Most of the studies evaluating the performance of these scores have concluded that all scoring systems exhibited suboptimal calibration.^{20,25,26,27,29}

Contrary to our observation, good calibration and good discrimination of the scoring systems with accurate prediction of mortality was observed in most of the studies from developed countries.^{22,23,28,29,30,31}Gemke et al²⁸ demonstrated that PRISM III at 12 hours (p=0.21, AUC=0.78) and PRISM III at 24 hours (p=0.214, AUC=0.78) and PIM (p=0.77, AUC=0.74) had good calibration and good discrimination.²⁸ Similar findings were noted by Korean²³ and Spanish²⁷ studies. Studies which were conducted in developing countries also showed that PRISM III had good calibration and good discrimination (p=0.16, AUC=0.89).³³

It is desirable that scoring systems should be devised in such a way that they work in both developed and developing countries. This may involve or adapting existing scoring systems in a way that may not affect their current functioning in the developed world but may appropriately modify their use within the developing world. The modification could take into account difference in the patient profile, difference in PICU practice and importantly, difference in resource allocation.

CONCLUSION: PIM II and PRISM III under predicted the mortality in the pediatric intensive care unit and also had poor calibration with good discrimination. Likely reasons for poor calibration and underestimation in this present study could be difference in patient’s profile, greater load of severity of illness being managed with lesser resources both physical and human; and differences in the quality of care.

Although we had poor calibration, when the results were taken as whole both the scores exhibited good capacity to discriminate between survivors and non survivors and can be used as a tool with comparable performance for prognostic evaluation of pediatric patients admitted in a PICU setup.

GRAPH 1: AGE

GRAPH 2: Pre ICU stay

GRAPH 3: PRIMARY ORGANS AFFECTED / ILLNESSES WITH OUTCOME

FIG. 1: RECEIVER OPERATIVE CHARACTERSTICS (ROC) CURVE

BIBLIOGRAPHY:
1.    Skrobik Y, Kavanagh B. Scoring systems for the critically ill: use, misuse And abuse. Can J. Anaesth 2006: 539: 432-6.
2.    Gunning K, Rowan K. ABC of intensive care outcome data and scoring Systems. BMJ 1999; 319: 241-4.
3.    Hall JB, Schmidt GA, Wood LDH. Principles of critical care 3rd edition; McGraw-Hill publications: 63-78.
4.    Gullo A, Lumb PD. Intensive and Critical Care Medicine-reflections, Recommendations and perspectives (WFSICCM); Springer publications.
5.    Le Call JR, Lemeshow S, Leleu G, et al. Customized probability models for early severe sepsis in adult intensive care patients. Intensive Care Unit Scoring Group JAMA 1995; 273: 644-50.
6.    Schwartz S, Cullen DJ. How many intensive care beds does your hospital need? Crit Care Med 1974; 9: 625-9.
7.    Chalfin DB, Cohen IL, Lambrinos J. The economics and cost-Effectiveness of critical care medicine. Intensive Care Med 1995; 21:952-61.
8.    Parviainen, I, Herranen A, Holm A, et al. Results and costs of intensive care in a tertiary university hospital from 1996-2000. Acta Anaesthesiol Scand 2004; 48: 55-60.
9.    Spillman BC, Lubitz J. The effect of longevity on spending for acute and Long-term care. N. Engl J Med 2000; 342:1409-15.
10.    Lubitz J. Greenberg LG, Gorina Y, et al. Three decades of health care use by the elderly, 1965-1998. Health Aff 2001; 20: 19-32.
11.    Baker SP, O’Neill B: The injury severity score: An update. J Trauma 1976; 16: 882-5.
12.    Norris C, Jacobs P, Rapport J, et al. ICU and non-ICU cost per day. Can J Anaest 1995; 42: 192-6.
13.    Subbe C. Recognition and assessment of critical illness.  Anaesthesia and Intensive care medicine 2007:8: 21-3.
14.    Subbe CP, Kruger M, Rutherford P L. Gemmel, Validation of a modified Early Warning Score in medical admissions.  QJM 2001; 94:521-526.
15.    Ruttimann UE. Statistical approaches to development and validation of Predictive instruments.Crit Care Clin 1994; 10: 19-35.
16.    Lemeshow S, Teres D, Avrunin JS, et al. Refining intensive care unit Outcome perdition by using changing probabilities of mortality. Crit Care Med 1988; 16: 470-7.
17.    Khilnani P. Indian Society of Critical Care Medicine (Pediatric Section) and Indian Academy of Pediatrics. Indian Pediatrics 2002; 39: 43-50.
18.    Hosmer DW, Lemeshow S. Applied logistic regression. New York: John Wiley; 1989.
19.    Zweig MH, Campbell G. ROC plots a fundamental evaluation tool in clinical medicine.  Clin Med 1993; 39: 561-77.
20.    Thukral A. Lodha R, Irshad M, Arora NK, Performance of pediatric mortality (PRISM), pediatric index of mortality (PIM) and PIM2 in a pediatric intensive care unit in developing country. PediatrCrit Care Med 2006; 7 (4): 356-61.
21.    Khilnani P, Sarma D, Singh R, Uttam R, Rajdev S, Makkar A, et al. Demographic profile and outcome analysis of a tertiary level pediatric Intensive care unit.Ind J Pediatr 2004; 71: 587-91.
22.    Choi KMS, Ng DKK, Wong SF, Kwok KL, Chow PY, Chan CH, et al. assessment of the Paediatric index of mortality (PIM) and the Paediatric risk of mortality (PRISM) III score of prediction of mortality in a pediatric intensive care unit in Hong Kong. Hong Kong Med J 2005; II (2): 97-103.
23.    Hwang HS, Lee NY, Han SB, Kwak GY, Lee SY, Chung SY, et al. Performance effectiveness of pediatric index of mortality  (PIM 2) and Pediatrict risk mortality III (PRISM III) in pediatric patients with in single institution:  Retrospective study. Korean J Pediatr 2008; 51(II): 1158-64.
24.    Leteurte S Leclerc F, Wirth J, Noizet O, Magnenant E, Sadik A, et al. Can generic paediatric mortality scores calculated 4 hours after admission be used as inclusion criteria for clinical trials.  Crit Care 2004; 8: R185-93.
25.    Ozer EA, Kizilgunesler A, Sariogulu B, Halicioglu O, Sutcuoglu S, Yaprak I. The comparison of PRISM and PIM scoring systems for mortality risk in infantile intensive care.  J Tropical Podiatry 2004; 50 (6): 334-8.
26.    Qureshi AU,Ali AS, Ahmed TM, Comparison of three prognostic scores (PRISM, PELOD and PIM 2) at paediatric intensive care unit under pakistani circumstances. J Ayub Med Coll Abbottabad 2007; 19 (2): 49-53.
27.    Eulmesekian PG, Perez A, Mines PG, Ferrero H. Validation of Paediatric index of Mortality 2 (PIM 2) in a single paediatric intensive care unit of Argentina.  Paediatr Crit Care Med 2007; 8(1): 54-7.
28.    Gemke RJ, van Vught J. Scoring systems in pediatric intensive care PRISM III versus PIM.Intens Care Med 202; 28 (2): 204-7.
29.    Prieto ES, Lopez-Herce CJ, Rey GC, Medina VA, Concha TA, Martinez CP. Prognostic indexes of mortality in pediatric intensive care units.  An Pediatr (Barc). 2007; 66 (4): 345-50.
30.    Slater A, Shann F, ANZICS Paediatric Study Group.  The suitability of the Pediatric Index of Mortality (PIM), PIM2, the Paediatric Risk of Mortality (PRISM), and PRISM III for monitoring the quality of pediatric intensive care in Australia and New Zealand.  Pediatr Crit Care Med 2004; 5 (5): 447-54.
31.    Martha VF, Ramos Garcia PC, Piva JP, Einloft PR, Bruno F, Rampon V. Comparison of two prognostic scores (PRISM and PIM) at a pediatric intensive care unit.  J Pediatr (Rio J) 2005; 81 (3): 259-64.
32.    Wells SM, Riera Fanego JF, Leyt DK, Dan GM, Lipman, Poor discriminatory performance of PRISM score in South African Intensive Care Unit. Crit Med 1996; 24: 1507-13.
33.    Wang JN, Wu JM, Chen YJ.  Validity of the updated pediatric risk of mortality score (PRISM III) in predicting the probability of mortality in a pediatric intensive care unit. Acta Paediatr Taiwan.  2001; 42 (6): 333-7.