Project Details

Prospective Evaluation of a Machine Learning Approach to Predicting Absence of Serious Bacterial Infection

Blake Martin, MD

Summary

BACKGROUND: Over a third of children admitted to the PICU are diagnosed with a serious bacterial infection (SBI). Accordingly, PICU providers often empirically prescribe antibiotics to critically ill pediatric patients while they are undergoing testing for infection. Unfortunately, this results in many children without subsequently identified SBI receiving broad-spectrum antibiotics for several days while microbiologic studies are pending. This increases the patient’s risk for antibiotic-associated adverse effects including nephrotoxicity, clostridium difficile colitis, and prolonged hospital length of stay. GAP: There are no clinically validated predictive models which utilize readily-available electronic health record (EHR) data to predict which children admitted to the PICU are at low risk for SBI and for whom unnecessary antibiotic exposure could be minimized. HYPOTHESIS: Silent, prospective validation of our two previously developed machine learning-based SBI predictive models (which utilize readily-available EHR inputs such as vital signs and lab values) will demonstrate negative predictive values ≥95% for ruling out SBI, will correctly identify ≥20% of the SBI-negative children who are subsequently given unnecessary antibiotics, and will demonstrate the potential to spare these children a median of ≥3.0 antibiotic-days per patient. We also hypothesize that a pilot qualitative needs assessment on antibiotic decision-making using interviews with PICU clinicians will identify key themes able to inform the future development of trusted and actionable, model-based antibiotic decision-making clinical decision support (CDS) tools. METHODS: We are performing a prospective cohort study of children 3-months-to-18-years-old admitted to our PICU over a 1-year period (9/1/2023 – 8/31/2024). During this period we are prospectively evaluating in silent fashion (model results not surfaced to clinicians) the performance of two previously developed ML-based predictive models trained to identify SBI-negative children for whom antibiotics could potentially be reduced. Additionally, we performed 15, 1-on-1, 60-minute interviews with PICU clinicians involved in antibiotic decision-making (nurses, physicians, and advanced practice providers) to understand the decisional needs of the end-users of a future CDS tool based on these SBI predictive models. During the interviews, topics covered included 1) antibiotic decision-making processes; 2) data required for decision-making; 3) desired CDS tool capabilities, display, EHR location, and function; and 4) barriers and facilitators to CDS tool use. RESULTS: Between September 1st, 2023, and November 1st, 2023, the predictive models produced 83,455 SBI predictions for 686 critically ill children admitted to the CHCO PICU. We are continuing to gather data on the infectious and clinical outcomes of the included patients to enable SBI classification and determine SBI model performance. We are also tracking the clinical outcomes of all study patients including antibiotic receipt and antibiotic adverse effects. Additionally, of the 15 interviews with PICU clinicians which we completed, 5 were attendings, 3 were fellows, 3 were nurses, and 4 were advanced practice providers. Qualitative analysis of the interview transcripts identified three predominant themes regarding optimal design of an antibiotic decision-making CDS tool: presence of both active (triggered alert at the time of antibiotic ordering) and passive (available on demand) CDS, display of the key model inputs driving SBI status predictions, and limiting unnecessary interruptions in clinical workflow. Pending results include prospective SBI model, analysis of the clinical outcomes of patients who receive unnecessary antibiotics, and determination of the potential quantity of antibiotic treatment-days potentially avoidable via model predictions. IMPACT: This project represents a paradigm shift in personalized antibiotic decision-making. If successful, we will have laid the groundwork for implementation of machine learning-based CDS tools able to aid PICU clinicians in minimizing unnecessary antibiotic exposure and associated adverse effects. Optional/Additional Comments: Our definition of serious bacterial infection (SBI) is adapted from the National Healthcare Safety Network and includes positive culture from any normally sterile body fluid/site (e.g. bacteremia, meningitis, pyelonephritis) as well as certain culture-negative conditions (culture negative septic shock, and pneumonia diagnosed via chest radiograph). The machine learning predictive models we will be prospectively evaluating include a Random Forest model and a penalized logistic regression model. Each uses 40 EHR data element inputs with imputation of missing values. The predictive models produce SBI probability predictions every 2 hours for the first 24 hours of PICU admission when the vast majority of antibiotic decisions are made.

Publications: