With large language models that take notes during patient visits and algorithms that identify disease, artificial intelligence has begun to prove its worth as an assistant for physicians. But a new study from Stanford Medicine shows the potential of AI as a facilitator — one that helps doctors and nurses connect to achieve more efficient, effective patient care.
The study, which published in JAMA Internal Medicine last month, describes an AI-based model in use at Stanford Hospital that predicts when a patient is declining and flags the patient's physicians and nurses. Ron Li, MD, a clinical associate professor of medicine and medical informatics director for digital health who is the senior author on the study, said the alert system helps clinicians connect more efficiently and effectively as well as intervene to prevent patients from deteriorating and landing in the intensive care unit.
Li, who worked with informatics postdoctoral scholar and lead author Robert Gallo, MD, on the evaluation, discussed their team's approach to harnessing the algorithm and how it fosters clinician connection in a ceaselessly buzzing hospital environment. Lisa Shieh, MD, PhD, clinical professor of medicine; Margaret Smith, director of operations for primary care and population health; and Jerri Westphal, nursing informatics manager, also helped lead the study and the implementation of the AI system.
What is a deterioration model and how does AI fit in?
The algorithm is a prediction model that pulls data — such as vital signs, information from electronic health records and lab results — in near-real time to predict whether a patient in the hospital is about to suffer a health decline. Physicians aren't able to monitor all of these data points for every patient all of the time, so the model runs in the background, looking at these values about every 15 minutes. It then uses artificial intelligence to calculate a risk score on the probability the patient is going to deteriorate, and if the patient seems like they might be declining, the model sends an alert to the care team.
What's the benefit of having such a model run in a hospital?
The big question I want to answer is, "How do we use AI to build a more resilient health system in high-stakes situations?" There are many ways to do that, but one core characteristic for a resilient system is strong communication channels. This model is powered by AI, but the action it triggers, the intervention, is basically a conversation that otherwise may not have happened.
Nurses and physicians have conversations and handoffs when they change shifts, but it's difficult to standardize these communication channels due to busy schedules and other hospital dynamics. The algorithm can help standardize it and draw clinicians' attention to a patient who may need additional care. Once the alert comes into the nurse and physician simultaneously, it initiates a conversation about what the patient needs to ensure they don't decline to the point of requiring a transfer to the ICU.
Tell me about how your team implemented and evaluated the model.
We integrated this model, which we did not create, into our workflow, but with a few tweaks. Originally, it sent an alert when the patient was already deteriorating, which we didn't find very helpful. We adjusted the model to focus on predicting ICU transfers and other indicators of health decline.
We wanted to ensure the nursing team was heavily involved and felt empowered to initiate conversations with physicians about adjusting a patient's care. When we evaluated the tool, which we had running for almost 10,000 patients, we saw a significant improvement in clinical outcomes — a 10.4% decrease in deterioration events, which we defined as transfers to the ICU, rapid response team events, or codes — among a subset of 963 patients with risk scores within a "regression discontinuity window," which basically means they're at the cusp of being high risk. These are patients whose clinical trajectory may not be as obvious to the medical team. For that group of patients, this model was especially helpful for encouraging physicians and nurses to collaborate to determine which patients need extra tending.
How have nurses and physicians responded to the integration of this new model?
The model is far from perfect. The reactions have overall been positive, but there is concern about alert fatigue, since not all alerts are flagging a real decline. When the model was validated on data from patients prior to implementation, we calculated that about 20% of patients flagged by the model did end up experiencing a deterioration event within six to 18 hours. At this point, even though it's not a completely accurate model, it's accurate enough to warrant a conversation. It shows that the algorithm doesn't have to be perfect for it to be effective.
With that said, we want to improve the accuracy; you need to do that to improve trust. That's what we're working on now.
