While immune checkpoint inhibitors (ICIs) can provide lifesaving treatment for patients with cancer, they have also been found to cause immune-related adverse events (irAEs) — side effects that can impact almost every organ in the body to varying degrees. The frequency and severity of irAEs in real-world datasets are not well understood, making it difficult to combine cases effectively across institutions and gain insights into the optimal management of these patients. Since current approaches to investigate irAEs are done manually and are inefficient, researchers from Mass General Brigham have incorporated the use of a prebuilt large language model (LLM) to identify information relating to instances of irAEs in hospital settings.
The LLM identified the most common hospitalizing irAEs, including ICI-induced colitis, hepatitis, pneumonitis, and ICI-induced myocarditis, which can be fatal. The LLM was compared to the performance of International Classification of Disease (ICD) codes, which retrospectively identify irAEs, and demonstrated higher accuracy. Results are published in The Journal of Clinical Oncology.
"Not only did the LLM demonstrate higher accuracy in detecting irAEs compared to ICD codes, it identified additional cases of irAEs not picked up via manual adjudication, with an excellent specificity/sensitivity and at only 9.53 seconds per chart," said corresponding author Kerry Reynolds, MD, Director of the Severe Immunotherapy Complications Program at Mass General Cancer Center. "As a free and open-source model, the LLM pipeline opens up this field, enabling other institutions to quickly recreate similar databases and has the potential to ignite collaboration in unprecedented ways."
This study analyzed 10 years of data in a gold standard, manually curated dataset from patients who were hospitalized after receiving ICI therapy.
"The results demonstrated that the model consistently achieved sensitivities and specificities above 90% across the four irAEs which is excellent," said Reynolds. "Historically, collaboration in the field of irAEs has been concentrated among large academic centers, leaving smaller community hospitals with less opportunity to contribute. This study has the potential to change that. The LLM presented in this study requires minimal computational resources and can be run on a local machine, we are eager to share it with the broader community."
Authorship: Virginia H. Sun, Julius C. Heemelaar, Vineet K. Raghu, Azin Ghamari, Giselle A. Suero-Abreu, Tomas G. Neilan, Jor Sam Ho, Jessica Wu, Hannah K. Gilman, Ibrahim Hadzic, Meghan J. Mooradian, Steven M. Blum, Ryan J. Sullivan, Chia-Yun Wu, Leyre Zubiri, Nicole R. LeBoeuf, Daniel A. Zlotoff, Ayo S Falade, Benjamin Medoff, Shilpa Grover, Kelley Grealish, Nora Hathaway, Alexandra-Chloe Villani, Meghan E. Sise and Michael Dougan.
Paper cited: Sun VH et al. "Enhancing precision in detecting severe immune-related adverse events: comparative analysis of large language models and ICD codes in patient records" Journal of Clinical Oncology DOI: 10.1200/JCO.24.00326
