Risk calculators are used to evaluate disease risk for millions of patients, making their accuracy crucial. But when national models are adapted for local populations, they often deteriorate, losing accuracy and interpretability. Investigators from Brigham and Women's Hospital, a founding member of the Mass General Brigham healthcare system, used advanced machine learning to increase the accuracy of a national cardiovascular risk calculator while preserving its interpretability and original risk associations. Their results showed higher accuracy overall in an electronic health records cohort from Mass General Brigham and reclassified roughly one in ten patients into a different risk category to facilitate more precise treatment decisions. The results are published in JAMA Cardiology.
"Risk calculators are incredibly important as they are an integral part of the conversation between providers and patients on risk prevention," said first author Aniket Zinzuwadia, MD, a resident physician in Internal Medicine at Brigham and Women's Hospital. "But sometimes, when applying these global calculators to local populations, there is variability inherent to the nature of an area—whether that is different demographic characteristics, different physician practice patterns, or different risk factors—so we wanted to find a way to tailor the foundational cardiovascular disease risk model to local populations in a safe way that builds upon what is already being done."
The American Heart Association released the Predicting Risk of Cardiovascular Disease Events (PREVENT) calculator in 2023 for adults ages 30-79. This new and improved tool helps predict the likelihood of a person developing a heart attack, stroke, or heart failure in 10 years and in 30 years. While the PREVENT equations have done well at assessing risk at a national level, the researchers wanted to test if their technique could better calibrate the risk assessment for more local populations.
In the study, researchers used electronic health record data from 95,326 Mass General Brigham patients who were 55 or older in 2007 and who had at least one lipid or blood pressure measurement between 1997-2006 and at least one encounter with the hospital system between 2007-2016. The team used XGBoost, an open-source machine learning library, to recalibrate PREVENT's equations while still preserving the associations of known risk factors with the outcomes observed in the original model. The results demonstrated greater accuracy and the reclassification of one out of ten patients in this population.
"This could theoretically represent a group of patients that might not have been prescribed statin therapies in the original application of the model, for example, but who might have benefited from them," said Zinzuwadia.
While more steps are needed before this technique could be applied to patient care, the team would like to see how it performs in the local populations of other healthcare systems and, eventually, for clinicians and researchers to use the tool to tailor global risk models.
"A major challenge of applying AI to medical research is ensuring that machine learning models are not just flexible, but also transparent, reliable, and grounded in domain knowledge," said co-senior author Olga Demler, PhD, an associate biostatistician at Brigham and Women's Hospital's Division of Preventive Medicine. "Our approach shows that it is possible to avoid the 'black box' nature of AI applications and may offer a path forward where sophisticated algorithms can retain their flexibility while producing guarantees of their performance."
Authorship: Additional authors include Olga Mineeva, Chunying Li, Zareen Farukhi, Franco Giulianini, Brian Cade, Lin Chen, Elizabeth Karlson, Nina Paynter, and Samia Mora.
Disclosures: Samia Mora has served as a consultant to Pfizer for work outside the current study. Olga Demler and Nina Paynter have received funding from Kowa Research Institute for work unrelated to the current study. Aniket Zinzuwadia has served as an employee of Heartbeat Health for work outside the current study.
Funding: Researchers were supported by the National Heart, Lung, and Blood Institute (K24 HL136852, R21 HL156174, R21HL167173, K01HL135342, and R21125962), the American Heart Association (17IGMV33860009),
the Swiss Federal Institute of Technology (ETH, Zurich, Switzerland), Dataspectrum4CVD from the Swiss Data Science Center/Personalized Health & Related Technologies, Zurich, Switzerland, and the National Human Genome Research Institute (U01HG008685).
Paper cited: Zinzuwadia A, et al. "Tailoring risk prediction models to local populations: Machine learning recalibration preserving interpretability of PREVENT." JAMA Cardiology, DOI: 10.1001/jamacardio.2024.2912
