Leadership-class supercomputers dedicated to open science are not built to last forever. In fact, they have a limited lifespan by design. No matter how powerful they may be on launch day, advancements in computing technology and changing computing needs will push them closer to obsolescence with each passing year until a new model is installed and the old one is ushered out of the data center.
But that didn't happen as planned for the IBM Power System AC922 Summit supercomputer at the Department of Energy's Oak Ridge Leadership Computing Facility, or OLCF. Although the 200-petaflop Summit reached its fifth year of service in 2023 - after its successor, the exascale-class Frontier, had already been deployed as a replacement machine - its many plugs were not pulled as originally scheduled. Instead, program managers at DOE and staff at the OLCF, a DOE Office of Science user facility located at Oak Ridge National Laboratory, had other ideas.
"Summit has been a remarkably successful supercomputer, and there was no reason to limit that success to just five years. It was still the fourth-fastest supercomputer in the U.S. at that time, we still had plenty of scientists who wanted to take advantage of Summit's computing power, and we would still have our data center facilities available in 2024. So, it made a lot of sense to keep Summit running into 2024 for another year of production," said Bronson Messer, the OLCF's director of science.
This resulted in a new DOE Office of Science-backed allocation program called SummitPLUS, which was announced in September 2023 and accepted applications through the next month. By December, 108 projects were selected to receive over 19 million hours of compute time from January 2024 to October.
Allocations were awarded to researchers from academia, government laboratories, federal agencies and industry with projects in a variety of scientific domains. What did we learn during Summit's bonus year of scientific discovery? Here are five sample projects with impactful results.
Lighting the way to a new Alzheimer's treatment
Amyloid fibrils are protein structures that can accumulate in body tissues and organs to form plaques associated with chronic degenerative diseases such as Alzheimer's, Parkinson's and Type 2 diabetes. Medications can reduce the levels of these anomalous protein aggregates, thereby slowing their development, but no efficacious treatment currently exists to specifically target these amyloid fibrils.
Amyloid fibrils, shown in this video aggregating in a complex neuronal signaling environment, have been targeted in a variety of degenerative diseases. Credit: Quantum Biology Laboratory, Howard University. Please see YouTube page for full video information.
But what if amyloid fibrils were not merely signs of pathology but rather a protective response? That's been the study of theoretical physicist Philip Kurian and his team in the Quantum Biology Laboratory at Howard University as they examine the quantum interactions of light and matter in proteins.
A variety of diseases, such as cancers or dementias, are characterized by internal cellular environments rife with oxidative stress - the production of free radicals from metabolic processes. These free radicals can emit photons across the ultraviolet, visible and infrared portions of the electromagnetic spectrum.
"Our work shows that amyloid fibrils can actually provide an enormous neuroprotective response due to a quantum optical effect, known as single-photon superradiance, by absorbing high-energy UV photons rapidly and downconverting them to longer, safer wavelengths," Kurian said.
Kurian and his team used Summit to make predictions for the thermal quantum yields - the ratio of the number of photons emitted to those absorbed at thermal equilibrium - of cytoskeletal filaments and amyloid fibrils of increasing lengths. The researchers found that networks of the amino acid tryptophan in amyloid fibrils rapidly superabsorb UV photons due to collective long-range interactions across these architectures, and the downconversion reemits them at longer, less damaging wavelengths.
"Our access to Summit has enabled the numerical simulation of huge mega-architectures of quantum optical networks in eukaryotic protein fibers. This has given us unprecedented access to understanding the role of quantum effects in neural and aneural pathways for information processing, and ultimately the computational capacity of life, by using more physical degrees of freedom offered by the interaction of the electromagnetic field and living matter," Kurian said.
Predicting the signatures of neutron stars and black holes
The universe is full of enigmatic phenomena that we can only observe from afar and attempt to decipher by gathering measurable clues: electromagnetic radiation, gravitational waves, cosmic rays, or neutrinos, for example. Although we may not always be sure what, exactly, we're looking at, supercomputers can help astrophysicists figure out where we should be looking to learn more. By simulating cosmic events and predicting their emission signatures, researchers can search for telltale signs to find their sources.
A team of researchers from the University of Toronto, the California Institute of Technology, the University of Maryland and Columbia University used Summit to better understand and predict the electromagnetic emissions - radio waves, X-rays and gamma rays - from neutron stars and black holes.
Neutron stars are found in a variety of forms: as very highly magnetized magnetars that burst regularly; as pulsars that look like lighthouses in the sky due to their fast rotation and the beam of emissions that periodically reaches Earth; and as binary systems, in which two orbiting neutron stars and/or black holes interact and emit both gravitational and electromagnetic waves.
"With our GPU-accelerated code and the GPUs of Summit, we are able to - for the first time -simulate all three of these objects in enough detail to calculate the emissions from different proposed mechanisms, enabling us to study neutron stars, black holes and binaries," said Bart Ripperda, an assistant professor at the University of Toronto's Canadian Institute for Theoretical Astrophysics and principal investigator for the project.
In particular, the team predicted new electromagnetic emissions signatures for the merger of two massive black holes, enabling multi-messenger study (detection of various emissions) of these cosmological events with upcoming space-based gravitational wave detectors such as NASA's planned LISA space probe .
The simulations required very high grid resolution to resolve the small-scale physics of the interaction of the plasma (the gas of charged particles) that powers the radiation (the photons) that determines the observable emissions.
"Compared to existing simulations, a factor of 1,000 in resolution is required to capture these processes accurately in 3D simulations. Additionally, incorporating the plasma-radiation interaction is essential to understanding the emission from first principles," Ripperda said. "This makes our simulations extremely computationally expensive, and they are only feasible when using thousands of GPUs, such as those available on Summit."
Seeking a path for large-molecule drug design
Large molecules generally have a difficult time passing through cell membranes, and this difficulty has made them poor candidates for new drug designs despite their potential to perform a wider variety of biological functions than traditional, small-molecule drugs. However, there is an exception to that generalization: cyclosporine A, an immunosuppressant usually given to transplant patients to prevent organ rejection.
How does cyclosporine A succeed at passing through cell membranes where other large molecules fail? Answering that question could inform general principles for understanding membrane permeability and lead to the design of new, potentially more effective large-molecule drugs.
"We're studying the different ways cyclosporine can twist and bend in different conformations because its conformational diversity plays an important part in its ability to permeate the membrane," said Edward Lindberg, a graduate student and member of assistant professor and principal investigator Thanh Do's research group at the University of Tennessee, Knoxville. "We are hoping to illuminate a network of these conformations that is consistent with general chemistry principles and spectroscopic data of cyclosporine and some related molecules."
Using Summit, the team produced the first molecular dynamics simulations of cyclosporine A to capture the major conformation in aqueous environments predicted by nuclear magnetic resonance, or NMR, and neutron diffraction, as well as a low-energy conformation found in high-temperature NMR. Their methodology incorporates redefinitions of important simulation parameters that improve upon existing strategies for sampling the conformational space of cyclosporine.
"Our use of leadership computing resources is primarily for running large-scale molecular dynamics simulations," Lindberg said. "Our project requires that many simulations of cyclosporine and similar molecules be run, preferably at the same time. This can require an incredible number of computational resources to run a simulation that's long enough to obtain statistically significant results."
Building a quieter and more efficient airplane
Startup aerospace company Whisper Aero has a vision for the future of regional air travel: quiet, efficient, and decarbonized jets powered by its patented electric ducted fans, or EDFs. Furthermore, its engineers have designed a Whisper Jet aircraft concept that integrates small EDFs into the leading edge of its wings to form a jetfoil. But refining and testing those designs for a prototype vehicle would have taken much longer if not for Summit and its ability to conduct largescale, high-resolution computational fluid dynamics simulations of airflow to predict the aircraft's aerodynamics.
"We were able to run much larger design-of-experiment studies on Summit than those we were typically constrained to carry out with traditional, CPU-based supercomputers," said Vineet Ahuja, Whisper Aero's head of flight sciences and an associate fellow at the American Institute of Aeronautics and Astronautics. "As a result, aerodynamic design has shown an incredible turnaround time relative to where we were just a few years back in our pre-Summit days. This speedup ultimately benefits vehicle development cycle times, which are reduced by over 20%."
With its SummitPLUS allocation, the Whisper Aero team was able to investigate vertical takeoff and landing and short takeoff and landing vehicle concepts that leverage Whisper's propulsors.
"We're doing a lot of developmental work to see how our technology actually transfers to that configuration because most people doing vertical takeoff and landing right now use open propellers," Ahuja said. "This entire field of electric aviation is undefined and uncharted territory. So, innovating at a rapid pace is extremely important. Summit enables us to model significant changes between propulsor size, efficiency and noise by harnessing the acceleration of those GPU codes. This capability enables us to iterate designs very, very quickly."
Probing the mechanics of aerosol virus transmission
When the COVID-19 pandemic struck, and the SARS-CoV-2 virus's method of infection was identified as respiratory - inhaling infectious liquid particles - the mechanisms of airborne transmission became a global concern. But even as "droplets" became a new buzzword, reaching peak popularity as a Google search term in April 2020, some scientists focused their attention on aerosols instead.
Aerosols can travel long distances and remain suspended in the air for hours or even days, a durability that poses significant infection risk for areas with poor ventilation, where infectious particles can accumulate.
"Medical dogma has long focused on droplets as the main transmission route for respiratory viruses, but airborne transmission via aerosols also plays a significant role in spreading disease, as evidenced by super-spreader events," said Rommie Amaro, endowed chair and professor in the Department of Molecular Biology at the University of California San Diego. "Intervention and mitigation decisions, such as the relative importance of surface cleaning or when to wear a mask, have unfortunately hinged on a weak understanding of aerosol transmission - to the detriment of public health."
The Amaro Lab is based at UC San Diego and develops and applies state-of-the-art computational methods to investigate the structure, function and dynamics of complex biological systems such as pathogen-bearing aerosols. A central challenge to understanding airborne transmission has been the inability of experimental science to reliably probe the structure and dynamics of viruses once they are inside respiratory aerosol particles. Simulations offer a way to predict these properties.
"Understanding the mechanisms behind the airborne transmission of disease is a crucial area of research for current and future pathogens. Computational microscopy is a powerful tool capable of overcoming the significant experimental limitations present in the study of submicron aerosols, particularly regarding those containing viruses," Amaro said.
Although new experimental results have highlighted the importance of pH (how acidic or alkaline a substance is) in the airborne viability of SARS-CoV-2 and suggest that pH is the driving force behind its viral decay, these studies lack a clear mechanistic explanation of why it happens.
For its work on Summit, Amaro's team ran all-atom molecular dynamics simulations of the Delta variant of SARS-CoV-2 within a submicron (280 nanometers) respiratory aerosol - a system comprising over 1 billion atoms. They were able to measure the immediate structural and morphological impacts of a rapidly increasing pH environment in the aerosol and evaluate the protective influence of respiratory fluid components in a pH 7 versus pH 12 environment.
"With our complementary smaller-scale simulations, this work will enable the study of aerosols across multiple scales of size, resolution, pH and time. Our work shows how leadership-class computing facilities will shed new light on scientific challenges that until now have been intractable," Amaro said.
UT-Battelle manages ORNL for DOE's Office of Science, the single largest supporter of basic research in the physical sciences in the United States. DOE's Office of Science is working to address some of the most pressing challenges of our time. For more information, visit energy.gov/science . - Coury Turczyn
