David Smith, front, and Dave Martinez work on a computer server submerged in liquid as part of testing at the High Performance Computing center at Sandia National Laboratories. The immersion system uses a nonconductive liquid coolant that enables 100% of the heat generated to be captured. The reflection of the computer demonstrates the high liquid level. (Photo by Craig Fritz) Click on the thumbnail for a high-resolution image.
Keeping electrical circuits dry is generally considered a vital safety measure, but at Sandia National Laboratories' high-performance computing center, technicians adjust live circuits submerged in liquid.
"It's pretty cool - and a little unusual - to walk into a computing lab and see electrical hardware submerged in fluid," said Sandia engineer David Damm.
Complete immersion of computer components appears to be the most effective way to cool them, explains David J. Martinez, engineering program project lead for Sandia's Infrastructure Computing Services.
"We place whole computers - their power cables, everything - in a liquefied solution," Martinez said. "We take an entire rack and drop it into fluid contained in big immersion tanks."
Martinez believes that this direct contact between a liquid coolant and electrified equipment could reshape the future designs of data centers.
Reshaping data-center architectures
By submerging all parts of the computing servers in a liquid coolant that doesn't conduct electricity, 100% of the generated heat can be captured, almost entirely eliminating the need for the power-hungry fans and chillers used in conventional cooling systems.
David Smith makes network connections to a computer server submerged in liquid as part of testing at the high performance computing center at Sandia. (Video by Craig Fritz) Click the image to watch or download the video.
Martinez estimates that this type of cooling system, using liquid approximately the viscosity of cooking oil, would cut energy consumption by 70%. Unlike water-chill systems that require evaporation to lower operating temperatures, no water is lost; the coolant gives up its heat to the open air, given the right temperature differential.
Sandia's forward-thinking approach arises from projections that the increasing water and electrical power demands for cooling in high-performance computing eventually will exceed the resources of small towns and become an unsupportable burden.
To address this, Sandia is formally testing a commercial non-conductive liquid system from the Barcelona-based company Submer throughout the summer. Early positive results led Martinez to envision a reduction in the two big problem areas of power and water use that are expected to hinder the development of faster, more powerful machines.
Hoan Luu, left, and David Damm works on a computer server submerged in liquid as part of testing at the High Performance Computing center at Sandia National Laboratories. The immersion system uses a non-conductive liquid coolant which enables 100% of the heat generated to be captured. Save for the silver handles at the top, all is under liquid. (Photo by Craig Fritz) Click on the thumbnail for a high-resolution image.
Promising innovations and partnerships
According to Submer promotional literature, their single-phase immersion cooling system uses a biodegradable, non-toxic, non-flammable and non-corrosive liquid. It boasts being 95% more efficient than traditional cooling technologies, while 1,400 times a better thermal conductor than air and eight times less electrically conductive.
Negotiating through ADACEN, a local data company, Martinez convinced Submer to partner with Sandia. Submer would provide fluid and equipment and Sandia would provide its expertise in system cooling and testing to perform a thorough technical evaluation of the system. Meanwhile, Martinez, who for 25 years has worked diligently to bring about the next stages of computer cooling, is happy to reduce Sandia's costs for power and water and provide a stage for a potential revolution for the industry.
In partnership with ADACEN, code is being written specifically for the new task by Hoonify Technologies, composed of five former Sandia researchers who have taken entrepreneurial leave from the Labs.
In 2020, Submer, in collaboration with Intel Corporation, introduced the idea of a liquid coolant that would bathe all components of a computing system to provide even cooling and allow higher operating temperatures.
As a bonus, the warmer temperature of the returning liquid can offer some heat to adjacent buildings during winter months and can be used to heat labs and showers in the summer through heat-exchange processes.
Sandia will conduct comprehensive tests on this system, evaluating hardware reliability and infrastructure support cost and effort. A case study is expected to be released this fall to assess the pros and cons of the technology.
Sandia National Laboratories is a multimission laboratory operated by National Technology and Engineering Solutions of Sandia LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy's National Nuclear Security Administration. Sandia Labs has major research and development responsibilities in nuclear deterrence, global security, defense, energy technologies and economic competitiveness, with main facilities in Albuquerque, New Mexico, and Livermore, California.
