Here the intervention teams are preparing to reclose after successful repairs (Image: CERN)
At 1 a.m. on Monday, 17 July, the LHC beams were dumped due to an electrical perturbation. Approximately 300 milliseconds later, several magnets lost their superconducting state ("quenched"). During a quench, the magnet warms up, which in turn warms and pressurizes the liquid helium that surrounds it.
While not common, this sequence of events is expected to happen to protect the superconducting cable of the magnet when an electrical glitch occurs; the mechanical stress exerted on different parts of the magnet can be quite strong.
Among the magnets that quenched on 17 July were the inner triplet magnets located to the left of Point 8 of the LHC, which play a crucial role in focusing the beams for the LHCb experiment. Unfortunately, this time, the quenches led to a helium leak in these magnets and stopped regular LHC operations.
Scroll through the photo diary below to re-live the ten day race against the clock to successfully repair the leak.
The reason for the electrical glitch that caused the security systems in the LHC to dump the beam and several magnets to quench was found: a tree on the Swiss side (about 55 km from CERN in the Canton of Vaud) fell on the power lines and perturbed the power system.
Ten hours later, on entering the tunnel, the investigating team found that the cryostats* of the triplet magnets near Point 8 were partly covered in ice. Tests quickly confirmed that a small amount of helium had escaped through a leak and filled the insulation vacuum.
Action was taken immediately: the adjacent magnets were electrically isolated, circuits were locked-off and grounded, and the quench heaters for this sector switched off. Additionally, to be able to work on the triplet, the 3 km of superconducting magnets in the affected sector were stabilised at a temperature of 20 K, instead of their usual 2 K (-271°C).
*All LHC superconducting magnets are housed in cryostats. Under normal operation the external wall of the cryostat is at room temperature, whilst the magnet operates at 2 K. The cryostat is designed to maintain the magnet at such a low temperature by minimising in-flow of heat - and insulation vacuum is essential to achieve that.
The exact position of the helium leak in the 50 m long cryostat was still unknown. By Tuesday 18 July, vibration and acoustic tests had been performed. Attaching accelerometers and microphones, the intervening team detected a clear signal at the interconnection zone between the first quadrupole magnet (Q1) and the second quadrupole magnet (Q2). Additional x-ray scans showed that the spacing of the bellows ridges on one of the pipes in the superconducting magnets appeared stretched. Bellows are employed in the physical connections between two magnets, giving flexibility. In this case the stretched bellows was on the M2 pipe, which contains the instrumentation connections.
The intervening teams agreed that the Q1-Q2 interconnection between the two quadrupole magnets would have to be opened for further investigation and repairs. To make a safe intervention possible, the sector around the magnets was emptied of the liquid helium. In parallel, an electrical quality assessment showed that the electrical circuits of the triplet were fine - the problem was thus elsewhere.
Teams of experts from different CERN groups (safety, vacuum, cryogenics, magnets, engineering, powering, magnet protection, survey, beam instrumentation, operations) discussed how to tackle a problem never encountered before on a 15-year-old string of magnets, creating a procedure on the spot.
The whole triplet cryostat reached room temperature. The external bellows and inner thermal shields at the Q1-Q2 interconnection were removed to inspect the inner helium lines.
The leaky bellows in the M2 pipe.
The teams located the suspicious M2 bellows and indeed, there was a crack on it: with a length of 1.6 mm, it was the source of the helium leak. An action plan was put into place: remove the broken bellows, replace it, go through all necessary tests, close again and start the cool down…
…all in under 10 days. Otherwise, a complete warm-up of the affected LHC sector could not be prevented, and this would stop the whole LHC physics programme for 2023.
While the broken bellows was cut out, the vacuum team conducted pressure and leak tests on spare bellows to test their resilience and provide a replacement unit for the tunnel repair.
Experts led by Sandrine Le Naour and Said Atieh discussed the possible repair solutions on site.
The new bellows was installed. On the far left: threading the instrumentation through the new bellow. In the middle: many hands make light work! On the right: skilled welders do their magic.
Graeme Barlow looking at the open interconnection, with the various pipes inside visible. The M lines allow the helium to be transported between magnets (M1 contains the busbar for electrical connection, M2 contains the instrumentation connections, and M4 has a cryogenic function). In the middle sits the beampipe where the particles circulate. The M2 bellows is just visible between the M1 and the beampipe.
The vacuum and mechanical teams discussed the action plan during ongoing repairs.
Often, two teams were at work at the same time: on the left, reinstalling beam position monitor (BPM) cables, on the right, starting the leak test on the new bellows.
The vacuum team installing the leak test tooling.
During the opening of the Q1-Q2 interconnection, the beam position monitoring (BPM) cables had to be removed. Here, the cable reinstallation is ongoing.
Sandrine Le Naour (far right) assessing the progress. She coordinated the mechanical interventions to open the magnet interconnections and then had to prepare the careful reclosure.
Wim Maan and Marcel Knoch checking the tightness of the final weld.
M2 bellows fully repaired. The bellows is surrounded by external shells to support and guide it when the helium is pressurised during different operational phases of the LHC.
The bellows is repaired and the leak test was successful within the ten day deadline. Although there's still plenty to do to reclose the interconnection, the light at end of the tunnel is in sight! After the teams repump the vacuum and cool down the magnets, the LHC can restart.
The LHC operations team is confident to see the first beam back in early September.
