Article courtesy of Brookhaven National Laboratory
We all remember the impact of stay-at-home-orders on our everyday lives in spring 2020. However, it was not only restaurants, salons, flower shops, and bookstores that had to close their doors. National user research facilities shut down most operations, closing the doors to thousands of visiting scientists, and bringing research on new batteries, pharmaceutical drugs, and many other materials to a grinding halt, at a time when the country needed these facilities more than ever. So, seven user research facilities decided to form a team of experts, the Remote Access Working Group (RAWG), to figure out how these facilities could keep the science going even when the researchers couldn't access them in person.
The solution is as simple as it is difficult. Research facilities that serve visiting researchers have to create an environment in which experiments can be run from afar - with nearly no human interaction. Scientists have dubbed this new way of doing research remote experimentation. While each facility started the unexpected journey to remote experimentation on their own, the RAWG has brought all the different ideas together to help each facility overcome the numerous challenges encountered along the way.
Most challenges result from the nature of how these facilities operate. All seven facilities are neutron or light sources funded by the U.S. Department of Energy (DOE) Office of Science. This means they generate highly intense beams of neutrons or x-rays that visiting scientists use to study the inner workings of materials. These visiting researchers, or users, collaborate with facility staff to study everything from ancient mummies to novel quantum materials, generating new knowledge daily.
The Desolation of COVID-19
In a world before COVID-19, these user facilities were a hub for research teams. Scientists traveled to them, used unique tools to study their materials, worked with brilliant people on all kinds of scientific questions, then left the facility with new data that could answer these questions. With the ongoing pandemic, travelling to a facility in a different state-let alone a different country-is not an option. And with this, the well-established cycle of creating new knowledge was broken.
To re-start this cycle without going back to the old ways, each facility was confronted with a host of challenges that ranged from how to control an experiment from afar to how to get the samples to the facility in the first place. This was just the tip of the iceberg of issues the pandemic created. The RAWG's mission is to share experiences and solutions for these issues among the facilities.
The Fellowship of Remote Experimentation
The RAWG was built upon the existing collaboration of the five DOE light source facilities. Their directors meet twice a year to discuss common challenges so that they can form teams to tackle various issues. So, it was only natural to join forces again when COVID-19 hit.
The light sources - namely the Advanced Light Source (ALS), the Advanced Photon Source (APS), the Linac Coherent Light Source (LCLS), the National Synchrotron Light Source II (NSLS-II), and the Stanford Synchrotron Radiation Lightsource (SSRL) - were soon joined by the two neutron sources at Oak Ridge National Laboratory (ORNL).
Once the team was assembled, it was time to identify the challenges and start the journey to find their solutions.
Defining the lay of the land: Surveys
"The facility directors saw that remote experiments were a challenge for all of us during the pandemic, but that they weren't going to go away after the pandemic. So, we needed to make a serious coordinated effort to understand the problems and work together on solutions," said Dula Parkinson, a beamline scientist, deputy for photon science operations at ALS, and RAWG member. "When we started sharing experiences across facilities, it was good to hear that others were struggling with the same challenges we were. We pretty quickly identified some of the urgent ones where we could work together."
The team shared anything from ideas to documents, but this shift to remote experimentation affected every aspect of facility operations. So, they needed more information and more points of view.
"We ran several surveys to get an understanding of what issues staff and users saw," said Ritimukta Sarangi, a senior scientist in hard x-ray spectroscopy from SSRL and RAWG member. "Our surveys really shed light on some issues that weren't clear in the beginning. For example, most users felt their experimental needs could be met by remote experimentation, while most facility staff members were more critical of that option. The results also revealed a shift in activities for light source staff. They changed from developing and maintaining instrumentation for user science to actively participating in user experiments."
A virtual seat at the table: Forum and website
"We realized early on that communication was key. If we wanted to learn from each other's progress, we needed a place to share our experiences. We, at NSLS-II, had a virtual weekly meeting for staff to show their projects to other staff members. It worked well for us. So, the RAWG decided to create a series of meetings with staff from all facilities for this purpose," said Lisa Miller, a biochemist, manager of the user services, communication, education and outreach office at NSLS-II, and RAWG member.
Image courtesy: APS, Argonne National Laboratory
The ongoing monthly Neutron & Light Source Forum is a chance for staff from all facilities to present their current projects connected to remote experimentation. Each session has a different speaker from one of the facilities and ensures that the presentation or demonstration is followed by a discussion involving experts from other facilities.
"The balance between presentation and discussion is important. We can all learn from each other. The forum offers a direct exchange between staff from our facilities that had not existed in this form before. We record the sessions. Together with the slides we host the presentations on our website. Our website is becoming a great resource for everyone working on this topic. Additionally, we also host all RAWG presentations on a YouTube channel," said Alex Wallace, an engineer, head of the experiment control system development department at LCLS, and a RAWG member.
Preparing the gear: Controlling the beamlines
"In the forefront of all of this stood the question of how can we run an experiment at a beamline without actually being there. Sure, the beamline is controlled by a computer but that is usually in walking distance of the instrument," said Stuart Wilkins, a physicist, manager of the data science and systems integration program at NSLS-II, and RAWG member. "Now, we needed to create a solution that would allow a user to login into that computer from anywhere while keeping it safe, ensuring that every part of the experiment - a motor, a heat source, a detector, anything - worked flawlessly, and making it work 24 hours a day."
All facilities used secure remote access software to connect to their instrumentation. This allowed the researchers to see and interact with programs such as measurement software running on each other's computers. By sharing their experiences during the forums, the different facilities could identify the best software solutions for their beamlines.
"Running real experiments remotely requires also effective real-time communication between the staff and the users at their home institution. Here, again, all facilities experimented with different methods of video conferences, audio connections, screen sharing, and video feeds," said Wallace. "Finding the right cameras, microphones, video tools, and so on is important. So, we ensured that the experiences and solutions were shared between all the facilities."
New travel companions: Robots
"Replacing what a human can do at the beamline isn't easy, even something as simple as changing a sample. Every user's samples look different, and finding a robotic solution that can handle different types of samples is a real challenge. Our experts started by adapting solutions from APS; now, we are sharing new solutions with the other facilities. Once the robots can handle the different samples, we have to integrate its controls with the rest of the beamline, but we are all learning from each other how best to do this. This is a great collaboration; it only started because of the work of the RAWG," said Kevin Stone, a staff scientist in the materials science division at SSRL and RAWG member.
At light source facilities, automation was a new trend for most beamlines prior to the pandemic because of the challenges created by varying sample shapes. However, even before the current push for remote access, developments in automation offered advantages to all beamlines in some way. The pandemic just forced these facilities to invest in more automation than before.
"There's always pressure to be more efficient with our limited resources. Pre-pandemic, our staff handled approximately 1,500 users, 800 experiments, and 10,000 samples annually. For that volume of work, automation is required wherever possible. By applying robots and automation in the right places, we increased our productivity and allowed scientists to focus on science instead of 3 a.m. sample changes," said Harley Skorpenske, a science support group leader at Oak Ridge's neutron sources and RAWG member. "During the pandemic, a common theme was 'shifting of the burden.' In this case, our 1,500 users became remote, and our staff conducted all on-site activities. We always appreciated the sleep automation permitted, but now it's absolutely critical to our operations."
There and back again: Sample shipping
"One really big issue was sample shipping, tracking, and handling during experiments, which has traditionally been overseen by the user with tools provided by the facility to help with transportation and safe handling. During shelter-in-place orders and remote operations, these responsibilities fall on the shoulders of beamline staff, who are ill-equipped to handle them. It is a huge undertaking when there are hundreds of samples per beamline in a year," said Sarangi. "The forum was great to bounce ideas for solutions. We learned from and adapted the shipping, handling, and communication process from crystallography beamlines. They have been optimized for remote access operations for some time."
Crystallography beamlines study tiny frozen protein crystals to understand their structure on the atomic scale. Since these crystals are always roughly the same shape, researchers concentrated their development efforts into automated experimentation with fast robots even before the pandemic. Most other beamlines measure data on samples in a variety of shapes, forms, and physical states (e.g., liquid, solid, hot, cold, etc.). These beamlines invested in other instrumentation for their experiments, while remote access tools were only partially developed.
"When a system is stressed, you gain a new perspective on its weaknesses and strengths. As a neutron facility, we have an obligation to track our samples due to radiation. For over a decade our lab staff have developed and supported our sample tracking and shipping protocols, and it has been absolute godsend during COVID. I think, the value of sharing our success with the RAWG forum, and the broadening of our perspective on the collective issues presented by the pandemic cannot be overstated," said Skorpenske.
Digging through the treasure: Data & Visualization
The last-but definitely not least-important area is data. It is important to ensure that user get their data and can produce results as fast as possible. This means not only after the experiment, but also while it is still running, in case they want to adjust the next run," said Alec Sandy, an associate division director at APS and RAWG member. "Many facilities accelerated ongoing work on streaming data to portals for ready access by users. We had exciting presentations about this in the forum."
Data from neutron and light sources can come in many different forms. So, the solution for distributing data to a wide range of users needs to be able to handle all the different formats and offer various ways to access it securely.
"Making the data available is only part of the puzzle. Helping researchers visualize their data is essential too. With the right online tools, fast visualization during an experiment can enhance the success of the experiment. In order to move this field forward, we are working together with the Light Source Directors Computing Steering Committee, which was founded to work on all data- and computing-related issues prior to the pandemic," said Suresh Narayanan, a beamline group leader at APS and RAWG member.
The Return of the User - Hybrid Operations
Even as some of these challenges have been solved at the different facilities, for many of them the journey towards new solutions continues. The RAWG, together with staff from all facilities, continues to keep the experiments running so that science moves forward.
The monthly forum series will continue to enhance the exchange between all facilities throughout 2022, while the RAWG will continue to work with other overarching groups such as the computing group to move remote access forward.
Overall, all facilities have concluded that remote experimentation will stay in place well beyond COVID-19 due to its many advantages. It opens the doors for more collaboration with researchers who can't travel due to personal or professional reasons. It allows user facilities to reach more researchers, which in turn will open pathways for more exciting science in the future. This is the beginning of a new era of hybrid operations at neutron and light source facilities.
Members of the RAWG were supported by their facilities. All facilities - the Advanced Light Source (ALS) at U.S. Department of Energy's (DOE's) Lawrence Berkeley National Laboratory, the Advanced Photon Source (APS) at DOE's Argonne National Laboratory, the Stanford Synchrotron Radiation Lightsource (SSRL) and the Linac Coherent Light Source (LCLS) at DOE's SLAC National Accelerator Laboratory, the National Synchrotron Light Source II (NSLS-II) at DOE's Brookhaven National Laboratory, and the High Flux Isotope Reactor (HIFR) and Spallation Neutron Source (SNS) at DOE's Oak Ridge National Laboratory - are U.S. Department of Energy Office of Science User Facilities.
