Frontiers in High Performance Computing (Oct 2024)
ExaFEL: extreme-scale real-time data processing for X-ray free electron laser science
- Johannes P. Blaschke,
- Robert Bolotovsky,
- Aaron S. Brewster,
- Jeffrey Donatelli,
- Antoine DuJardin,
- Wu-chun Feng,
- Vidya Ganapati,
- Wilko Kroeger,
- Derek Mendez,
- Derek Mendez,
- Peter McCorquodale,
- Seema Mirchandaney,
- Christopher P. O'Grady,
- Daniel W. Paley,
- Amedeo Perazzo,
- Frederic P. Poitevin,
- Billy K. Poon,
- Vinay B. Ramakrishnaiah,
- Nicholas K. Sauter,
- Niteya Shah,
- Niteya Shah,
- Elliott Slaughter,
- Christine Sweeney,
- Daniel Tchoń,
- Monarin Uervirojnangkoorn,
- Felix Wittwer,
- Michael E. Wall,
- Chun Hong Yoon,
- Iris D. Young
Affiliations
- Johannes P. Blaschke
- National Energy Research Scientific Computing Center (NERSC), Lawrence Berkeley National Laboratory, Berkeley, CA, United States
- Robert Bolotovsky
- Molecular Biophysics and Integrated Bioimaging Division, Lawrence Berkeley National Laboratory, Berkeley, CA, United States
- Aaron S. Brewster
- Molecular Biophysics and Integrated Bioimaging Division, Lawrence Berkeley National Laboratory, Berkeley, CA, United States
- Jeffrey Donatelli
- Center for Advanced Mathematics for Energy Research Applications (CAMERA), Lawrence Berkeley National Laboratory, Berkeley, CA, United States
- Antoine DuJardin
- SLAC National Accelerator Laboratory, Menlo Park, CA, United States
- Wu-chun Feng
- Department of Computer Science, Virginia Polytechnic Institute and State University, Blacksburg, VA, United States
- Vidya Ganapati
- Molecular Biophysics and Integrated Bioimaging Division, Lawrence Berkeley National Laboratory, Berkeley, CA, United States
- Wilko Kroeger
- SLAC National Accelerator Laboratory, Menlo Park, CA, United States
- Derek Mendez
- Molecular Biophysics and Integrated Bioimaging Division, Lawrence Berkeley National Laboratory, Berkeley, CA, United States
- Derek Mendez
- SLAC National Accelerator Laboratory, Menlo Park, CA, United States
- Peter McCorquodale
- Applied Numerical Algorithms Group (ANAG), Lawrence Berkeley National Laboratory, Berkeley, CA, United States
- Seema Mirchandaney
- SLAC National Accelerator Laboratory, Menlo Park, CA, United States
- Christopher P. O'Grady
- SLAC National Accelerator Laboratory, Menlo Park, CA, United States
- Daniel W. Paley
- Molecular Biophysics and Integrated Bioimaging Division, Lawrence Berkeley National Laboratory, Berkeley, CA, United States
- Amedeo Perazzo
- SLAC National Accelerator Laboratory, Menlo Park, CA, United States
- Frederic P. Poitevin
- SLAC National Accelerator Laboratory, Menlo Park, CA, United States
- Billy K. Poon
- Molecular Biophysics and Integrated Bioimaging Division, Lawrence Berkeley National Laboratory, Berkeley, CA, United States
- Vinay B. Ramakrishnaiah
- Computer, Computational and Statistical Sciences Division, Los Alamos National Laboratory, Los Alamos, NM, United States
- Nicholas K. Sauter
- Molecular Biophysics and Integrated Bioimaging Division, Lawrence Berkeley National Laboratory, Berkeley, CA, United States
- Niteya Shah
- Computer, Computational and Statistical Sciences Division, Los Alamos National Laboratory, Los Alamos, NM, United States
- Niteya Shah
- Virginia Polytechnic Institute and State University, Blacksburg, VA, United States
- Elliott Slaughter
- SLAC National Accelerator Laboratory, Menlo Park, CA, United States
- Christine Sweeney
- Computer, Computational and Statistical Sciences Division, Los Alamos National Laboratory, Los Alamos, NM, United States
- Daniel Tchoń
- Molecular Biophysics and Integrated Bioimaging Division, Lawrence Berkeley National Laboratory, Berkeley, CA, United States
- Monarin Uervirojnangkoorn
- SLAC National Accelerator Laboratory, Menlo Park, CA, United States
- Felix Wittwer
- National Energy Research Scientific Computing Center (NERSC), Lawrence Berkeley National Laboratory, Berkeley, CA, United States
- Michael E. Wall
- Computer, Computational and Statistical Sciences Division, Los Alamos National Laboratory, Los Alamos, NM, United States
- Chun Hong Yoon
- SLAC National Accelerator Laboratory, Menlo Park, CA, United States
- Iris D. Young
- Molecular Biophysics and Integrated Bioimaging Division, Lawrence Berkeley National Laboratory, Berkeley, CA, United States
- DOI
- https://doi.org/10.3389/fhpcp.2024.1414569
- Journal volume & issue
-
Vol. 2
Abstract
ExaFEL is an HPC-capable X-ray Free Electron Laser (XFEL) data analysis software suite for both Serial Femtosecond Crystallography (SFX) and Single Particle Imaging (SPI) developed in collaboration with the Linac Coherent Lightsource (LCLS), Lawrence Berkeley National Laboratory (LBNL) and Los Alamos National Laboratory. ExaFEL supports real-time data analysis via a cross-facility workflow spanning LCLS and HPC centers such as NERSC and OLCF. Our work therefore constitutes initial path-finding for the US Department of Energy's (DOE) Integrated Research Infrastructure (IRI) program. We present the ExaFEL team's 7 years of experience in developing real-time XFEL data analysis software for the DOE's exascale supercomputers. We present our experiences and lessons learned with the Perlmutter and Frontier supercomputers. Furthermore we outline essential data center services (and the implications for institutional policy) required for real-time data analysis. Finally we summarize our software and performance engineering approaches and our experiences with NERSC's Perlmutter and OLCF's Frontier systems. This work is intended to be a practical blueprint for similar efforts in integrating exascale compute resources into other cross-facility workflows.
Keywords
- exascale
- Single Particle Imaging
- Serial Femtosecond Crystallography
- hardware acceleration
- data-intensive
- interfacility
