The Astrophysical Journal (Jan 2024)
Anomaly Detection and Approximate Similarity Searches of Transients in Real-time Data Streams
- P. D. Aleo,
- A. W. Engel,
- G. Narayan,
- C. R. Angus,
- K. Malanchev,
- K. Auchettl,
- V. F. Baldassare,
- A. Berres,
- T. J. L. de Boer,
- B. M. Boyd,
- K. C. Chambers,
- K. W. Davis,
- N. Esquivel,
- D. Farias,
- R. J. Foley,
- A. Gagliano,
- C. Gall,
- H. Gao,
- S. Gomez,
- M. Grayling,
- D. O. Jones,
- C.-C. Lin,
- E. A. Magnier,
- K. S. Mandel,
- T. Matheson,
- S. I. Raimundo,
- V. G. Shah,
- M. D. Soraisam,
- K. M. de Soto,
- S. Vicencio,
- V. A. Villar,
- R. J. Wainscoat
Affiliations
- P. D. Aleo
- ORCiD
- Department of Astronomy, University of Illinois at Urbana-Champaign , 1002 W. Green Street, IL 61801, USA ; [email protected]; Center for AstroPhysical Surveys , National Center for Supercomputing Applications, Urbana, IL 61801, USA
- A. W. Engel
- ORCiD
- Pacific Northwest National Laboratory , 902 Battelle Boulevard, Richland, WA 99354, USA
- G. Narayan
- ORCiD
- Department of Astronomy, University of Illinois at Urbana-Champaign , 1002 W. Green Street, IL 61801, USA ; [email protected]; Center for AstroPhysical Surveys , National Center for Supercomputing Applications, Urbana, IL 61801, USA
- C. R. Angus
- ORCiD
- DARK, Niels Bohr Institute, University of Copenhagen , Jagtvej 128, 2200 Copenhagen, Denmark; Astrophysics Research Centre, School of Mathematics and Physics, Queen’s University Belfast , Belfast BT7 1NN, UK
- K. Malanchev
- ORCiD
- Department of Astronomy, University of Illinois at Urbana-Champaign , 1002 W. Green Street, IL 61801, USA ; [email protected]; McWilliams Center for Cosmology, Department of Physics, Carnegie Mellon University , Pittsburgh, PA 15213, USA
- K. Auchettl
- ORCiD
- Department of Astronomy and Astrophysics, University of California , Santa Cruz, CA 95064, USA; School of Physics, The University of Melbourne , VIC 3010, Australia
- V. F. Baldassare
- ORCiD
- Department of Physics & Astronomy, Washington State University , Pullman, WA 99164, USA
- A. Berres
- ORCiD
- Department of Astronomy, University of Illinois at Urbana-Champaign , 1002 W. Green Street, IL 61801, USA ; [email protected]
- T. J. L. de Boer
- ORCiD
- Institute for Astronomy, University of Hawaii , 2680 Woodlawn Drive, Honolulu, HI 96822, USA
- B. M. Boyd
- ORCiD
- Institute of Astronomy and Kavli Institute for Cosmology , Madingley Road, Cambridge, CB3 0HA, UK
- K. C. Chambers
- ORCiD
- Institute for Astronomy, University of Hawaii , 2680 Woodlawn Drive, Honolulu, HI 96822, USA
- K. W. Davis
- ORCiD
- Department of Astronomy and Astrophysics, University of California , Santa Cruz, CA 95064, USA
- N. Esquivel
- ORCiD
- NSF’s National Optical-Infrared Astronomy Research Laboratory , 950 North Cherry Avenue, Tucson, AZ 85719, USA
- D. Farias
- ORCiD
- DARK, Niels Bohr Institute, University of Copenhagen , Jagtvej 128, 2200 Copenhagen, Denmark
- R. J. Foley
- ORCiD
- Department of Astronomy and Astrophysics, University of California , Santa Cruz, CA 95064, USA
- A. Gagliano
- ORCiD
- The NSF AI Institute for Artificial Intelligence and Fundamental Interactions , USA; Center for Astrophysics—Harvard & Smithsonian , Cambridge, MA 02138, USA
- C. Gall
- ORCiD
- DARK, Niels Bohr Institute, University of Copenhagen , Jagtvej 128, 2200 Copenhagen, Denmark
- H. Gao
- ORCiD
- Institute for Astronomy, University of Hawaii , 2680 Woodlawn Drive, Honolulu, HI 96822, USA
- S. Gomez
- ORCiD
- Space Telescope Science Institute , Baltimore, MD 21218, USA
- M. Grayling
- ORCiD
- Institute of Astronomy and Kavli Institute for Cosmology , Madingley Road, Cambridge, CB3 0HA, UK
- D. O. Jones
- ORCiD
- Institute for Astronomy, University of Hawai’i , 640 N. A’ohoku Place, Hilo, HI 96720, USA
- C.-C. Lin
- ORCiD
- Institute for Astronomy, University of Hawaii , 2680 Woodlawn Drive, Honolulu, HI 96822, USA
- E. A. Magnier
- ORCiD
- Institute for Astronomy, University of Hawaii , 2680 Woodlawn Drive, Honolulu, HI 96822, USA
- K. S. Mandel
- ORCiD
- Institute of Astronomy and Kavli Institute for Cosmology , Madingley Road, Cambridge, CB3 0HA, UK
- T. Matheson
- ORCiD
- NSF’s National Optical-Infrared Astronomy Research Laboratory , 950 North Cherry Avenue, Tucson, AZ 85719, USA
- S. I. Raimundo
- ORCiD
- DARK, Niels Bohr Institute, University of Copenhagen , Jagtvej 128, 2200 Copenhagen, Denmark; Department of Physics and Astronomy, University of Southampton , Highfield, Southampton SO17 1BJ, UK
- V. G. Shah
- ORCiD
- Department of Astronomy, University of Illinois at Urbana-Champaign , 1002 W. Green Street, IL 61801, USA ; [email protected]
- M. D. Soraisam
- ORCiD
- NSF’s National Optical-Infrared Astronomy Research Laboratory , Gemini North, 670 N. A'ohoku Place, Hilo, HI 96720, USA
- K. M. de Soto
- ORCiD
- Center for Astrophysics—Harvard & Smithsonian , Cambridge, MA 02138, USA
- S. Vicencio
- ORCiD
- NSF’s National Optical-Infrared Astronomy Research Laboratory , 950 North Cherry Avenue, Tucson, AZ 85719, USA
- V. A. Villar
- ORCiD
- Center for Astrophysics—Harvard & Smithsonian , Cambridge, MA 02138, USA
- R. J. Wainscoat
- ORCiD
- Institute for Astronomy, University of Hawaii , 2680 Woodlawn Drive, Honolulu, HI 96822, USA
- DOI
- https://doi.org/10.3847/1538-4357/ad6869
- Journal volume & issue
-
Vol. 974,
no. 2
p. 172
Abstract
We present Lightcurve Anomaly Identification and Similarity Search ( LAISS ), an automated pipeline to detect anomalous astrophysical transients in real-time data streams. We deploy our anomaly detection model on the nightly Zwicky Transient Facility (ZTF) Alert Stream via the ANTARES broker, identifying a manageable ∼1–5 candidates per night for expert vetting and coordinating follow-up observations. Our method leverages statistical light-curve and contextual host galaxy features within a random forest classifier, tagging transients of rare classes ( spectroscopic anomalies), of uncommon host galaxy environments ( contextual anomalies), and of peculiar or interaction-powered phenomena ( behavioral anomalies). Moreover, we demonstrate the power of a low-latency (∼ms) approximate similarity search method to find transient analogs with similar light-curve evolution and host galaxy environments. We use analogs for data-driven discovery, characterization, (re)classification, and imputation in retrospective and real-time searches. To date, we have identified ∼50 previously known and previously missed rare transients from real-time and retrospective searches, including but not limited to superluminous supernovae (SLSNe), tidal disruption events, SNe IIn, SNe IIb, SNe I-CSM, SNe Ia-91bg-like, SNe Ib, SNe Ic, SNe Ic-BL, and M31 novae. Lastly, we report the discovery of 325 total transients, all observed between 2018 and 2021 and absent from public catalogs (∼1% of all ZTF Astronomical Transient reports to the Transient Name Server through 2021). These methods enable a systematic approach to finding the “needle in the haystack” in large-volume data streams. Because of its integration with the ANTARES broker, LAISS is built to detect exciting transients in Rubin data.
Keywords
- Supernovae
- Transient detection
- Astronomical methods
- Time domain astronomy
- Time series analysis
- Astrostatistics techniques
