aws-s3-integrity-check: an open-source bash tool to verify the integrity of a dataset stored on Amazon S3

Sonia García-Ruiz; Regina Hertfelder Reynolds; Melissa Grant-Peters; Emil Karl Gustavsson; Aine Fairbrother-Browne; Zhongbo Chen; Jonathan William Brenton; Mina Ryten

doi:10.46471/gigabyte.87

GigaByte (Aug 2023)

aws-s3-integrity-check: an open-source bash tool to verify the integrity of a dataset stored on Amazon S3

Sonia García-Ruiz ,
Regina Hertfelder Reynolds ,
Melissa Grant-Peters ,
Emil Karl Gustavsson ,
Aine Fairbrother-Browne ,
Zhongbo Chen ,
Jonathan William Brenton ,
Mina Ryten

Affiliations

Sonia García-Ruiz: ORCiD; Department of Genetics and Genomic Medicine Research & Teaching, UCL GOS Institute of Child Health, London, UK, NIHR Great Ormond Street Hospital Biomedical Research Centre, University College London, London, UK
Regina Hertfelder Reynolds: ORCiD; Department of Genetics and Genomic Medicine Research & Teaching, UCL GOS Institute of Child Health, London, UK, NIHR Great Ormond Street Hospital Biomedical Research Centre, University College London, London, UK
Melissa Grant-Peters: ORCiD; Department of Genetics and Genomic Medicine Research & Teaching, UCL GOS Institute of Child Health, London, UK, NIHR Great Ormond Street Hospital Biomedical Research Centre, University College London, London, UK
Emil Karl Gustavsson: ORCiD; Department of Genetics and Genomic Medicine Research & Teaching, UCL GOS Institute of Child Health, London, UK, NIHR Great Ormond Street Hospital Biomedical Research Centre, University College London, London, UK
Aine Fairbrother-Browne: ORCiD; Department of Genetics and Genomic Medicine Research & Teaching, UCL GOS Institute of Child Health, London, UK, Department of Medical and Molecular Genetics, School of Basic and Medical Biosciences, King’s College London, London, UK, Department of Neurodegenerative Disease, Queen Square Institute of Neurology, UCL, London, UK
Zhongbo Chen: ORCiD; Department of Genetics and Genomic Medicine Research & Teaching, UCL GOS Institute of Child Health, London, UK, Department of Neurodegenerative Disease, Queen Square Institute of Neurology, UCL, London, UK
Jonathan William Brenton: ORCiD; Department of Genetics and Genomic Medicine Research & Teaching, UCL GOS Institute of Child Health, London, UK, NIHR Great Ormond Street Hospital Biomedical Research Centre, University College London, London, UK
Mina Ryten: ORCiD; Department of Genetics and Genomic Medicine Research & Teaching, UCL GOS Institute of Child Health, London, UK, NIHR Great Ormond Street Hospital Biomedical Research Centre, University College London, London, UK

DOI: https://doi.org/10.46471/gigabyte.87

Abstract

Read online

Amazon Simple Storage Service (Amazon S3) is a widely used platform for storing large biomedical datasets. Unintended data alterations can occur during data writing and transmission, altering the original content and generating unexpected results. However, no open-source and easy-to-use tool exists to verify end-to-end data integrity. Here, we present aws-s3-integrity-check, a user-friendly, lightweight, and reliable bash tool to verify the integrity of a dataset stored in an Amazon S3 bucket. Using this tool, we only needed ∼114 min to verify the integrity of 1,045 records ranging between 5 bytes and 10 gigabytes and occupying ∼935 gigabytes of the Amazon S3 cloud. Our aws-s3-integrity-check tool also provides file-by-file on-screen and log-file-based information about the status of each integrity check. To our knowledge, this tool is the only open-source one that allows verifying the integrity of a dataset uploaded to the Amazon S3 Storage quickly, reliably, and efficiently. The tool is freely available for download and use at https://github.com/SoniaRuiz/aws-s3-integrity-check and https://hub.docker.com/r/soniaruiz/aws-s3-integrity-check.

Published in GigaByte

ISSN: 2709-4715 (Online)
Publisher: GigaScience Press
Country of publisher: Hong Kong
LCC subjects: Science: Mathematics: Instruments and machines: Electronic computers. Computer science
Website: https://gigabytejournal.com/

About the journal