Sequential Data Processing for IMERG Satellite Rainfall Comparison and Improvement Using LSTM and ADAM Optimizer

Abstract

This study introduces a systematic methodology whereby different technologies were utilized to download, pre-process, and interactively compare the rainfall datasets from the Integrated Multi-Satellite Retrievals for Global Precipitation Mission (IMERG) satellite and rain gauges. To efficiently handle the large volume of data, we developed automated shell scripts for downloading IMERG data and storing it, along with rain gauge data, in a relational database system. Hypertext pre-processor (pHp) programs were built to visualize the result for better analysis. In this study, the performance of IMERG estimations over the east coast of Peninsular Malaysia for the duration of 10 years (2011–2020) against rain gauge observation data is evaluated. Moreover, this study aimed to improve the daily IMERG estimations with long short-term memory (LSTM) developed with Python. Findings show that the LSTM with Adaptive Moment Estimation (ADAM) optimizer trained against the mean square error (MSE) loss enhances the accuracy of satellite estimations. At the point-to-pixel scale, the correlation between satellite estimations and ground observations was increased by about 15%. The bias was reduced by 81–118%, MAE was reduced by 18–59%, the root-mean-square error (RMSE) was reduced by 1–66%, and the Kling–Gupta efficiency (KGE) was increased by approximately 200%. The approach developed in this study establishes a comprehensive and scalable data processing and analysis pipeline that can be applied to diverse datasets and regions encountering similar domain-specific challenges.

Keywords

• IMERG satellite rainfall
• SQL relational database
• LSTM
• ADAM
• Python
• pHp