Using Barlow Twins to Create Representations From Cloud-Corrupted Remote Sensing Time Series

Abstract

Satellite-based monitoring is a key tool for supporting global food security and natural resource management but is challenged by cloud corruption and lack of labeled training data. To address these issues, self-supervised learning (SSL) techniques have been developed that first learn representations from almost limitless available unlabeled data, before using labeled samples for a specific downstream task. As the learned representations detect, integrate, and compress information in the dataset in a fully unsupervised manner, the downstream tasks require only small labeled datasets. In this study, we present spectral–temporal Barlow Twins (ST-BT), a new pixelwise SSL architecture that generates useful representations designed to be invariant to extensive cloudiness. We demonstrate that ST-BT representations enable stable and high F1 scores on the downstream task of crop classification even with cloud cover reaching 50% of available dates and using only a few labeled samples. The ST-BT representations achieve maximum F1 scores of 0.94 and 0.90 on the two benchmark classification datasets used. These results indicate that ST-BT can create useful representations of pixelwise multispectral Sentinel-2 timeseries despite cloud corruption.

Keywords

• Crops
• remote sensing
• self-supervised learning
• time series analysis