Statistical regularization for trend detection: an integrated approach for detecting long-term trends from sparse tropospheric ozone profiles

Abstract

Detecting a tropospheric ozone trend from sparsely sampled ozonesonde profiles (typically once per week) is challenging due to the short-lived anomalies in the time series resulting from ozone's high temporal variability. To enhance trend detection, we have developed a sophisticated statistical approach that utilizes a geoadditive model to assess ozone variability across a time series of vertical profiles. Treating the profile time series as a set of individual time series on discrete pressure surfaces, a class of smoothing spline ANOVA (analysis of variance) models is used for the purpose of jointly modeling multiple correlated time series (on separate pressure surfaces) by their associated seasonal and interannual variabilities. This integrated fit method filters out the unstructured variation through a statistical regularization (i.e., a roughness penalty) by taking advantage of the additional correlated data points available on the pressure surfaces above and below the surface of interest. We have applied this technique to the trend analysis of the vertically correlated time series of tropospheric ozone observations from (1) IAGOS (In-service Aircraft for a Global Observing System) commercial aircraft profiles above Europe and China throughout 1994–2017 and (2) NOAA GML's (Global Monitoring Laboratory) ozonesonde records at Hilo, Hawaii, (1982–2018) and Trinidad Head, California (1998–2018). We illustrate the ability of this technique to detect a consistent trend estimate and its effectiveness in reducing the associated uncertainty in the profile data due to the low sampling frequency. We also conducted a sensitivity analysis of frequent IAGOS profiles above Europe (approximately 120 profiles per month) to determine how many profiles in a month are required for reliable long-term trend detection. When ignoring the vertical correlation, we found that a typical sampling strategy (i.e. four profiles per month) might result in 7 % of sampled trends falling outside the 2σ uncertainty interval derived from the full dataset with an associated 10 % of mean absolute percentage error. Based on a series of sensitivity studies, we determined optimal sampling frequencies for (1) basic trend detection and (2) accurate quantification of the trend. When applying the integrated fit method, we find that a typical sampling frequency of four profiles per month is adequate for basic trend detection; however, accurate quantification of the trend requires 14 profiles per month. Accurate trend quantification can be achieved with only 10 profiles per month if a regular sampling frequency is applied. In contrast, the standard separated fit method, which ignores the vertical correlation between pressure surfaces, requires 8 profiles per month for basic trend detection and 18 profiles per month for accurate trend quantification. While our method improves trend detection from sparse datasets, the key to substantially reducing the uncertainty is to increase the sampling frequency.