A user-oriented forecast verification metric: Weighted Percent Correct

Abstract

The simple skill measure known as Percent Correct (PC) has been widely used in Australian climate services to measure the accuracy of public forecasts of the probability of above or below median conditions. Since it is simple and intuitive, it has been well accepted by users who are not familiar with the complexities of forecast verification. However, a major disadvantage of PC is that it only measures the categorical outcome of events and ignores the magnitude of anomalies in both the forecast distributions and observed outcomes. In this paper, we introduce the Weighted Percent Correct (WPC) verification metric. Instead of just focusing on the categorical outcome, WPC includes the magnitude of the observed anomaly (departure from the median for above/below median forecasts), which is then applied as a weighting to the PC score. The WPC has many similarities to the PC, with a score of 100 % indicating a perfect forecast, and a score of 50 % or below indicating an unskilled forecast (as applied to above/below median). The advantage of the WPC for users is that it downplays very small anomalies in the observations, but highlights areas with significant anomalies that were correctly forecast. This is a particular advantage for places like Australia, where large areas are seasonally dry, and most anomalies during those dry seasons are so small that they are of little relevance to users and their applications. Some examples are given that show the advantages of WPC over PC, and to highlight how it is also intuitive, easy to use and simple to communicate – aspects that stakeholders need to make decisions.

Keywords

• verification
• probability forecast
• climate outlook
• skill
• percent consistency