New Forecasting Metrics Evaluated in Prophet, Random Forest, and Long Short-Term Memory Models for Load Forecasting

Abstract

Data mining is vital for smart grids because it enhances overall grid efficiency, enabling the analysis of large volumes of data, the optimization of energy distribution, the identification of patterns, and demand forecasting. Several performance metrics, such as the MAPE and RMSE, have been created to assess these forecasts. This paper presents new performance metrics called Evaluation Metrics for Performance Quantification (EMPQ), designed to evaluate forecasting models in a more comprehensive and detailed manner. These metrics fill the gap left by established metrics by assessing the likelihood of over- and under-forecasting. The proposed metrics quantify forecast bias through maximum and minimum deviation percentages, assessing the proximity of predicted values to actual consumption and differentiating between over- and under-forecasts. The effectiveness of these metrics is demonstrated through a comparative analysis of short-term load forecasting for residential customers in Dubai. This study was based on high-resolution smart meter data, weather data, and voluntary survey data of household characteristics, which permitted the subdivision of the customers into several groups. The new metrics were demonstrated on the Prophet, Random Forest (RF), and Long Short-term Memory (LSTM) models. EMPQ help to determine that the LSTM model exhibited a superior performance with a maximum deviation of approximately 10% for day-ahead and 20% for week-ahead forecasts in the “AC-included” category, outperforming the Prophet model, which had deviation rates of approximately 44% and 42%, respectively. EMPQ also help to determine that the RF excelled over LSTM for the ‘bedroom-number’ subcategory. The findings highlight the value of the proposed metrics in assessing model performance across diverse subcategories. This study demonstrates the value of tailored forecasting models for accurate load prediction and underscores the importance of enhanced performance metrics in informing model selection and supporting energy management strategies.

Keywords

• smart grid
• load forecasting
• machine learning
• deep learning
• time series
• performance metrics