Near Real-time Map Building with Multi-class Image Set Labeling and Classification of Road Conditions Using Convolutional Neural Networks

Abstract

Road Weather Information Systems (RWIS) provide real-time weather information at point locations and are often used to produce road weather forecasts and provide input for pavement forecast models. Compared to the prevalant street cameras, however, RWIS are sometimes limited in availability. Thus, extraction of road conditions data by computer vision can provide a complementary observational data source if it can be done quickly and on large scales. In this paper, we leverage state-of-the-art convolutional neural networks (CNN) in labeling images taken by street and highway cameras located across North America. The final training set included 47,000 images labeled with five classes: dry, wet, snow/ice, poor, and offline. The experiments tested different configurations of six CNNs. The EfficientNet-B4 framework was found to be most suitable to this problem, achieving validation accuracy of 90.6%, although EfficientNet-B0 achieved an accuracy of 90.3% with half the execution time. The classified images were then used to construct a map showing real-time road conditions at various camera locations. The proposed approach is presented in three parts: i) application pipeline, ii) description of the deep learning frameworks, iii) the dataset labeling process and the classification metrics.