Spatial–Temporal Transformer Networks for Traffic Flow Forecasting Using a Pre-Trained Language Model

Abstract

Most current methods use spatial–temporal graph neural networks (STGNNs) to analyze complex spatial–temporal information from traffic data collected from hundreds of sensors. STGNNs combine graph neural networks (GNNs) and sequence models to create hybrid structures that allow for the two networks to collaborate. However, this collaboration has made the model increasingly complex. This study proposes a framework that relies solely on original Transformer architecture and carefully designs embeddings to efficiently extract spatial–temporal dependencies in traffic flow. Additionally, we used pre-trained language models to enhance forecasting performance. We compared our new framework with current state-of-the-art STGNNs and Transformer-based models using four real-world traffic datasets: PEMS04, PEMS08, METR-LA, and PEMS-BAY. The experimental results demonstrate that our framework outperforms the other models in most metrics.

Keywords

• traffic flow forecasting
• spatial–temporal dependency
• Transformer
• LLMs