Application of attention mechanism-based LSTM neural network in stratigraphy identification

Abstract

This paper proposes a lithology recognition system based on a Long Short-Term Memory (LSTM) neural network, which incorporates multi-head masked attention mechanisms and the Transformer architecture. Initially, an equipment-formation coupling model is established based on contact mechanics and rock mechanics to determine the foundation for formation classification. Furthermore, a data collection strategy is proposed to ensure the real-time data collection at construction sites, thereby guaranteeing the authenticity and reliability of the data. To improve prediction accuracy, an advanced data cleaning method is also developed to eliminate invalid data. Additionally, the LSTM neural network combines multi-head attention mechanisms and the Transformer architecture, further enhancing the efficiency and accuracy of formation prediction. The Transformer architecture consists of an encoder and a decoder, where the encoder uses multiple Transformer encoder layers to learn time-series features based on time-window shifts, and leverages multi-head masked attention mechanisms to better capture key features in the input sequence. Subsequently, the decoder combines the time-series features extracted from each Transformer encoder layer using a one-dimensional Convolutional Neural Network (1D-CNN) and inputs them into the LSTM neural network for formation prediction. The validation results show that the proposed LSTM model achieves up to 98% accuracy in complex formation recognition tasks, outperforming other machine learning models such as Random Forest and Support Vector Machine (SVM). The model excels in identifying complex formation boundaries.

Keywords

• Rotary drilling rig
• Stratigraphy identification
• Recurrent neural network
• Long short term memory neural network
• Multi-head mask attention
• Transformer architecture