Neural network approach to ensuring visual coherence in augmented reality flight simulators

Abstract

In 2023, the leading US aerospace corporation Lockheed Martin announced the simultaneous development of several extended/augmented reality (XR/AR) simulators for pilots of TF-50, F-16, F-22, and F-35 without being a pioneer in this area of focus, in 2022 similar projects were launched by Boeing and the leading British aeronautical equipment manufacturer BAE Systems. In January 2024 the US Air Force invested in the development of pilot AR simulators based on Microsoft Hololens augmented reality smart glasses. At the same time, Apple began bulk sales of the Apple Vision Pro AR headset. It is difficult to doubt that in 2024 a variety of new aviation simulators will appear using this device. The rapid development of a new generation of aerospace simulator technology, i.e., XR/AR simulators, is accompanied by a boom in research in the field of visual coherence (VC) of augmented reality scenes: virtual objects in these scenes should be virtually identical with real ones. It is VC that provides new capabilities of AR simulators, which fundamentally distinguish from conventional flight simulators with virtual reality. Recently, VC has been increasingly provided by neural network methods, thereby, the most important aspects of VC are lighting conditions, so the major share of research is focused on transferring these conditions (location of light sources and their color tone) from the real world to the virtual one, but the great body of the known approaches are characterized by the lack of versatility and the need to perform manual procedures. These disadvantages are not found in the spectral transplantation method based on twodimensional spectral image conversions, which, however, requires determining the size of the spectrum part being transplanted from the real picture of the world to a virtual object. This article is devoted to the development of a neural network model for the mechanism of selecting the optimal size of a spectral transplant.

Keywords

• flight simulators
• augmented reality
• visual coherence