A Neural Networks Approach to Detecting Lost Heritage in Historical Video

Abstract

Documenting Cultural Heritage through the extraction of 3D measures with photogrammetry is fundamental for the conservation of the memory of the past. However, when the heritage has been lost the only way to recover this information is the use of historical images from archives. The aim of this study is to experiment with new ways to search for architectural heritage in video material and to save the effort of the operator in the archive in terms of efficiency and time. A workflow is proposed to automatically detect lost heritage in film footage using Deep Learning to find suitable images to process with photogrammetry for its 3D virtual reconstruction. The performance of the network was tested on two case studies considering different architectural scenarios, the Tour Saint Jacques which still exists for the tuning of the networks, and Les Halles to test the algorithms on a real case of an architecture which has been destroyed. Despite the poor quantity and low quality of the historical images available for the training of the network, it has been demonstrated that, with few frames, it was possible to reach the same results in terms of performance of a network trained on a large dataset. Moreover, with the introduction of new metrics based on time intervals the measure of the real time saving in terms of human effort was achieved. These findings represent an important innovation in the documentation of destroyed monuments and open new ways to recover information about the past.

Keywords

• machine learning
• deep learning
• neural networks
• object detection
• video processing and classification
• photogrammetry