Virtual Restoration of Ancient Mold-Damaged Painting Based on 3D Convolutional Neural Network for Hyperspectral Image

Abstract

Painted cultural relics hold significant historical value and are crucial in transmitting human culture. However, mold is a common issue for paper or silk-based relics, which not only affects their preservation and longevity but also conceals the texture, patterns, and color information, hindering cultural value and heritage. Currently, the virtual restoration of painting relics primarily involves filling in the RGB based on neighborhood information, which might cause color distortion and other problems. Another approach considers mold as noise and employs maximum noise separation for its removal; however, eliminating the mold components and implementing the inverse transformation often leads to more loss of information. To effectively acquire virtual restoration for mold removal from ancient paintings, the spectral characteristics of mold were analyzed. Based on the spectral features of mold and the cultural relic restoration philosophy of maintaining originality, a 3D CNN artifact restoration network was proposed. This network is capable of learning features in the near-infrared spectrum (NIR) and spatial dimensions to reconstruct the reflectance of visible spectrum, achieving the virtual restoration for mold removal of calligraphic and art relics. Using an ancient painting from the Qing Dynasty as a test subject, the proposed method was compared with the Inpainting, Criminisi, and inverse MNF transformation methods across three regions. Visual analysis, quantitative evaluation (the root mean squared error (RMSE), mean absolute percentage error (MAPE), mean absolute error (MEA), and a classification application were used to assess the restoration accuracy. The visual results and quantitative analyses demonstrated that the proposed 3D CNN method effectively removes or mitigates mold while restoring the artwork to its authentic color in various backgrounds. Furthermore, the color classification results indicated that the images restored with 3D CNN had the highest classification accuracy, with overall accuracies of 89.51%, 92.24%, and 93.63%, and Kappa coefficients of 0.88, 0.91, and 0.93, respectively. This research provides technological support for the digitalization and restoration of cultural artifacts, thereby contributing to the preservation and transmission of cultural heritage.

Keywords

• hyperspectral remote sensing
• ancient paintings’ mold
• mold spectral characteristic
• restoration
• 3D CNN