Deepfake detection method based on patch-wise lighting inconsistency

Abstract

The rapid development and widespread dissemination of deepfake techniques has caused increased concern.The malicious application of deepfake techniques also poses a potential threat to the society.Therefore, how to detect deepfake content has become a popular research topic.Most of the previous deepfake detection algorithms focused on capturing subtle forgery traces at pixel level and have achieved some results.However, most of the deepfake algorithms ignore the lighting information before and after generation, resulting in some lighting inconsistency between the original face and the forged face, which provided the possibility of using lighting inconsistency to detect deepfake.A corresponding algorithm was designed from two perspectives: introducing lighting inconsistency information and designing a network structure module for a specific task.For the introduction of lighting task, a new network structure was derived by designing the corresponding channel fusion method to provide more lighting inconsistency information to the network feature extraction layer.In order to ensure the portability of the network structure, the process of feature channel fusion was placed before the network extraction information, so that the proposed method can be fully planted to common deepfake detection networks.For the design of the network structure, a deepfake detection method was proposed for lighting inconsistency based on patch-similarity from two perspectives: network structure and loss function design.For the network structure, based on the characteristic of inconsistency between the forged image tampering region and the background region, the extracted features were chunked in the network feature layer and the feature layer similarity matrix was obtained by comparing the patch-wise cosine similarity to make the network focus more on the lighting inconsistency.On this basis, based on the feature layer similarity matching scheme, an independent ground truth and loss function was designed for this task in a targeted manner by comparing the input image with the untampered image of this image for patch-wise authenticity.It is demonstrated experimentally that the accuracy of the proposed method is significantly improved for deepfake detection compared with the baseline method.

Keywords

• deepfake detection
• lighting inconsistency
• color constancy
• patch-similarity comparison