Facial Physical Adversarial Example Performance Prediction Algorithm Based on Multi-modal Feature Fusion

Abstract

Facial physical adversarial attack(FPAA) refers to a method that an attacker pasting or wearing physical adversary examples,such as printed glasses,paper,to make the face recognition system to recognize his face as the face of a specific target,or make the face recognition system unable to recognize his face under the camera.The existing performance evaluation process of the FPAA can be affected by multiple environmental factors and require multiple manual operations,resulting in very low efficiency of performance evaluation.In order to reduce the workload of evaluating the performance of facial physical adversarial examples,combined with the multimodality between digital images and environmental factors,a multimodal feature fusion prediction algorithm(MFFP) is proposed.Specifically,different networks are used to extract the features of attacker's face images,victim's face images and facial digital adversarialexample images,and the proposed environmental feature extraction network is used to extract the features of environmental factors.A multimodal feature fusion network is proposed to fuse these features.The output of the multimodal feature fusion network is the cosine similarity performance between the predicted facial physical adversarial example image and the victim image.MFFP algorithm achieves a regression mean square error of 0.003 under the experimental scenario of unknown environment and unknown FPAA,which is better than the performance of the baseline.It verifies the accuracy of MFFP algorithm for predicting of the performance of FPAA.Moreover,it verifies that MFFP can quickly evaluate the performance of FPAA,while greatly reduce the workload of manual operation.

Keywords

• artificial intelligence security|adversarial example|facial physical adversarial attack|performance prediction|multimodal feature fusion