Web Application Page Element Recognition and Visual Script Generation Based on Machine Vision

Abstract

In order to provide richer interactive response effect,the visualization elements of the Web application is becoming more complex and diverse.The traditional test based on DOM cannot match the new requirement to test Web application.A new generation test based on computer vision provides an efficient way for the complex elements in Web application.This test for the Web based on computer vision mainly depends on template matching technique to recognize the page elements,so that it can ge-nerate visualization test script.However,the subtle changes of the page elements’ appearance can lead to the failure of template matching technique,so that the Web page elements cannot be recognized and the visualization test script cannot be generated.How to improve the accuracy of Web page element recognition based on machine vision and make it still applicable in complex conditions is a challenging task.Object detection based on deep learning is a research hotspot in the field of today’s computer vision and machine learning.It has been shown from the deep data characteristics gained through the large sample learning that it can recognize the target accurately and has stronger generalization ability.Therefore,this paper targets the Web application,starts from the visual characteristics of the page elements,and proposes a Web page elements recognition method based on deep lear-ning.This method uses YOLOv3 algorithmic structure model based on the target testing to automatically localize the position and boundary of an element,recognize the type of Web page elements as well as describe its function.On the base of the Web page elements recognition,it can automatically generate visualization test script for the Web application.To verify the accuracy of the page elements recognition based on computer vision,experiments are set to test between different versions of the same Web application,and between different Web applications to analyze its accuracy.The results show that the average recall rate of machine vision-based Web page element recognition model is 75.6%.It provides foundation for the Web application’s visualization test script.

Keywords

• web application testing|web application page element recognition|visual test script|automatic generation of test script|machine vision