Approaches to detection of malware using custom network protocols

Abstract

The majority of malware communicate with their components over the network. The purpose of this interaction can be downloading the necessary modules, transferring stolen information, etc. Often such interaction is based on the most common protocols - HTTP and HTTPS, but there are quite a lot of facts that malware creators implement their own protocols. The purpose of this study is to analyze approaches to implementing custom protocols that exist among malware developers and methods for its detecting in network traffic. Possible reasons that encourage malware creators to do this are described. A brief overview of the variants of such protocols, from primitive to very complex, is given, including examples. The paper also describes some approaches for detection such activity in network traffic: signature analysis, methods based on statistics and machine learning, neural networks. The main advantages and disadvantages of these methods are considered. The result of the analysis of these methods is the conclusion that the most promising approach to this problem is the use of neural networks, but when implementing neural networks in practice, certain difficulties may arise associated with their training and further support, which most likely will need to be overcome on their own.

Keywords

• malware, network protocols, encryption, detection, neural networks.