Rethinking the Weakness of Stream Ciphers and Its Application to Encrypted Malware Detection

Abstract

One critical vulnerability of stream ciphers is the reuse of an encryption key. Since most stream ciphers consist of only a key scheduling algorithm and an Exclusive OR (XOR) operation, an adversary may break the cipher by XORing two captured ciphertexts generated under the same key. Various cryptanalysis techniques based on this property have been introduced in order to recover plaintexts or encryption keys; in contrast, this research reinterprets the vulnerability as a method of detecting stream ciphers from the ciphertexts it generates. Patterns found in the values (characters) expressed across the bytes of a ciphertext make the ciphertext distinguishable from random and are unique to each combination of ciphers and encryption keys. We propose a scheme that uses these patterns as a fingerprint, which is capable of detecting all ciphertexts of a given length generated by an encryption pair. The scheme can be utilized to detect a specific type of malware that exploits a stream cipher with a stored key such as the DarkComet Remote Access Trojan (RAT). We show that our scheme achieves 100% accuracy for messages longer than 13 bytes in about 17 μsec, providing a fast and highly accurate tool to aid in encrypted malware detection.

Keywords

• Encryption
• Intrusion Detection
• malware
• network security
• stream ciphers