A Kernel Rootkit Detection Approach Based on Virtualization and Machine Learning

Abstract

OS kernel is the core part of the operating system, and it plays an important role for OS resource management. A popular way to compromise OS kernel is through a kernel rootkit (i.e., malicious kernel module). Once a rootkit is loaded into the kernel space, it can carry out arbitrary malicious operations with high privilege. To defeat kernel rootkits, many approaches have been proposed in the past few years. However, existing methods suffer from some limitations: 1) most methods focus on user-mode rootkit detection; 2) some methods are limited to detect obfuscated kernel modules; and 3) some methods introduce significant performance overhead. To address these problems, we propose VKRD, a kernel rootkit detection system based on the hardware assisted virtualization technology. Compared with previous methods, VKRD can provide a transparent and an efficient execution environment for the target kernel module to reveal its run-time behavior. To select the important run-time features for training our detection models, we utilize the TF-IDF method. By combining the hardware assisted virtualization and machine learning techniques, our kernel rootkit detection solution could be potentially applied in the cloud environment. The experiments show that our system can detect windows kernel rootkits with high accuracy and moderate performance cost.

Keywords

• OS
• kernel rootkit
• virtualization
• machine learning