VISO: Characterizing malicious behaviors of virtual machines with unsupervised clustering

Abstract

Cloud computing has become one of the most dominant computation platforms in recent years. Security threats could be one of the major stunning blocks on this evolution road. While system vendors and cloud tenants benefit much from sharing resources in the cloud environment, security breaches can cause more significant damages of the cloud ecosystem than personal computers. Virtualization techniques facilitate the movement of intrusion detection system to cloud-host operating systems with virtual machine management by observing behaviors of virtual machines (VMs). However, a VM-based detection system inherits the semantic gap problem: it is needed the ability to reveal (malicious) behaviors of VMs from observed data. We propose an automatic and systematic analysis framework for charactering malware behaviors using unsupervised clustering. This framework consists of three phases: (1) unsupervised clustering on behaviors of VMs, (2) supervised classification rule derivation, and (3) online system detection. Specifically, we collect and cluster system call distributions of VMs within a small period as samples, identify clusters that contain only samples from malicious VMs, and derive detection rules by extracting features of these malicious clusters. VMs that have been observed their system call distributions falling into a malicious cluster are considered to be malicious. We have integrated the presented framework with OpenStack and develop a prototype online monitoring system, called VISO. We conduct several experiments against common attacks, showing the effectiveness of VISO on clustering, classifying and detecting malicious behaviors of VMs.