This paper proposes an anomalous behavior detection model based on cloud computing. Virtual Machines (VMs) are one of the key components of cloud Infrastructure as a Service (IaaS). The security of such VMs is critical to IaaS security. Many studies have been done on cloud computing security issues, but research into VM security issues, especially regarding VM network traffic anomalous behavior detection, remains inadequate. More and more studies show that communication among internal nodes exhibits complex patterns. Communication among VMs in cloud computing is invisible. Researchers find such issues challenging, and few solutions have been proposed—leaving cloud computing vulnerable to network attacks. This paper proposes a model that uses Software-Defined Networks (SDN) to implement traffic redirection. Our model can capture inter-VM traffic, detect known and unknown anomalous network behaviors, adopt hybrid techniques to analyze VM network behaviors, and control network systems. The experimental results indicate that the effectiveness of our approach is greater than 90%, and prove the feasibility of the model.

Network security requirements based on virtual network technologies in IaaS platforms and corresponding solutions were reviewed. A dynamic network security architecture was proposed, which was built on the technologies of software defined networking, Virtual Machine (VM) traffic redirection, network policy unified management, software defined isolation networks, vulnerability scanning, and software updates. The proposed architecture was able to obtain the capacity for detection and access control for VM traffic by redirecting it to configurable security appliances, and ensured the effectiveness of network policies in the total life cycle of the VM by configuring the policies to the right place at the appropriate time, according to the impacts of VM state transitions. The virtual isolation domains for tenants’ VMs could be built flexibly based on VLAN policies or Netfilter/Iptables firewall appliances, and vulnerability scanning as a service and software update as a service were both provided as security supports. Through cooperation with IDS appliances and automatic alarm mechanisms, the proposed architecture could dynamically mitigate a wide range of network-based attacks. The experimental results demonstrate the effectiveness of the proposed architecture.