Journal Home > Volume 18 , Issue 4
Tsinghua Science and Technology 2013, 18(4): 406-417 https://doi.org/10.1109/TST.2013.6574679
Open Access | Issue | Published: 05 August 2013

TIFAflow: Enhancing Traffic Archiving System with Flow Granularity for Forensic Analysis in Network Security

Show Author's Information Zhen Chen( ),  Lingyun Ruan,  Junwei Cao,  Yifan Yu,  Xin Jiang
Research Institute of Information Technology and Tsinghua National Laboratory for Information Science and Technology (TNList), Tsinghua University, Beijing 100084, China
Department of Automation, Research Institute of Information Technology and Tsinghua National Laboratory for Information Science and Technology (TNList), Tsinghua University, Beijing 100084, China
Department of Computer Science of Purdue University, West Lafayette, IN 47907, USA
Department of Electronic Engineering and Tsinghua National Laboratory for Information Science and Technology (TNList), Tsinghua University, Beijing 100084, China
Department of Computer Science and Technology, Research Institute of Information Technology and Tsinghua National Laboratory for Information Science and Technology (TNList), Tsinghua University, Beijing 100084, China
Keywords:
network security, cloud computing, forensic analysis, hadoop distributed file system, traffic archival, phishing attack, bitmap database, NoSQL
Cite this article:
Chen Z, Ruan L, Cao J, et al. TIFAflow: Enhancing Traffic Archiving System with Flow Granularity for Forensic Analysis in Network Security. Tsinghua Science and Technology, 2013, 18(4): 406-417. https://doi.org/10.1109/TST.2013.6574679
Download citation
EndNote(RIS)
BibTeX

407

Views

7

Downloads

Citations

10

Crossref

N/A

WoS

16

Scopus

0

CSCD

Abstract Full text About this article

The archiving of Internet traffic is an essential function for retrospective network event analysis and forensic computer communication. The state-of-the-art approach for network monitoring and analysis involves storage and analysis of network flow statistic. However, this approach loses much valuable information within the Internet traffic. With the advancement of commodity hardware, in particular the volume of storage devices and the speed of interconnect technologies used in network adapter cards and multi-core processors, it is now possible to capture 10 Gbps and beyond real-time network traffic using a commodity computer, such as n2disk. Also with the advancement of distributed file system (such as Hadoop, ZFS, etc.) and open cloud computing platform (such as OpenStack, CloudStack, and Eucalyptus, etc.), it is practical to store such large volume of traffic data and fully in-depth analyse the inside communication within an acceptable latency. In this paper, based on well-known TimeMachine, we present TIFAflow, the design and implementation of a novel system for archiving and querying network flows. Firstly, we enhance the traffic archiving system named TImemachine+FAstbit (TIFA) with flow granularity, i.e., supply the system with flow table and flow module. Secondly, based on real network traces, we conduct performance comparison experiments of TIFAflow with other implementations such as common database solution, TimeMachine and TIFA system. Finally, based on comparison results, we demonstrate that TIFAflow has a higher performance improvement in storing and querying performance than TimeMachine and TIFA, both in time and space metrics.


menu
Abstract
Full text
Outline
About this article

TIFAflow: Enhancing Traffic Archiving System with Flow Granularity for Forensic Analysis in Network Security

Show Author's information Zhen Chen( ), Lingyun Ruan, Junwei Cao, Yifan Yu, Xin Jiang
Research Institute of Information Technology and Tsinghua National Laboratory for Information Science and Technology (TNList), Tsinghua University, Beijing 100084, China
Department of Automation, Research Institute of Information Technology and Tsinghua National Laboratory for Information Science and Technology (TNList), Tsinghua University, Beijing 100084, China
Department of Computer Science of Purdue University, West Lafayette, IN 47907, USA
Department of Electronic Engineering and Tsinghua National Laboratory for Information Science and Technology (TNList), Tsinghua University, Beijing 100084, China
Department of Computer Science and Technology, Research Institute of Information Technology and Tsinghua National Laboratory for Information Science and Technology (TNList), Tsinghua University, Beijing 100084, China

Abstract

The archiving of Internet traffic is an essential function for retrospective network event analysis and forensic computer communication. The state-of-the-art approach for network monitoring and analysis involves storage and analysis of network flow statistic. However, this approach loses much valuable information within the Internet traffic. With the advancement of commodity hardware, in particular the volume of storage devices and the speed of interconnect technologies used in network adapter cards and multi-core processors, it is now possible to capture 10 Gbps and beyond real-time network traffic using a commodity computer, such as n2disk. Also with the advancement of distributed file system (such as Hadoop, ZFS, etc.) and open cloud computing platform (such as OpenStack, CloudStack, and Eucalyptus, etc.), it is practical to store such large volume of traffic data and fully in-depth analyse the inside communication within an acceptable latency. In this paper, based on well-known TimeMachine, we present TIFAflow, the design and implementation of a novel system for archiving and querying network flows. Firstly, we enhance the traffic archiving system named TImemachine+FAstbit (TIFA) with flow granularity, i.e., supply the system with flow table and flow module. Secondly, based on real network traces, we conduct performance comparison experiments of TIFAflow with other implementations such as common database solution, TimeMachine and TIFA system. Finally, based on comparison results, we demonstrate that TIFAflow has a higher performance improvement in storing and querying performance than TimeMachine and TIFA, both in time and space metrics.

Keywords: network security, cloud computing, forensic analysis, hadoop distributed file system, traffic archival, phishing attack, bitmap database, NoSQL

References(42)

[1]
B. Wardman, G. Shukla, and G. Warner, Identifying vulnerable websites by analysis of common strings in phishing URLs, in eCrime Researchers Summit, eCRIME ’09, Tacoma, USA, 2009.
DOI
[2]
S. Li and R. Schmitz, A novel anti-phishing framework based on honeypots, in eCrime Researchers Summit, eCRIME’09, Tacoma, USA, 2009.
[3]
R. Layton, P. Watters, and R. Dazeley, Automatically determining phishing campaigns using the USCAP methodology, in eCrime Researchers Summit (eCrime), 2010.
DOI
[4]
S. Sheng, B. Wardman, G. Warner, L. Cranor, J. Hong, and C. Zhang, An empirical analysis of phishing blacklists, in Proc. Sixth Conference on Email and AntiSpam (CEAS 2009), Mountain View, California, USA, 2009.
[5]
Z. Chen, C. Lin, U. Ni, D. H. Ruan, B. Zheng, Y. X. Jiang, X. H. Deng, Y. Wang, A. A. Luo, B. Zhu, Y. Yue, and F. Y. Ren, AntiWorm NPU-based parallel bloom filters for TCP/IP content processing in Giga-Ethernet LAN, in Local Computer Networks, 30th Anniversary, Sydney, Australia, 2005, pp. 748-755.
DOI
[6]
D. H. Ruan, Z. Chen, C. Lin, J. Ni, and P. D. Unysundn, Handling high speed traffic measurement using network processors, in Communication Technology, ICCT ’06, Guilin, 2006, pp. 1-5.
[7]
S. H. Huang, Z. Chen, A. A. Luo, X. Jiang, K. Wang, H. Zheng, and X. H. Peng, Proxy-based security audit system for remote desktop access, in Computer Communications and Networks, ICCCN 2009, San Francisco, USA, 2006, pp. 1095-2055.
[8]
C. Cranor, T. Johnson, and O. Spatscheck, Gigascope: A stream database for network applications, in 2003 ACM SIGMOD International Conference on Management of Data, New York, USA, 2003, pp. 647-651.
DOI
[9]
M. Sullivan and A. Heybey, Tribeca: A system for managing large databases of network traffic, in Proceedings of the USENIX Annual Technical Conference (NO 98), ew Orleans, Louisiana, June 1998.
[10]
L. Deri, V. Lorenzetti, and S. Mortimer, Collection and exploration of large data monitoring sets using bitmap databases, in Traffic Monitoring and Analysis(TMA), Jan 2010, pp. 73-86.
DOI
[11]
P. Desnoyers and P. Shenoy, Hyperion: High volume stream archival for retrospective querying, in USENIX Annual Technical Conference, Santa Clara, USA, 2007.
[12]
S. Kornexl, V. Paxson, H. Dreger, A. Feldmann, and R. Sommer, Building a time machine for efficient recording and retrieval of high-volume network traffic, in Proceedings of the 5th ACM SIGCOMM Conference on Internet Measurement, Berkeley, CA, USA, 2005, p. 23.
DOI
[13]
G. Maier, R. Sommer, H. Dreger, A. Feldmann, V. Paxson, and F. Schneider, Enriching network security analysis with time travel, in Proceedings of the ACM SIGCOMM 2008 Conference on Data Communication, New York, USA, 2008, pp. 183-194.
DOI
[14]
L. Deri, A. Cardigliano, and F. Fusco, 10 Gbit line rate packet-to-disk using n2disk, in Proc. of 2013 Traffic Monitoring and Analysis Workshop, Turin, Italy, 2013.
DOI
[15]
TimeMachine project in Bro, http://tracker.bro.org/time-machine/, 2013.
[16]
J. Li, S. Ding, M. Xu, F. Han, X. Guan, and Z. Chen, TIFA: Enabling real-time querying and storage of massive stream data, in Networking and Distributed Computing (ICNDC), 2011 Second International Conference on IEEE, Beijing, China, 2011.
DOI
[17]
K. Wu, E. W. Bethel, J. Chen, H. Childs, E. Cormier-Michel, C. Geddes, J. Gu, H. Hagen, B. Hamann, W. Koegler, J. Lauret, J. Meredith, P. Messmer, E. Otoo, V. Perevoztchikov, A. Poskanzer, , O. Rbel, A. Shoshani, A. Sim, K. Stockinger, G. Weber, and W.-M. Zhang, FastBit: Interactively searching massive data, in Proc. of SciDAC 2009, 2009.
DOI
[18]
B. Claise, Cisco systems netflow services export version 9, RFC 3954, 2004.
[19]
P. Phaal, S. Panchen, and N. McKee, InMon corporations sflow: A method for monitoring traffic in switched and routed networks, RFC 3176, 2001, pp. 1-31.
[20]
P. Haag, Watch your flows with nfsen and nfdump, in 50th RIPE Meeting, Stockholm, Sweden, 2005.
[21]
M. Fullmer and S. Roming, The OSU flowtools packetage and cisco netFlow logs, in Proc. of 19th Intl. Conference on Scientific and Statistical Database Management, Banff, Canada, 2007.
[22]
NEye, An open source netflow collector, http://neye.unsupported.info, 2004.
[23]
J. P. Navarro, B. Nickless, and L. Winkler, Combining cisco NetFlow exports with relational database technology for usage statistics, intrusion detection, and network forensics, in Proceedings of the 14th Large Installation Systems Administration Conference (LISA 2000), 2000, pp. 285-290.
[24]
F. Fusco, X. Dimitropoulos, and M. Vlachos, PcapIndex: An index for network packet traces with legacy compatibility, ACM SIGCOMM Computer Communication Review, vol. 42, no. 1, pp. 47-53, January 2012.
DOI Google Scholar
[25]
F. Fusco, M. Vlachos, X. Dimitropoulos, and L. Deri, Indexing million of packets per second using GPUs, in Proc. of the 13th ACM SIGCOMM Conference on Internet Measurement, IMC 2013, 2013.
DOI
[26]
F. Fusco and L. Deri, High speed network traffic analysis with commodity multi-core systems, in Proceedings of the 10th ACM SIGCOMM Conference on Internet Measurement, New York, USA, 2010, pp. 218-224.
DOI
[27]
Cavium Network Processor OCTEON 58XX, http://www.cavium.com/OCTEON-Plus_CN58XX.html, 2013.
[28]
J. Meng, X. Chen, Z. Chen, C. Lin, B. Mu, and L. Ruan, Towards high-performance ipsec on cavium OCTEON platform, in Trusted Systems, 2011, pp. 37-46.
DOI
[29]
Tilera manycore network processor, http://www.tilera.com/products/processors/TILEPRO64, 2013.
[30]
S. Ding, Z. Chen, and Z. Liu, Parallelizing FIB lookup in content centric networking, in Networking and Distributed Computing (ICNDC), Hangzhou, China, 2012, pp. 6-10.
DOI
[31]
V. Paxson, Bro: A system for detecting network intruders in real-time, Computer Networks, vol. 31, no. 23, pp. 2435-2463, 1998.
DOI Google Scholar
[32]
F. Fusco, M. P Stoecklin, and M. Vlachos, NET-FLi: On-the-fly compression, archiving and indexing of streaming network traffic, Proceedings of the VLDB Endowment, vol. 3, no. 1-2, pp. 1382-1393, 2010.
DOI Google Scholar
[33]
Z. Chen, X. Shi, L. Y. Raan, F. Xie, and J. Li, High speed traffic archiving system for flow granularity storage and querying, in Computer Communications and Networks (ICCCN), Munich, Germany, 2012, pp. 1-5.
DOI
[34]
X. Chen, B. Mu, and Z. Chen, NetSecu: A collaborative network security platform for in-network security, in Proc. the 3rd International Conference on Communications and Mobile Computing (CMC), Qingdao, China, 2011, pp. 59-64.
DOI
[35]
B. P. Mu, X. M. Chen, and Z. Chen, A collaborative network security management system in metropolitan area network, in Communications and Mobile Computing (CMC), 2011 Third International Conference, Qingdao, China, 2011, pp. 45-50.
DOI
[36]
F. C. Deng, A. A. Luo, Y. K. Zhang, Z. Chen, X. H. Peng, X, Jiang, and D. S. Peng, TNC-UTM: A holistic solution to secure enterprise networks, in Young Computer Scientists, ICYCS 2008, Hunan, China, 2008, pp. 2240-2245
DOI
[37]
Z. Chen, F. Y. Han, J. W. Cao, and S. Shen, Cloud computing-based forensic analysis for collaborative network security management system, Tsinghua Science and Technology, vol. 18, no. 1, pp. 40-50, 2013.
DOI Google Scholar
[38]
F. Han, Z. Chen, H. Xu, and Y. Liang, Garlic: A distributed botnets suppression system, in Distributed Computing Systems Workshops (ICDCSW), Macau, China, 2012, pp. 634-639.
DOI
[39]
A. Luo, L. Chuang, Z. Chen, X. Peng, and P. D. Ungsunan, TNC-compatible NAC system implemented on network processor, in Local Computer Networks, Dublin, Ireland, 2007, pp. 1096-1075.
DOI
[40]
Z. Ying, F. Deng, Z. Chen, Y. Xue, and C. Lin, UTM-CM: A practical control mechanism solution for UTM system, in Communications and Mobile Computing (CMC), Shenzhen, China, 2010, pp. 86-90.
[41]
L. Deri and F. Fusco, MicroCloud-based network traffic monitoring, in Proc. of the Intern. Symposium on Integrated Network Management, IM 2013, 2013.
[42]
T. Y. Li, F. Y. Han, S. Ding, and Z. Chen, LARX: Large-scale anti-phishing by retrospective data-exploring based on a cloud computing platform, in Computer Communications and Networks (ICCCN), 2011 Proceedings of 20th International Conference on IEEE, Maui, USA, 2011, pp. 1-5.
DOI
Publication history
Copyright
Acknowledgements
Rights and permissions

Publication history

Received: 15 July 2013
Accepted: 15 July 2013
Published: 05 August 2013
Issue date: August 2013

Copyright

© The author(s) 2013

Acknowledgements

This work was supported by the National Key Basic Research and Development (973) Program of China (Nos. 2012CB315801 and 2011CB302805), the National Natural Science Foundation of China A3 Program (No. 61161140320) and the National Natural Science Foundation of China (No. 61233016). This work was also supported by Intel Research Councils UPO program with title of security Vulnerability Analysis based on Cloud Platform with Intel IA Architecture. We also thank for Shuai Ding, Fuye Han, Jun Li, Yong Liang, Er-Long Min, and Xi Shi for their work in NSlab-Saturn team.

Rights and permissions

Return