Mobile Internet Big Data Platform in China Unicom

Wenliang Huang; Zhen Chen; Wenyu Dong; Hang Li; Bin Cao; Junwei Cao

doi:10.1109/TST.2014.6733212

Tsinghua Science and Technology 2014, 19(1): 95-101 https://doi.org/10.1109/TST.2014.6733212

Open Access | Issue | Published: 07 February 2014

Mobile Internet Big Data Platform in China Unicom

Show Author's Information Hide Author's Information Wenliang Huang, Zhen Chen(

), Wenyu Dong, Hang Li, Bin Cao, Junwei Cao

China Unicom Groups, No. 21 Financial Street, Xicheng District, Beijing 100140, China

Research Institute of Information Technology (RIIT) and Tsinghua National Lab for Information Science and Technology (TNList), Tsinghua University, Beijing 100084, China

Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China

Department of Computer Science and Technology, PLA Univ. of Info. & Eng., Zhengzhou 450001, China

Keywords:

big data platform, China Unicom, 3G wireless network, Hadoop Distributed File System (HDFS), mobile Internet, network forensic, data warehouse, HBase

Cite this article:

Huang W, Chen Z, Dong W, et al. Mobile Internet Big Data Platform in China Unicom. Tsinghua Science and Technology, 2014, 19(1): 95-101. https://doi.org/10.1109/TST.2014.6733212

Download citation

EndNote(RIS)

BibTeX

455

Views

Downloads

Citations

Crossref

N/A

WoS

Scopus

CSCD

Abstract Full text About this article

Abstract

China Unicom, the largest WCDMA 3G operator in China, meets the requirements of the historical Mobile Internet Explosion, or the surging of Mobile Internet Traffic from mobile terminals. According to the internal statistics of China Unicom, mobile user traffic has increased rapidly with a Compound Annual Growth Rate (CAGR) of 135%. Currently China Unicom monthly stores more than 2 trillion records, data volume is over 525 TB, and the highest data volume has reached a peak of 5 PB. Since October 2009, China Unicom has been developing a home-brewed big data storage and analysis platform based on the open source Hadoop Distributed File System (HDFS) as it has a long-term strategy to make full use of this Big Data. All Mobile Internet Traffic is well served using this big data platform. Currently, the writing speed has reached 1 390 000 records per second, and the record retrieval time in the table that contains trillions of records is less than 100 ms. To take advantage of this opportunity to be a Big Data Operator, China Unicom has developed new functions and has multiple innovations to solve space and time constraint challenges presented in data processing. In this paper, we will introduce our big data platform in detail. Based on this big data platform, China Unicom is building an industry ecosystem based on Mobile Internet Big Data, and considers that a telecom operator centric ecosystem can be formed that is critical to reach prosperity in the modern communications business.

Full text

Abstract

Full text

Outline

About this article

Mobile Internet Big Data Platform in China Unicom

Show Author's information Hide Author's Information Wenliang Huang, Zhen Chen(

), Wenyu Dong, Hang Li, Bin Cao, Junwei Cao

China Unicom Groups, No. 21 Financial Street, Xicheng District, Beijing 100140, China

Research Institute of Information Technology (RIIT) and Tsinghua National Lab for Information Science and Technology (TNList), Tsinghua University, Beijing 100084, China

Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China

Department of Computer Science and Technology, PLA Univ. of Info. & Eng., Zhengzhou 450001, China

Abstract

Keywords: big data platform, China Unicom, 3G wireless network, Hadoop Distributed File System (HDFS), mobile Internet, network forensic, data warehouse, HBase

References(17)

[1]

J. Manyika, M. Chui, B. Brown, J. Bughin, R. Dobbs, C. Roxburgh, and A. H. Byers, Big data: The next frontier for innovation, competition, and productivity, Technical report, McKinsey Global Institute, 2011.

[2]

M. Meeker and L. Wu, Mary Meeker’s 2013 Internet Trends report, http://www.kpcb.com/insights/2013-internet-trends, 2013.

[3]

Cisco Inc., Visual networking index: Forecast and methodology, 2010-2015, http://apo.org.au/research/cisco-visual-networking-index-forecast-and-methodology-2010-2015, 2011.

[4]

T. Li, F. Y. Han, S. Ding, and Z. Chen, LARX: Large-scale anti-phishing by retrospective data-exploring based on a cloud computing platform, in Proc. 20th International Conference on Computer Communications and Networks (ICCCN), Hawaii, USA, 2011, pp. 1-5.

DOI

[5]

Z. Chen, F. Y. Han, J. W. Cao, X. Jiang, and S. Chen, Cloud computing-based forensic analysis for collaborative network security management system, Tsinghua Science and Technology, vol. 18, no. 1, pp. 40-50, 2013.

DOI Google Scholar

[6]

Z. Chen, L. Y. Ruan, J. W. Cao, Y. Yu, and X. Jiang, TIFAflow: Enhancing traffic archiving system with flow granularity for forensic analysis in network security, Tsinghua Science and Technology, vol. 18, no. 4, pp. 406-417, 2013.

DOI Google Scholar

[7]

S. Kornexl, V. Paxson, H. Dreger, A. Feldmann, and R. Sommer, Building a time machine for efficient recording and retrieval of high-volume network traffic, in Proc. 5th ACM SIGCOMM Conference on Internet Measurement, 2005.

DOI

[8]

G. Maier, R. Sommer, H. Dreger, A. Feldmann, V. Paxson, and F. Schneider, Enriching network security analysis with time travel, in Proc. ACM SIGCOMM 2008 Conference on Data Communication, New York, USA, 2008, pp. 183-194.

DOI

[9]

J. Li, S. Ding, M. Xu, F. Y. Han, X. Guan, and Z. Chen, TIFA: Enabling real-time querying and storage of massive stream data, in Proc. International Conference on Networking and Distributed Computing (ICNDC), Osaka, Japan, 2011, pp. 61-64.

DOI

[10]

K. Wu, S. Ahern, E. W. Bethel, J. Chen, H. Childs, E. Cormier-Michel, C. Geddes, J. Gu, H. Hagen, B. Hamann, W. Koegler, J. Lauret, J. Meredith, P. Messmer, E. Otoo, V. Perevoztchikov, A. Poskanzer, , O. Rbel, A. Shoshani, A. Sim, K. Stockinger, G. Weber, and W. M. Zhang, FastBit: Interactively searching massive data, Journal of Physics: Conference Series, vol. 180, no. 1, 2009.

DOI Google Scholar

[11]

L. Deri and F. Fusco, MicroCloud-based network traffic monitoring, in IFIP/IEEE International Symposium on Integrated Network Management (IM), 2013.

[12]

L. Deri and F. Fusco, Real-time MicroCloud-based flow aggregation for fixed and mobile networks, in 9th IEEE International Wireless Communications and Mobile Computing Conference (IWCMC), 2013, pp. 96-101.

DOI

[13]

Y. Lee and Y. Lee, Toward scalable internet traffic measurement and analysis with Hadoop, ACM SIGCOMM Computer Communication Review, vol. 43, no. 1, pp. 5-13, 2012.

DOI Google Scholar

[14]

Y. Lee, W. Kang, and H. Son, An internet traffic analysis method with mapreduce, in Network Operations and Management Symposium Workshops (NOMS Wksps), 2010, pp. 357-361.

DOI

[15]

F. Qian, Z. Wang, Y. Gao, J. Huang, A. Gerber, Z. Mao, S. Sen, and O. Spatscheck, Periodic transfers in mobile applications: Network-wide origin, impact, and optimization, in Proc. 21st International Conference on World Wide Web, Lyon, France, 2012, pp. 51-60.

DOI

[16]

J. L. Gailly and M. Adler, The ZLIB library, http://www.zlib.org/, 2013.

[17]

M. F. Oberhumer, Lzo documentation, http:// www.oberhumer.com/opensource/lzo/lzodoc.php, 2011.

About this article

Publication history

Acknowledgements

Rights and permissions

Publication history

Received: 09 January 2014

Accepted: 10 January 2014

Published: 07 February 2014

Issue date: February 2014

Copyright

Acknowledgements

This work was supported in part by the National Key Basic Research and Development (973) Program of China (Nos. 2013CB228206 and 2012CB315801), the National Natural Science Foundation of China (Nos. 61233016 and 61140320). This work was also supported by the Intel Research Council under the title of "Security Vulnerability Analysis Based on Cloud Platform with Intel IA Architecture" .