Journal Home > Volume 21 , Issue 1

Cloud monitoring is of a source of big data that are constantly produced from traces of infrastructures, platforms, and applications. Analysis of monitoring data delivers insights of the system’s workload and usage pattern and ensures workloads are operating at optimum levels. The analysis process involves data query and extraction, data analysis, and result visualization. Since the volume of monitoring data is big, these operations require a scalable and reliable architecture to extract, aggregate, and analyze data in an arbitrary range of granularity. Ultimately, the results of analysis become the knowledge of the system and should be shared and communicated. This paper presents our cloud service architecture that explores a search cluster for data indexing and query. We develop REST APIs that the data can be accessed by different analysis modules. This architecture enables extensions to integrate with software frameworks of both batch processing (such as Hadoop) and stream processing (such as Spark) of big data. The analysis results are structured in Semantic Media Wiki pages in the context of the monitoring data source and the analysis process. This cloud architecture is empirically assessed to evaluate its responsiveness when processing a large set of data records under node failures.


menu
Abstract
Full text
Outline
About this article

A Cloud Service Architecture for Analyzing Big Monitoring Data

Show Author's information Samneet Singh( )Yan Liu( )
Department of Electrical and Computer Science, Concordia University, Montreal, Canada.

Abstract

Cloud monitoring is of a source of big data that are constantly produced from traces of infrastructures, platforms, and applications. Analysis of monitoring data delivers insights of the system’s workload and usage pattern and ensures workloads are operating at optimum levels. The analysis process involves data query and extraction, data analysis, and result visualization. Since the volume of monitoring data is big, these operations require a scalable and reliable architecture to extract, aggregate, and analyze data in an arbitrary range of granularity. Ultimately, the results of analysis become the knowledge of the system and should be shared and communicated. This paper presents our cloud service architecture that explores a search cluster for data indexing and query. We develop REST APIs that the data can be accessed by different analysis modules. This architecture enables extensions to integrate with software frameworks of both batch processing (such as Hadoop) and stream processing (such as Spark) of big data. The analysis results are structured in Semantic Media Wiki pages in the context of the monitoring data source and the analysis process. This cloud architecture is empirically assessed to evaluate its responsiveness when processing a large set of data records under node failures.

Keywords: big data, cloud computing, software architecture, REST API, semantic web

References(13)

[1]
Aceto G., Botta A., Donato W. De, and Pescape A., Cloud monitoring: A Survey, Computer Networks, vol. 57, no. 9, pp. 2093–2115, 2013.
[2]
Wilkes J., More Google cluster data, Google research blog, 2011.
[3]
Reiss C., Wilkes J., and Hellerstein J. L., Google cluster-usage traces: Format + schema, Technical report, Google Inc., Mountain View, CA, USA, 2011.
[4]
Di S., Kondo D., and Franck C., Characterizing cloud applications on a Google data center, in 42nd International Conference on Parallel Processing (ICPP), Lyon, France, 2013.
[5]
Mishra A. K., Hellerstein J. L., Cirne W., and Das C. R., Towards characterizing cloud backend workloads: Insights from Google compute clusters, SIGMETRICS Perform. Eval. Rev., vol. 37, no. 4, pp. 34–41, 2010.
[6]
Kuć R., Apache Solr 4 Cookbook. Packt Publishing Ltd, 2013.
[7]
Barrett D. J., MediaWiki. OReilly Media, Inc., 2008.
[8]
Krötzsch M., Vrandečić D., and Völkel M., Semantic mediawiki, in The Semantic Web-ISWC 2006, Springer, 2006, pp. 935–942.
DOI
[9]
Hibler M., Ricci R., Stoller L., Duerig J., Guruprasad S., Stack T., Webb K., and Lepreau J., Large-scale virtualization in the emulab network testbed, in USENIX Annual Technical Conference, 2008, pp. 113–128.
[10]
Ward J. S. and Barker A., Observing the clouds: A survey and taxonomy of cloud monitoring, Journal of Cloud Computing, vol. 3, no. 1, pp. 1–30, 2014.
[11]
Tsai W., Bai X., and Huang Y., Software-as-a-service (saas): Perspectives and challenges, Science China Information Sciences, vol. 57, no. 5, pp. 1–15, 2014.
[12]
Xu K., Zhu M., Hu G., Zhu L., Zhong Y., Liu Y., Wu J., and Wang N., Towards evolvable internet architecture design constraints and models analysis, Science China Information Sciences, vol. 57, no. 11, pp. 1–24, 2014.
[13]
Li L., Rolling Window time series prediction using MapReduce, PhD dissertation, University of Sydney, 2014.
DOI
Publication history
Copyright
Acknowledgements
Rights and permissions

Publication history

Received: 09 September 2015
Revised: 25 October 2015
Accepted: 06 December 2015
Published: 04 February 2016
Issue date: February 2016

Copyright

© The author(s) 2016

Acknowledgements

This  work  was  supported  by  the  Discovery  grant  No. RGPIN 2014-05254 from Natural Science & Engineering Research Council (NSERC), Canada.

Rights and permissions

Return