LogoML: An Open Machine Learning Library for Distributed Big Data Analytics

Implementation of distributed machine learning algorithms in MapReduce faces three challenges: low execution efficiency of iterative algorithms, poor data scalability, and high complexity in decomposing a complicated algorithm into the rigid structure of Map and Reduce operations. The last challenge makes many useful algorithms for big data analysis unavailable in current distributed systems. In this paper, we propose LogoML, a library of machine learning algorithms for distributed big data analysis. LogoML is implemented on top of the Local Operation and Global Operation (LOGO) distributed computing framework and the Random Sample Partition (RSP) data representation model. Unlike MapReduce algorithms, the algorithms in LogoML are sequential in nature. With LogoML, a data analysis task is carried out with two core operations in the LOGO distributed computing framework. The Local Operation (LO) operation executes a data analytic algorithm in multiple nodes or virtual machines in parallel to process a set of random sample data blocks and generate a set of local results. The Global Operation (GO) operation executes an ensemble algorithm to aggregate the local results into the final ensemble result. Through innovative design and implementation, the LogoML library exhibits the following merits in distributed big data analysis: It has rich algorithms because sequential algorithms are easy to implement in LogoML, is efficient in execution of iterative algorithms and scalable to big data and its open structure allows the user to add new algorithms to the library. The performance, efficiency, and data scalability of LogoML algorithms are demonstrated with the experiment results of both synthetic and real-datasets.