DSparse: A Distributed Training Method for Edge Clusters Based on Sparse Update

Edge machine learning creates a new computational paradigm by enabling the deployment of intelligent applications at the network edge. It enhances application efficiency and responsiveness by performing inference and training tasks closer to data sources. However, it encounters several challenges in practice. The variance in hardware specifications and performance across different devices presents a major issue for the training and inference tasks. Additionally, edge devices typically possess limited network bandwidth and computing resources compared with data centers. Moreover, existing distributed training architectures often fail to consider the constraints of resources and communication efficiency in edge environments. In this paper, we propose DSparse, a method for distributed training based on sparse update in edge clusters with various memory capacities. It aims at maximizing the utilization of memory resources across all devices within a cluster. To reduce memory consumption during the training process, we adopt sparse update to prioritize the updating of selected layers on the devices in the cluster, which not only lowers memory usage but also reduces the data volume of parameters and the time required for parameter aggregation. Furthermore, DSparse utilizes a parameter aggregation mechanism based on multi-process groups, subdividing the aggregation tasks into AllReduce and Broadcast types, thereby further reducing the communication frequency for parameter aggregation. Experimental results using the MobileNetV2 model on the CIFAR-10 dataset demonstrate that DSparse reduces memory consumption by an average of 59.6% across seven devices, with a 75.4% reduction in parameter aggregation time, while maintaining model precision.