2024 article

Sign-Based Gradient Descent With Heterogeneous Data: Convergence and Byzantine Resilience

Jin, R., Liu, Y., Huang, Y., He, X., Wu, T., & Dai, H. (2024, January 12). IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS.

By: R. Jin *, Y. Liu*, Y. Huang *, X. He *, T. Wu n  & H. Dai n 

author keywords: Byzantine resilience; communication efficiency; data heterogeneity; federated learning (FL); sign-based gradient descent
TL;DR: A novel magnitude-driven stochastic-sign-based gradient compressor is proposed to address the non-convergence issue of signSGD, and the convergence of the proposed method is established in the presence of arbitrary data heterogeneity. (via Semantic Scholar)
Source: Web Of Science
Added: March 11, 2024

Communication overhead has become one of the major bottlenecks in the distributed training of modern deep neural networks. With such consideration, various quantization-based stochastic gradient descent (SGD) solvers have been proposed and widely adopted, among which signSGD with majority vote shows a promising direction because of its communication efficiency and robustness against Byzantine attackers. However, signSGD fails to converge in the presence of data heterogeneity, which is commonly observed in the emerging federated learning (FL) paradigm. In this article, a sufficient condition for the convergence of the sign-based gradient descent method is derived, based on which a novel magnitude-driven stochastic-sign-based gradient compressor is proposed to address the non-convergence issue of signSGD. The convergence of the proposed method is established in the presence of arbitrary data heterogeneity. The Byzantine resilience of sign-based gradient descent methods is quantified, and the error-feedback mechanism is further incorporated to boost the learning performance Experimental results on the MNIST dataset, the CIFAR-10 dataset, and the Tiny-ImageNet dataset corroborate the effectiveness of the proposed methods.