2023 journal article
Federated TD Learning Over Finite-Rate Erasure Channels: Linear Speedup Under Markovian Sampling
IEEE CONTROL SYSTEMS LETTERS, 7, 2461–2466.
author keywords: Servers; Markov processes; Quantization (signal); Function approximation; Approximation algorithms; Reinforcement learning; Supervised learning; Machine learning; large-scale systems; communication networks
TL;DR:
This work proposes and analyze QFedTD - a quantized federated temporal difference learning algorithm with linear function approximation that highlights the effect of quantization and erasures on the convergence rate and establishes a linear speedup w.r.t. the number of agents under Markovian sampling.
(via Semantic Scholar)
ieeexplore.ieee.org (publisher PDF)
Source: Web Of Science
Added: August 7, 2023
Federated learning (FL) has recently gained much attention due to its effectiveness in speeding up supervised learning tasks under communication and privacy constraints. However, whether similar speedups can be established for reinforcement learning remains much less understood theoretically. Towards this direction, we study a federated policy evaluation problem where agents communicate via a central aggregator to expedite the evaluation of a common policy. To capture typical communication constraints in FL, we consider finite capacity up-link channels that can drop packets based on a Bernoulli erasure model. Given this setting, we propose and analyze QFedTD - a quantized federated temporal difference learning algorithm with linear function approximation. Our main technical contribution is to provide a finite-sample analysis of QFedTD that (i) highlights the effect of quantization and erasures on the convergence rate; and (ii) establishes a linear speedup w.r.t. the number of agents under Markovian sampling. Notably, while different quantization mechanisms and packet drop models have been extensively studied in the FL, distributed optimization, and networked control systems literature, our work is the first to provide a non-asymptotic analysis of their effects in multi-agent and federated reinforcement learning.