More Iterations per Second, Same Quality -- Why Asynchronous Algorithms may Drastically Outperform Traditional Ones

In this paper, we consider the convergence of a very general asynchronous-parallel algorithm called ARock, that takes many well-known asynchronous algorithms as special cases (gradient descent, proximal gradient, Douglas Rachford, ADMM, etc.). In asynchronous-parallel algorithms, the computing nodes simply use the most recent information that they have access to, instead of waiting for a full update from all nodes in the system. This means that nodes do not have to waste time waiting for information, which can be a major bottleneck, especially in distributed systems. When the system has $p$ nodes, asynchronous algorithms may complete $\Theta(\ln(p))$ more iterations than synchronous algorithms in a given time period (“more iterations per second”). Although asynchronous algorithms may compute more iterations per second, there is error associated with using outdated information. How many more iterations in total are needed to compensate for this error is still an open question. The main results of this paper aim to answer this question. We prove, loosely, that as the size of the problem becomes large, the number of additional iterations that asynchronous algorithms need becomes negligible compared to the total number (“same quality” of the iterations). Taking these facts together, our results provide solid evidence of the potential of asynchronous algorithms to vastly speed up certain distributed computations.

💡 Research Summary

This paper investigates the convergence properties of a highly general asynchronous‑parallel algorithm called ARock, which subsumes many well‑known asynchronous methods such as asynchronous block gradient descent, forward‑backward splitting, proximal point, Douglas‑Rachford, ADMM, and others. The authors focus on the fundamental trade‑off that asynchronous algorithms enjoy: they can perform many more iterations per unit time because they never wait for a global synchronization barrier, yet each iteration may be based on stale (out‑of‑date) information. The central question addressed is whether the extra iterations gained by avoiding synchronization outweigh the loss of accuracy caused by using outdated data.

Main Contributions

1. Synchronization Penalty Model – The paper first formalizes a realistic model of a synchronous parallel implementation in which all (p) computing nodes must read the same current iterate, compute their local updates, and then wait for every node before the next iteration can start. Under a simple load‑balancing assumption, the expected time per synchronous round grows as (\Theta(\log p)). Consequently, a synchronous algorithm completes (\Theta(\log p)) fewer epochs per second than an asynchronous counterpart.

2. ARock Algorithm Definition – ARock is presented as an asynchronous block‑coordinate fixed‑point iteration. Given a non‑expansive operator (T:H\to H) with contraction constant (0<r<1) (so that (S=I-T) is a contraction), the algorithm updates a single block (i(k)) at iteration (k) using a possibly delayed iterate (\hat x_k = x_{k-\tilde j(k)}). The block index sequence ({i(k)}) is assumed IID uniform over the (m) blocks and independent of the history of iterates and delays.

3. Lyapunov‑Based Convergence Analysis – The authors construct a Lyapunov function
   \