A Note on Parallel Algorithmic Speedup Bounds

A parallel program can be represented as a directed acyclic graph. An important performance bound is the time to execute the critical path through the graph. We show how this performance metric is related to Amdahl speedup and the degree of average parallelism. These bounds formally exclude superlinear performance.

💡 Research Summary

The paper presents a rigorous analysis of parallel algorithm speed‑up limits by modeling a parallel program as a directed acyclic graph (DAG). In this representation each node corresponds to a unit of work (or sub‑task) and each edge encodes a control or communication dependency. Two fundamental execution times are defined: T₁, the time required to execute all nodes sequentially on a single processor, and T∞, the time needed to traverse the critical path when an unlimited number of processors are available. T∞ is simply the depth of the DAG and constitutes an absolute lower bound on any parallel execution because no schedule can finish faster than the longest dependency chain.

Leiserson’s classic lower bounds are restated: (3) Tₚ ≥ T₁/p, which captures the intuitive notion that work cannot be completed in less than the total work divided equally among p processors; and (4) Tₚ ≥ T∞, which asserts that the critical path length dominates execution time regardless of processor count. Substituting (3) into the speed‑up definition Sₚ = T₁/Tₚ yields the ideal linear speed‑up Sₚ = p. In practice, overhead, load imbalance, and communication cause sub‑linear speed‑up, i.e., Sₚ ≤ p. The paper also acknowledges the theoretical possibility of super‑linear speed‑up (Sₚ > p) but shows that bound (3) precludes it for any realistic DAG.

The authors then introduce the average parallelism metric A = W/T∞, where W is the total work (the sum of node execution times) and T∞ is the critical‑path time. They prove that A is identical to Leiserson’s S∞ (the asymptotic speed‑up as p → ∞). An illustrative example uses a DAG with 18 unit‑time nodes and a critical path of nine nodes, giving W = 18, T∞ = 9, and thus A = 2. This matches S∞ = 2. The paper contrasts this with Amdahl’s law, which would predict an asymptotic speed‑up of 1/σ = 4 for the same graph because four nodes are deemed serial (σ = 4/18). The discrepancy highlights that Amdahl’s bound, based solely on a serial fraction, can be overly optimistic compared with the tighter bound provided by average parallelism.

To address super‑linear claims, the paper proves that bound (3) is equivalent to assuming a depth‑one DAG, i.e., all work placed on a single level. Since any non‑trivial DAG has depth greater than one, it cannot be flattened arbitrarily, and thus the maximal speed‑up cannot exceed the number of processors (linear scaling). Apparent super‑linear behavior may arise only when the baseline for linearity is chosen incorrectly (e.g., comparing small‑p runs to larger‑p runs without accounting for the work distribution), but the efficiency Eₚ = Sₚ/p never exceeds 1 under the model.

In summary, the paper unifies three perspectives on parallel performance: the critical‑path bound (T∞), the work‑division bound (T₁/p), and the average parallelism A = W/T∞. It demonstrates that average parallelism coincides with the asymptotic speed‑up and provides a more precise, graph‑aware alternative to Amdahl’s law. Moreover, it offers a formal proof that super‑linear speed‑up is impossible for any DAG‑structured computation, reinforcing the importance of using correct theoretical baselines when evaluating parallel scalability.