Ira: Efficient Transaction Replay for Distributed Systems

In primary-backup replication, consensus latency is bounded by the time for backup nodes to replay (re-execute) transactions proposed by the primary. In this work, we present Ira, a framework to accelerate backup replay by transmitting compact \emph{hints} alongside transaction batches. Our key insight is that the primary, having already executed transactions, possesses knowledge of future access patterns which is exactly the information needed for optimal replay. We use Ethereum for our case study and present a concrete protocol, Ira-L, within our framework to improve cache management of Ethereum block execution. The primaries implementing Ira-L provide hints that consist of the working set of keys used in an Ethereum block and one byte of metadata per key indicating the table to read from, and backups use these hints for efficient block replay. We evaluated Ira-L against the state-of-the-art Ethereum client reth over two weeks of Ethereum mainnet activity ($100,800$ blocks containing over $24$ million transactions). Our hints are compact, adding a median of $47$ KB compressed per block ($\sim5%$ of block payload). We observe that the sequential hint generation and block execution imposes a $28.6%$ wall-time overhead on the primary, though the direct cost from hints is $10.9%$ of execution time; all of which can be pipelined and parallelized in production deployments. On the backup side, we observe that Ira-L achieves a median per-block speedup of $25\times$ over baseline reth. With $16$ prefetch threads, aggregate replay time drops from $6.5$ hours to $16$ minutes ($23.6\times$ wall-time speedup).

💡 Research Summary

Ira introduces a hint‑based replay framework aimed at reducing the latency of backup nodes in primary‑backup replication systems, where consensus speed is often limited by the time required for backups to re‑execute transaction batches. The central observation is that the primary, having already executed the batch, possesses complete knowledge of the future access pattern of keys. By encoding this knowledge into a compact hint and transmitting it alongside the transaction data, backups can prefetch required state and apply near‑optimal cache replacement policies, dramatically cutting I/O stalls.

The paper first formalizes the problem: in standard SMR (State Machine Replication) the primary sends only the raw transaction payloads, forcing backups to reconstruct the execution path without any foresight. Conventional cache policies such as LRU make eviction decisions based solely on past accesses, leading to unnecessary cache misses, especially in workloads where state I/O dominates computation. The authors point out that Belady’s MIN algorithm—optimal for cache replacement—requires exactly the future access information that the primary already has.

Ira’s architecture consists of two components. The primary instrumentedly records every key read or written during batch execution, forming an access set A. It then builds a hint H that includes A and a one‑byte metadata per key indicating the storage table (e.g., main state table vs. change‑set table). Optionally, richer hints could carry ordering information or even the actual values, but the baseline implementation keeps hints minimal to reduce bandwidth. H is compressed (median 47 KB per Ethereum block, about 5 % of the block payload) and sent together with the block.

Backups receive H before replay. Using the hint, they launch a prefetch phase where multiple threads (up to 16 in the evaluation) load all keys in A into their local cache. Because the cache now contains the exact data needed for the upcoming transactions, the replay phase proceeds with virtually no additional I/O, allowing the backup to follow an eviction strategy that mirrors Belady’s optimal policy. Hints are advisory; if a hint is missing, corrupted, or malicious, the backup simply falls back to the standard replay path, preserving correctness at the cost of performance.

The authors instantiate the framework for Ethereum, naming the concrete protocol Ira‑L. Ethereum’s execution model (EVM) heavily relies on contract storage accesses (SLOAD/SSTORE), which dominate block execution time. A detailed measurement campaign re‑executed 100,800 consecutive main‑net blocks (≈24 M transactions) using the Rust client reth, instrumented via the EVM inspector. The trace revealed that I/O accounts for roughly 68 % of total execution time, and storage operations constitute the bulk of that I/O. This makes Ethereum an ideal target for cache‑centric optimizations.

Implementation details: the primary modifies reth to capture every key‑value lookup in MDBX, builds the per‑block hint, compresses it with Zstandard, and appends it to the block proposal. The backup extends reth with a “Hint Prefetcher” that parses H, spawns up to 16 prefetch threads, and loads the keys before invoking the existing replay engine. The hint generation adds a 28.6 % wall‑time overhead on the primary, but only 10.9 % of that is the direct cost of hint construction; the remainder is the normal block execution, which can be pipelined in production. On the backup side, the median per‑block speedup is 25×; with 16 prefetch threads the aggregate replay time for the two‑week dataset drops from 6.5 hours to 16 minutes, a 23.6× wall‑time improvement.

The paper discusses limitations. Hint transmission consumes extra bandwidth (≈5 % of block size) and introduces modest CPU/memory overhead on the primary. Since hints are advisory, a malicious primary could degrade backup performance, suggesting the need for integrity checks (e.g., signatures). The current design is tailored to Ethereum’s flat key‑value storage; adapting to systems with hierarchical or multi‑level storage (e.g., LSM‑trees) may require richer hint structures. Future work includes improving compression, dynamic cache sizing based on hint statistics, and extending the approach to other replication scenarios such as Raft‑based databases, streaming replication, or distributed key‑value stores.

In conclusion, Ira demonstrates that leveraging the primary’s already‑available future access information can transform the replay bottleneck from an I/O‑bound operation into a largely compute‑bound one. The experimental results on real Ethereum main‑net data validate the practicality of the approach, achieving order‑of‑magnitude speedups with modest overhead. Because hints do not affect correctness, Ira can be incrementally deployed in existing replication pipelines, offering a compelling optimization for any system where state I/O dominates transaction processing.