HyperOffload: Graph-Driven Hierarchical Memory Management for Large Language Models on SuperNode Architectures

The rapid evolution of Large Language Models (LLMs) towards long-context reasoning and sparse architectures has pushed memory requirements far beyond the capacity of individual device HBM. While emerging supernode architectures offer terabyte-scale shared memory pools via high-bandwidth interconnects, existing software stacks fail to exploit this hardware effectively. Current runtime-based offloading and swapping techniques operate with a local view, leading to reactive scheduling and exposed communication latency that stall the computation pipeline. In this paper, we propose the SuperNode Memory Management Framework (\textbf{HyperOffload}). It employs a compiler-assisted approach that leverages graph-driven memory management to treat remote memory access as explicit operations in the computation graph, specifically designed for hierarchical SuperNode architectures. Unlike reactive runtime systems, SuperNode represents data movement using cache operators within the compiler’s Intermediate Representation (IR). This design enables a global, compile-time analysis of tensor lifetimes and execution dependencies. Leveraging this visibility, we develop a global execution-order refinement algorithm that statically schedules data transfers to hide remote memory latency behind compute-intensive regions. We implement SuperNode within the production deep learning framework MindSpore, adding a remote memory backend and specialized compiler passes. Evaluation on representative LLM workloads shows that SuperNode reduces peak device memory usage by up to 26% for inference while maintaining end-to-end performance. Our work demonstrates that integrating memory-augmented hardware into the compiler’s optimization framework is essential for scaling next-generation AI workloads.

💡 Research Summary

The paper addresses the growing memory bottleneck of large language models (LLMs) as they evolve toward longer contexts, multimodal capabilities, and sparsely activated MoE architectures. While emerging SuperNode hardware—characterized by terabyte‑scale shared memory pools interconnected by ultra‑high‑bandwidth links—offers a promising solution, existing software stacks rely on runtime‑driven offloading, swapping, and prefetching that operate with only a local view of the computation graph. Such reactive approaches cannot anticipate future memory needs, leading to frequent “bubbles” where communication stalls the compute pipeline and to sub‑optimal utilization of the high‑speed interconnect.

HyperOffload (the authors’ proposed framework) fundamentally re‑thinks memory management by elevating remote memory accesses to first‑class operators within the compiler’s intermediate representation (IR). Implemented on top of MindSpore, the framework introduces explicit remote‑cache‑load and remote‑cache‑store nodes into the MindIR graph. During compilation, a dedicated analysis pass computes the lifetime of every tensor, the dependencies among operators, and the bandwidth/latency characteristics of the SuperNode’s shared memory pool. With this global visibility, HyperOffload applies a Global Execution‑Order Refinement algorithm that statically schedules data transfers so that they are hidden behind compute‑intensive regions. The algorithm uses a cost model that balances transfer time, compute time, and memory pressure, and it can reorder independent operators to maximize overlap.

Key technical contributions include:

1. Remote Cache Operations as Graph Primitives – By making offload/reload explicit graph nodes, the compiler can reason about memory movement just like any other computation, enabling deterministic planning and eliminating redundant allocations.
2. Graph‑Driven Execution‑Order Optimization – The static reordering algorithm removes the need for runtime‑driven prefetching, thereby eradicating the runtime‑induced bubbles observed in prior work.
3. Unified Hierarchical Memory Execution Model – The model treats on‑device HBM, on‑package caches, and the remote shared pool as a single logical hierarchy, allowing seamless switching between them for both inference and training workloads.

The authors evaluate HyperOffload on a 384‑NPUs SuperNode (Huawei Ascend 910) using several representative LLMs, including LLaMA‑3‑8B and GPT‑NeoX‑20B. For inference, HyperOffload reduces peak device memory usage by up to 26 % while keeping end‑to‑end latency at 5.5 seconds, compared to 15 seconds when using a state‑of‑the‑art runtime prefetcher (a 2.7× slowdown). The compute‑communication overlap improves from ~50 % to >85 %, and the number of memory‑compaction stalls drops dramatically. In training experiments with models ranging from 1.5 B to 100 B parameters, the framework successfully offloads activation checkpoints and optimizer states to the shared pool, preventing out‑of‑memory failures and adding less than 5 % overhead to total training time.

Overall, HyperOffload demonstrates that integrating memory‑augmented hardware capabilities into the compiler’s optimization pipeline is essential for scaling next‑generation AI workloads. By treating remote memory as a programmable resource rather than an after‑thought, the framework unlocks the full potential of SuperNode architectures, delivering both higher scalability and better performance without requiring user‑level code changes. Future directions include extending the approach to multi‑node clusters, handling consistency across distributed memory pools, and adapting the methodology to other hardware ecosystems such as GPU‑based SuperNodes.