Chemora: A PDE Solving Framework for Modern HPC Architectures

Modern HPC architectures consist of heterogeneous multi-core, many-node systems with deep memory hierarchies. Modern applications employ ever more advanced discretisation methods to study multi-physics problems. Developing such applications that explore cutting-edge physics on cutting-edge HPC systems has become a complex task that requires significant HPC knowledge and experience. Unfortunately, this combined knowledge is currently out of reach for all but a few groups of application developers. Chemora is a framework for solving systems of Partial Differential Equations (PDEs) that targets modern HPC architectures. Chemora is based on Cactus, which sees prominent usage in the computational relativistic astrophysics community. In Chemora, PDEs are expressed either in a high-level \LaTeX-like language or in Mathematica. Discretisation stencils are defined separately from equations, and can include Finite Differences, Discontinuous Galerkin Finite Elements (DGFE), Adaptive Mesh Refinement (AMR), and multi-block systems. We use Chemora in the Einstein Toolkit to implement the Einstein Equations on CPUs and on accelerators, and study astrophysical systems such as black hole binaries, neutron stars, and core-collapse supernovae.

💡 Research Summary

The paper “Chemora: A PDE Solving Framework for Modern HPC Architectures” addresses the significant challenge of developing high-performance computing applications for solving complex systems of Partial Differential Equations (PDEs) on modern, heterogeneous HPC systems. The authors identify a critical bottleneck: creating such applications requires deep expertise in three distinct domains—physics, numerical mathematics (discretization), and computer science (performance optimization)—a combination rarely found within a single research group. Furthermore, mainstream programming languages are inherently serial and too low-level to conveniently express the core science, while parallel programming models like MPI and OpenMP become cumbersome at extreme scale.

As a solution, the paper introduces Chemora, a framework designed to separate these concerns and enable efficient collaboration across disciplines. Chemora is built upon the Cactus software framework, which is widely used in computational relativistic astrophysics. Cactus facilitates collaboration through its modular “thorn” architecture, where independent components implementing physics, discretization schemes, or infrastructure services (like parallel I/O) can be combined by end-users without central coordination. The successful Einstein Toolkit community, with over 100 members, exemplifies this model, relying on community-agreed standards for interoperability.

The core innovation of Chemora is its use of high-level problem specification coupled with automated code generation. Physicists can express PDE systems in a high-level, LaTeX-like language or directly in Mathematica, focusing solely on the continuous equations. Numerical analysts can separately define discretization stencils, supporting methods like Finite Differences, Discontinuous Galerkin Finite Elements (DGFE), and Adaptive Mesh Refinement (AMR). This declarative specification is then processed by the Kranc code generation system.

Kranc automatically transforms the high-level description into highly optimized, low-level code (C, C++, CUDA, OpenCL) tailored for specific modern HPC architectures, including CPUs and accelerators like GPUs. This automation tackles the “Ninja Performance Gap”—the large performance difference between naively written and highly optimized code—by applying domain-specific optimizations such as loop tiling, memory layout transformations, and efficient GPU kernel generation. This approach liberates computer scientists from manually rewriting intricate physics kernels for each new architecture.

By cleanly separating the concerns of physics, mathematics, and implementation, Chemora streamlines the development process. It allows experts in each field to work independently and productively. The framework has been successfully used within the Einstein Toolkit to implement the Einstein equations and simulate astrophysical systems like black hole binaries and neutron stars. In conclusion, Chemora presents a powerful paradigm shift in scientific HPC software development, moving from manual, integrated coding to declarative, separated specification. This enhances collaboration, accelerates adaptation to evolving hardware, and ultimately empowers a broader community of scientists to leverage cutting-edge HPC resources for frontier research.