PLaID++: A Preference Aligned Language Model for Targeted Inorganic Materials Design

Reinforcement Learning from Verifiable Rewards (RLVR) has emerged as a promising approach to improve correctness in LLMs, however, in many scientific problems, the objective is not necessarily to produce the correct answer, but instead to produce a diverse array of candidates which satisfy a set of constraints. We study this challenge in the context of materials generation. To this end, we introduce PLaID++, an LLM post-trained for stable and property-guided crystal generation. We find that performance hinges on our crystallographic representation and reward formulation. First, we introduce a compact, symmetry-informed Wyckoff text representation which improves computational efficiency and encourages generalization from physical priors. Second, we demonstrate that temperature scaling acts as an entropy regularizer which counteracts mode collapse and encourages exploration. By encoding symmetry constraints directly into text and guiding model outputs towards desirable chemical space, PLaID++ generates structures that are thermodynamically stable, unique, and novel at a $\sim$50% greater rate than prior methods and conditionally generates structures with desired space group properties. Our work demonstrates the potential of adapting post-training techniques from natural language processing to materials design, paving the way for targeted and efficient discovery of novel materials.

💡 Research Summary

PLaID++ addresses the challenge of generating diverse, constraint‑satisfying inorganic crystal structures by adapting large‑language‑model (LLM) post‑training techniques to materials design. The authors identify two major obstacles in existing LLM‑based generators: (1) inefficient or absent encoding of crystallographic symmetry, leading to high token counts and poor generalization, and (2) mode collapse during reinforcement learning, which reduces diversity when the model over‑optimizes for stability.

To solve these problems, the paper introduces a compact, symmetry‑aware Wyckoff text representation. Instead of encoding full fractional coordinates for every atom, only the asymmetric unit’s atomic species and Wyckoff positions are written as a short textual string. This reduces the average token length from ~215 to ~186 (≈14 % reduction) on the MP‑20 dataset, while preserving all symmetry information. Because each atom is tied to a Wyckoff site, any change propagates through the space‑group symmetry operations, forcing the model to learn physical priors rather than memorizing individual coordinates.

The second innovation is Reinforcement Learning from Interatomic Potentials (RLIP), a reinforcement‑learning framework that replaces human preference data with a fast machine‑learning interatomic potential (EquiformerV2). The potential predicts relaxed formation energies, providing a proxy for thermodynamic stability (energy‑above‑hull, E_hull). Preference pairs are constructed across three stability buckets (stable ≤ 0 eV/atom, metastable ≤ 0.08 eV/atom, unstable > 0.08 eV/atom) and also encode novelty/uniqueness (via Pymatgen StructureMatcher) and space‑group correctness. Direct Preference Optimization (DPO) is then applied, optimizing a KL‑regularized objective that directly maximizes the likelihood of preferred structures without explicit reward or value networks.

A crucial practical detail is the use of temperature scaling as an entropy regularizer. Across successive DPO iterations, the sampling temperature is gradually increased. Higher temperature encourages the model to explore more diverse Wyckoff configurations, boosting novelty and uniqueness, while a modest loss in stability is tolerated because the reward signal still prefers lower E_hull. Empirically, this schedule prevents the model from collapsing onto a narrow set of “stable but repetitive” crystals, a problem observed when training on raw coordinate strings.

The experimental pipeline proceeds as follows: a pre‑trained Qwen‑2.5 7B model is first fine‑tuned on the MP‑20 dataset (45 k inorganic crystals) using 4‑bit quantization and LoRA adapters, separately for coordinate‑based and Wyckoff‑based text. Then, 10 k unconditional samples and 1 k samples for each of seven representative space groups are generated, evaluated by EquiformerV2, and labeled for stability, novelty, and space‑group match. Preference pairs are built from these labels, and iterative DPO is performed where the previous policy serves as the reference model (π_ref = π_{θ‑1}).

Results show substantial improvements over prior methods (VAE‑based CD‑VAE, diffusion models like DiffCSP, and earlier LLM approaches such as CrystalLLM). Specifically:

• Stability – the proportion of generated crystals with E_hull ≤ 0 eV/atom rises by ~115 % compared to supervised fine‑tuning alone, reaching ~70 % of all samples.
• Diversity – the fraction of duplicate structures drops below 20 %, and the novelty score (fraction of structures not present in the training set) increases from 0.42 to 0.68.
• Space‑group conditioning – when prompted with a target space‑group number, the model produces the correct symmetry in ~68 % of cases, a >50 % relative gain over baseline LLMs.
• Efficiency – the Wyckoff representation reduces token length and memory usage, yielding ~14 % faster inference and ~22 % lower GPU memory consumption.

The authors discuss limitations: the study is confined to crystals with ≤20 atoms, and the MLIP reward, while fast, is still an approximation to DFT‑level energies; thus experimental validation remains necessary. Moreover, the temperature schedule is hand‑crafted; future work could integrate adaptive entropy control or Bayesian optimization to balance stability and diversity automatically.

In conclusion, PLaID++ demonstrates that embedding crystallographic symmetry directly into a compact textual format, combined with a physics‑grounded reinforcement learning loop (RLIP) and entropy‑regularized DPO, enables large language models to generate stable, novel, and symmetry‑controlled inorganic materials at a rate substantially higher than existing generative approaches. This work opens a pathway for leveraging the scalability of LLMs in scientific discovery, suggesting that similar symmetry‑aware representations and MLIP‑based rewards could be extended to more complex materials systems, alloy design, and beyond.