MatSpray: Fusing 2D Material World Knowledge on 3D Geometry

MatSpray: Fusing 2D Material World Knowledge on 3D Geometry Philipp Langsteiner philipp.langsteiner@uni-tuebingen.de Jan-Niklas Dihlmann jan-niklas.dihlmann@uni-tuebingen.de Hendrik Lensch hendrik.lensch@uni-tuebingen.de Figure 1. MatSpray Overview we utilize 2D material world knowlegde from 2D diffusion models to reconstruct 3D relightable objects. Given multi-view images of a target object, we first generate per-view PBR material predictions (base color, roughness, metallic) using any 2D diffusion-based material model. These 2D estimates are then integrated into a 3D Gaussian Splatting reconstruction via Gaussian ray tracing. Finally, a neural refinement stage applies a softmax-based restriction to enforce multi-view consistency and enhance the physical accuracy of the materials. The resulting 3D assets feature high-quality, fully relightable PBR materials under novel illumination. Project page: https://matspray.jdihlmann.com/ Abstract Manual modeling of material parameters and 3D geometry is a time consuming yet essential task in the gaming and film industries. While recent advances in 3D reconstruction have enabled accurate approximations of scene geometry and appearance, these methods often fall short in relighting scenarios due to the lack of precise, spatially varying ma- terial parameters. At the same time, diffusion models oper- ating on 2D images have shown strong performance in pre- dicting physically based rendering (PBR) properties such as albedo, roughness, and metallicity. However, transfer- ring these 2D material maps onto reconstructed 3D geom- etry remains a significant challenge. We propose a frame- work for fusing 2D material data into 3D geometry using a combination of novel learning-based and projection-based approaches. We begin by reconstructing scene geometry via Gaussian Splatting. From the input images, a diffusion model generates 2D maps for albedo, roughness, and metal- lic parameters. Any existing diffusion model that can con- vert images or videos to PBR materials can be applied. The predictions are further integrated into the 3D representa- tion either by optimizing an image-based loss or by directly projecting the material parameters onto the Gaussians us- ing Gaussian ray tracing. To enhance fine-scale accuracy and multi-view consistency, we further introduce a light- weight neural refinement step (Neural Merger), which takes ray-traced material features as input and produces detailed adjustments. Our results demonstrate that the proposed methods outperform existing techniques in both quantitative metrics and perceived visual realism. This enables more accurate, relightable, and photorealistic renderings from 1 arXiv:2512.18314v1 [cs.CV] 20 Dec 2025 reconstructed scenes, significantly improving the realism and efficiency of asset creation workflows in content pro- duction pipelines. Project page: https://matspray. jdihlmann.com/ 1. Introduction Editing and relighting real scenes captured with casual cam- eras is central to many vision and graphics applications. While modern neural 3D reconstruction methods can pro- duce impressive geometry and appearance from images, they often entangle illumination with appearance, yielding textures or coefficients that are not physically meaningful for relighting. Classical inverse rendering requires strong assumptions about lighting and exposure and remains frag- ile when materials vary spatially. In parallel, recent 2D ma- terial predictors learn rich priors from large-scale data and can produce plausible material maps from images, yet they operate in 2D and are not directly consistent across views or attached to a 3D representation. We introduce a method to transfer 2D material pre- dictions onto a 3D Gaussian representation to obtain re- lightable assets with spatially varying base color, rough- ness, and metallic parameters. The approach projects 2D material maps to 3D via efficient ray-traced assignment, refines materials with a small MLP to reduce multi-view inconsistencies, and supervises rendered material maps di- rectly with the 2D predictions to preserve plausible pri- ors while discouraging baked-in lighting. This combina- tion yields cleaner albedo, more accurate roughness, and in- formed metallic estimates, enabling higher-quality relight- ing compared to pipelines that learn only appearance. Our contributions are: • World Material Fusion. A plug-and-play pipeline that, to our knowledge, is the first to fuse swappable diffusion- based 2D PBR priors (“world material knowledge”) with 3D Gaussian material optimization via Gaussian ray trac- ing and PBR consistent supervision to obtain relightable assets. • Neural Merger. A softmax neural merger that aggregates per-Gaussian, multi-view material estimates, suppresses baked-in lighting, and enforces cross-view consistency while stabilizing joint environment map optimization. • Faster Reconstruction. A simple projection and op- timization scheme that reconstructs high-quality re-

This content is AI-processed based on open access ArXiv data.