3D Software Synthesis Guided by Constraint-Expressive Intermediate Representation

Graphical user interface (UI) software has undergone a fundamental transformation from traditional two-dimensional (2D) desktop/web/mobile interfaces to spatial three-dimensional (3D) environments. While existing work has made remarkable success in automated 2D software generation, such as HTML/CSS and mobile app interface code synthesis, the generation of 3D software still remains under-explored. Current methods for 3D software generation usually generate the 3D environments as a whole and cannot modify or control specific elements in the software. Furthermore, these methods struggle to handle the complex spatial and semantic constraints inherent in the real world. To address the challenges, we present Scenethesis, a novel requirement-sensitive 3D software synthesis approach that maintains formal traceability between user specifications and generated 3D software. Scenethesis is built upon ScenethesisLang, a domain-specific language that serves as a granular constraint-aware intermediate representation (IR) to bridge natural language requirements and executable 3D software. It serves both as a comprehensive scene description language enabling fine-grained modification of 3D software elements and as a formal constraint-expressive specification language capable of expressing complex spatial constraints. By decomposing 3D software synthesis into stages operating on ScenethesisLang, Scenethesis enables independent verification, targeted modification, and systematic constraint satisfaction. Our evaluation demonstrates that Scenethesis accurately captures over 80% of user requirements and satisfies more than 90% of hard constraints while handling over 100 constraints simultaneously. Furthermore, Scenethesis achieves a 42.8% improvement in BLIP-2 visual evaluation scores compared to the state-of-the-art method.

💡 Research Summary

The paper addresses the under‑explored problem of automatically generating three‑dimensional (3D) user‑interface software from natural‑language (NL) specifications. While 2D UI synthesis has matured with model‑based, template‑driven, and constraint‑based techniques, existing 3D approaches either generate an entire scene end‑to‑end or rely on simplistic scene‑graph representations that cannot express the rich, continuous spatial, physical, and semantic constraints required by real‑world applications. These limitations lead to two major challenges: (C1) lack of compositional control and maintainability—any minor change forces a full regeneration of the scene; and (C2) inability to handle complex constraints such as distance thresholds, collision avoidance, and domain‑specific rules simultaneously.

To overcome these challenges, the authors introduce Scenethesis, a requirement‑sensitive 3D software synthesis framework built around a novel domain‑specific language called ScenethesisLang. ScenethesisLang serves as a granular, constraint‑aware intermediate representation (IR) that bridges NL requirements and executable Unity‑compatible 3D software. It combines a scene‑description language (objects, transforms, materials) with a formal spatial‑constraint language capable of expressing continuous values, multiple simultaneous relationships, and logical compositions (e.g., “all fire extinguishers must be within 2 m of any workstation while keeping evacuation paths at least 1.5 m wide”).

The system decomposes the synthesis task into four independent, verifiable stages:

1. Requirement Formalization – An LLM with few‑shot prompting first classifies the scene (indoor/outdoor) and expands the user query with inferred hidden constraints (e.g., lighting, accessibility). The expanded prompt is split into region‑wise sub‑prompts, each of which is translated into ScenethesisLang statements, making implicit physical laws (gravity, boundary limits) explicit.

2. Asset Synthesis – Object declarations in the IR are resolved to concrete 3D assets. The pipeline uses a hybrid strategy: it retrieves high‑quality models from curated repositories when possible, and falls back to text‑to‑3D generation for missing items, balancing visual fidelity and coverage.

3. Spatial Constraint Solving – Placement of objects is formulated as a continuous constraint‑satisfaction problem (CSP). The authors propose the Rubik Spatial Constraint Solver, an iterative “twist‑and‑fix” algorithm inspired by solving a Rubik’s cube. Local adjustments propagate globally, allowing the solver to satisfy hundreds of constraints without the exponential blow‑up typical of SAT/SMT solvers.

4. Software Synthesis – Solved layouts and assets are assembled into Unity‑compatible scenes, complete with C# scripts, metadata, and APIs for programmatic manipulation. The generated artifacts embed traceability links back to the original ScenethesisLang specifications, enabling round‑trip engineering and targeted post‑generation edits.

The evaluation uses a newly constructed dataset of 50 diverse, highly detailed user queries (average length 508 words) covering various room types and functional requirements. Results show that Scenethesis correctly captures more than 80 % of user requirements even under a strict matching threshold, and satisfies over 90 % of hard constraints while handling more than 100 simultaneous constraints. Visual quality is measured with the BLIP‑2 model; Scenethesis achieves a 42.8 % improvement over the state‑of‑the‑art baseline Holodeck. The advantage holds across multiple LLM backbones, demonstrating robustness to the underlying language model.

Key contributions are:

• ScenethesisLang, a formal DSL that unifies scene description and expressive spatial‑constraint specification, supporting continuous values and complex logical compositions.
• A four‑stage modular pipeline that aligns with classic software‑engineering principles (modularity, inspectability, correctness, controllability), allowing independent development and verification of each stage.
• The Rubik Spatial Constraint Solver, which offers practical scalability for dense constraint sets through local‑to‑global refinement, avoiding the combinatorial explosion of traditional CSP solvers.
• Comprehensive empirical evidence showing superior requirement coverage, constraint satisfaction, and visual fidelity compared with existing end‑to‑end and scene‑graph‑based methods.

In summary, Scenethesis demonstrates that introducing a constraint‑expressive intermediate representation and a principled, staged synthesis process can bring 3D UI generation into the realm of maintainable, testable software engineering, bridging the gap between high‑level user intent and low‑level 3D implementation.