Boosting MLLM Spatial Reasoning with Geometrically Referenced 3D Scene Representations

While Multimodal Large Language Models (MLLMs) have achieved remarkable success in 2D visual understanding, their ability to reason about 3D space remains limited. To address this gap, we introduce geometrically referenced 3D scene representations (GR3D). Given a set of input images, GR3D annotates objects in the images with unique IDs and encodes their 3D geometric attributes as textual references indexed by these IDs. This representation enables MLLMs to interpret 3D cues using their advanced language-based skills in mathematical reasoning, while concurrently analyzing 2D visual features in a tightly coupled way. We present a simple yet effective approach based on GR3D, which requires no additional training and is readily applicable to different MLLMs. Implemented in a zero-shot setting, our approach boosts GPT-5’s performance on VSI-Bench by 8% overall and more than 11% on tasks that rely heavily on spatial layout understanding. Qualitative studies further demonstrate that GR3D empowers MLLMs to perform complex spatial reasoning with highly sparse input views.

💡 Research Summary

The paper addresses a critical gap in current multimodal large language models (MLLMs): while they excel at 2‑D visual understanding, they struggle with reasoning about three‑dimensional space. Existing attempts to endow MLLMs with 3‑D capabilities either fine‑tune them on limited 3‑D datasets or feed auxiliary modalities such as point clouds, but these approaches suffer from data scarcity, costly training, and shallow spatial comprehension.

To overcome these limitations, the authors introduce Geometrically Referenced 3‑D scene representations (GR3D). GR3D is a dual‑modal representation that tightly couples 2‑D visual cues with explicit 3‑D geometric information. The pipeline consists of three main stages:

1. 3‑D Scene Reconstruction – Using state‑of‑the‑art neural reconstruction models (e.g., DUSt3R, VGGT), a set of uncalibrated, unposed images is converted into dense depth maps, camera intrinsics/extrinsics, and a global point cloud. Because neural reconstructions are defined only up to an unknown scale, the authors recover a metric scale by comparing the reconstructed heights of a few reference objects (ceilings, countertops, desks) with typical real‑world dimensions.

2. Object‑Level Geometry Extraction – 2‑D semantic segmentation (Mask2Former trained on ADE20K) is back‑projected onto the point cloud. A voxel‑grid aggregation yields spatially connected clusters with consistent semantic labels, which are treated as object candidates. For each cluster the system fits simple geometric primitives: axis‑aligned bounding boxes for most objects, cylinders for round tables, etc., using RANSAC or learning‑based fitting. The resulting parameters (center coordinates, side lengths, radii, heights) are formatted as human‑readable text strings, each prefixed with a unique object ID.

3. Text‑Image Association – Object centers are projected into each image using the recovered camera matrices. The corresponding ID is overlaid on the image at the projected pixel location. To avoid annotating occluded objects, the projected depth is compared against the per‑pixel depth map; if the object’s depth exceeds the measured depth, the ID is omitted. This yields a set of annotated images where every visible object is marked with its ID, and a parallel textual list that describes the same objects’ 3‑D geometry.

The final input to an MLLM consists of (a) the annotated images, (b) the textual geometric list, and (c) a task‑specific prompt that asks a spatial question (e.g., “What is the distance from the chair to the stainless trash bin?”). Because the geometric data are expressed in natural language, the MLLM can leverage its strong mathematical and logical reasoning abilities—already demonstrated on chain‑of‑thought style problems—to compute distances, infer directions, and reason about relative positions without any additional fine‑tuning.

The authors evaluate the approach on the Visual‑Spatial Intelligence Benchmark (VSI‑Bench), a large‑scale dataset containing over 5,000 question‑answer pairs derived from 288 indoor videos. They reconstruct scenes from subsampled frames using VGGT, generate semantic masks with Mask2Former, and apply the GR3D pipeline described above. In a zero‑shot setting (no training on VSI‑Bench or related data), GPT‑5 equipped with GR3D achieves an 8 % absolute gain in overall accuracy compared to the baseline GPT‑5 without GR3D. The improvement is even more pronounced on tasks that heavily depend on layout understanding, where accuracy rises by more than 11 %. Ablation studies confirm that both the ID‑based image annotation and the depth‑based occlusion filtering are essential; removing either component degrades performance substantially.

A notable strength of GR3D is its robustness to sparse visual input. When only a few viewpoints are available, the global 3‑D coordinate system still provides spatial anchors, allowing the model to infer the positions of unobserved objects based on their relationships to the annotated ones. This contrasts with prior methods that rely on a bird’s‑eye‑view (BEV) image or require dense multi‑view coverage.

In summary, the paper makes three key contributions:

1. GR3D Representation – A novel, training‑free way to present 3‑D scene geometry to language models by linking textual descriptions with ID‑annotated images.
2. Zero‑Shot Spatial Reasoning Boost – Demonstrating that simply feeding GR3D to an off‑the‑shelf MLLM dramatically improves performance on a comprehensive spatial reasoning benchmark.
3. Sparse‑View Capability – Showing that GR3D enables reliable reasoning even when the visual input is highly limited, a scenario where most existing 3‑D‑aware models fail.

The work opens a promising direction for leveraging the powerful language‑centric reasoning of MLLMs in 3‑D domains without the need for costly fine‑tuning, and suggests future extensions such as richer primitive fitting, real‑time reconstruction pipelines, and applications in robotics, AR/VR, and embodied AI.