Building a Mind Palace: Structuring Environment-Grounded Semantic Graphs for Effective Long Video Analysis with LLMs

Long-form video understanding with Large Vision Language Models is challenged by the need to analyze temporally dispersed yet spatially concentrated key moments within limited context windows. In this work, we introduce VideoMindPalace, a new framework inspired by the “Mind Palace”, which organizes critical video moments into a topologically structured semantic graph. VideoMindPalace organizes key information through (i) hand-object tracking and interaction, (ii) clustered activity zones representing specific areas of recurring activities, and (iii) environment layout mapping, allowing natural language parsing by LLMs to provide grounded insights on spatio-temporal and 3D context. In addition, we propose the Video MindPalace Benchmark (VMB), to assess human-like reasoning, including spatial localization, temporal reasoning, and layout-aware sequential understanding. Evaluated on VMB and established video QA datasets, including EgoSchema, NExT-QA, IntentQA, and the Active Memories Benchmark, VideoMindPalace demonstrates notable gains in spatio-temporal coherence and human-aligned reasoning, advancing long-form video analysis capabilities in VLMs.

💡 Research Summary

The paper tackles the long‑standing challenge of understanding long‑form egocentric videos with large vision‑language models (VLMs). While such videos often span many minutes, the most informative moments are temporally dispersed yet spatially concentrated in a few activity zones. Existing approaches either densely caption every frame or sample uniformly, leading to information overload and poor temporal coherence when feeding the data to large language models (LLMs) with limited context windows.

To address this, the authors introduce VideoMindPalace, a framework inspired by the “mind palace” mnemonic technique. The core idea is to compress a video into a three‑layer hierarchical semantic graph that can be directly consumed by a text‑only LLM. The layers are:

1. Human‑Object Layer – Using RT‑DETR for object detection and ByteTrack for multi‑object tracking, the system extracts per‑frame object categories, bounding boxes, and tracking IDs. Hand keypoints are combined with object proximity to infer hand‑object interaction edges, forming a sub‑graph of human‑object relations for each frame.

2. Activity‑Zone Layer – By leveraging RGB frames together with camera pose information, the method clusters spatially coherent regions where actions repeatedly occur (e.g., kitchen counter, sofa). Each zone is associated with the Human‑Object sub‑graph of the frames that fall inside it, thereby aggregating all interactions that happen in that zone.

3. Scene‑Layout Layer – A higher‑level graph captures the topological arrangement of rooms or large spaces. Using a 3‑D mapping model, relative distances and connectivity (doors, corridors) between zones/rooms are encoded as edges.

The complete graph is serialized into JSON and supplied as a prompt to an LLM. The LLM, guided by predefined templates, can query node attributes (object type, 3‑D coordinates) and edge relations (temporal order, spatial adjacency) to answer natural‑language questions such as “Where did I use the cup?” or “Is there an unobstructed path between the kitchen and the living room?”.

To evaluate human‑like reasoning, the authors propose the Video MindPalace Benchmark (VMB), which consists of three question families:

• Enhanced Spatial Localization – Requires precise spatial relationships between objects.
• Contextual Temporal Reasoning – Demands event‑based answers that mimic human memory sequencing.
• Layout‑Aware Sequential Reasoning – Tests understanding of navigation and spatial layout across rooms.

VMB mixes multiple‑choice and open‑ended formats, providing a more nuanced assessment than prior egocentric benchmarks (Ego4D‑QA, AMB).

Experimental Findings

• On VMB, VideoMindPalace achieves a 7.3 percentage‑point gain over the strongest prior VLM baselines.
• On established datasets (EgoSchema, NExT‑QA, IntentQA, Active Memories Benchmark) it improves accuracy by 4–6 pp, with especially large gains in spatio‑temporal consistency metrics.
• Ablation studies reveal that removing the activity‑zone clustering drops performance by ~3 pp, while omitting the layout layer harms spatial localization, confirming the complementary role of each graph tier.

Limitations & Future Work
The current hand‑object interaction detection relies on 2‑D cues, limiting depth accuracy. Complex multi‑room layouts sometimes cause layout‑graph errors. The pipeline is offline; real‑time deployment would require faster graph updates and tighter integration with LLMs for interactive reasoning. Future directions include 3‑D point‑cloud‑based tracking, incremental graph construction, and joint training of perception modules with LLMs to close the modality gap.

In summary, VideoMindPalace offers a novel, memory‑inspired representation that efficiently encodes the essential spatio‑temporal structure of long videos, enabling LLMs to perform human‑aligned reasoning with far fewer tokens and higher fidelity. The accompanying VMB benchmark sets a new standard for evaluating such capabilities in egocentric video AI.