- Source
- arXiv
- Published
- Runtime
- 0:00
- Snippets
- 4
A conversation between
Self in Space: Benchmarking Self-Awareness and Spatial Cognition in UAV Embodied Intelligence
§02
Snippets
-
Current multimodal LLMs exhibit fundamental limitations in modeling dynamic, agent-centered processes—self-awareness remains implicit despite being essential for embodied spatial reasoning.
Self-awareness, not just scene understanding, is critical for autonomous systems to operate reliably in complex real-world environments.
-
SIS-Bench introduces a unified self-in-space benchmark organizing evaluation along two dimensions (space and self) across three cognitive levels (perception, memory, reasoning) with 4,856 QA pairs from real-world UAV videos.
Benchmarks that measure both environmental and self-awareness enable fairer comparison and development of truly embodied AI systems.
-
Performance degrades significantly across cognitive levels; spatial cognition outpaces self-awareness, revealing that current models handle static scenes better than dynamic, agent-centric understanding.
This performance gap pinpoints a concrete weakness—models need richer representations of motion and agency, not just visual content.
-
Motion-aware representations that fuse optical flow with visual features consistently improve both spatial cognition and self-awareness, and generalize to downstream UAV decision-making tasks.
Modeling agent motion is a practical, generalizable technique that lifts performance across multiple tasks, suggesting self-dynamics are learnable and beneficial.
§03
Synthesis
The Problem: UAVs Don't Know Themselves
Current AI systems for drones—built on multimodal large language models (MLLMs)—excel at describing their surroundings but fail at understanding their own role in the scene. A drone might identify "a building 50 meters away" without grasping "I am moving toward it at 5 meters per second." This self-awareness gap matters because embodied agents need coherent models of themselves and their environment to act intelligently. Existing benchmarks focus almost entirely on spatial understanding, leaving self-awareness implicit and untested.
The authors introduce SIS-Bench, a benchmark that forces explicit evaluation of both spatial cognition and self-awareness in UAV scenarios. The key insight: these two dimensions are distinct, and current MLLMs handle them unevenly.
How SIS-Bench Works
The benchmark organizes evaluation along two axes:
Dimensions: "Space" (understanding the environment) versus "Self" (understanding the agent's state, motion, and perspective).
Hierarchy: Three cognitive levels—perception (recognizing what is), memory (recalling what was), and reasoning (inferring what will be).
This creates a grid: a question might ask about spatial perception ("What's in front of the drone?"), self-awareness memory ("Where was the drone 3 seconds ago?"), or spatial reasoning ("Will the drone collide with that tree?").
The benchmark contains 4,856 question-answer pairs across 13 tasks, derived from 1,646 real-world UAV videos. The authors built it through a "task-conditioned construction pipeline"—essentially, they extracted video clips, used expert annotators to verify quality, and generated questions aligned to specific task types.
What They Found
Testing current MLLMs reveals a stark imbalance: models perform significantly better on spatial tasks than self-awareness tasks. Performance also degrades predictably as cognitive demands increase—reasoning is harder than memory, which is harder than perception. This suggests MLLMs treat the agent as invisible to their reasoning process.
The Fix: Motion-Aware Representations
Rather than accept this limitation, the authors propose incorporating the drone's own motion into the model's representations. They fuse optical flow (which captures pixel-level motion patterns) with visual features. This simple addition—making the model "see" motion—improves both perception and memory performance on spatial and self-awareness tasks. Crucially, gains transfer to downstream decision-making tasks for UAVs, suggesting the improvement is genuinely useful, not a benchmark artifact.
Why It Matters
Embodied AI requires agents to model themselves as dynamic entities, not passive observers. A robot or drone that doesn't understand its own momentum, viewpoint, or trajectory will struggle with navigation, obstacle avoidance, and collaborative tasks. This work makes that problem visible through a concrete benchmark and demonstrates that motion-aware learning is a practical step forward. The benchmark itself will likely become a standard evaluation tool for embodied spatial reasoning, pushing future MLLM development to close the self-awareness gap.
Mine your own.
Lode is a workbench, not a feed. Paste a YouTube URL. The model proposes a transcript, a set of quote-grounded snippets, a synthesis essay, and the fan-out. You decide what stays.