Geometry across depth & timeSingle egocentric RGB stream
Two complementary selection problems. Fuse geometry from multiple representation depths into the navigation policy, and retain useful geometric history under a bounded memory budget. Open full size ↗
55.7%R2R-CE success rateVal-Unseen · main benchmark
54.1%RxR-CE success rateVal-Unseen · main benchmark
39.7%Lower mean GFM-KV memory900K vs. Hybrid Inc. 8+48 · R2R-CE
0Extra navigation samplesBeyond standard benchmark training data
01 / Overview
Geometry to reason. Memory to navigate.
Single RGBQwen3.5-4BFrozen VGGT-1B
Vision-language navigation requires an embodied agent to align language with visual observations while maintaining a spatially coherent understanding of its environment. Geometry foundation models expose representations throughout their processing hierarchy, but using only a terminal feature can leave complementary geometric information inaccessible to the navigation policy.
AdaGeoVLN selects geometry across both representation depth and navigation time. Hierarchical GFM–VLM fusion couples earlier, intermediate, and later VGGT representations to successive stages of the navigation policy. Navigation-aware memory retains historical VGGT global-attention key–value states according to instruction relevance, geometric confidence, and transition novelty.
Experiments on R2R-CE and RxR-CE show strong performance with a single RGB stream and no additional navigation-specific external training data. Controlled ablations support the value of multi-depth fusion, while navigation-aware retention improves SR and SPL at a similar measured KV footprint and reduces memory relative to larger-memory temporal retention.
02 / Method
Select across depth. Retain across time.
The policy combines a Qwen3.5-4B vision-language model with a frozen VGGT-1B geometry encoder. Two mechanisms determine how geometry enters reasoning and remains available at future steps.
A / Representation depth
Hierarchical geometry fusion
Geometry from three VGGT depths is aligned with successive language-decoder stages. Each branch uses normalization, 2 × 2 spatial grouping, a depth-specific projection, a token-wise gate, and a learned scale.
VGGT depth 11→Decoder stage 0
VGGT depth 17→Decoder stage 1
VGGT depth 23→Decoder stage 2
Residual updates are added to image-token states after decoder layers L₀, L₁, and L₂; text-token states are not directly overwritten. No predefined semantic role is assigned to an individual VGGT depth.
B / Navigation time
Navigation-aware geometric memory
Historical and current VGGT global-attention KV states compete for a fixed per-layer budget. Three complementary signals determine which patch tokens to retain:
01
Instruction relevance
Maximum centered cosine similarity between projected shallow tokens and instruction segments; cached at insertion.
02
Geometric confidence
The minimum of calibrated depth and point-map confidence, requiring both predictors to be reliable.
03
Transition novelty
Nearest-other-frame descriptor novelty across all observed frames, refreshed at each step. Relevance and confidence enter the score separately.
Per-layer z-score normalization combines the three signals with fixed equal weights. GHOST-derived layer allocations distribute the total budget; camera/register tokens have priority within it. Selection follows the current VGGT pass and affects the next observation.
Hierarchical geometry fusion. Frozen VGGT features enrich visual tokens at successive VLM decoder stages, followed by the language head that generates navigation action tokens. Open full size ↗Navigation-aware geometric memory. Candidate states are scored using instruction relevance, geometric confidence, and transition novelty. Each layer retains a bounded set of KV states for the next step. Open full size ↗
Input: RGB observations + language instructionTraining: R2R + RxR · 8× H100 · approximately 100 hoursInference: no odometry or simulator depth
03 / Benchmark results
From instructions to successful navigation.
Results on 1,839 R2R-CE and 3,669 RxR-CE Val-Unseen episodes. The tables preserve the paper’s sensing assumptions and external training data so the comparisons can be read in context.
↑ Higher is better ↓ Lower is better
Table I · R2R-CE Val-Unseen
Method
Observation
NE ↓
OS ↑
SR ↑
SPL ↑
External data
HPN+DN
Pano. + odo. + depth
6.31
40.0
36.0
34.0
—
Sim2Sim
Pano. + odo. + depth
6.07
52.0
43.0
36.0
—
VLN⟳BERT
Pano. + odo. + depth
5.74
53.0
44.0
39.0
—
Ego²-Map
Pano. + odo. + depth
5.54
56.0
47.0
41.0
—
DreamWalker
Pano. + odo. + depth
5.53
59.0
49.0
44.0
—
AO-Planner
Pano. + depth
5.55
59.0
47.0
33.0
—
g3D-LF
RGB + odo. + depth
5.70
59.5
47.2
34.6
—
Seq2Seq
RGB + depth
7.77
37.0
25.0
22.0
—
NaVid-4D
RGB + depth
5.99
55.7
43.8
37.1
—
NavMorph
RGB + depth
5.75
56.9
47.9
33.2
—
NaVid
Single RGB
5.47
49.1
37.4
35.9
953K
Sim2Real
Single RGB
5.95
55.8
44.9
30.4
0K
StreamVLN*
Single RGB
5.98
51.3
45.6
42.3
—
Uni-NaVid
Single RGB
5.58
53.3
47.0
42.7
3,577K
NaVILA*
Single RGB
5.37
57.6
49.7
45.5
12,574K
JanusVLN*
Single RGB
5.17
58.0
52.8
49.2
0K
AdaGeoVLN
Single RGB
5.27
60.7
55.7
51.4
0K
Table II · RxR-CE Val-Unseen
Method
Observation
NE ↓
SR ↑
SPL ↑
nDTW ↑
External data
VLN⟳BERT
Pano. + odo. + depth
8.98
27.0
22.6
46.7
—
AO-Planner
Pano. + depth
7.06
43.3
30.5
50.1
—
Seq2Seq
RGB + depth
12.10
13.9
11.9
30.8
—
NavMorph
RGB + depth
8.85
30.8
22.8
44.2
—
Sim2Real
Single RGB
8.79
36.7
25.5
18.1
0K
StreamVLN
Single RGB
6.72
48.6
42.5
60.2
—
Uni-NaVid
Single RGB
6.24
48.7
40.9
—
3,577K
NaVILA
Single RGB
6.77
49.3
44.0
58.8
13,132K
JanusVLN*
Single RGB
6.46
51.4
44.3
59.1
0K
AdaGeoVLN
Single RGB
5.64
54.1
44.7
61.8
0K
NE is navigation error in meters; OS is oracle success; SR is success rate; SPL is success weighted by path length; nDTW is normalized dynamic time warping. All metrics except NE use a 0–100 scale. Pano.: panoramic observations; odo.: odometry. External data counts samples beyond R2R/RxR; — denotes an unreported total or metric. R2R-CE StreamVLN* is the oracle-navigation ablation; RxR-CE uses standard StreamVLN, whose total external sample count including EnvDrop is not separately quantified. NaVILA* excludes human-following data; JanusVLN* uses no additional external data.
+2.9 / +2.7 pts
SR gains over JanusVLN* on R2R-CE and RxR-CE, respectively. SPL improves by 2.2 and 0.4 points. On RxR-CE, nDTW increases by 2.7 points and NE decreases by 0.82 m. All use a single RGB stream, with no additional navigation training samples beyond R2R/RxR.
Does representation depth matter?
The repeated-deep control uses the same three fusion locations, but injects the terminal VGGT representation at every location.
Table III · Geometry fusion · R2R-CE Val-Unseen
Geometry configuration
SR ↑
SPL ↑
OS ↑
NE ↓
No geometry · Qwen3.5 SFT
46.5
42.3
56.0
5.19
Single deep · G₂₃ → L₂
48.9
44.1
56.0
5.28
Deep × 3 · G₂₃
42.1
36.9
56.7
6.64
Hierarchical · G₁₁ / G₁₇ / G₂₃
55.7
51.4
60.7
5.27
Hierarchical fusion improves SR/SPL by 9.2/9.1 points over no geometry, 6.8/7.3 over single-deep fusion, and 13.6/14.5 over Deep × 3 at matched fusion locations. The geometry-free policy retains a slightly lower NE (5.19 versus 5.27 m).
04 / Memory efficiency
Keep useful geometry. Bound the history.
Selected retention strategies on R2R-CE Val-Unseen compare navigation accuracy with measured GFM-KV and allocated GPU memory. Token budgets and measured memory footprints are reported separately.
39.7% less geometric KV memory
Mean retained VGGT global-attention KV memory on R2R-CE Val-Unseen.
Hybrid Inc. (8+48)5,767.55 MB
900K retention budget3,475.92 MB
55.7% SR · 51.4% SPL
Nav. 900K improves SR/SPL by 0.4/1.4 points over Hybrid Inc. (8+48), which achieves 55.3% SR / 50.0% SPL.
What does the budget mean?
Aggregate layer-token slots
900,000
VGGT global-attention layers
24
Mean allocated GPU memory
−20.2%
Mean GFM-KV memory
−39.73%
Nav. 900K mean allocated GPU memory
19,275.87 MB
The 900K budget counts token slots across all 24 layers, rather than unique scene tokens. Each layer independently retains its highest-scoring candidates, with special camera/register tokens given retention priority.
Relative to Hybrid Inc. (8+48), mean allocated GPU memory decreases from 24,161.94 to 19,275.87 MB (20.22%); mean GFM-KV memory decreases from 5,767.55 to 3,475.92 MB (39.73%).
Figure 4 · Geometry fusion and retention. SR and SPL versus mean allocated GPU memory, with navigation-aware budgets of 600K, 800K, and 900K. Savings compare Nav. 900K with Hybrid Inc. (8+48). Open full size ↗
Accuracy and memory across retention strategies
Table IV · Retention strategies · R2R-CE Val-Unseen
Retention
SR ↑
SPL ↑
OS ↑
NE ↓
Mean KV (MB) ↓
Mean GPU (MB) ↓
Hybrid Inc. (8+24)
55.1
50.3
62.4
5.30
3,595.53
21,263.14
Hybrid Inc. (8+48)
55.3
50.0
61.5
5.21
5,767.55
24,161.94
GHOST
50.0
48.0
54.9
5.55
6,758.17
23,075.50
InfiniteVGGT
54.1
49.6
60.3
5.30
6,854.04
22,066.32
Nav. 600K
53.1
49.2
59.2
5.42
2,340.81
18,133.58
Nav. 800K
54.2
50.2
59.5
5.38
3,105.28
18,903.07
Nav. 900K
55.7
51.4
60.7
5.27
3,475.92
19,275.87
At a similar measured KV footprint, Nav. 900K improves SR/SPL by 0.6/1.1 points over Hybrid Inc. (8+24), with 3.33% less mean GFM-KV memory and 9.35% less mean allocated GPU memory. It achieves the highest SR and SPL among the selected configurations; Hybrid Inc. (8+24) retains the highest OS, while Hybrid Inc. (8+48) has the lowest NE.
Hybrid Inc. retains the first 8 observations plus the most recent 24 or 48. Configured layer-token capacity, KV storage, and allocated GPU memory are distinct quantities. Increasing the navigation-aware budget from 600K to 900K improves SR/SPL by 2.6/2.2 points for an additional 1,142.29 MB of mean allocated GPU memory.
Contribution of the retention signals
Table V · Leave-one-signal-out ablation · 900K · R2R-CE
Retention score
SR ↑
SPL ↑
OS ↑
NE ↓
KV (MB)
All signals
55.7
51.4
60.7
5.27
3,475.92
Without transition
54.3
50.3
60.1
5.41
3,475.96
Without confidence
53.6
49.6
59.4
5.48
3,475.85
Without instruction
54.2
50.1
59.2
5.45
3,475.76
Removing any signal reduces SR and SPL while mean KV memory varies by only 0.20 MB. Removing confidence causes the largest SR decrease (2.1 points), followed by instruction relevance (1.5) and transition novelty (1.4). These paired runs support each signal’s contribution, not the optimality of the equal weights.
05 / Navigation in action
Spatial reasoning along the route.
Four Habitat episodes, one from each scene, compare AdaGeoVLN against JanusVLN and Qwen3.5 SFT under the same navigation instruction.
VLN-CE visualization 1
Instruction: Take a right after the pool table and take your first left and walk into the room and wait in between the two rooms.
Qwen3.5 SFT✕JanusVLN✕AdaGeoVLNOurs✓
VLN-CE visualization 2
Instruction: Walk into the hall ahead. Walk down the flight of stairs with an iron railing. Continue to the bottom of the stairs and stop near the bench at the bottom next to a closed door.
Qwen3.5 SFT✕JanusVLN✕AdaGeoVLNOurs✓
VLN-CE visualization 3
Instruction: Exit the laundry room and walk up the stairs to the left. Continue past the carpeted stairs and turn right. Turn right again and enter the living room. Wait near the coffee table.
Qwen3.5 SFT✕JanusVLN✕AdaGeoVLNOurs✓
VLN-CE visualization 4
Instruction: Walk past the shelf and down the hallway to the left of the double doors. At the end of the hallway turn right and stop in front of the end table.
Qwen3.5 SFT✕JanusVLN✕AdaGeoVLNOurs✓
Figure 5 · Simulation rollouts and humanoid deployment. Top: AdaGeoVLN reaches the bedroom doorway after staircase traversal, while Qwen3.5 SFT and JanusVLN fail in this example. Bottom: two representative indoor Unitree G1 navigation sequences. Open full size ↗
06 / Real-world deployment
AdaGeoVLN on Unitree G1.
A ZED X Mini supplies the RGB stream. An onboard NVIDIA Jetson AGX Orin handles image preprocessing and communication; policy inference runs remotely on an NVIDIA RTX A6000.
