Scal3R Learning Efficient Multi-Relative Pose Query for Scalable Online 3D Reconstruction

Chin-Yang Lin1,2 Yang-Che Sun1 Cheng Sun2 Fu-En Yang2 Min-Hung Chen2 Yen-Yu Lin1 Wei-Chen Chiu1 Yu-Lun Liu1
1National Yang Ming Chiao Tung University      2NVIDIA
ECCV 2026
Paper arXiv Video Code
TL;DR: Scalable online 3D reconstruction on kilometer-scale sequences, with only ~1% extra parameters on a frozen backbone trained in 8 hours on a single GPU.

Abstract

Online 3D reconstruction models perform poorly on long videos. This happens because regressing poses relative to a fixed first-frame anchor forces extrapolation far beyond the training distribution. Small drifts accumulate and amplify into significant geometric collapse. However, we observe that per-frame depth remains stable throughout this failure. The backbone's local geometry remains intact; only the global pose head breaks down. Motivated by this decoupling, we introduce Scal3R. This approach reformulates online reconstruction as multi-reference relative pose querying. We use lightweight learnable tokens, which make up about ~1% of the parameters, and inject them into a completely frozen backbone via asymmetric attention. This setup queries poses relative to multiple past keyframes. An online pose-graph optimization system with loop closure suppresses long-range drift. Scal3R reaches convergence in 8 hours on a single GPU. It reduces the average ATE by over 60% on KITTI compared to the online baseline. It also achieves state-of-the-art performance across Virtual KITTI, Sintel, TUM-Dynamic, ScanNet, and 7-Scenes.


Online long-sequence reconstruction

Input stream
BEV Trajectory
3D Reconstruction

Scal3R reconstructs globally consistent 3D geometry online from long driving sequences with minimal trajectory drift. Pick a scene from the strip below to view its input stream, bird's-eye-view (BEV) trajectory, and the resulting 3D reconstruction. On KITTI and vKITTI scenes, toggle the 3D Reconstruction panel between the rendered Video and an Interactive 3D point cloud with the estimated camera-pose trajectory (drag to orbit).



Method

step 1 / 4
1
Frozen image path
Images flow through the frozen encoder into the decoder as image tokens.
2
Trainable pose queries
Reference camera tokens pass one tiny trainable MLP and become pose query tokens (~1%).
3
Frozen decoder
Asymmetric attention injection inside the completely frozen 3R decoder (CUT3R, STream3R).
4
Two heads, one loss
Point Head untouched: pointmaps identical, no loss; a small Pose Head learns relative poses: the only training signal.
Overview of the Scal3R framework. For an incoming frame, a frozen encoder extracts dense image tokens. Historical camera tokens from selected reference frames are projected using lightweight trainable MLPs to generate relative pose tokens. These tokens are concatenated and sent into a completely frozen 3D reconstruction decoder. Our asymmetric attention injection mechanism ensures that relative pose tokens serve only as queries to extract geometric cues, while image tokens perform self-attention exclusively among themselves. This preserves the original high-quality point cloud generation through the frozen Point Head, while the trainable Relative Pose Head predicts robust multi-reference relative transformations. Finally, an online inference backend (PGO and loop closure) aggregates these local constraints to produce a globally consistent trajectory.
Multi-reference relative pose querying
Instead of regressing every pose against a single first-frame anchor, each incoming frame queries its relative pose to K reference keyframes in parallel. Every query stays a short-range, in-distribution prediction; a revisited place simply re-enters the buffer as one more reference, and online pose-graph optimization fuses all constraints into a globally consistent trajectory.
Frame Keyframe, kept as reference Current frame Relative-pose query Loop closure
step 1 / 4
1
Keep a few references
Keyframes are cached in a small reference buffer as the stream flows in.
2
Query, don’t extrapolate
K short-range queries per frame, always in-distribution.
3
Close loops for free
A revisit is just one more reference, with no architectural change.
4
Fuse, online
The pose graph re-solves as frames arrive.
Asymmetric attention injection
Pose tokens ride the frozen stack, and inside every layer they may only ask questions: injected as extra queries (Q′), they read the image tokens’ keys and values and never write back. The pretrained image representations stay intact.
Image tokens · frozen backbone Relative pose tokens · trainable ❄️Frozen 🔥Trainable
Transformer Layer Transformer Layer Transformer Layer ❄️ ❄️ ❄️ ••• Image Tokens 🔥 Relative Pose Tokens → pointmaps (Point Head ❄️) Pose Head 🔥 inside one frozen layer image tokens pose tokens WV ❄️ WK ❄️ WQ ❄️ V K Q Q′ frozen WQ → extra queries Q′ Attention untouched ❄️ Attention read-only 🔥 Q′ reads image K & V nothing ever flows back The new tokens read a frozen expert. They can’t break it.
step 1 / 4
1
Ride the frozen stack
Pose tokens climb every frozen layer alongside the image tokens.
2
Same frozen projections
Inside a layer, both streams pass through the same frozen WV, WK, WQ.
3
Emerge as queries only
Image tokens yield V, K, Q; pose tokens yield only extra queries Q′.
4
Read, never write back
Q′ reads the image’s K & V; image self-attention runs untouched.

Qualitative comparison

Ours (Scal3R)
CUT3R
Point cloud: Ours (Scal3R)
Point cloud coming soon
Point cloud: CUT3R
Point cloud coming soon

Rendered 3D reconstructions of Ours (Scal3R) versus the backbone baseline on long driving sequences. On every scene you can toggle the baseline backbone between CUT3R and STream3R. Pick a scene from the strip below; the selected thumbnail plays its input video.



Loop closure

Before loop closure · ATE 49.8 m

Loop closure falls out of the same design: when the camera revisits a place, one archived keyframe is re-injected as an extra reference, a single long-range constraint, and the map snaps shut. Watch the ATE fall from 49.8 m to 6.4 m as the map closes.

Robustness analysis

Per-frame ATE on KITTI (m)
Per-frame ATE curves sorted in ascending order across all KITTI sequences. Scal3R (CUT3R and STream3R backbones) maintains the lowest per-frame ATE throughout the entire evaluation range, while competing methods such as MUSt3R and Point3R suffer catastrophic divergence at moderate sequence lengths, confirming superior robustness under challenging long-sequence conditions.

BibTeX

@inproceedings{lin2026scal3r,
    title     = {{Scal3R}: Learning Efficient Multi-Relative Pose Query for Scalable Online 3D Reconstruction},
    author    = {Lin, Chin-Yang and Sun, Yang-Che and Sun, Cheng and Yang, Fu-En and Chen, Min-Hung and Lin, Yen-Yu and Chiu, Wei-Chen and Liu, Yu-Lun},
    booktitle = {European Conference on Computer Vision (ECCV)},
    year      = {2026}
}