RegimeVGGTLayer-Wise Spatially Preserving Redundancy Removal
for Visual Geometry Grounded Transformer

Jinhao You1,*Shuo Lyu1,*†Zhuohang Lyu1,*Tanxuan Li1,*Zibo Zhao1,*Jiaxiang Hu2Kai Tang3Yichen Guo3
1 University of Pennsylvania2 University of California, Irvine3 Nanyang Technological University

* Equal contribution   † Corresponding author

University of PennsylvaniaUC IrvineNanyang Technological University Singapore

Not every transformer layer needs the same computation.
RegimeVGGT accelerates multi-view 3D reconstruction by adapting compression to each layer’s role—while preserving the cross-frame updates that camera pose needs.

NeurIPS 2026

Explore the reconstruction. Choose a scene to explore fresh RegimeVGGT reconstructions from real RGB views. Drag to rotate, zoom into the geometry, and show the predicted camera path.
About this visualization

Geometry comes from predicted depths and camera poses. Display points are confidence-filtered and voxel-downsampled to at most 300,000 points. A rigid rotation, centering and uniform scaling fit each scene to the viewer. The camera path uses model predictions; up to 40 camera frustums are drawn for readability. Ground-truth geometry is not used. These qualitative demos are separate from the paper benchmark and the original comparison below.

Abstract

Visual Geometry Grounded Transformer (VGGT) recovers dense 3D structure from multi-view images in a single forward pass, but quadratic cross-frame attention limits its scalability. Our spectral, probing, and causal analyses reveal three distinct layer regimes: shallow layers lack cross-view structure, middle layers drive geometric alignment, and deep layers are redundant for dense geometry yet remain essential for camera pose. RegimeVGGT follows this structure with two complementary mechanisms: saliency-guided token merging and selectively protected K/V downsampling. Applied to frozen VGGT weights without retraining, it reduces inference time from 477.3 s to 71.5 s on ScanNet-50 at 1,000 input frames, with a small change in reconstruction Chamfer Distance.

01Different layers, different roles

Where does a visual geometry transformer actually need cross-view computation? We study VGGT through attention spectra, feature probes, and causal interventions. The answer varies with depth, suggesting that compression should vary with it.

Layer heterogeneity. Attention spectra provide one view of the model’s internal structure; probing and causal interventions establish the different functional roles.
Layers 1–10

Shallow

Cross-view structure is still weak. Redundant computation can be compressed more aggressively.

Layers 11–18

Middle

Cross-view alignment takes shape. Conservative compression protects geometric reasoning.

Layers 19–24

Deep

Dense geometry is redundant, but cross-frame updates remain important for camera pose.

02Compress tokens. Preserve spatial structure.

RegimeVGGT uses a U-shaped compression policy: stronger in the shallow and deep regimes, gentler in the middle. Two mechanisms reduce attention computation along complementary axes, using the pretrained model without retraining.

RegimeVGGT overview. Layer-aware token merging and protected K/V downsampling reduce redundancy while retaining spatial coverage and the pose-critical path.

Saliency-guided banded merging

Geometry- and edge-salient patches guide which tokens to preserve. Redundant source patches are merged, then dense tokens are restored through merge–unmerge for subsequent processing.

Selectively protected K/V downsampling

A phase-shifted spatial grid reduces patch keys and values while retaining a full-density reference frame and uncompressed camera/register tokens. Patch queries continue to update, including in the deep layers.

Layer-wise inference policy
RegimeLayersSource-patch merge ratioK/V sampling factor
Shallow1–100.991.5
Middle11–180.501.3
Deep19–240.991.7

Merge ratios apply to eligible source patches, not all tokens. Camera/register tokens and a reference frame remain protected.

03Scaling to 1,000 views

On ScanNet-50, RegimeVGGT processes 1,000 input frames in 71.5 seconds, compared with 477.3 seconds for the memory-optimized VGGT* baseline—a 6.68× speedup. Reconstruction CD changes from 0.463 to 0.465, while the reported trajectory error is lower.

ScanNet-50 · 1,000 input frames
MethodTime ↓CD ↓ATE ↓
VGGT*477.3 s0.4630.117
FastVGGT113.5 s0.4640.111
RegimeVGGT71.5 s0.4650.092

Reported paper results. One NVIDIA H800 80 GB; input resolution 518 × 392. Lower is better for all three metrics. VGGT* is the memory-optimized baseline.

Latency and memory have different trade-offs: maximum peak memory is 51.8 GB for RegimeVGGT versus 45.3 GB for VGGT* in this setting.

04A plausible reconstruction can hide pose errors

What if only camera and register queries update inside the deep band? In the example below, the point cloud still retains recognizable room structure, but the camera trajectory deteriorates substantially. Evaluating dense geometry alone would miss this failure.

RegimeVGGT

RegimeVGGT point cloud at 500 framesRegimeVGGT trajectory at 500 frames: ATE 0.251 and ARE 6.192

Deep-asymmetric

Deep-asymmetric point cloud at 500 frames retains recognizable room structureDeep-asymmetric trajectory at 500 frames: ATE 1.119 and ARE 31.966
Geometry and pose at 500 frames. One qualitative ScanNet case. Trajectory annotations report ATE increasing from 0.251 to 1.119 and ARE from 6.192 to 31.966 under deep-asymmetric updates.

This motivates retaining patch-query updates while selectively reducing K/V context. At 1,000 frames, all three methods in this qualitative example show substantial drift and visible reconstruction degradation.

Original three-method comparison
Frames

VGGT baseline

VGGT baseline point cloud at 500 frames

RegimeVGGT ours

RegimeVGGT point cloud at 500 frames

Deep-asymmetric

Deep-asymmetric point cloud at 500 frames

Swipe to compare all three methods →

A scene, through three models. A qualitative ScanNet example at 500 input frames. Switch between reconstructed geometry and camera trajectories. Full figure ↗

05Resources & citation

Read the complete paper for the method, experiments, and analysis. The RegimeVGGT repository provides the implementation.

BibTeX

@article{you2026regimevggt,
  title={RegimeVGGT: Layer-Wise Spatially Preserving
    Redundancy Removal for Visual Geometry Grounded Transformer},
  author={You, Jinhao and Lyu, Shuo and Lyu, Zhuohang and
    Li, Tanxuan and Zhao, Zibo and Hu, Jiaxiang and
    Tang, Kai and Guo, Yichen},
  journal={arXiv preprint arXiv:2606.18439},
  year={2026}
}