Understanding Geometric Representations in Self-Supervised ViTs via Subspace Intervention

Abstract

We introduce a controlled subspace intervention framework to investigate how self-supervised Vision Transformers (ViTs) encode dense geometric information. While linear probing is widely used to assess geometric representations, it treats features as a black box, failing to disentangle the underlying topology. To address this issue, we decompose the weights of converged linear probes to isolate the low-rank subspaces containing explicit geometric signals using Singular Value Decomposition (SVD). Our perspective yields three key insights: (1) Pre-training objectives determine how features are encoded — DINOv2 aligns spatial features for efficient linear extraction, while Masked Autoencoders (MAE) tend to disperse these signals, requiring a broader spatial context. (2) Explicit geometric representations are highly compressible, suggesting dense predictive heads could potentially be constrained to low-rank subspaces with minimal performance loss. (3) The layer-wise task affinity suggests that geometric precision peaks at intermediate layers before yielding to semantic abstraction in the final layers. By connecting internal encoding mechanics with downstream performance, these findings provide a basis for effective feature selection and lightweight decoder design.

The Diagnostic Framework

From black-box probing to inspecting the feature topology

Rather than reading only the accuracy of a downstream probe, we (A) establish a readability gap with a three-tier probing hierarchy, then (B) perform a post-hoc subspace intervention that decomposes the converged linear weights to isolate the directions that carry explicit geometry.

A · Readability Gap

Three-tier probing hierarchy

A Linear probe measures explicitly rectified geometry; a MLP adds point-wise non-linearity (isolating local entanglement); a DPT decoder adds a global receptive field (isolating spatial fragmentation). The gaps between three-tier probes quantify how entangled vs. dispersed the geometric signal is.

B · Subspace Intervention

SVD on converged probe weights

We factor W = UΣV^⊤ and keep the top-k right singular vectors as the aligned basis. Projecting features onto S_k, R_k, or the residual S_k^⊥, then reading them through the frozen head, attributes any change purely to the geometric capacity of that subspace.

Key Insights

Three takeaways

01 / ENCODING

Pre-training shapes how geometry is represented

Self-distillation (DINOv2) aligns explicit geometry into a linearly accessible, low-rank coordinate system. Masked reconstruction (MAE) disperses the signal across broader spatial context, needing a global receptive field to decode.

02 / COMPRESSIBILITY

Geometry from a Low-Rank Subspace

Across every paradigm, a low-rank, task-aligned subspace is sufficient to preserve and reconstruct explicitly decodable geometry. Constraining dense predictive heads to this subspace incurs minimal performance loss, motivating lightweight decoders.

03 / TASK AFFINITY

Geometry peaks in the middle

Geometric precision concentrates in intermediate layers, then declines as deeper layers shift toward semantic abstraction, revealing a layer-wise affinity that argues against a terminal-only feature readout.

Results & Analysis

Quantitative results

Table 1 · Linear Readability Gap

**DINOv2 is nearly linearly readable** (0.9157 from a linear probe alone), whereas MAE relies on a global DPT decoder (+0.0989 over linear) — the gain bars (one shared scale) expose how much each probe tier adds: DINOv2 barely benefits, while MAE's large jumps reveal masked reconstruction's local entanglement and spatial dispersion. Metric: scale-aware depth accuracy on NYU Depth v2.
Backbone	Performance · SA-δ₁ ↑			Accessibility Gaps
Backbone	Linear	1×1 MLP	DPT	Local Entang.Δ Linear→MLP	Spatial Frag.Δ MLP→DPT
DINOv2-L	0.9157	0.9325	0.9483	+0.0168	+0.0158
iBOT-L	0.8198	0.8376	0.8524	+0.0178	+0.0148
MAE-L	0.6033	0.6390	0.7022	+0.0357	+0.0632

Table 2 · Cross-Layer Energy Allocation

**Relative geometric contribution E_i (%)** for four representative depths (each row sums to 100%). The bars are read against the dashed 25% uniform line, and σ is the spread around it — **lower σ = more uniform**. DINOv2 (σ=11.5) concentrates >72% of its energy in the intermediate layers (l₁₂, l₁₈) then drops sharply, whereas iBOT (σ=2.7) and MAE (σ=5.8) distribute the signal more uniformly.
Model	l₆	l₁₂	l₁₈	l₂₄	Layer distributionbars vs. 25% uniform (dashed) · σ = spread
DINOv2	17.2	35.8	36.7	10.3	σ 11.5 · concentrated
iBOT	26.9	28.4	22.4	22.3	σ 2.7 · most uniform
MAE	19.5	28.4	32.7	19.4	σ 5.8 · mild peak

Figures

Analysis gallery

Absolute recovery and recovery efficiency versus subspace rank k — **Fig. 2 · Compressibility.** (a) The task-aligned subspace recovers most of the signal well above the residual/random noise floor. (b) Normalized to each model's full-rank baseline, MAE saturates fastest (>98% by k=32), while DINOv2 converges slower, linearly encoding richer fine-grained geometry.

Subspace stability: similarity of extracted bases across three random seeds versus rank — **Fig. 8 · Stability.** Subspace similarity across three random probe initializations. High consistency at low ranks (k ≤ 16) rules out optimization artifacts; the stochastic tail (k ≥ 64) reflects a lack of stable signal in tail dimensions.

Qualitative depth predictions from the global linear probe across selected subspace ranks for DINOv2, MAE and iBOT — **Fig. 3 · Qualitative subspace interventions.** Depth predictions from the global linear probe across ranks k. DINOv2 (top) recovers a coherent scene layout at an extremely low rank (k=8); MAE (middle) needs higher capacity (k ≥ 64) to resolve object boundaries; iBOT (bottom) is intermediate.

Energy distribution across singular directions for DINOv2, iBOT and MAE — **Fig. 4 · Energy distribution.** Spectral-weighted energy density per singular vector (dashed line = uniform baseline, 0.25). DINOv2 shows multimodal peaks — geometry is localized in intermediate layers; iBOT and MAE overlap near the uniform baseline, diffusely spreading the signal across the hierarchy.

Layer-wise subspace analysis: performance evolution across layers and subspace ranks — **Fig. 5 · Layer-wise subspace analysis.** The entanglement gap (linear vs. non-linear) and the rank-dispersion gap (k=8) reveal a distinct rank-sensitivity transition in DINOv2's intermediate layers, contrasting with the continuous sensitivity of MAE and iBOT.

Qualitative task affinity in DINOv2: surface normal, depth and semantic segmentation decoded at layers 18, 21 and 24 — **Fig. 6 · Qualitative task affinity (DINOv2).** Predictions decoded from frozen features at layers 18, 21 and 24. Surface normals, depth and semantic segmentation peak at *different* layers — distinct affinities set by the task objective.

Supplementary Update Added Sep 2026

Does the mid-layer peak hold for DINOv3?

The question we were asked most often: does the late-layer decline of geometry we report for DINOv2 (Takeaway 03, Fig. 5a) also appear in DINOv3? To answer it we re-ran the layer-wise subspace analysis on DINOv3 ViT-L/16 under exactly the same protocol as Fig. 5: frozen features, a linear depth probe trained per layer on NYU Depth v2, SVD of the converged weights, and re-reading every layer through the fixed head at ranks k = 8 … full.

Layer-wise subspace analysis on DINOv3 ViT-L/16: depth accuracy (SA-delta-1) versus layer index for the full-rank linear probe and for subspace ranks k = 8, 16, 32, 64 and 128; the full-rank curve plateaus near 0.91 in the final layers while the k = 8 curve collapses — **Fig. S1 · Layer-wise subspace analysis on DINOv3 (ViT-L/16).** Same protocol, metric (SA-δ₁) and rank sweep as Fig. 5; the non-linear DPT curve is omitted. The full-rank linear probe (bold) climbs steadily through the network and holds at ≈0.91 over the final blocks — no late-stage decline. The low-rank readouts, however, spread apart: k = 8 (dotted) collapses in the deep layers, so the rank-diffusion gap (shaded) widens markedly toward the output.

1 · Full rank holds

Depth stays linearly readable to the last layer

DINOv3's full-rank probe reaches 0.89 about two-thirds of the way through the network and then plateaus at 0.90–0.91 for the remaining blocks. The same probe on DINOv2 (Fig. 5a) peaks and then gives back 0.044 by the last layer; on DINOv3 the peak-to-last-layer drop is 0.005.

0.925 → 0.881Δ −0.044DINOv2-L · peak → last

0.913 → 0.909Δ −0.005DINOv3-L · peak → last

2 · But rank diffuses

Deep-layer geometry is no longer low-rank

The k = 8 subspace tracks the full-rank curve only in the shallow layers (gap < 0.05), then collapses to 0.08 in the deep layers before partially recovering (0.47) at the output, and the intermediate ranks (k = 16–128) fan out widely. Depth in DINOv3's late layers remains linearly decodable, but it is dispersed over many more directions than in DINOv2.

3 · Reading

Tied to the pre-training recipe, not to ViTs in general

The mid-layer peak of Takeaway 03 is therefore paradigm-specific: DINOv3's final layers remain safe to read depth from at full rank, while the low-rank compressibility of Takeaway 02 holds only in its intermediate layers. DINOv3's Gram anchoring, designed to prevent dense-feature degradation, is a natural candidate for the difference, though confirming it would need a controlled ablation.

Citation

BibTeX

cite this work

@inproceedings{Zhou2026UnderstandingGR,
  title     = {Understanding Geometric Representations in Self-Supervised Vision Transformers via Subspace Intervention},
  author    = {Zhou, Weichen and Zou, Yawen and Gu, Chunzhi and Dong, Ran and Xie, Haoran and Zhang, Chao},
  booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
  year      = {2026}
}