Retire the correlation gate: the shared spectrum governs recovery

A controlled truncation refutes the project's central go/no-go rule.
Projecting the recovering synthetic fields to rank r holds the field
correlation at 0.902-0.929 while recovery moves 6.2% -> 12.9% -> 95.6%
across ranks 4, 8, 16. A field past the supposed 0.9 threshold recovers
13%, so correlation neither predicts nor forbids recovery and the width
of the shared spectrum is what moves it.

The gate becomes a joint condition on correlation and shared width,
measured by principal angles against a scene-shuffled null. Neither
suffices alone: 18 shared directions at 0.508 fails, 11 at 0.902 fails.

With the old gate retired, natural data was finally searched: 0.0000
against 0.0039 chance. The old verdict was right, its reasoning was not.

Also closes route D by measurement. rho_IT ~ sqrt(4 log N / N) rises as N
falls, and at N = 16 through 96 the deepest state a strong searcher
reaches is deeper than the truth in 3/3 replicates at every size.

Free gains: eigenvalue-weighted projection over a wide basis with
128-dim text vectors takes the correlation 0.656 -> 0.716 and shared
width 10 -> 16. Hubness refuted as an inflation hypothesis.

Moving the per-image segmentation eigendecomposition onto the GPU cut
batch time from 130s to 1.9s.

Co-Authored-By: Claude <noreply@anthropic.com>

1 files changed, 52 insertions, 22 deletions

diff --git a/EXTENDED_ABSTRACT.md b/EXTENDED_ABSTRACT.md
index 1a75b8c..1e96cae 100644
--- a/EXTENDED_ABSTRACT.md
+++ b/EXTENDED_ABSTRACT.md
@@ -71,14 +71,26 @@ third-order energy that appeared to pass, and a value-alignment ceiling
 of six correct entries in ten that combining two signals lifted to ten.
 Each was settled by computing the bound rather than inferring it.
 
-The gate yields a cheap predictor. Recovery tracks a single measurable
-quantity, the correlation between the two relation fields at the true
-pairing, and the phase boundary sits where correlated-matching theory
-puts it: at five hundred scenes the information-theoretic threshold is
-near 0.29 and the polynomial-algorithm threshold near 0.9. Nothing we
-built worked below 0.9, and recovery appeared as soon as fields crossed
-it. Any new representation can therefore be priced in minutes, before a
-single search is run.
+The gate also produced our longest-lived mistake, and correcting it is
+the paper's second result. Recovery appeared to track one measurable
+quantity — the correlation between the two relation fields at the true
+pairing — with the phase boundary where correlated-matching theory puts
+it, near 0.9 at five hundred scenes. We priced representations against
+that number for months. **It does not govern recovery.** Truncating the
+synthetic fields that recover to rank eight leaves the correlation at
+0.906, past the supposed threshold, and recovery at 13%; the same fields
+at rank sixteen recover 96%. Correlation is nearly fixed across the
+ladder while recovery crosses its whole range.
+
+The theory said so in advance and we misread it. Matching thresholds are
+derived for exchangeable full-rank noise, where each of the N²/2 field
+entries constrains the pairing independently; a rank-*r* shared component
+supplies about *rN*, an eighth as many at 256 scenes with *r* near ten.
+The corrected gate is a joint condition on the correlation and on the
+width of the shared spectrum — the number of directions along which the
+two modalities agree, measured by principal angles against a
+scene-shuffled null. Neither alone suffices: 18 shared directions at
+correlation 0.508 fails, and 11 at 0.902 fails.
 
 ## A closed world settles the mechanism
 
@@ -120,9 +132,9 @@ frequency rankings diverge in the tail.
 
 Ported to Visual Genome under the same recipe — DINOv2 patch features,
 spectral segmentation, distributional word statistics — the field
-correlation reaches 0.656, against the 0.9 recovery needs. No recovery
-run is warranted, and the trajectory that got there is more informative
-than the number.
+correlation reaches 0.716 and the shared spectrum 15 directions, against
+0.928 and 26 for the synthetic world that recovers. The trajectory that
+got there is more informative than either number.
 
 Discretisation cost more than half the signal: hard class codes give
 0.166 where continuous states give 0.489, because fields need only
@@ -137,20 +149,38 @@ underperform corpus-fitted statistics, 0.19 against 0.656, so in-context
 features beat isolated crops and corpus-specific vectors beat a language
 model's pooled states.
 
+The width diagnosis localises the remaining gap somewhere the correlation
+alone never pointed. Both modalities are individually rich on
+photographs — vision effective rank 40, text 48 — and they agree on 15
+directions, so **each side is rich about something and they are rich
+about different things.** That intersection barely moves under anything
+tunable on the vision side: near-tripling the segments costs one shared
+direction, and handing the pipeline annotated boxes instead of
+unsupervised segmentation buys one. The text side moves it from 10 to 17
+and saturates. Enlarging the population does not help either, 23 to 19
+shared directions from 256 to 1024 scenes. The ceiling behaves like a
+property of what the two corpora are about rather than of how either is
+encoded, which makes corpus choice a first-class variable: the closed
+world reaches 26 because its captions state exactly the world state.
+
 ## What is not shown
 
 The closed-world result uses object states built by hand; no
 self-supervised recipe we tested — orbit contrastive, masked pixel
 reconstruction, their hybrid, masked feature prediction, or slot
 attention at two resolutions — produced their equivalent, each failing
-by a distinct measured mechanism. Natural data sits 0.24 below the
-recovery threshold. No frozen language model has been driven end to end
-on natural data by a recovered correspondence, though a paired control
-establishes that the interface works once the correspondence is right.
-
-The untried lever is the one the closed world identified as a protocol
-requirement: two-sided multi-view observation. Visual Genome offers one
-photograph per scene, so orbits must come from augmentation or from
-corpora that carry them natively. Whether that closes 0.656 to 0.9 is
-the next measurement, and the gate will answer it before any search is
-run.
+by a distinct measured mechanism. Natural data recovers no better than
+chance. No frozen language model has been driven end to end on natural
+data by a recovered correspondence, though a paired control establishes
+that the interface works once the correspondence is right.
+
+The lever the closed world named as a protocol requirement — two-sided
+multi-view observation — has now been tried on photographs and returned
+nothing: four augmentation orbits leave the correlation at 0.6559 either
+way. Visual Genome offers one photograph per scene, so this tests
+augmentation rather than genuine re-observation, and corpora carrying
+native orbits remain untested. But the width result reframes what to
+look for in them. The question is no longer whether a lever closes a
+scalar gap; it is whether a pair of corpora describes enough of the same
+world for their shared spectrum to be wide, and that is a property to
+select corpora on rather than a deficiency to engineer around.