Retire the correlation gate: the shared spectrum governs recovery

A controlled truncation refutes the project's central go/no-go rule.
Projecting the recovering synthetic fields to rank r holds the field
correlation at 0.902-0.929 while recovery moves 6.2% -> 12.9% -> 95.6%
across ranks 4, 8, 16. A field past the supposed 0.9 threshold recovers
13%, so correlation neither predicts nor forbids recovery and the width
of the shared spectrum is what moves it.

The gate becomes a joint condition on correlation and shared width,
measured by principal angles against a scene-shuffled null. Neither
suffices alone: 18 shared directions at 0.508 fails, 11 at 0.902 fails.

With the old gate retired, natural data was finally searched: 0.0000
against 0.0039 chance. The old verdict was right, its reasoning was not.

Also closes route D by measurement. rho_IT ~ sqrt(4 log N / N) rises as N
falls, and at N = 16 through 96 the deepest state a strong searcher
reaches is deeper than the truth in 3/3 replicates at every size.

Free gains: eigenvalue-weighted projection over a wide basis with
128-dim text vectors takes the correlation 0.656 -> 0.716 and shared
width 10 -> 16. Hubness refuted as an inflation hypothesis.

Moving the per-image segmentation eigendecomposition onto the GPU cut
batch time from 130s to 1.9s.

Co-Authored-By: Claude <noreply@anthropic.com>

1 files changed, 107 insertions, 89 deletions

diff --git a/CANDIDATE_REGISTER.md b/CANDIDATE_REGISTER.md
index efed50e..eff860e 100644
--- a/CANDIDATE_REGISTER.md
+++ b/CANDIDATE_REGISTER.md
@@ -1,107 +1,122 @@
 # Candidate register
 
-The obstruction on natural data decomposes into five factors, and every
-candidate attacks exactly one of them. Field correlation stands at 0.656
-against the 0.9 that polynomial recovery needs, so the register is
-ordered by expected movement per unit of effort rather than by elegance.
+*Reordered 2026-08-01 after the rank experiment. See `RANK_RESULTS.md`.*
+
+The register used to be ordered by how much each candidate was expected to
+raise the field correlation, because correlation was the go/no-go statistic.
+That statistic has been shown not to govern recovery: a synthetic field
+truncated to rank 8 correlates at 0.906 and recovers 13%, while the same field
+at rank 16 recovers 96%. **The target is now a pair — the correlation and the
+width of the shared spectrum — and candidates are ordered by which one they
+move.**
 
 Status is one of: **done** (measured, number recorded), **running**,
-**queued** (specified, not started), **open** (idea, not specified).
+**queued** (specified, not started), **open** (idea, not specified),
+**closed** (measured and eliminated).
+
+Current position: correlation 0.716, shared directions 15, against a
+recovering synthetic reference at 0.928 and 26.
 
-## A. Raise the field correlation
+## A. Widen the shared spectrum
 
-The binding constraint. Everything else is worth less than a tenth of
-what this is worth if it moves.
+The binding constraint, and the one only identified today. Both modalities are
+individually rich — vision effective rank 40, text 48 — and agree on 15
+directions. Candidates here are judged on the shared count, not the
+correlation.
+
+| Candidate | Cost | Status |
+|---|---|---|
+| Eigenvalue-weighted projection, wide basis | low | done: shared 10 to 15, ρ 0.656 to 0.692 |
+| Text vector dimension | low | done: shared 10 to 17, saturates; ρ peaks 0.716 at 128 |
+| More segments per scene (6 to 16) | low | closed: shared 15 to 14 |
+| Annotated boxes instead of segmentation (oracle) | low | closed: shared 15, buys one direction |
+| More scenes (256 to 1024) | low | closed: shared 23 to 19 at matched width |
+| Augmentation orbits, four views | medium | closed: no gain, ρ 0.6559 either way |
+| Larger vision backbone (DINOv2-large) | medium | running: predicted useless, see below |
+| Third-order moments in the set kernel | low | queued |
+| Corpora with naturally wide overlap | high | queued |
+| Intervention responses: occlude, re-encode, use the change | high | queued |
+| Corpus-trained text encoder instead of PPMI vectors | medium | queued |
+| Structured phrase encoding: head and modifiers apart | low | queued |
+| Nonlinear content projection (kernel discriminant) | low | closed: 0.62 against 0.69 linear |
+
+The backbone row is a live prediction test rather than a hope. The shared count
+does not move when segmentation is replaced by ground-truth boxes, when
+segments are nearly tripled, or when the vision field's own effective rank
+doubles, so a larger backbone is predicted to leave it unchanged. The run is
+finishing anyway because the project's rule is to measure the bound rather than
+infer it, and a cheap refutation of my own prediction is worth more than the
+GPU time.
+
+**Corpora with naturally wide overlap is the row today's result argues for and
+the one never attempted.** The synthetic world reaches 26 shared directions
+because its captions state exactly the world state; Visual Genome reaches 15
+because a region description and a patch descriptor overlap on about that many
+aspects of a photograph. If the ceiling is a property of what the two corpora
+are about, the lever is corpus choice: dense descriptions, product listings
+carrying photographs and full specifications, screenshots paired with
+accessibility trees, scientific figures with their captions.
+
+## B. Raise the correlation
+
+Still necessary — 18 shared directions at ρ = 0.508 also fails — but no longer
+sufficient, and no longer the thing to optimise alone.
 
 | Candidate | Cost | Status |
 |---|---|---|
 | Continuous states instead of class codes | low | done: 0.166 to 0.489 |
 | Content projection, between-scene over within-scene | low | done: 0.489 to 0.656 |
-| Augmentation orbits, four views | medium | done: no gain, 0.6559 either way |
-| Larger vision backbone (DINOv2-large, -giant) | medium | queued |
+| Eigenvalue weighting plus wider text vectors | low | done: 0.656 to 0.716 |
+| Degree-residual fields | low | done: +0.015, keep, composes freely |
+| Hubness inflation of the statistic | low | closed: refuted, degree model is 1-5% of variance |
 | Multi-scale patch features, several receptive fields | medium | queued |
-| Corpus-trained text encoder instead of PPMI vectors | medium | queued |
-| Structured phrase encoding: head and modifiers apart | low | queued |
-| Intervention responses: occlude a region, re-encode, use the change | high | queued |
-| Native multi-view corpora: product photographs, video frames | high | queued |
-| Nonlinear content projection (kernel discriminant) | low | open |
 | Adaptive segment count with quality filtering | low | open |
 
-The intervention row is the one the closed world argued for hardest and
-the one never run on natural data. It needs real forward passes -- masked
-re-encoding on both sides -- not cached-feature algebra, which is a
-tautology.
-
-## B. Reshape the landscape
-
-Decoys exist because they match second-order statistics. Either raise
-them or widen the true basin.
+## C. Reshape the landscape — closed
 
 | Candidate | Cost | Status |
 |---|---|---|
-| Third-order terms, tr(M^3) | low | done: deepens truth and decoys alike |
-| Rank-truncation ladder, coarse to sharp | low | done: at chance, all ranks |
-| Convex-concave path following (PATH, FAQ family) | medium | queued |
-| Entropic doubly-stochastic relaxation | low | done: stalls far above truth |
-| Degree normalisation of the fields | low | queued |
-| Hubness correction (CSLS) on field similarities | low | queued |
-| Sparsified fields, top-k neighbours only | low | queued |
-| Functional term: held-out generalisation of the induced map | medium | running |
-
-The rank ladder settles the category. At the natural-data field quality
-every rank from four to full returns chance accuracy, as does full-rank
-spectral initialisation with refinement -- 0.0039 against 0.0039 chance.
-**Landscape reshaping does not substitute for field correlation:** below
-the polynomial threshold the deficit is information the algorithm class
-cannot use, not basins it cannot find, so smoothing has nothing to
-recover. Path following is the canonical landscape-deformation method for
-quadratic assignment and remains untried, but this result lowers its
-expected value at the current correlation and it should wait until the
-correlation moves. The functional term
-is landscape reshaping by orthogonal information: a decoy that copies
-relational statistics still has to induce a map that generalises.
-
-## C. Search harder
+| Third-order terms, tr(M³) | low | done: deepens truth and decoys alike |
+| Rank-truncation ladder, coarse to sharp | low | closed: at chance, all ranks |
+| Entropic doubly-stochastic relaxation | low | closed: stalls far above truth |
+| Convex-concave path following (PATH, FAQ family) | medium | closed by inference |
 
-| Candidate | Cost | Status |
-|---|---|---|
-| Spectral initialisation plus exact refinement | low | done: 53.5% to 95.3% composed |
-| Parallel tempering with exact swap deltas | medium | done: equilibrates into the decoy shelf |
-| Multiple spectral inits over eta, keep best by energy | low | queued |
-| Seeded expansion from high-confidence pairs | medium | queued |
-| Belief propagation on the assignment factor graph | high | queued |
-| Cluster moves: swap blocks rather than single pairs | low | queued |
-| Path relinking and crossover over permutations | medium | open |
-| Branch and bound, exact on small blocks | medium | open |
-| Sum-of-squares or low-degree hierarchies | high | open |
-
-Accepting exponential cost does not rescue the hard phase at fixed size:
-tempering already is exponential-time and fails by equilibrating, not by
-running out of budget. Exponential methods pay only where the block is
-small enough to finish, which is why C and D are the same programme seen
-from two sides.
-
-## D. Shrink the problem
-
-The phase boundary is joint in correlation and population size, so a
-corpus need not be matched at once.
+Below the recoverable region the deficit is information the algorithm class
+cannot use, not basins it cannot find, so smoothing has nothing to recover.
+Path following remains the canonical landscape-deformation method for quadratic
+assignment and remains untried, but it should not be tried until the shared
+spectrum moves.
+
+## D. Shrink the problem — closed, and the reasoning was backwards
 
 | Candidate | Cost | Status |
 |---|---|---|
-| Largest recoverable N at the natural field quality | low | running |
-| Diversity-selected bootstrap subset instead of random | low | queued |
-| Hierarchical: cluster first, match within cluster | medium | queued |
-| Many small blocks, merge by consistency voting | medium | queued |
-
-Diversity selection is the cheapest idea in the register and possibly
-the best value: decoys arise from scenes that resemble each other, and
-choosing a mutually dissimilar bootstrap set thins them at no
-algorithmic cost. It composes with small blocks by construction, since
-if only a few hundred pairs are needed they should be the most
-distinguishable few hundred.
+| Largest recoverable N at natural field quality | low | closed: none, 16 through 256 |
+| Diversity-selected bootstrap subset | low | closed by the above, and not protocol-legal |
+| Hierarchical: cluster first, match within cluster | medium | closed |
+| Many small blocks, merge by consistency voting | medium | closed |
+
+The premise was that the hard phase is joint in correlation and population
+size, so small blocks would need less correlation. The sign is the other way:
+ρ_IT ≈ √(4 log N / N) **rises** as N falls — 0.29 at 256 scenes, 0.51 at 64,
+0.83 at 16 — so small blocks need *more* correlation, not less. The gate
+confirms it directly: at N = 16, 24, 32, 48 and 64, the best state a strong
+searcher reaches is deeper than the truth in three replicates of three. When
+the truth is not the optimum the algorithm class is irrelevant, which is also
+why accepting exponential cost buys nothing.
+
+Diversity selection additionally turns out not to be protocol-legal as
+imagined: choosing dissimilar images and dissimilar captions independently
+yields two sets that are not in bijection, so the selection cannot be realised
+without the pairing it is meant to discover. It survives only inside the
+unbalanced formulation in F.
 
 ## E. Add orthogonal signal
 
+Unaffected by today's result and now relatively more attractive, because
+information that does not pass through the relation field is not subject to the
+shared-spectrum ceiling.
+
 | Candidate | Cost | Status |
 |---|---|---|
 | Held-out generalisation of the induced map | medium | running |
@@ -113,22 +128,25 @@ distinguishable few hundred.
 ## F. Change the formulation
 
 Real corpora are not in bijection, so the permutation formulation is a
-scaffold that has to be replaced eventually regardless of performance.
+scaffold that has to be replaced eventually regardless of performance. Today's
+closure of D makes this more urgent, since the unbalanced formulation is now
+the only home for selection-based ideas.
 
 | Candidate | Cost | Status |
 |---|---|---|
 | Unbalanced coupling: partial mass, unmatched items allowed | medium | queued |
+| Diversity selection inside the unbalanced formulation | medium | queued |
 | Many-to-many soft assignment | medium | open |
 | Part-level matching across the corpus rather than scene-level | medium | open |
 | Joint factor-level and scene-level, alternating | medium | open |
 
 ## Order of attempt
 
-Cheap and untried first, because the register's value is in eliminating
-branches quickly: diversity selection, degree normalisation, hubness
-correction, sparsified fields, multiple spectral inits, structured phrase
-encoding, higher-order joint tables. Then the medium tier led by path
-following and the functional term. The two expensive rows worth pre-
-committing to are intervention responses and a natively multi-view
-corpus, because the closed world named both as protocol requirements
-rather than optimisations.
+Third-order moments first: it is the cheapest untried thing that raises field
+rank by construction, since a set kernel carrying third moments spans more
+directions than one carrying two. Then structured phrase encoding and the
+corpus-trained text encoder, because the text side is the one that moved the
+shared count at all. Then the two rows worth pre-committing to regardless of
+cost — intervention responses, and a corpus chosen for naturally wide overlap —
+the second of which is now the register's central bet rather than one candidate
+among many.