Retire the correlation gate: the shared spectrum governs recovery

A controlled truncation refutes the project's central go/no-go rule.
Projecting the recovering synthetic fields to rank r holds the field
correlation at 0.902-0.929 while recovery moves 6.2% -> 12.9% -> 95.6%
across ranks 4, 8, 16. A field past the supposed 0.9 threshold recovers
13%, so correlation neither predicts nor forbids recovery and the width
of the shared spectrum is what moves it.

The gate becomes a joint condition on correlation and shared width,
measured by principal angles against a scene-shuffled null. Neither
suffices alone: 18 shared directions at 0.508 fails, 11 at 0.902 fails.

With the old gate retired, natural data was finally searched: 0.0000
against 0.0039 chance. The old verdict was right, its reasoning was not.

Also closes route D by measurement. rho_IT ~ sqrt(4 log N / N) rises as N
falls, and at N = 16 through 96 the deepest state a strong searcher
reaches is deeper than the truth in 3/3 replicates at every size.

Free gains: eigenvalue-weighted projection over a wide basis with
128-dim text vectors takes the correlation 0.656 -> 0.716 and shared
width 10 -> 16. Hubness refuted as an inflation hypothesis.

Moving the per-image segmentation eigendecomposition onto the GPU cut
batch time from 130s to 1.9s.

Co-Authored-By: Claude <noreply@anthropic.com>

1 files changed, 73 insertions, 0 deletions

diff --git a/LAB_NOTES.md b/LAB_NOTES.md
index e6aa208..5b38e5f 100644
--- a/LAB_NOTES.md
+++ b/LAB_NOTES.md
@@ -584,6 +584,79 @@ battery -- larger backbones and finer segmentation over the same corpus
 -- with three array tasks starting immediately against a queue of 810,
 9,624 GPU-hours remaining on the account.
 
+## 2026-08-01: shrinking the population is backwards, and the truth is not the optimum
+
+Population size was the last non-correlation lever, on the reasoning that
+the hard phase is joint in correlation and size, so small blocks would
+need less correlation. The sign is the other way. The
+information-theoretic threshold is ρ_IT ≈ √(4 log N / N), which **rises**
+as N falls: 0.29 at 256 scenes, 0.51 at 64, 0.83 at 16.
+
+The gate settles it directly rather than by citation. At N = 16, 24, 32,
+48, 64 and 96, the deepest state a strong searcher reaches — spectral
+start plus 200 random restarts, each run to a local optimum under exact
+steepest descent — is deeper than the truth, in three replicates out of
+three at every size. Where the truth is not the optimum, no searcher of
+any cost finds it and the algorithm class stops mattering, which also
+disposes of "accept exponential cost". Route D closes.
+
+## 2026-08-01: the correlation was never the statistic that governs recovery
+
+The project's central gate — correlation at truth must reach about 0.9,
+price any representation in minutes without running a search — is wrong,
+and a controlled truncation shows it. Project the synthetic fields that
+recover to rank *r* and match blind. Correlation stays at 0.902, 0.906,
+0.928 across ranks 4, 8, 16 while recovery goes 6.2%, 12.9%, **95.6%**.
+A field past the supposed threshold recovers 13%.
+
+We misread the theory rather than being failed by it. Matching thresholds
+assume exchangeable full-rank noise, where each of N²/2 entries
+constrains the pairing independently; a rank-*r* shared component
+supplies about *rN*, an eighth as many at 256 scenes with *r* near ten.
+The gate becomes a joint condition on correlation and on the width of the
+shared spectrum, measured by principal angles between the two fields'
+leading eigenspaces against a scene-shuffled null. Neither suffices
+alone: 18 shared directions at correlation 0.508 fails, 11 at 0.902
+fails.
+
+With the old gate retired the excuse for never searching natural data
+went with it. Five blind trials on the best natural field recover
+**0.0000 against 0.0039 chance** — refinement moves away from the truth,
+which is what descent does when the truth is not the optimum. The old
+verdict was right; its reasoning was not.
+
+Diagnosis: on photographs both modalities are individually rich — vision
+effective rank 40, text 48 — and agree on 15 directions. Each side is
+rich about something and they are rich about different things. The
+intersection barely responds to anything on the vision side: 6 to 16
+segments costs one direction, annotated boxes buy one, third-order
+moments in the set kernel raise the vision field's own rank by seven and
+the shared count by one. The text side moves it from 10 to 17 and
+saturates; more scenes do not help (23 to 19 from 256 to 1024).
+
+Free gains along the way: eigenvalue-weighted projection over a wide
+basis plus 128-dimensional text vectors take the correlation from 0.656
+to 0.716 and the shared count from 10 to 16. Hubness was refuted as an
+inflation hypothesis — the additive degree model is 1–5% of field
+variance and removing it slightly *raises* the correlation.
+
+See `RANK_RESULTS.md`.
+
+## 2026-08-01: two operational lessons
+
+**Never `scancel -u` on Delta.** Clearing three failed array tasks that
+way also killed four DAGFormer H200 jobs queued nineteen minutes earlier
+on the same shared account. Restored from `sacct`'s recorded WorkDir and
+SubmitLine, losing queue position only. Rule and recovery recipe written
+into `DELTA_HPC.md`.
+
+**The vision extraction was CPU-bound on its own eigendecomposition,**
+one dense symmetric problem of side grid² per image, costing more than
+the forward pass that produced the features. Moving it to the accelerator
+already holding the model took the batch time from 130 s to 1.9 s, a 68×
+speedup that makes the whole backbone battery cheap rather than
+overnight.
+
 ## Current experimental gate
 
 The node-level ordering-and-basin gate is passed on VG by content-projected