Retire the correlation gate: the shared spectrum governs recovery

A controlled truncation refutes the project's central go/no-go rule.
Projecting the recovering synthetic fields to rank r holds the field
correlation at 0.902-0.929 while recovery moves 6.2% -> 12.9% -> 95.6%
across ranks 4, 8, 16. A field past the supposed 0.9 threshold recovers
13%, so correlation neither predicts nor forbids recovery and the width
of the shared spectrum is what moves it.

The gate becomes a joint condition on correlation and shared width,
measured by principal angles against a scene-shuffled null. Neither
suffices alone: 18 shared directions at 0.508 fails, 11 at 0.902 fails.

With the old gate retired, natural data was finally searched: 0.0000
against 0.0039 chance. The old verdict was right, its reasoning was not.

Also closes route D by measurement. rho_IT ~ sqrt(4 log N / N) rises as N
falls, and at N = 16 through 96 the deepest state a strong searcher
reaches is deeper than the truth in 3/3 replicates at every size.

Free gains: eigenvalue-weighted projection over a wide basis with
128-dim text vectors takes the correlation 0.656 -> 0.716 and shared
width 10 -> 16. Hubness refuted as an inflation hypothesis.

Moving the per-image segmentation eigendecomposition onto the GPU cut
batch time from 130s to 1.9s.

Co-Authored-By: Claude <noreply@anthropic.com>

1 files changed, 351 insertions, 0 deletions

diff --git a/artifacts/vg_5k/shared_rank_final.json b/artifacts/vg_5k/shared_rank_final.json
new file mode 100644
index 0000000..3ade344
--- /dev/null
+++ b/artifacts/vg_5k/shared_rank_final.json
@@ -0,0 +1,351 @@
+{
+  "protocol": "Effective rank from the participation ratio of each field's spectrum; shared directions from principal angles between the two leading eigenspaces. No pairing enters beyond the alignment the fields already carry.",
+  "width": 32,
+  "rows": [
+    {
+      "label": "synth-RECOVERS-95pct",
+      "size": 256,
+      "correlation": 0.9284809883663888,
+      "visual_effective_rank": 14.50854049599196,
+      "text_effective_rank": 11.832081486516831,
+      "shared_directions_cos_gt_0.7": 27,
+      "shuffled_null": 1.0,
+      "shared_above_null": 26.0,
+      "principal_cosines_top8": [
+        1.0,
+        1.0,
+        1.0,
+        1.0,
+        1.0,
+        0.999,
+        0.999,
+        0.999
+      ],
+      "spectrum_profile": [
+        {
+          "index": 0,
+          "best_text_overlap": 0.9988184979219944
+        },
+        {
+          "index": 1,
+          "best_text_overlap": 0.9038254548424662
+        },
+        {
+          "index": 2,
+          "best_text_overlap": 0.7738637793117231
+        },
+        {
+          "index": 3,
+          "best_text_overlap": 0.7898924505967239
+        },
+        {
+          "index": 4,
+          "best_text_overlap": 0.9633572761985347
+        },
+        {
+          "index": 5,
+          "best_text_overlap": 0.9667594083538564
+        },
+        {
+          "index": 6,
+          "best_text_overlap": 0.8581068112833727
+        },
+        {
+          "index": 7,
+          "best_text_overlap": 0.7720408457258224
+        },
+        {
+          "index": 8,
+          "best_text_overlap": 0.9451869286586472
+        },
+        {
+          "index": 9,
+          "best_text_overlap": 0.9346387523699597
+        },
+        {
+          "index": 10,
+          "best_text_overlap": 0.9744620235608439
+        },
+        {
+          "index": 11,
+          "best_text_overlap": 0.8287855141725994
+        },
+        {
+          "index": 12,
+          "best_text_overlap": 0.8450929971524885
+        },
+        {
+          "index": 13,
+          "best_text_overlap": 0.7475087759782241
+        },
+        {
+          "index": 14,
+          "best_text_overlap": 0.5363062788775304
+        },
+        {
+          "index": 15,
+          "best_text_overlap": 0.5851587348701632
+        }
+      ]
+    },
+    {
+      "label": "synth-v0-fails-5pct",
+      "size": 256,
+      "correlation": 0.507656498620682,
+      "visual_effective_rank": 31.301831479053796,
+      "text_effective_rank": 27.887166453738427,
+      "shared_directions_cos_gt_0.7": 19,
+      "shuffled_null": 1.0,
+      "shared_above_null": 18.0,
+      "principal_cosines_top8": [
+        0.999,
+        0.99,
+        0.986,
+        0.983,
+        0.981,
+        0.979,
+        0.975,
+        0.971
+      ],
+      "spectrum_profile": [
+        {
+          "index": 0,
+          "best_text_overlap": 0.9711463046324016
+        },
+        {
+          "index": 1,
+          "best_text_overlap": 0.5862398136648852
+        },
+        {
+          "index": 2,
+          "best_text_overlap": 0.40411629072393973
+        },
+        {
+          "index": 3,
+          "best_text_overlap": 0.3782147186014658
+        },
+        {
+          "index": 4,
+          "best_text_overlap": 0.49623067040441227
+        },
+        {
+          "index": 5,
+          "best_text_overlap": 0.38858486646786783
+        },
+        {
+          "index": 6,
+          "best_text_overlap": 0.4204576749257697
+        },
+        {
+          "index": 7,
+          "best_text_overlap": 0.4531244365205851
+        },
+        {
+          "index": 8,
+          "best_text_overlap": 0.5071647502947677
+        },
+        {
+          "index": 9,
+          "best_text_overlap": 0.43985083861605223
+        },
+        {
+          "index": 10,
+          "best_text_overlap": 0.4285920808891206
+        },
+        {
+          "index": 11,
+          "best_text_overlap": 0.416942537981572
+        },
+        {
+          "index": 12,
+          "best_text_overlap": 0.36849147041514496
+        },
+        {
+          "index": 13,
+          "best_text_overlap": 0.3540029990986704
+        },
+        {
+          "index": 14,
+          "best_text_overlap": 0.41250502299139413
+        },
+        {
+          "index": 15,
+          "best_text_overlap": 0.34226328782237314
+        }
+      ]
+    },
+    {
+      "label": "nat-baseline-0.656",
+      "size": 256,
+      "correlation": 0.6558829559111284,
+      "visual_effective_rank": 18.735297664859708,
+      "text_effective_rank": 13.915532152821992,
+      "shared_directions_cos_gt_0.7": 11,
+      "shuffled_null": 1.0,
+      "shared_above_null": 10.0,
+      "principal_cosines_top8": [
+        0.993,
+        0.971,
+        0.954,
+        0.938,
+        0.922,
+        0.868,
+        0.845,
+        0.826
+      ],
+      "spectrum_profile": [
+        {
+          "index": 0,
+          "best_text_overlap": 0.8308561390616345
+        },
+        {
+          "index": 1,
+          "best_text_overlap": 0.6425639957390433
+        },
+        {
+          "index": 2,
+          "best_text_overlap": 0.6401247564780709
+        },
+        {
+          "index": 3,
+          "best_text_overlap": 0.5245819731835861
+        },
+        {
+          "index": 4,
+          "best_text_overlap": 0.5324575387488661
+        },
+        {
+          "index": 5,
+          "best_text_overlap": 0.504776020548726
+        },
+        {
+          "index": 6,
+          "best_text_overlap": 0.5017207831704809
+        },
+        {
+          "index": 7,
+          "best_text_overlap": 0.26453910202691044
+        },
+        {
+          "index": 8,
+          "best_text_overlap": 0.3831674979777484
+        },
+        {
+          "index": 9,
+          "best_text_overlap": 0.34779386653289623
+        },
+        {
+          "index": 10,
+          "best_text_overlap": 0.2366745915766192
+        },
+        {
+          "index": 11,
+          "best_text_overlap": 0.1973585194151871
+        },
+        {
+          "index": 12,
+          "best_text_overlap": 0.21439811143331272
+        },
+        {
+          "index": 13,
+          "best_text_overlap": 0.2925437897693861
+        },
+        {
+          "index": 14,
+          "best_text_overlap": 0.09814723704154468
+        },
+        {
+          "index": 15,
+          "best_text_overlap": 0.20755663736183314
+        }
+      ]
+    },
+    {
+      "label": "nat-best-0.716",
+      "size": 256,
+      "correlation": 0.7157944505313559,
+      "visual_effective_rank": 39.69132496762981,
+      "text_effective_rank": 47.98422953390366,
+      "shared_directions_cos_gt_0.7": 16,
+      "shuffled_null": 1.0,
+      "shared_above_null": 15.0,
+      "principal_cosines_top8": [
+        0.987,
+        0.978,
+        0.96,
+        0.941,
+        0.934,
+        0.922,
+        0.899,
+        0.89
+      ],
+      "spectrum_profile": [
+        {
+          "index": 0,
+          "best_text_overlap": 0.8406807387214822
+        },
+        {
+          "index": 1,
+          "best_text_overlap": 0.6748753041856854
+        },
+        {
+          "index": 2,
+          "best_text_overlap": 0.7174150319587717
+        },
+        {
+          "index": 3,
+          "best_text_overlap": 0.4742164704027202
+        },
+        {
+          "index": 4,
+          "best_text_overlap": 0.7105187496542062
+        },
+        {
+          "index": 5,
+          "best_text_overlap": 0.5435228035336661
+        },
+        {
+          "index": 6,
+          "best_text_overlap": 0.5173316630294161
+        },
+        {
+          "index": 7,
+          "best_text_overlap": 0.3720956303455928
+        },
+        {
+          "index": 8,
+          "best_text_overlap": 0.4031225977235423
+        },
+        {
+          "index": 9,
+          "best_text_overlap": 0.5909531802492844
+        },
+        {
+          "index": 10,
+          "best_text_overlap": 0.373881786434362
+        },
+        {
+          "index": 11,
+          "best_text_overlap": 0.2816693575587397
+        },
+        {
+          "index": 12,
+          "best_text_overlap": 0.21822240451244762
+        },
+        {
+          "index": 13,
+          "best_text_overlap": 0.3929212823566225
+        },
+        {
+          "index": 14,
+          "best_text_overlap": 0.4024276537497661
+        },
+        {
+          "index": 15,
+          "best_text_overlap": 0.2659834432053085
+        }
+      ]
+    }
+  ],
+  "reading": "The narrow modality is the one to upgrade. A shared count far below both effective ranks means each side is rich but they are rich about different things."
+}
\ No newline at end of file