Retire the correlation gate: the shared spectrum governs recovery

A controlled truncation refutes the project's central go/no-go rule.
Projecting the recovering synthetic fields to rank r holds the field
correlation at 0.902-0.929 while recovery moves 6.2% -> 12.9% -> 95.6%
across ranks 4, 8, 16. A field past the supposed 0.9 threshold recovers
13%, so correlation neither predicts nor forbids recovery and the width
of the shared spectrum is what moves it.

The gate becomes a joint condition on correlation and shared width,
measured by principal angles against a scene-shuffled null. Neither
suffices alone: 18 shared directions at 0.508 fails, 11 at 0.902 fails.

With the old gate retired, natural data was finally searched: 0.0000
against 0.0039 chance. The old verdict was right, its reasoning was not.

Also closes route D by measurement. rho_IT ~ sqrt(4 log N / N) rises as N
falls, and at N = 16 through 96 the deepest state a strong searcher
reaches is deeper than the truth in 3/3 replicates at every size.

Free gains: eigenvalue-weighted projection over a wide basis with
128-dim text vectors takes the correlation 0.656 -> 0.716 and shared
width 10 -> 16. Hubness refuted as an inflation hypothesis.

Moving the per-image segmentation eigendecomposition onto the GPU cut
batch time from 130s to 1.9s.

Co-Authored-By: Claude <noreply@anthropic.com>

6 files changed, 65 insertions, 0 deletions

diff --git a/logs/backbone_battery.log b/logs/backbone_battery.log
new file mode 100644
index 0000000..e69de29
--- /dev/null
+++ b/logs/backbone_battery.log
diff --git a/logs/obj_base_gpu.log b/logs/obj_base_gpu.log
new file mode 100644
index 0000000..de196aa
--- /dev/null
+++ b/logs/obj_base_gpu.log
@@ -0,0 +1,3 @@
+Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
+`torch_dtype` is deprecated! Use `dtype` instead!
+
segment:   0%|          | 0/250 [00:00<?, ?it/s]
segment:   0%|          | 1/250 [00:02<10:41,  2.58s/it]
segment:   1%|          | 2/250 [00:04<08:49,  2.13s/it]
segment:   1%|          | 3/250 [00:06<08:11,  1.99s/it]
segment:   2%|▏         | 4/250 [00:08<07:52,  1.92s/it]
segment:   2%|▏         | 5/250 [00:09<07:41,  1.88s/it]
segment:   2%|▏         | 6/250 [00:11<07:02,  1.73s/it]
segment:   3%|▎         | 7/250 [00:12<06:28,  1.60s/it]
segment:   3%|▎         | 8/250 [00:13<06:06,  1.52s/it]
segment:   4%|▎         | 9/250 [00:15<05:50,  1.45s/it]
segment:   4%|▍         | 10/250 [00:16<05:40,  1.42s/it]
segment:   4%|▍         | 11/250 [00:17<05:31,  1.39s/it]
segment:   5%|▍         | 12/250 [00:19<05:26,  1.37s/it]
segment:   5%|▌         | 13/250 [00:20<05:23,  1.37s/it]
segment:   6%|▌         | 14/250 [00:21<05:20,  1.36s/it]
segment:   6%|▌         | 15/250 [00:23<05:16,  1.35s/it]
segment:   6%|▋         | 16/250 [00:24<05:14,  1.34s/it]
segment:   7%|▋         | 17/250 [00:25<05:12,  1.34s/it]
segment:   7%|▋         | 18/250 [00:27<05:10,  1.34s/it]
segment:   8%|▊         | 19/250 [00:28<05:07,  1.33s/it]
segment:   8%|▊         | 20/250 [00:29<05:06,  1.33s/it]
segment:   8%|▊         | 21/250 [00:31<05:04,  1.33s/it]
segment:   9%|▉         | 22/250 [00:32<05:01,  1.32s/it]
segment:   9%|▉         | 23/250 [00:33<04:59,  1.32s/it]
segment:  10%|▉         | 24/250 [00:35<04:58,  1.32s/it]
segment:  10%|█         | 25/250 [00:36<04:57,  1.32s/it]
segment:  10%|█         | 26/250 [00:37<04:56,  1.32s/it]
segment:  11%|█         | 27/250 [00:39<04:56,  1.33s/it]
segment:  11%|█         | 28/250 [00:40<04:55,  1.33s/it]
segment:  12%|█▏        | 29/250 [00:41<04:54,  1.33s/it]
segment:  12%|█▏        | 30/250 [00:43<04:53,  1.33s/it]
segment:  12%|█▏        | 31/250 [00:44<04:51,  1.33s/it]
segment:  13%|█▎        | 32/250 [00:45<04:48,  1.32s/it]
segment:  13%|█▎        | 33/250 [00:47<04:46,  1.32s/it]
segment:  14%|█▎        | 34/250 [00:48<04:44,  1.32s/it]
segment:  14%|█▍        | 35/250 [00:49<04:43,  1.32s/it]
segment:  14%|█▍        | 36/250 [00:51<04:43,  1.32s/it]
segment:  15%|█▍        | 37/250 [00:52<04:41,  1.32s/it]
segment:  15%|█▌        | 38/250 [00:53<04:40,  1.32s/it]
segment:  16%|█▌        | 39/250 [00:55<04:39,  1.32s/it]
segment:  16%|█▌        | 40/250 [00:56<04:36,  1.32s/it]
segment:  16%|█▋        | 41/250 [00:57<04:34,  1.31s/it]
segment:  17%|█▋        | 42/250 [00:59<04:33,  1.32s/it]
segment:  17%|█▋        | 43/250 [01:00<04:32,  1.32s/it]
segment:  18%|█▊        | 44/250 [01:01<04:31,  1.32s/it]
segment:  18%|█▊        | 45/250 [01:03<04:34,  1.34s/it]
segment:  18%|█▊        | 46/250 [01:04<04:31,  1.33s/it]
segment:  19%|█▉        | 47/250 [01:05<04:27,  1.32s/it]
segment:  19%|█▉        | 48/250 [01:06<04:25,  1.32s/it]
segment:  20%|█▉        | 49/250 [01:08<04:23,  1.31s/it]
segment:  20%|██        | 50/250 [01:09<04:21,  1.31s/it]
segment:  20%|██        | 51/250 [01:10<04:21,  1.31s/it]
segment:  21%|██        | 52/250 [01:12<04:20,  1.31s/it]
segment:  21%|██        | 53/250 [01:13<04:19,  1.32s/it]
segment:  22%|██▏       | 54/250 [01:14<04:17,  1.32s/it]
segment:  22%|██▏       | 55/250 [01:16<04:16,  1.32s/it]
segment:  22%|██▏       | 56/250 [01:17<04:15,  1.32s/it]
segment:  23%|██▎       | 57/250 [01:18<04:14,  1.32s/it]
segment:  23%|██▎       | 58/250 [01:20<04:13,  1.32s/it]
\ No newline at end of file
diff --git a/logs/obj_large.log b/logs/obj_large.log
new file mode 100644
index 0000000..44fda9d
--- /dev/null
+++ b/logs/obj_large.log
@@ -0,0 +1,3 @@
+Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
+`torch_dtype` is deprecated! Use `dtype` instead!
+
segment:   0%|          | 0/313 [00:00<?, ?it/s]
segment:   0%|          | 1/313 [00:33<2:54:15, 33.51s/it]
segment:   1%|          | 2/313 [01:02<2:39:35, 30.79s/it]
segment:   1%|          | 3/313 [01:35<2:43:37, 31.67s/it]
segment:   1%|▏         | 4/313 [03:41<5:56:15, 69.18s/it]
segment:   2%|▏         | 5/313 [05:25<6:59:54, 81.80s/it]
segment:   2%|▏         | 6/313 [06:43<6:51:46, 80.48s/it]
segment:   2%|▏         | 7/313 [07:37<6:05:24, 71.65s/it]
segment:   3%|▎         | 8/313 [08:40<5:50:37, 68.98s/it]
segment:   3%|▎         | 9/313 [09:35<5:27:25, 64.62s/it]
segment:   3%|▎         | 10/313 [10:33<5:16:19, 62.64s/it]
segment:   4%|▎         | 11/313 [11:46<5:30:23, 65.64s/it]
segment:   4%|▍         | 12/313 [12:45<5:19:20, 63.66s/it]
segment:   4%|▍         | 13/313 [13:49<5:19:01, 63.81s/it]
segment:   4%|▍         | 14/313 [15:20<5:59:27, 72.13s/it]
segment:   5%|▍         | 15/313 [17:25<7:16:35, 87.91s/it]
segment:   5%|▌         | 16/313 [19:18<7:52:16, 95.41s/it]
segment:   5%|▌         | 17/313 [21:20<8:30:07, 103.40s/it]
segment:   6%|▌         | 18/313 [23:31<9:09:44, 111.81s/it]
segment:   6%|▌         | 19/313 [25:46<9:41:50, 118.74s/it]
segment:   6%|▋         | 20/313 [28:00<10:02:23, 123.36s/it]
segment:   7%|▋         | 21/313 [30:04<10:01:04, 123.51s/it]
segment:   7%|▋         | 22/313 [32:13<10:06:32, 125.06s/it]
segment:   7%|▋         | 23/313 [34:50<10:50:35, 134.61s/it]
segment:   8%|▊         | 24/313 [37:17<11:06:29, 138.37s/it]
segment:   8%|▊         | 25/313 [39:40<11:11:26, 139.88s/it]
segment:   8%|▊         | 26/313 [41:26<10:20:42, 129.76s/it]
\ No newline at end of file
diff --git a/logs/projection_sweep.log b/logs/projection_sweep.log
new file mode 100644
index 0000000..6efda04
--- /dev/null
+++ b/logs/projection_sweep.log
@@ -0,0 +1,44 @@
+fit 1200 vision / 1200 text, evaluated 256
+linear     keep=4     shrink=0.01   rho=0.6453
+linear     keep=8     shrink=0.01   rho=0.6495
+linear     keep=16    shrink=0.01   rho=0.6471
+linear     keep=32    shrink=0.01   rho=0.6227
+linear     keep=64    shrink=0.01   rho=0.6143
+linear     keep=4     shrink=0.05   rho=0.6525
+linear     keep=8     shrink=0.05   rho=0.6559
+linear     keep=16    shrink=0.05   rho=0.6519
+linear     keep=32    shrink=0.05   rho=0.6274
+linear     keep=64    shrink=0.05   rho=0.6198
+linear     keep=4     shrink=0.2    rho=0.6607
+linear     keep=8     shrink=0.2    rho=0.6582
+linear     keep=16    shrink=0.2    rho=0.6513
+linear     keep=32    shrink=0.2    rho=0.6292
+linear     keep=64    shrink=0.2    rho=0.6231
+linear     keep=4     shrink=1.0    rho=0.6579
+linear     keep=8     shrink=1.0    rho=0.6405
+linear     keep=16    shrink=1.0    rho=0.6280
+linear     keep=32    shrink=1.0    rho=0.6142
+linear     keep=64    shrink=1.0    rho=0.6106
+weighted^0.5 keep=8     shrink=0.05   rho=0.6698
+weighted^1.0 keep=8     shrink=0.05   rho=0.6651
+weighted^0.5 keep=16    shrink=0.05   rho=0.6826
+weighted^1.0 keep=16    shrink=0.05   rho=0.6721
+weighted^0.5 keep=32    shrink=0.05   rho=0.6874
+weighted^1.0 keep=32    shrink=0.05   rho=0.6753
+weighted^0.5 keep=64    shrink=0.05   rho=0.6922
+weighted^1.0 keep=64    shrink=0.05   rho=0.6768
+weighted^0.5 keep=8     shrink=0.2    rho=0.6728
+weighted^1.0 keep=8     shrink=0.2    rho=0.6692
+weighted^0.5 keep=16    shrink=0.2    rho=0.6835
+weighted^1.0 keep=16    shrink=0.2    rho=0.6752
+weighted^0.5 keep=32    shrink=0.2    rho=0.6884
+weighted^1.0 keep=32    shrink=0.2    rho=0.6780
+weighted^0.5 keep=64    shrink=0.2    rho=0.6918
+weighted^1.0 keep=64    shrink=0.2    rho=0.6788
+kernel g=0.25 keep=8     shrink=0.05   rho=0.6239
+kernel g=0.25 keep=16    shrink=0.05   rho=0.6199
+kernel g=0.25 keep=32    shrink=0.05   rho=0.6044
+kernel g=1.0 keep=8     shrink=0.05   rho=0.5589
+kernel g=1.0 keep=16    shrink=0.05   rho=0.5472
+kernel g=1.0 keep=32    shrink=0.05   rho=0.5251
+{"variant": "weighted^0.5", "keep": 64, "shrinkage": 0.05, "correlation": 0.6922159706308013}
diff --git a/logs/rank_causal.log b/logs/rank_causal.log
new file mode 100644
index 0000000..3635e3c
--- /dev/null
+++ b/logs/rank_causal.log
@@ -0,0 +1,8 @@
+rank=4     rho=0.9017  recovery=0.062  eff_rank=44.9
+rank=8     rho=0.9058  recovery=0.129  eff_rank=71.3
+rank=16    rho=0.9281  recovery=0.956  eff_rank=95.7
+rank=32    rho=0.9282  recovery=0.938  eff_rank=109.5
+rank=64    rho=0.9284  recovery=0.958  eff_rank=106.3
+rank=128   rho=0.9285  recovery=0.961  eff_rank=106.0
+rank=256   rho=0.9285  recovery=0.958  eff_rank=106.0
+{"chance": 0.00390625}
diff --git a/logs/small_n_gate.log b/logs/small_n_gate.log
new file mode 100644
index 0000000..deeb617
--- /dev/null
+++ b/logs/small_n_gate.log
@@ -0,0 +1,7 @@
+N         rho   rho_IT    E(truth)     E(best) acc(best) gate
+16      0.705    0.833        2.95        2.37     0.271 0/3 truth-deepest
+24      0.703    0.728        3.24        2.29     0.194 0/3 truth-deepest
+32      0.679    0.658        3.69        2.38     0.073 0/3 truth-deepest
+48      0.706    0.568        3.48        2.65     0.125 0/3 truth-deepest
+64      0.700    0.510        3.24        2.15     0.151 0/3 truth-deepest
+96      0.698    0.436        3.65        2.57     0.115 0/3 truth-deepest