summaryrefslogtreecommitdiff
path: root/EXTENDED_ABSTRACT.md
diff options
context:
space:
mode:
Diffstat (limited to 'EXTENDED_ABSTRACT.md')
-rw-r--r--EXTENDED_ABSTRACT.md74
1 files changed, 52 insertions, 22 deletions
diff --git a/EXTENDED_ABSTRACT.md b/EXTENDED_ABSTRACT.md
index 1a75b8c..1e96cae 100644
--- a/EXTENDED_ABSTRACT.md
+++ b/EXTENDED_ABSTRACT.md
@@ -71,14 +71,26 @@ third-order energy that appeared to pass, and a value-alignment ceiling
of six correct entries in ten that combining two signals lifted to ten.
Each was settled by computing the bound rather than inferring it.
-The gate yields a cheap predictor. Recovery tracks a single measurable
-quantity, the correlation between the two relation fields at the true
-pairing, and the phase boundary sits where correlated-matching theory
-puts it: at five hundred scenes the information-theoretic threshold is
-near 0.29 and the polynomial-algorithm threshold near 0.9. Nothing we
-built worked below 0.9, and recovery appeared as soon as fields crossed
-it. Any new representation can therefore be priced in minutes, before a
-single search is run.
+The gate also produced our longest-lived mistake, and correcting it is
+the paper's second result. Recovery appeared to track one measurable
+quantity — the correlation between the two relation fields at the true
+pairing — with the phase boundary where correlated-matching theory puts
+it, near 0.9 at five hundred scenes. We priced representations against
+that number for months. **It does not govern recovery.** Truncating the
+synthetic fields that recover to rank eight leaves the correlation at
+0.906, past the supposed threshold, and recovery at 13%; the same fields
+at rank sixteen recover 96%. Correlation is nearly fixed across the
+ladder while recovery crosses its whole range.
+
+The theory said so in advance and we misread it. Matching thresholds are
+derived for exchangeable full-rank noise, where each of the N²/2 field
+entries constrains the pairing independently; a rank-*r* shared component
+supplies about *rN*, an eighth as many at 256 scenes with *r* near ten.
+The corrected gate is a joint condition on the correlation and on the
+width of the shared spectrum — the number of directions along which the
+two modalities agree, measured by principal angles against a
+scene-shuffled null. Neither alone suffices: 18 shared directions at
+correlation 0.508 fails, and 11 at 0.902 fails.
## A closed world settles the mechanism
@@ -120,9 +132,9 @@ frequency rankings diverge in the tail.
Ported to Visual Genome under the same recipe — DINOv2 patch features,
spectral segmentation, distributional word statistics — the field
-correlation reaches 0.656, against the 0.9 recovery needs. No recovery
-run is warranted, and the trajectory that got there is more informative
-than the number.
+correlation reaches 0.716 and the shared spectrum 15 directions, against
+0.928 and 26 for the synthetic world that recovers. The trajectory that
+got there is more informative than either number.
Discretisation cost more than half the signal: hard class codes give
0.166 where continuous states give 0.489, because fields need only
@@ -137,20 +149,38 @@ underperform corpus-fitted statistics, 0.19 against 0.656, so in-context
features beat isolated crops and corpus-specific vectors beat a language
model's pooled states.
+The width diagnosis localises the remaining gap somewhere the correlation
+alone never pointed. Both modalities are individually rich on
+photographs — vision effective rank 40, text 48 — and they agree on 15
+directions, so **each side is rich about something and they are rich
+about different things.** That intersection barely moves under anything
+tunable on the vision side: near-tripling the segments costs one shared
+direction, and handing the pipeline annotated boxes instead of
+unsupervised segmentation buys one. The text side moves it from 10 to 17
+and saturates. Enlarging the population does not help either, 23 to 19
+shared directions from 256 to 1024 scenes. The ceiling behaves like a
+property of what the two corpora are about rather than of how either is
+encoded, which makes corpus choice a first-class variable: the closed
+world reaches 26 because its captions state exactly the world state.
+
## What is not shown
The closed-world result uses object states built by hand; no
self-supervised recipe we tested — orbit contrastive, masked pixel
reconstruction, their hybrid, masked feature prediction, or slot
attention at two resolutions — produced their equivalent, each failing
-by a distinct measured mechanism. Natural data sits 0.24 below the
-recovery threshold. No frozen language model has been driven end to end
-on natural data by a recovered correspondence, though a paired control
-establishes that the interface works once the correspondence is right.
-
-The untried lever is the one the closed world identified as a protocol
-requirement: two-sided multi-view observation. Visual Genome offers one
-photograph per scene, so orbits must come from augmentation or from
-corpora that carry them natively. Whether that closes 0.656 to 0.9 is
-the next measurement, and the gate will answer it before any search is
-run.
+by a distinct measured mechanism. Natural data recovers no better than
+chance. No frozen language model has been driven end to end on natural
+data by a recovered correspondence, though a paired control establishes
+that the interface works once the correspondence is right.
+
+The lever the closed world named as a protocol requirement — two-sided
+multi-view observation — has now been tried on photographs and returned
+nothing: four augmentation orbits leave the correlation at 0.6559 either
+way. Visual Genome offers one photograph per scene, so this tests
+augmentation rather than genuine re-observation, and corpora carrying
+native orbits remain untested. But the width result reframes what to
+look for in them. The question is no longer whether a lever closes a
+scalar gap; it is whether a pair of corpora describes enough of the same
+world for their shared spectrum to be wide, and that is a property to
+select corpora on rather than a deficiency to engineer around.