World Alignment: unpaired cross-modal correspondence by relational identifiability

Method: scene states are sets of part states; relation fields are built
within each modality and are invariant to how each side labels its own
features; the cross-modal bridge is a coupling searched under an energy
that is a closed-form functional of one matrix; solving is spectral
initialisation followed by exact local refinement.

Evidence: in a procedurally generated closed world, blind recovery of a
hidden image-caption correspondence reaches 95.3% at 256 scenes against
0.39% chance, and the recovered pairs transfer to 200 held-out scenes at
93.0% exact retrieval with random-pair and shuffled-image controls at or
near chance. Cross-modal value correspondence is derived from disjoint
corpora rather than declared. On Visual Genome the field correlation
reaches 0.656 against the 0.9 that polynomial recovery needs, with the
deficit attributed away from segmentation and discretisation.

Protocol: no image-text pair enters any objective, optimiser,
initialisation, or model selection; hidden pairs score orderings only.

Co-Authored-By: Claude <noreply@anthropic.com>

1 files changed, 156 insertions, 0 deletions

diff --git a/EXTENDED_ABSTRACT.md b/EXTENDED_ABSTRACT.md
new file mode 100644
index 0000000..1a75b8c
--- /dev/null
+++ b/EXTENDED_ABSTRACT.md
@@ -0,0 +1,156 @@
+# World Alignment: recovering cross-modal correspondence from corpora that share no example
+
+## The correspondence can be searched rather than bought
+
+Every multimodal system in use buys its cross-modal correspondence with
+paired data. CLIP pays in hundreds of millions of image-text pairs; a
+frozen-backbone connector pays less but still pays; unsupervised
+captioning pays covertly, through a supervised detector's label space.
+We ask what happens when nothing is paid: two corpora, one of images and
+one of text, collected separately, overlapping in the world they describe
+but sharing no example and carrying no link between them.
+
+The premise is that a correspondence need not be learned if it is already
+determined. Two sufficient descriptions of one world must satisfy the
+same relational structure, and that structure is a constraint on the
+unknown correspondence — enough of it, and the correspondence becomes
+identifiable. The method that follows treats the cross-modal bridge as a
+variable to be solved for under that constraint, never as weights to be
+fitted, which is what separates it from both contrastive pairing and any
+cross-modal predictive objective.
+
+## States are sets, relations are fields, and the bridge is a coupling
+
+Four commitments define the method, each forced by a measured failure of
+its alternative.
+
+**A scene state is a set of part states, not a pooled vector.** Pooled
+embeddings of independently trained encoders share a cross-modal
+subspace of roughly sixteen population directions — projecting onto the
+top sixteen raises relational agreement from 0.306 to 0.367, and
+whitening the full spectrum collapses it to 0.03. Sixteen coarse
+dimensions cannot separate five hundred instances, so no energy over
+pooled states passes a local-ordering test, and none did across eighteen
+conditions.
+
+**Relations are computed within each modality and compared as fields.**
+Writing a scene as a set of part states, its relation to another scene is
+a moment kernel of the two sets; the field of all such relations is the
+object the two modalities must agree on. Fields are invariant to how each
+side labels its own features, so no shared vocabulary is presupposed —
+a fact we established by accident, when an oracle class alignment and a
+frequency-rank alignment produced bit-identical fields while agreeing on
+none of twenty-three classes.
+
+**The energy is a functional of one matrix.** With `M` the elementwise
+product of the permuted text field and the visual field, the pairwise
+term is a constant less `2·sum(M)` and the all-triple term a constant
+less `2·tr(M³)/6`, because traces of powers survive simultaneous
+row-column permutation. One batched matrix product scores hundreds of
+candidate couplings over every triple, and the same identity opens
+`tr(M^k)` for any order.
+
+**Solving is spectral first, then local.** A spectral solver reads the
+coarse correspondence out of the eigenstructure in polynomial time and
+never becomes trapped, but rounds badly; exact steepest descent on the
+closed-form energy rounds well but cannot find the basin from a random
+start. Composed, they recover; separately, they do not — 53.5% and 7.0%
+alone against 95.3% together.
+
+## The gate decides before the search runs
+
+A candidate energy is admitted only if the true configuration is the
+deepest state a strong searcher reaches. This sounds procedural and is
+the single most productive rule in the project, because the intuitive
+substitutes are wrong in a specific way: **descent retention measures the
+search operator, not the energy.** Sampled-proposal descent kept the
+truth for every candidate energy tested at five hundred scenes; long
+tempering on the same energies reached 10% below it. Three times a limit
+looked intrinsic and proved to belong to the solver — retention, a
+third-order energy that appeared to pass, and a value-alignment ceiling
+of six correct entries in ten that combining two signals lifted to ten.
+Each was settled by computing the bound rather than inferring it.
+
+The gate yields a cheap predictor. Recovery tracks a single measurable
+quantity, the correlation between the two relation fields at the true
+pairing, and the phase boundary sits where correlated-matching theory
+puts it: at five hundred scenes the information-theoretic threshold is
+near 0.29 and the polynomial-algorithm threshold near 0.9. Nothing we
+built worked below 0.9, and recovery appeared as soon as fields crossed
+it. Any new representation can therefore be priced in minutes, before a
+single search is run.
+
+## A closed world settles the mechanism
+
+A procedurally generated world makes representational sufficiency a
+construction rather than a hope: scenes are sets of object groups with
+spatial relations, captions state exactly the discrete state in a
+43-word vocabulary, and renders resample layout so that what views share
+is precisely the world state. Vision-only and text-only splits are
+disjoint scenes.
+
+Three unimodal operations — appearance grouping, distance-transform
+watershed for touching objects, and one-dimensional clustering of radial
+extent for size classes — take the field correlation from 0.570 to
+0.928, and blind recovery from 5.1% to **95.3% exact scene
+identification** at 256 scenes against 0.39% chance, replicated at 96.1%
+and 96.1% over two further hidden shuffles and 84.7% at 600 scenes.
+
+Recovery would be a curiosity if it could not be spent. Treated as
+pseudo-pair supervision for a ridge map and applied to 200 held-out
+scenes that took no part in matching, it reaches **93.0% exact
+retrieval** against 0.5% chance, with colour-set F1 of 0.989 and count
+multisets exact 97.5% of the time. The random-pair control lands exactly
+on chance and the shuffled-image control drops colour F1 to 0.295, so
+the output is conditioned on the image and extrapolates beyond the
+matched population. Matching is a bootstrap for the map, not the
+deployment mechanism, which is why population size binds far less than
+the recovery curve alone suggests.
+
+Cross-modal value correspondence is derived, not declared. Text factor
+families come from distributional structure, vision classes from
+clustering raw appearance, and the two are paired by marginal frequency
+rank and colour-by-shape joint structure — read from 6,000 text-only and
+6,000 vision-only scenes with no shared instance. The derivation is
+exactly right on ten of ten colour terms, and survives a world where
+captions name only half the objects present and the pixel and text
+frequency rankings diverge in the tail.
+
+## Natural data is not there yet, and the deficit is located
+
+Ported to Visual Genome under the same recipe — DINOv2 patch features,
+spectral segmentation, distributional word statistics — the field
+correlation reaches 0.656, against the 0.9 recovery needs. No recovery
+run is warranted, and the trajectory that got there is more informative
+than the number.
+
+Discretisation cost more than half the signal: hard class codes give
+0.166 where continuous states give 0.489, because fields need only
+scene similarity and quantising states first is attrition the synthetic
+world hid by being discrete already. Segmentation is not the wall on
+photographs — annotated boxes buy 0.061 over unsupervised spectral
+segmentation, the inverse of the closed world where segmentation was the
+entire gap, since flat untextured scenes are adversarial for object
+discovery in a way photographs are not. Content projection is the
+largest lever at 0.489 to 0.656. General-purpose frozen encoders
+underperform corpus-fitted statistics, 0.19 against 0.656, so in-context
+features beat isolated crops and corpus-specific vectors beat a language
+model's pooled states.
+
+## What is not shown
+
+The closed-world result uses object states built by hand; no
+self-supervised recipe we tested — orbit contrastive, masked pixel
+reconstruction, their hybrid, masked feature prediction, or slot
+attention at two resolutions — produced their equivalent, each failing
+by a distinct measured mechanism. Natural data sits 0.24 below the
+recovery threshold. No frozen language model has been driven end to end
+on natural data by a recovered correspondence, though a paired control
+establishes that the interface works once the correspondence is right.
+
+The untried lever is the one the closed world identified as a protocol
+requirement: two-sided multi-view observation. Visual Genome offers one
+photograph per scene, so orbits must come from augmentation or from
+corpora that carry them natively. Whether that closes 0.656 to 0.9 is
+the next measurement, and the gate will answer it before any search is
+run.