path: root/EXTENDED_ABSTRACT.md

# World Alignment: recovering cross-modal correspondence from corpora that share no example

## The correspondence can be searched rather than bought

Every multimodal system in use buys its cross-modal correspondence with
paired data. CLIP pays in hundreds of millions of image-text pairs; a
frozen-backbone connector pays less but still pays; unsupervised
captioning pays covertly, through a supervised detector's label space.
We ask what happens when nothing is paid: two corpora, one of images and
one of text, collected separately, overlapping in the world they describe
but sharing no example and carrying no link between them.

The premise is that a correspondence need not be learned if it is already
determined. Two sufficient descriptions of one world must satisfy the
same relational structure, and that structure is a constraint on the
unknown correspondence — enough of it, and the correspondence becomes
identifiable. The method that follows treats the cross-modal bridge as a
variable to be solved for under that constraint, never as weights to be
fitted, which is what separates it from both contrastive pairing and any
cross-modal predictive objective.

## States are sets, relations are fields, and the bridge is a coupling

Four commitments define the method, each forced by a measured failure of
its alternative.

**A scene state is a set of part states, not a pooled vector.** Pooled
embeddings of independently trained encoders share a cross-modal
subspace of roughly sixteen population directions — projecting onto the
top sixteen raises relational agreement from 0.306 to 0.367, and
whitening the full spectrum collapses it to 0.03. Sixteen coarse
dimensions cannot separate five hundred instances, so no energy over
pooled states passes a local-ordering test, and none did across eighteen
conditions.

**Relations are computed within each modality and compared as fields.**
Writing a scene as a set of part states, its relation to another scene is
a moment kernel of the two sets; the field of all such relations is the
object the two modalities must agree on. Fields are invariant to how each
side labels its own features, so no shared vocabulary is presupposed —
a fact we established by accident, when an oracle class alignment and a
frequency-rank alignment produced bit-identical fields while agreeing on
none of twenty-three classes.

**The energy is a functional of one matrix.** With `M` the elementwise
product of the permuted text field and the visual field, the pairwise
term is a constant less `2·sum(M)` and the all-triple term a constant
less `2·tr(M³)/6`, because traces of powers survive simultaneous
row-column permutation. One batched matrix product scores hundreds of
candidate couplings over every triple, and the same identity opens
`tr(M^k)` for any order.

**Solving is spectral first, then local.** A spectral solver reads the
coarse correspondence out of the eigenstructure in polynomial time and
never becomes trapped, but rounds badly; exact steepest descent on the
closed-form energy rounds well but cannot find the basin from a random
start. Composed, they recover; separately, they do not — 53.5% and 7.0%
alone against 95.3% together.

## The gate decides before the search runs

A candidate energy is admitted only if the true configuration is the
deepest state a strong searcher reaches. This sounds procedural and is
the single most productive rule in the project, because the intuitive
substitutes are wrong in a specific way: **descent retention measures the
search operator, not the energy.** Sampled-proposal descent kept the
truth for every candidate energy tested at five hundred scenes; long
tempering on the same energies reached 10% below it. Three times a limit
looked intrinsic and proved to belong to the solver — retention, a
third-order energy that appeared to pass, and a value-alignment ceiling
of six correct entries in ten that combining two signals lifted to ten.
Each was settled by computing the bound rather than inferring it.

The gate yields a cheap predictor. Recovery tracks a single measurable
quantity, the correlation between the two relation fields at the true
pairing, and the phase boundary sits where correlated-matching theory
puts it: at five hundred scenes the information-theoretic threshold is
near 0.29 and the polynomial-algorithm threshold near 0.9. Nothing we
built worked below 0.9, and recovery appeared as soon as fields crossed
it. Any new representation can therefore be priced in minutes, before a
single search is run.

## A closed world settles the mechanism

A procedurally generated world makes representational sufficiency a
construction rather than a hope: scenes are sets of object groups with
spatial relations, captions state exactly the discrete state in a
43-word vocabulary, and renders resample layout so that what views share
is precisely the world state. Vision-only and text-only splits are
disjoint scenes.

Three unimodal operations — appearance grouping, distance-transform
watershed for touching objects, and one-dimensional clustering of radial
extent for size classes — take the field correlation from 0.570 to
0.928, and blind recovery from 5.1% to **95.3% exact scene
identification** at 256 scenes against 0.39% chance, replicated at 96.1%
and 96.1% over two further hidden shuffles and 84.7% at 600 scenes.

Recovery would be a curiosity if it could not be spent. Treated as
pseudo-pair supervision for a ridge map and applied to 200 held-out
scenes that took no part in matching, it reaches **93.0% exact
retrieval** against 0.5% chance, with colour-set F1 of 0.989 and count
multisets exact 97.5% of the time. The random-pair control lands exactly
on chance and the shuffled-image control drops colour F1 to 0.295, so
the output is conditioned on the image and extrapolates beyond the
matched population. Matching is a bootstrap for the map, not the
deployment mechanism, which is why population size binds far less than
the recovery curve alone suggests.

Cross-modal value correspondence is derived, not declared. Text factor
families come from distributional structure, vision classes from
clustering raw appearance, and the two are paired by marginal frequency
rank and colour-by-shape joint structure — read from 6,000 text-only and
6,000 vision-only scenes with no shared instance. The derivation is
exactly right on ten of ten colour terms, and survives a world where
captions name only half the objects present and the pixel and text
frequency rankings diverge in the tail.

## Natural data is not there yet, and the deficit is located

Ported to Visual Genome under the same recipe — DINOv2 patch features,
spectral segmentation, distributional word statistics — the field
correlation reaches 0.656, against the 0.9 recovery needs. No recovery
run is warranted, and the trajectory that got there is more informative
than the number.

Discretisation cost more than half the signal: hard class codes give
0.166 where continuous states give 0.489, because fields need only
scene similarity and quantising states first is attrition the synthetic
world hid by being discrete already. Segmentation is not the wall on
photographs — annotated boxes buy 0.061 over unsupervised spectral
segmentation, the inverse of the closed world where segmentation was the
entire gap, since flat untextured scenes are adversarial for object
discovery in a way photographs are not. Content projection is the
largest lever at 0.489 to 0.656. General-purpose frozen encoders
underperform corpus-fitted statistics, 0.19 against 0.656, so in-context
features beat isolated crops and corpus-specific vectors beat a language
model's pooled states.

## What is not shown

The closed-world result uses object states built by hand; no
self-supervised recipe we tested — orbit contrastive, masked pixel
reconstruction, their hybrid, masked feature prediction, or slot
attention at two resolutions — produced their equivalent, each failing
by a distinct measured mechanism. Natural data sits 0.24 below the
recovery threshold. No frozen language model has been driven end to end
on natural data by a recovered correspondence, though a paired control
establishes that the interface works once the correspondence is right.

The untried lever is the one the closed world identified as a protocol
requirement: two-sided multi-view observation. Visual Genome offers one
photograph per scene, so orbits must come from augmentation or from
corpora that carry them natively. Whether that closes 0.656 to 0.9 is
the next measurement, and the gate will answer it before any search is
run.