path: root/EXTENDED_ABSTRACT.md

# World Alignment: recovering cross-modal correspondence from corpora that share no example

## The correspondence can be searched rather than bought

Every multimodal system in use buys its cross-modal correspondence with
paired data. CLIP pays in hundreds of millions of image-text pairs; a
frozen-backbone connector pays less but still pays; unsupervised
captioning pays covertly, through a supervised detector's label space.
We ask what happens when nothing is paid: two corpora, one of images and
one of text, collected separately, overlapping in the world they describe
but sharing no example and carrying no link between them.

The premise is that a correspondence need not be learned if it is already
determined. Two sufficient descriptions of one world must satisfy the
same relational structure, and that structure is a constraint on the
unknown correspondence — enough of it, and the correspondence becomes
identifiable. The method that follows treats the cross-modal bridge as a
variable to be solved for under that constraint, never as weights to be
fitted, which is what separates it from both contrastive pairing and any
cross-modal predictive objective.

## States are sets, relations are fields, and the bridge is a coupling

Four commitments define the method, each forced by a measured failure of
its alternative.

**A scene state is a set of part states, not a pooled vector.** Pooled
embeddings of independently trained encoders share a cross-modal
subspace of roughly sixteen population directions — projecting onto the
top sixteen raises relational agreement from 0.306 to 0.367, and
whitening the full spectrum collapses it to 0.03. Sixteen coarse
dimensions cannot separate five hundred instances, so no energy over
pooled states passes a local-ordering test, and none did across eighteen
conditions.

**Relations are computed within each modality and compared as fields.**
Writing a scene as a set of part states, its relation to another scene is
a moment kernel of the two sets; the field of all such relations is the
object the two modalities must agree on. Fields are invariant to how each
side labels its own features, so no shared vocabulary is presupposed —
a fact we established by accident, when an oracle class alignment and a
frequency-rank alignment produced bit-identical fields while agreeing on
none of twenty-three classes.

**The energy is a functional of one matrix.** With `M` the elementwise
product of the permuted text field and the visual field, the pairwise
term is a constant less `2·sum(M)` and the all-triple term a constant
less `2·tr(M³)/6`, because traces of powers survive simultaneous
row-column permutation. One batched matrix product scores hundreds of
candidate couplings over every triple, and the same identity opens
`tr(M^k)` for any order.

**Solving is spectral first, then local.** A spectral solver reads the
coarse correspondence out of the eigenstructure in polynomial time and
never becomes trapped, but rounds badly; exact steepest descent on the
closed-form energy rounds well but cannot find the basin from a random
start. Composed, they recover; separately, they do not — 53.5% and 7.0%
alone against 95.3% together.

## The gate decides before the search runs

A candidate energy is admitted only if the true configuration is the
deepest state a strong searcher reaches. This sounds procedural and is
the single most productive rule in the project, because the intuitive
substitutes are wrong in a specific way: **descent retention measures the
search operator, not the energy.** Sampled-proposal descent kept the
truth for every candidate energy tested at five hundred scenes; long
tempering on the same energies reached 10% below it. Three times a limit
looked intrinsic and proved to belong to the solver — retention, a
third-order energy that appeared to pass, and a value-alignment ceiling
of six correct entries in ten that combining two signals lifted to ten.
Each was settled by computing the bound rather than inferring it.

The gate also produced our longest-lived mistake, and correcting it is
the paper's second result. Recovery appeared to track one measurable
quantity — the correlation between the two relation fields at the true
pairing — with the phase boundary where correlated-matching theory puts
it, near 0.9 at five hundred scenes. We priced representations against
that number for months. **It does not govern recovery.** Truncating the
synthetic fields that recover to rank eight leaves the correlation at
0.906, past the supposed threshold, and recovery at 13%; the same fields
at rank sixteen recover 96%. Correlation is nearly fixed across the
ladder while recovery crosses its whole range.

The theory said so in advance and we misread it. Matching thresholds are
derived for exchangeable full-rank noise, where each of the N²/2 field
entries constrains the pairing independently; a rank-*r* shared component
supplies about *rN*, an eighth as many at 256 scenes with *r* near ten.
The corrected gate is a joint condition on the correlation and on the
width of the shared spectrum — the number of directions along which the
two modalities agree, measured by principal angles against a
scene-shuffled null. Neither alone suffices: 18 shared directions at
correlation 0.508 fails, and 11 at 0.902 fails.

## A closed world settles the mechanism

A procedurally generated world makes representational sufficiency a
construction rather than a hope: scenes are sets of object groups with
spatial relations, captions state exactly the discrete state in a
43-word vocabulary, and renders resample layout so that what views share
is precisely the world state. Vision-only and text-only splits are
disjoint scenes.

Three unimodal operations — appearance grouping, distance-transform
watershed for touching objects, and one-dimensional clustering of radial
extent for size classes — take the field correlation from 0.570 to
0.928, and blind recovery from 5.1% to **95.3% exact scene
identification** at 256 scenes against 0.39% chance, replicated at 96.1%
and 96.1% over two further hidden shuffles and 84.7% at 600 scenes.

Recovery would be a curiosity if it could not be spent. Treated as
pseudo-pair supervision for a ridge map and applied to 200 held-out
scenes that took no part in matching, it reaches **93.0% exact
retrieval** against 0.5% chance, with colour-set F1 of 0.989 and count
multisets exact 97.5% of the time. The random-pair control lands exactly
on chance and the shuffled-image control drops colour F1 to 0.295, so
the output is conditioned on the image and extrapolates beyond the
matched population. Matching is a bootstrap for the map, not the
deployment mechanism, which is why population size binds far less than
the recovery curve alone suggests.

Cross-modal value correspondence is derived, not declared. Text factor
families come from distributional structure, vision classes from
clustering raw appearance, and the two are paired by marginal frequency
rank and colour-by-shape joint structure — read from 6,000 text-only and
6,000 vision-only scenes with no shared instance. The derivation is
exactly right on ten of ten colour terms, and survives a world where
captions name only half the objects present and the pixel and text
frequency rankings diverge in the tail.

## Natural data is not there yet, and the deficit is located

Ported to Visual Genome under the same recipe — DINOv2 patch features,
spectral segmentation, distributional word statistics — the field
correlation reaches 0.716 and the shared spectrum 15 directions, against
0.928 and 26 for the synthetic world that recovers. The trajectory that
got there is more informative than either number.

Discretisation cost more than half the signal: hard class codes give
0.166 where continuous states give 0.489, because fields need only
scene similarity and quantising states first is attrition the synthetic
world hid by being discrete already. Segmentation is not the wall on
photographs — annotated boxes buy 0.061 over unsupervised spectral
segmentation, the inverse of the closed world where segmentation was the
entire gap, since flat untextured scenes are adversarial for object
discovery in a way photographs are not. Content projection is the
largest lever at 0.489 to 0.656. General-purpose frozen encoders
underperform corpus-fitted statistics, 0.19 against 0.656, so in-context
features beat isolated crops and corpus-specific vectors beat a language
model's pooled states.

The width diagnosis localises the remaining gap somewhere the correlation
alone never pointed. Both modalities are individually rich on
photographs — vision effective rank 40, text 48 — and they agree on 15
directions, so **each side is rich about something and they are rich
about different things.** That intersection barely moves under anything
tunable on the vision side: near-tripling the segments costs one shared
direction, and handing the pipeline annotated boxes instead of
unsupervised segmentation buys one. The text side moves it from 10 to 17
and saturates. Enlarging the population does not help either, 23 to 19
shared directions from 256 to 1024 scenes. The ceiling behaves like a
property of what the two corpora are about rather than of how either is
encoded, which makes corpus choice a first-class variable: the closed
world reaches 26 because its captions state exactly the world state.

## What is not shown

The closed-world result uses object states built by hand; no
self-supervised recipe we tested — orbit contrastive, masked pixel
reconstruction, their hybrid, masked feature prediction, or slot
attention at two resolutions — produced their equivalent, each failing
by a distinct measured mechanism. Natural data recovers no better than
chance. No frozen language model has been driven end to end on natural
data by a recovered correspondence, though a paired control establishes
that the interface works once the correspondence is right.

The lever the closed world named as a protocol requirement — two-sided
multi-view observation — has now been tried on photographs and returned
nothing: four augmentation orbits leave the correlation at 0.6559 either
way. Visual Genome offers one photograph per scene, so this tests
augmentation rather than genuine re-observation, and corpora carrying
native orbits remain untested. But the width result reframes what to
look for in them. The question is no longer whether a lever closes a
scalar gap; it is whether a pair of corpora describes enough of the same
world for their shared spectrum to be wide, and that is a property to
select corpora on rather than a deficiency to engineer around.