# WorldAlign An end-to-end test of whether independently pretrained vision and language representations can be aligned without image-text pairs, then used as the input interface to a frozen causal language model. For the project-level hypothesis, intended final system, non-goals, evidence standards, and current experimental gate, read [PROJECT_CONCEPT.md](PROJECT_CONCEPT.md). The chronological research decisions and falsified directions are maintained in [LAB_NOTES.md](LAB_NOTES.md). The first completed real-data run finds strong shared relational geometry and a small reproducible unpaired retrieval signal with Qwen2.5-1.5B, but the static GW/SWD energy does not yet recover a usable semantic bridge. The paired control confirms that the frozen-backbone interface itself works. See [RESULTS.md](RESULTS.md) for exact numbers and interpretation. A second hidden-permutation experiment uses 5,000 Visual Genome scenes with 16 visual/text views per node. It finds that explicitly visible relations improve blind graph matching to 6x chance at R@1, while adding QA weakens it. This isolates the relevant requirement as common relational information, not total information in either representation. See [VG_RESULTS.md](VG_RESULTS.md) and [VG_PROTOCOL.md](VG_PROTOCOL.md). The third experiment removes the bridge entirely. It performs iterative energy descent on free Qwen-side latent particles, conditioned by frozen DINO population relations. Five-caption observation orbits strengthen the correct-vs-shuffled energy gap. However, a convergence audit shows that further reduction of the fixed energy returns retrieval to chance, and the true paired configuration has higher energy than incorrect optimized states. The present energy is therefore falsified as an identifiable alignment objective; its early rank gain is only a trajectory transient. See [ENERGY_RESULTS.md](ENERGY_RESULTS.md) and [LAB_NOTES.md](LAB_NOTES.md). The current experiment closes the remaining escape route by restricting configurations to permutations of real frozen text states, then testing the ordering of the energy directly. The true assignment beats every random permutation globally but is never a local minimum: 7--33% of single swaps lower the energy, exact 2-swap descent walks away from the truth, and random-start descent finds on-manifold counterfeits below the true energy in every condition. Spectral audits locate the cause in the states rather than the objective: the cross-modal shared signal occupies roughly sixteen population directions, and the fine structure needed for instance identity is modality-private. A view-permutation replay then shows that fine-grained shared structure does exist one level down, between the sixteen region views inside each Visual Genome node (z = 46.6, coarse-frame view matching at 2.3x chance). The next candidate energy must therefore be defined over structured view-level states, not pooled node vectors. Two controls sharpen this: an Ollivier-Ricci flow more than doubles the global ranking separation while leaving every local verdict unchanged (geometry is a frame ingredient, not a resolver), and static view-level relation fields fail the same ordering gate (no strict local minimum among 5,000 nodes; frame-free matching at chance). See [MANIFOLD_RESULTS.md](MANIFOLD_RESULTS.md). The requirements decomposition for the next representation structure and alignment process is maintained in [STRUCTURE_DESIGN.md](STRUCTURE_DESIGN.md). The first two requirement batteries lift the ceiling. Whitening the within-scene view noise (fitted without pairs) triples the VG cross-modal relational correlation to 0.61 and collapses improving swaps to 0.2%; joint-context encoding shows the model layer's shared signal lives in its mixed states (rho 0.216 vs 0.128 isolated), not its attention weights. On the resulting states the basin audit passes for the first time: across nine VG conditions (two state families, three subset seeds, ten restarts each) every descent trajectory ends 5--19% above the true configuration. Flickr still fails by under 1% -- one visual view leaves nothing to whiten -- so two-sided multi-view observation is a protocol requirement. The passed gate licenses unpaired soft-coupling recovery, which is the next phase. See [BATTERY_RESULTS.md](BATTERY_RESULTS.md). A synthetic closed world with exact closure and from-scratch unimodal towers turns data sufficiency into a construction rather than an assumption. Its first measurement relocates the bottleneck: with perfect closure, global similarity matches natural data while relational sharing collapses six-fold, because InfoNCE's uniformity erases between-scene geometry. Sufficiency of the data does not transfer to sufficiency of the representation; the unimodal training objective is a phase-diagram axis. See [SYNTH_RESULTS.md](SYNTH_RESULTS.md). The synthetic world then produced the project's first end-to-end result. Fixing segmentation with unimodal perceptual operations (Gestalt appearance grouping, distance-transform watershed, extent-based size classes) raised the cross-modal field correlation from 0.57 to 0.93, past the threshold where polynomial matching becomes possible, and a spectral solver composed with exact energy refinement recovers **95.3%** of a hidden image-caption correspondence at N=256 -- no pairs, no declared dictionary, no learned cross-modal map, with the colour correspondence itself derived from unimodal marginal frequency ranks. The recovered correspondence then transfers: spent as pseudo-pairs for a ridge map, it reaches 93.0% exact retrieval on 200 held-out scenes that took no part in matching (0.5% chance), with the random-pair control at chance and the shuffled-image control collapsing colour F1 from 0.989 to 0.295. The machinery works end to end when representations are good enough; whether self-supervised learning produces such representations remains open. See [SYNTH_RESULTS.md](SYNTH_RESULTS.md). The first recovery attempt maps the search phase: parallel tempering, entropic Sinkhorn, and a spectral-band homotopy all fail to enter the true funnel, with tempering equilibrating exactly into the quench band 9--20% above the truth. The problem sits in the hard phase of planted assignment: statistically identifiable, algorithmically out of reach for local dynamics. The mathematical tool inventory for closing that gap -- symmetric group Fourier band-limiting, cavity methods, operator-algebraic intertwiners, sheaf-cohomological gluing, rigidity thresholds -- is mapped to measured obstacles in [STRUCTURE_DESIGN.md](STRUCTURE_DESIGN.md). See [RECOVERY_RESULTS.md](RECOVERY_RESULTS.md). The first experiment uses: - Flickr30k real images and captions; - a frozen image-only DINOv2 encoder; - a frozen text-only Qwen2.5 causal LM; - disjoint image-only and text-only training subsets; - Gromov-Wasserstein structure matching plus distributional and isometry energies; - a text-only-trained semantic-latent-to-prefix adapter; - a paired bridge trained on the same backbones as an interface upper bound. The paired validation/test data are never used for unsupervised training or checkpoint selection. ## Protocol ```text unpaired images -> frozen DINO -> visual features --\ > structural bridge unpaired captions -> frozen Qwen -> text features ---/ captions -> frozen Qwen text feature -> text-only prefix adapter image -> DINO -> bridge -> prefix adapter -> frozen Qwen -> caption ``` The unpaired split is stronger than shuffling captions: the image-only and text-only subsets contain disjoint Flickr image IDs. ## Commands ```bash python -m worldalign.prepare --output artifacts/manifest.json python -m worldalign.extract_vision \ --manifest artifacts/manifest.json \ --output artifacts/vision.pt \ --device cuda:1 python -m worldalign.extract_text \ --manifest artifacts/manifest.json \ --output artifacts/text.pt \ --device cuda:3 python -m worldalign.diagnose \ --manifest artifacts/manifest.json \ --vision artifacts/vision.pt \ --text artifacts/text.pt python -m worldalign.precompute_gw \ --manifest artifacts/manifest.json \ --vision artifacts/vision.pt \ --text artifacts/text.pt \ --output artifacts/gw.pt python -m worldalign.train_bridge \ --mode unpaired_gw \ --manifest artifacts/manifest.json \ --vision artifacts/vision.pt \ --text artifacts/text.pt \ --gw-cache artifacts/gw.pt \ --output artifacts/bridge_unpaired.pt \ --device cuda:1 python -m worldalign.train_bridge \ --mode paired \ --manifest artifacts/manifest.json \ --vision artifacts/vision.pt \ --text artifacts/text.pt \ --output artifacts/bridge_paired.pt \ --device cuda:3 python -m worldalign.train_prefix \ --manifest artifacts/manifest.json \ --text artifacts/text.pt \ --output artifacts/prefix.pt \ --device cuda:1 python -m worldalign.evaluate_prefix \ --manifest artifacts/manifest.json \ --vision artifacts/vision.pt \ --text artifacts/text.pt \ --prefix artifacts/prefix.pt \ --device cuda:1 python -m worldalign.evaluate \ --manifest artifacts/manifest.json \ --vision artifacts/vision.pt \ --text artifacts/text.pt \ --bridge artifacts/bridge_unpaired.pt \ --prefix artifacts/prefix.pt \ --device cuda:1 ``` The on-manifold assignment gate and the view-level probe: ```bash python -m worldalign.manifold_gate \ --dataset flickr --split test --samples 512 --text-mode orbit_mean \ --output artifacts/manifold_gate/flickr_test512_orbit.json python -m worldalign.manifold_gate \ --dataset vg --vg-text artifacts/vg_5k/text_features.pt \ --samples 512 --subset-seed 0 \ --output artifacts/manifold_gate/vg512_region_closed_seed0_scalar.json python -m worldalign.vg_view_probe \ --context-nodes 1000 --skip-floor \ --output artifacts/manifold_gate/vg_view_probe_full.json ``` Use `--help` on each command for smoke-test size controls. The default first run uses Qwen2.5-0.5B and DINOv2-small to validate the experimental signal. Once the protocol works, model names can be replaced without changing the method.