World Alignment: unpaired cross-modal correspondence by relational identifiability

Method: scene states are sets of part states; relation fields are built
within each modality and are invariant to how each side labels its own
features; the cross-modal bridge is a coupling searched under an energy
that is a closed-form functional of one matrix; solving is spectral
initialisation followed by exact local refinement.

Evidence: in a procedurally generated closed world, blind recovery of a
hidden image-caption correspondence reaches 95.3% at 256 scenes against
0.39% chance, and the recovered pairs transfer to 200 held-out scenes at
93.0% exact retrieval with random-pair and shuffled-image controls at or
near chance. Cross-modal value correspondence is derived from disjoint
corpora rather than declared. On Visual Genome the field correlation
reaches 0.656 against the 0.9 that polynomial recovery needs, with the
deficit attributed away from segmentation and discretisation.

Protocol: no image-text pair enters any objective, optimiser,
initialisation, or model selection; hidden pairs score orderings only.

Co-Authored-By: Claude <noreply@anthropic.com>

1 files changed, 244 insertions, 0 deletions

diff --git a/MANIFOLD_RESULTS.md b/MANIFOLD_RESULTS.md
new file mode 100644
index 0000000..4ca0098
--- /dev/null
+++ b/MANIFOLD_RESULTS.md
@@ -0,0 +1,244 @@
+# On-manifold assignment gate
+
+Run date: 2026-07-29
+
+## Question
+
+The free-particle energy was falsified off-manifold: optimization left the
+decodable language region and produced counterfeit states below the true
+configuration. This experiment removes that escape route by construction.
+Configurations are permutations of real frozen text states, so the
+language-only terms of the falsified energy (sliced distribution, prototype
+manifold) are constants, and the only varying terms are the cross-modal
+relation MSE and conditional KL. If the earlier failure was caused by
+leaving the manifold, the true assignment should now be an energy minimum.
+
+It is not. The falsification survives the restriction, and the gate
+localizes the cause in the states rather than the search space.
+
+## Protocol
+
+- Configuration space: permutations of held-out real text states. Hidden
+  pairs place the true assignment in the ranking and are never visible to
+  the energy.
+- Gate A, global ordering: energy of the true assignment against 1,000
+  uniform permutations and a similarity-ordered cyclic shift.
+- Gate B, local ordering: the exact energy change of every transposition of
+  the true assignment. The quadratic text terms cancel, so one matrix
+  product evaluates all N(N-1)/2 swaps; the closed form is unit-tested
+  against brute-force recomputation.
+- Gate C, basin audit: exact steepest 2-swap descent from the true
+  assignment and from random assignments, with a local-minimum certificate.
+  Random-start descent doubles as a search for on-manifold counterfeits.
+- Conditions: Flickr30k test/val (DINOv2-small CLS against Qwen2.5-0.5B
+  mean-pooled states, single caption and five-caption orbit means) and
+  Visual Genome 512-node subsets and the full 5,000 nodes (16-view means,
+  `region_closed` and `visible_relations` tiers, scalar and
+  distribution-valued relation channels).
+
+## The ordering passes globally and fails locally
+
+Every condition separates the true assignment from random permutations;
+no condition makes it a local minimum.
+
+| Condition | Random-perm z | Improving swaps | Kept by descent | E true | E counterfeit |
+|---|---:|---:|---:|---:|---:|
+| Flickr test 512, orbit | 18.7 | 7.4% | 13.9% | 4.17 | 3.26 |
+| Flickr test 512, single | 10.8 | 15.6% | 6.6% | 4.67 | 3.44 |
+| Flickr val 512, orbit | 18.0 | 6.8% | 11.9% | 4.10 | 3.26 |
+| Flickr test 1,000, orbit | 25.3 | 7.3% | 11.0% | 4.30 | 3.42 |
+| VG 512, region_closed | 4.9 | 23.5% | 2.7% | 4.58 | 2.73 |
+| VG 512, visible_relations | 9.2 | 17.9% | 3.7% | 4.14 | 2.27 |
+| VG 512, region bundle | 2.7 | 33.2% | 4.5% | 4.86 | 3.22 |
+| VG 5,000, region_closed | 18.3 | 21.6% | (descent disabled) | 4.69 | -- |
+| VG 5,000, visible_relations | 28.2 | 17.7% | (descent disabled) | 4.39 | -- |
+
+"Improving swaps" is the fraction of all transpositions of the true
+assignment that lower the energy: 9,723 of 130,816 in the best condition,
+2,697,600 of 12,497,500 at N=5,000. Scale does not repair local ordering.
+
+VG rows show subset seed 0; seeds 1 and 2 replicate every verdict. Across
+the twelve VG subset runs (3 seeds x 2 tiers x 2 channel modes), improving
+swaps span 15.7--33.2%, descent retention 2.7--22.5%, and a counterfeit
+below the true energy by 1.6--2.2 is found in 12 of 12 runs.
+
+The derangement curve shows where the energy has resolution. On Flickr
+orbit, permutations that displace 32 or more nodes are detected in 200 of
+200 samples; transpositions are missed in 9% of samples. On VG the same
+inversion holds (k=2: 18%, k=128: 0%). **The energy resolves coarse
+rearrangements and is blind to fine ones.**
+
+## Descent manufactures counterfeits from every start
+
+Steepest descent from the true assignment walks away from it in every
+condition, keeping 2.7--13.9% of nodes. Quenches from random assignments
+reach energies 0.9--1.9 below the true configuration at 0.0--0.4% accuracy.
+The counterfeit family is not an artifact of continuous relaxation; it is a
+property of the assignment landscape itself:
+
+E(true) > E(wrong local minimum), for every observed minimum.
+
+Five-caption orbits halve the improving-swap fraction (15.6% to 7.4%) and
+double descent retention (6.6% to 13.9%); explicitly visible relations cut
+it by a quarter (23.5% to 17.9%). Both help, neither changes the verdict.
+
+## The mechanism is spectral: about sixteen shared dimensions
+
+Three measurements locate the failure in the node states.
+
+First, the pooled text states are extremely anisotropic: 99.5% of all
+Flickr caption pairs have cosine above 0.8, and every VG
+`visible_relations` pair is above 0.9. Raw cosine carries almost no
+instance identity; the energy operates on standardized relation values,
+whose cross-modal Spearman correlation is 0.19 (single caption), 0.31
+(orbit mean), 0.21 (`region_closed`), 0.30 (`visible_relations`). Roughly
+90% of fine relational variance is modality-private.
+
+Second, the shared part is spectrally concentrated. Projecting both
+modalities onto their top population principal components before computing
+relations gives:
+
+| Kept dimensions | Cross-modal Spearman |
+|---:|---:|
+| 2 | 0.315 |
+| 16 | 0.367 |
+| 128 | 0.322 |
+| 384 | 0.315 |
+| untruncated | 0.306 |
+
+Two directions already carry more cross-modal relational signal than the
+full representation; the optimum near sixteen dimensions gains 0.06 over
+full; every dimension past that dilutes. Population whitening, which
+equalizes the spectrum, collapses the correlation to 0.03. **The shared
+world signal lives in a roughly sixteen-dimensional coarse subspace, and
+the fine structure that identification needs is modality-private.**
+
+Third, the worst improving swaps are pairs whose text states are nearly
+parallel while their scenes are unrelated: text cosine 0.88--0.94 against
+visual cosine -0.13--0.15. Example pair at text cosine 0.897: "A baseball
+player in red leaps into the air..." against "A woman in a blue shirt is
+walking across the street...". A sixteen-dimensional topical code cannot
+separate 512 instances, and no functional of these relation fields can.
+
+## Higher moments and view bundles buy nothing at node level
+
+The distribution-valued relation channels (std, q10, q90 of the 256
+view-pair cosines between two VG nodes) were a direct test of richer edge
+structure. Their cross-modal Spearman is -0.02, 0.05, and 0.11--0.16
+respectively, against 0.21--0.30 for the mean channel, and they are
+strongly entangled with it within each modality (rank correlation -0.82 to
+0.63). Adding them raises the improving-swap fraction from 23.5% to 33.2%:
+channels without shared signal act as noise. Model scale was already known
+to be inert here (Qwen 0.5B to 1.5B left relational diagnostics unchanged),
+and whitening is destructive, so the node-state toolbox is exhausted:
+every cheap transform either denoises within the sixteen shared dimensions
+or dilutes them. None expands them.
+
+## Region-level structure inside nodes is shared
+
+The constructive result. Both VG modalities observe the same sixteen
+regions per scene; the preparation RNG that hid the view correspondence is
+replayable from the seed and the private image IDs, and the replay
+reconstructs all 5,000 released phrase bundles exactly, so view-level truth
+is available for evaluation.
+
+On all 5,000 nodes:
+
+- Within-node relation fields align: Spearman between the visual and
+  truth-aligned text view-relation fields is 0.128 against -0.002 for
+  shuffled views, z = 46.6 across nodes.
+- Views become matchable through the coarse frame: ranking each view's
+  relation profile against a 1,000-node population frame and solving the
+  16-view assignment gives 14.5% accuracy against 6.25% chance (2.3x;
+  top-3 29.5% against 18.8%). Growing the frame from 200 to 1,000 nodes
+  raises accuracy from 12.9% to 14.5%. This uses the true node alignment
+  as the frame, so it is an upper bound on what a recovered coarse
+  alignment would enable.
+- Within-node relations alone reach 7.0% on a 200-node sample: the
+  population context, not the 16-view geometry, carries the signal.
+
+The fine-grained shared structure the node-level energy lacks does exist
+one level down, in the compositional structure the pooled states discard.
+
+## Ricci flow doubles the frame and fails the resolution
+
+A discrete Ollivier-Ricci flow was run as a control for the hypothesis
+that smoothing each modality's relational geometry could repair local
+ordering. Each modality's kNN graph evolved independently for ten
+iterations; identical heat-kernel channels were gated with and without the
+flow, so the flow's own effect is isolated.
+
+| Condition | Random-perm z | Improving swaps | Kept by descent | E true | E counterfeit |
+|---|---:|---:|---:|---:|---:|
+| Flickr orbit 512, diffusion only | 21.4 | 15.2% | 4.1% | 1.263 | 0.717 |
+| Flickr orbit 512, Ricci flow | 47.8 | 9.1% | 5.1% | 0.818 | 0.427 |
+| VG 512 region, diffusion only | 6.9 | 22.9% | 3.1% | 1.601 | 0.648 |
+| VG 512 region, Ricci flow | 22.5 | 13.9% | 1.2% | 1.405 | 0.650 |
+
+The flow is a genuinely good coarse-channel denoiser: it more than
+doubles the global separation in both datasets. It changes no verdict:
+9--14% of transpositions still improve the flowed energy, descent keeps
+1--5% of the truth, and every random quench finds a counterfeit at about
+half the true energy. **Better geometry buys a better frame, not a better
+resolver** -- the two roles dissociate exactly as the spectral audit
+predicted, so geometric flows belong in the coarse-frame layer of the next
+design and cannot substitute for finer states.
+
+## Static view-level energies fail the same gate
+
+The view-level continuation gates the structured states directly. Given
+the true coarse frame (upper-bound scaffolding), the true view assignment
+is a strict local minimum for none of 5,000 nodes: the within-node
+relational energy has 32.0% improving view-transpositions, the
+population-profile energy 28.3%, and their best combination 25.5%.
+Frame-free matching of internal relation fields (signature-initialized
+Hungarian plus exact batched 2-swap descent, with a value-scrambled
+capacity null and column centering) is dead at 512 nodes across three
+seeds: R@10 2.0--3.3% against 2.0% chance, Hungarian recovery at most
+0.6%, top-margin-decile precision 0 of 51, and the true-pair cost
+separation is z = 0.1--1.8. A small negative separation seen at the
+96-node smoke scale did not replicate and is recorded as noise.
+
+Static within-node relation fields therefore carry shared signal (the
+z = 46.6 alignment result) but not enough for identification even at
+sixteen candidates -- the view-level gate needs model-internal relations
+(joint-context attention) or true re-encoding interventions, which is the
+R3/R8 battery of `STRUCTURE_DESIGN.md`.
+
+## Conclusion
+
+The manifold restriction was necessary and is now insufficient for a
+different reason than the free-particle failure. The energy is not being
+cheated by off-manifold vectors; it is blind at the resolution where
+instance identity lives, because pooled node states expose only a coarse
+shared subspace. No matcher, objective reweighting, or scale change fixes
+an information ceiling in the states.
+
+The gate for the next experiment therefore moves down one level:
+
+1. a node state is a set of region views with its internal relation field,
+   not a pooled vector;
+2. the candidate energy scores view-level assignments conditioned on a
+   coarse frame, and must first pass this same transposition/descent gate
+   at the view level, with the replayed view truth as hidden evaluation;
+3. response operators under controlled interventions (view removal,
+   attribute edits) are the candidate source of shared fine structure
+   beyond static view relations, to be tested through the identical gate
+   before any descent run.
+
+## Main artifacts
+
+- `artifacts/manifold_gate/flickr_test512_{orbit,single}.json`
+- `artifacts/manifold_gate/flickr_val512_orbit.json`
+- `artifacts/manifold_gate/flickr_test1000_orbit.json`
+- `artifacts/manifold_gate/vg512_{region_closed,visible_relations}_seed{0,1,2}_{scalar,bundle}.json`
+- `artifacts/manifold_gate/vg5000_{region_closed,visible_relations}_scalar.json`
+- `artifacts/manifold_gate/resolution_audit.json`
+- `artifacts/manifold_gate/spectral_localization.json`
+- `artifacts/manifold_gate/bundle_channel_audit.json`
+- `artifacts/manifold_gate/vg_view_probe_smoke.json`
+- `artifacts/manifold_gate/vg_view_probe_full.json`
+- `artifacts/manifold_gate/ricci_flickr_test512_orbit.json`
+- `artifacts/manifold_gate/ricci_vg512_region_seed0.json`
+- `artifacts/manifold_gate/view_gate_full.json`
+- `artifacts/manifold_gate/view_gate_cost_seed{0,1,2}.pt`