World Alignment: unpaired cross-modal correspondence by relational identifiability

Method: scene states are sets of part states; relation fields are built
within each modality and are invariant to how each side labels its own
features; the cross-modal bridge is a coupling searched under an energy
that is a closed-form functional of one matrix; solving is spectral
initialisation followed by exact local refinement.

Evidence: in a procedurally generated closed world, blind recovery of a
hidden image-caption correspondence reaches 95.3% at 256 scenes against
0.39% chance, and the recovered pairs transfer to 200 held-out scenes at
93.0% exact retrieval with random-pair and shuffled-image controls at or
near chance. Cross-modal value correspondence is derived from disjoint
corpora rather than declared. On Visual Genome the field correlation
reaches 0.656 against the 0.9 that polynomial recovery needs, with the
deficit attributed away from segmentation and discretisation.

Protocol: no image-text pair enters any objective, optimiser,
initialisation, or model selection; hidden pairs score orderings only.

Co-Authored-By: Claude <noreply@anthropic.com>

1 files changed, 193 insertions, 0 deletions

diff --git a/ENERGY_RESULTS.md b/ENERGY_RESULTS.md
new file mode 100644
index 0000000..03e0690
--- /dev/null
+++ b/ENERGY_RESULTS.md
@@ -0,0 +1,193 @@
+# Energy-descent multimodal inference MVP
+
+Run date: 2026-07-29
+
+## Question
+
+Can an image condition a frozen language model through iterative energy
+descent in language latent space, without learning an image-to-language MLP or
+using image-text pairs in the objective?
+
+The finite samples in this experiment are particles used to estimate
+population energies. They are not assumed to be semantic graph nodes.
+
+## Protocol
+
+- Data: Flickr30k.
+- Vision: frozen `facebook/dinov2-small`.
+- Language: frozen `Qwen/Qwen2.5-0.5B`.
+- Image-only and text-only populations contain disjoint image IDs.
+- A language particle is a free 896-dimensional state initialized from a real
+  text-only activation.
+- Five independently written captions are encoded as a language observation
+  orbit and averaged. Their image correspondence is never exposed to the
+  energy or optimizer.
+- The optimized particles are decoded by a prefix interface trained only to
+  reconstruct captions from frozen Qwen text activations.
+- Paired validation/test captions are loaded only for evaluation. The
+  optimizer, energy, initialization, and stopping step do not use them.
+
+The base energy is
+
+\[
+E(Z;X)=
+\lambda_r E_{\mathrm{relation}}(X,Z)
++\lambda_c E_{\mathrm{conditional}}(X,Z)
++\lambda_d E_{\mathrm{text\ distribution}}(Z)
++\lambda_m E_{\mathrm{text\ manifold}}(Z).
+\]
+
+The cross-modal terms match standardized population relation fields and
+multiscale conditional-neighborhood distributions. The language-only terms
+match sliced text-population quantiles and distance to text-only prototypes.
+Only `Z` is optimized; there is no shared image-to-text function.
+
+## Necessary-condition diagnostics
+
+On 512 held-out test images, the correct language configuration has lower
+relation energy than a random permutation:
+
+| Language observation | Correct energy | Shuffled energy | Gap |
+|---|---:|---:|---:|
+| One caption | 1.653 | 1.969 | 0.316 |
+| Mean of five-caption orbit | 1.425 | 2.007 | **0.582** |
+
+Relational overcompleteness nearly doubles the correct-vs-shuffled energy gap.
+
+## Free latent descent
+
+Weak language-manifold regularization produces a large semantic-rank
+improvement but leaves the decodable language manifold. On the independent
+test split, manifold weight 10 gives:
+
+| Metric | Initial | Final |
+|---|---:|---:|
+| R@5 | 1.95% | 6.64% |
+| R@10 | 2.34% | 11.72% |
+| Median rank (256 candidates) | 125 | 103 |
+| Caption unigram F1 (100 samples) | 0.250 | 0.244 |
+
+The final prototype-manifold energy is 0.087 versus 0.0147 for real paired
+text states. Better relative geometry is therefore not sufficient for frozen
+LLM decoding.
+
+## Strong manifold point
+
+Manifold weight 30 was selected on validation because it keeps the final
+language-manifold energy closest to the real-text regime, rather than because
+of paired retrieval. On all 256 test particles:
+
+| Metric | Initial | Final |
+|---|---:|---:|
+| R@5 | 1.95% | 2.73% |
+| R@10 | 2.34% | 5.86% |
+| Median rank | 125 | 103 |
+| Caption unigram F1 | 0.2495 | 0.2623 |
+
+The paired F1 change is `+0.0128`, with a 10,000-resample percentile bootstrap
+interval of `[+0.0005, +0.0252]`.
+
+This apparent generation improvement is not image-conditioned, however.
+
+## Required controls
+
+| Energy | Final R@10 | Final median rank | Final caption F1 | F1 change |
+|---|---:|---:|---:|---:|
+| Full image + language energy | **5.86%** | **103** | 0.2623 | +0.0128 |
+| Language prior only | 3.13% | 120 | **0.2697** | +0.0203 |
+| Full energy, visual particles shuffled | 4.30% | 120 | 0.2500 | +0.0005 |
+
+The correct visual relation field is responsible for the rank improvement:
+prior-only and shuffled-vision controls remain near chance and have worse
+median rank. But the language-prior-only control improves caption F1 more than
+the full system. The generation gain is therefore a generic caption-prior
+cleanup effect, not evidence that the image selected the correct language
+state.
+
+## Functional-cycle energy
+
+A higher-level frozen-language energy was also tested:
+
+\[
+z\rightarrow
+\text{text-only prefix + frozen Qwen decode}
+\rightarrow\hat y
+\rightarrow\text{frozen Qwen encode}
+\rightarrow\hat z,
+\qquad
+E_{\mathrm{cycle}}=1-\cos(z,\hat z).
+\]
+
+With functional weight 10 on 128 validation particles, R@10 improves from
+9.38% to 17.97% and median rank from 57 to 43, but caption F1 falls from
+0.2508 to 0.2256. A stronger cycle weight suppresses both the rank gain and
+the degradation but still does not improve generation.
+
+Decode-reencode consistency is thus not a sufficient world energy: a generic
+or incorrect caption can be a stable self-consistent fixed point.
+
+## Convergence audit
+
+The total printed during the original 100-step run is not a fixed-objective
+convergence curve: relation and conditional weights are annealed throughout
+the run. At the `m30` endpoint, both cross-modal components were still changing
+and the tangent gradient was nonzero.
+
+To separate optimization from objective quality, the final weights were fixed
+at `(2.0, 0.2, 80.0, 30.0)` and the saved endpoint was optimized for another
+1,000 steps at one tenth of the original learning rate.
+
+| Metric | Saved endpoint | Continued descent |
+|---|---:|---:|
+| Fixed total energy | 2.767 | approximately 1.54 |
+| R@10 | 5.86% | approximately 3--4% |
+| Median rank | 103 | approximately 120--123 |
+
+Random R@10 among 256 candidates is 3.91%. Thus substantial additional energy
+reduction removes rather than strengthens the retrieval signal. Gradients also
+remain nonzero and Adam oscillates around the lower-energy region, so the
+original endpoint was not converged.
+
+More importantly, the true paired language configuration has fixed energy
+approximately 4.49, while incorrect optimized particles reach approximately
+1.54. The current objective therefore assigns lower energy to synthetic
+language-side configurations that imitate visual relation statistics than to
+the correct semantic configuration.
+
+This is an objective falsification, not merely an optimization failure.
+
+## Conclusion
+
+The necessary-condition diagnostic supports a narrower claim:
+
+> Overcomplete language observations make the correct visual-language
+> configuration more distinguishable from shuffled configurations.
+
+The descent experiment does not establish stable semantic alignment. Its early
+rank improvement is a trajectory transient: optimizing the current energy
+further returns retrieval to chance because the correct configuration is not
+an energy minimum.
+
+Before another descent experiment, a candidate energy should be tested directly
+on held-out real data and required to rank the true configuration below a large
+family of incorrect configurations. Paired data can remain evaluation-only in
+this diagnostic.
+
+The missing term is not another static geometry or language-fluency prior. It
+must be an externally verifiable predictive world energy. The next experiment
+should compare response operators under controlled interventions—for example,
+object removal, attribute change, counting, or spatial perturbation—so that an
+incorrect but fluent/self-consistent language state cannot achieve low energy.
+
+## Main artifacts
+
+- `artifacts/energy_free_orbit_test_m10.json`
+- `artifacts/energy_free_orbit_test_m30.json`
+- `artifacts/energy_orbit_prefix_test_m30_256.json`
+- `artifacts/energy_prior_only_test_m30.json`
+- `artifacts/energy_prefix_prior_only_test_m30_256.json`
+- `artifacts/energy_shuffled_visual_test_m30.json`
+- `artifacts/energy_prefix_shuffled_visual_test_m30_256.json`
+- `artifacts/energy_functional_val_f10.json`
+- `artifacts/energy_functional_prefix_val_f10_128.json`
+- `artifacts/energy_convergence_audit_m30.json`