# Energy-descent multimodal inference MVP

Run date: 2026-07-29

## Question

Can an image condition a frozen language model through iterative energy
descent in language latent space, without learning an image-to-language MLP or
using image-text pairs in the objective?

The finite samples in this experiment are particles used to estimate
population energies. They are not assumed to be semantic graph nodes.

## Protocol

- Data: Flickr30k.
- Vision: frozen `facebook/dinov2-small`.
- Language: frozen `Qwen/Qwen2.5-0.5B`.
- Image-only and text-only populations contain disjoint image IDs.
- A language particle is a free 896-dimensional state initialized from a real
  text-only activation.
- Five independently written captions are encoded as a language observation
  orbit and averaged. Their image correspondence is never exposed to the
  energy or optimizer.
- The optimized particles are decoded by a prefix interface trained only to
  reconstruct captions from frozen Qwen text activations.
- Paired validation/test captions are loaded only for evaluation. The
  optimizer, energy, initialization, and stopping step do not use them.

The base energy is

\[
E(Z;X)=
\lambda_r E_{\mathrm{relation}}(X,Z)
+\lambda_c E_{\mathrm{conditional}}(X,Z)
+\lambda_d E_{\mathrm{text\ distribution}}(Z)
+\lambda_m E_{\mathrm{text\ manifold}}(Z).
\]

The cross-modal terms match standardized population relation fields and
multiscale conditional-neighborhood distributions. The language-only terms
match sliced text-population quantiles and distance to text-only prototypes.
Only `Z` is optimized; there is no shared image-to-text function.

## Necessary-condition diagnostics

On 512 held-out test images, the correct language configuration has lower
relation energy than a random permutation:

| Language observation | Correct energy | Shuffled energy | Gap |
|---|---:|---:|---:|
| One caption | 1.653 | 1.969 | 0.316 |
| Mean of five-caption orbit | 1.425 | 2.007 | **0.582** |

Relational overcompleteness nearly doubles the correct-vs-shuffled energy gap.

## Free latent descent

Weak language-manifold regularization produces a large semantic-rank
improvement but leaves the decodable language manifold. On the independent
test split, manifold weight 10 gives:

| Metric | Initial | Final |
|---|---:|---:|
| R@5 | 1.95% | 6.64% |
| R@10 | 2.34% | 11.72% |
| Median rank (256 candidates) | 125 | 103 |
| Caption unigram F1 (100 samples) | 0.250 | 0.244 |

The final prototype-manifold energy is 0.087 versus 0.0147 for real paired
text states. Better relative geometry is therefore not sufficient for frozen
LLM decoding.

## Strong manifold point

Manifold weight 30 was selected on validation because it keeps the final
language-manifold energy closest to the real-text regime, rather than because
of paired retrieval. On all 256 test particles:

| Metric | Initial | Final |
|---|---:|---:|
| R@5 | 1.95% | 2.73% |
| R@10 | 2.34% | 5.86% |
| Median rank | 125 | 103 |
| Caption unigram F1 | 0.2495 | 0.2623 |

The paired F1 change is `+0.0128`, with a 10,000-resample percentile bootstrap
interval of `[+0.0005, +0.0252]`.

This apparent generation improvement is not image-conditioned, however.

## Required controls

| Energy | Final R@10 | Final median rank | Final caption F1 | F1 change |
|---|---:|---:|---:|---:|
| Full image + language energy | **5.86%** | **103** | 0.2623 | +0.0128 |
| Language prior only | 3.13% | 120 | **0.2697** | +0.0203 |
| Full energy, visual particles shuffled | 4.30% | 120 | 0.2500 | +0.0005 |

The correct visual relation field is responsible for the rank improvement:
prior-only and shuffled-vision controls remain near chance and have worse
median rank. But the language-prior-only control improves caption F1 more than
the full system. The generation gain is therefore a generic caption-prior
cleanup effect, not evidence that the image selected the correct language
state.

## Functional-cycle energy

A higher-level frozen-language energy was also tested:

\[
z\rightarrow
\text{text-only prefix + frozen Qwen decode}
\rightarrow\hat y
\rightarrow\text{frozen Qwen encode}
\rightarrow\hat z,
\qquad
E_{\mathrm{cycle}}=1-\cos(z,\hat z).
\]

With functional weight 10 on 128 validation particles, R@10 improves from
9.38% to 17.97% and median rank from 57 to 43, but caption F1 falls from
0.2508 to 0.2256. A stronger cycle weight suppresses both the rank gain and
the degradation but still does not improve generation.

Decode-reencode consistency is thus not a sufficient world energy: a generic
or incorrect caption can be a stable self-consistent fixed point.

## Convergence audit

The total printed during the original 100-step run is not a fixed-objective
convergence curve: relation and conditional weights are annealed throughout
the run. At the `m30` endpoint, both cross-modal components were still changing
and the tangent gradient was nonzero.

To separate optimization from objective quality, the final weights were fixed
at `(2.0, 0.2, 80.0, 30.0)` and the saved endpoint was optimized for another
1,000 steps at one tenth of the original learning rate.

| Metric | Saved endpoint | Continued descent |
|---|---:|---:|
| Fixed total energy | 2.767 | approximately 1.54 |
| R@10 | 5.86% | approximately 3--4% |
| Median rank | 103 | approximately 120--123 |

Random R@10 among 256 candidates is 3.91%. Thus substantial additional energy
reduction removes rather than strengthens the retrieval signal. Gradients also
remain nonzero and Adam oscillates around the lower-energy region, so the
original endpoint was not converged.

More importantly, the true paired language configuration has fixed energy
approximately 4.49, while incorrect optimized particles reach approximately
1.54. The current objective therefore assigns lower energy to synthetic
language-side configurations that imitate visual relation statistics than to
the correct semantic configuration.

This is an objective falsification, not merely an optimization failure.

## Conclusion

The necessary-condition diagnostic supports a narrower claim:

> Overcomplete language observations make the correct visual-language
> configuration more distinguishable from shuffled configurations.

The descent experiment does not establish stable semantic alignment. Its early
rank improvement is a trajectory transient: optimizing the current energy
further returns retrieval to chance because the correct configuration is not
an energy minimum.

Before another descent experiment, a candidate energy should be tested directly
on held-out real data and required to rank the true configuration below a large
family of incorrect configurations. Paired data can remain evaluation-only in
this diagnostic.

The missing term is not another static geometry or language-fluency prior. It
must be an externally verifiable predictive world energy. The next experiment
should compare response operators under controlled interventions—for example,
object removal, attribute change, counting, or spatial perturbation—so that an
incorrect but fluent/self-consistent language state cannot achieve low energy.

## Main artifacts

- `artifacts/energy_free_orbit_test_m10.json`
- `artifacts/energy_free_orbit_test_m30.json`
- `artifacts/energy_orbit_prefix_test_m30_256.json`
- `artifacts/energy_prior_only_test_m30.json`
- `artifacts/energy_prefix_prior_only_test_m30_256.json`
- `artifacts/energy_shuffled_visual_test_m30.json`
- `artifacts/energy_prefix_shuffled_visual_test_m30_256.json`
- `artifacts/energy_functional_val_f10.json`
- `artifacts/energy_functional_prefix_val_f10_128.json`
- `artifacts/energy_convergence_audit_m30.json`