diff options
Diffstat (limited to 'ENERGY_RESULTS.md')
| -rw-r--r-- | ENERGY_RESULTS.md | 193 |
1 files changed, 193 insertions, 0 deletions
diff --git a/ENERGY_RESULTS.md b/ENERGY_RESULTS.md new file mode 100644 index 0000000..03e0690 --- /dev/null +++ b/ENERGY_RESULTS.md @@ -0,0 +1,193 @@ +# Energy-descent multimodal inference MVP + +Run date: 2026-07-29 + +## Question + +Can an image condition a frozen language model through iterative energy +descent in language latent space, without learning an image-to-language MLP or +using image-text pairs in the objective? + +The finite samples in this experiment are particles used to estimate +population energies. They are not assumed to be semantic graph nodes. + +## Protocol + +- Data: Flickr30k. +- Vision: frozen `facebook/dinov2-small`. +- Language: frozen `Qwen/Qwen2.5-0.5B`. +- Image-only and text-only populations contain disjoint image IDs. +- A language particle is a free 896-dimensional state initialized from a real + text-only activation. +- Five independently written captions are encoded as a language observation + orbit and averaged. Their image correspondence is never exposed to the + energy or optimizer. +- The optimized particles are decoded by a prefix interface trained only to + reconstruct captions from frozen Qwen text activations. +- Paired validation/test captions are loaded only for evaluation. The + optimizer, energy, initialization, and stopping step do not use them. + +The base energy is + +\[ +E(Z;X)= +\lambda_r E_{\mathrm{relation}}(X,Z) ++\lambda_c E_{\mathrm{conditional}}(X,Z) ++\lambda_d E_{\mathrm{text\ distribution}}(Z) ++\lambda_m E_{\mathrm{text\ manifold}}(Z). +\] + +The cross-modal terms match standardized population relation fields and +multiscale conditional-neighborhood distributions. The language-only terms +match sliced text-population quantiles and distance to text-only prototypes. +Only `Z` is optimized; there is no shared image-to-text function. + +## Necessary-condition diagnostics + +On 512 held-out test images, the correct language configuration has lower +relation energy than a random permutation: + +| Language observation | Correct energy | Shuffled energy | Gap | +|---|---:|---:|---:| +| One caption | 1.653 | 1.969 | 0.316 | +| Mean of five-caption orbit | 1.425 | 2.007 | **0.582** | + +Relational overcompleteness nearly doubles the correct-vs-shuffled energy gap. + +## Free latent descent + +Weak language-manifold regularization produces a large semantic-rank +improvement but leaves the decodable language manifold. On the independent +test split, manifold weight 10 gives: + +| Metric | Initial | Final | +|---|---:|---:| +| R@5 | 1.95% | 6.64% | +| R@10 | 2.34% | 11.72% | +| Median rank (256 candidates) | 125 | 103 | +| Caption unigram F1 (100 samples) | 0.250 | 0.244 | + +The final prototype-manifold energy is 0.087 versus 0.0147 for real paired +text states. Better relative geometry is therefore not sufficient for frozen +LLM decoding. + +## Strong manifold point + +Manifold weight 30 was selected on validation because it keeps the final +language-manifold energy closest to the real-text regime, rather than because +of paired retrieval. On all 256 test particles: + +| Metric | Initial | Final | +|---|---:|---:| +| R@5 | 1.95% | 2.73% | +| R@10 | 2.34% | 5.86% | +| Median rank | 125 | 103 | +| Caption unigram F1 | 0.2495 | 0.2623 | + +The paired F1 change is `+0.0128`, with a 10,000-resample percentile bootstrap +interval of `[+0.0005, +0.0252]`. + +This apparent generation improvement is not image-conditioned, however. + +## Required controls + +| Energy | Final R@10 | Final median rank | Final caption F1 | F1 change | +|---|---:|---:|---:|---:| +| Full image + language energy | **5.86%** | **103** | 0.2623 | +0.0128 | +| Language prior only | 3.13% | 120 | **0.2697** | +0.0203 | +| Full energy, visual particles shuffled | 4.30% | 120 | 0.2500 | +0.0005 | + +The correct visual relation field is responsible for the rank improvement: +prior-only and shuffled-vision controls remain near chance and have worse +median rank. But the language-prior-only control improves caption F1 more than +the full system. The generation gain is therefore a generic caption-prior +cleanup effect, not evidence that the image selected the correct language +state. + +## Functional-cycle energy + +A higher-level frozen-language energy was also tested: + +\[ +z\rightarrow +\text{text-only prefix + frozen Qwen decode} +\rightarrow\hat y +\rightarrow\text{frozen Qwen encode} +\rightarrow\hat z, +\qquad +E_{\mathrm{cycle}}=1-\cos(z,\hat z). +\] + +With functional weight 10 on 128 validation particles, R@10 improves from +9.38% to 17.97% and median rank from 57 to 43, but caption F1 falls from +0.2508 to 0.2256. A stronger cycle weight suppresses both the rank gain and +the degradation but still does not improve generation. + +Decode-reencode consistency is thus not a sufficient world energy: a generic +or incorrect caption can be a stable self-consistent fixed point. + +## Convergence audit + +The total printed during the original 100-step run is not a fixed-objective +convergence curve: relation and conditional weights are annealed throughout +the run. At the `m30` endpoint, both cross-modal components were still changing +and the tangent gradient was nonzero. + +To separate optimization from objective quality, the final weights were fixed +at `(2.0, 0.2, 80.0, 30.0)` and the saved endpoint was optimized for another +1,000 steps at one tenth of the original learning rate. + +| Metric | Saved endpoint | Continued descent | +|---|---:|---:| +| Fixed total energy | 2.767 | approximately 1.54 | +| R@10 | 5.86% | approximately 3--4% | +| Median rank | 103 | approximately 120--123 | + +Random R@10 among 256 candidates is 3.91%. Thus substantial additional energy +reduction removes rather than strengthens the retrieval signal. Gradients also +remain nonzero and Adam oscillates around the lower-energy region, so the +original endpoint was not converged. + +More importantly, the true paired language configuration has fixed energy +approximately 4.49, while incorrect optimized particles reach approximately +1.54. The current objective therefore assigns lower energy to synthetic +language-side configurations that imitate visual relation statistics than to +the correct semantic configuration. + +This is an objective falsification, not merely an optimization failure. + +## Conclusion + +The necessary-condition diagnostic supports a narrower claim: + +> Overcomplete language observations make the correct visual-language +> configuration more distinguishable from shuffled configurations. + +The descent experiment does not establish stable semantic alignment. Its early +rank improvement is a trajectory transient: optimizing the current energy +further returns retrieval to chance because the correct configuration is not +an energy minimum. + +Before another descent experiment, a candidate energy should be tested directly +on held-out real data and required to rank the true configuration below a large +family of incorrect configurations. Paired data can remain evaluation-only in +this diagnostic. + +The missing term is not another static geometry or language-fluency prior. It +must be an externally verifiable predictive world energy. The next experiment +should compare response operators under controlled interventions—for example, +object removal, attribute change, counting, or spatial perturbation—so that an +incorrect but fluent/self-consistent language state cannot achieve low energy. + +## Main artifacts + +- `artifacts/energy_free_orbit_test_m10.json` +- `artifacts/energy_free_orbit_test_m30.json` +- `artifacts/energy_orbit_prefix_test_m30_256.json` +- `artifacts/energy_prior_only_test_m30.json` +- `artifacts/energy_prefix_prior_only_test_m30_256.json` +- `artifacts/energy_shuffled_visual_test_m30.json` +- `artifacts/energy_prefix_shuffled_visual_test_m30_256.json` +- `artifacts/energy_functional_val_f10.json` +- `artifacts/energy_functional_prefix_val_f10_128.json` +- `artifacts/energy_convergence_audit_m30.json` |