diff options
Diffstat (limited to 'RESULTS.md')
| -rw-r--r-- | RESULTS.md | 128 |
1 files changed, 128 insertions, 0 deletions
diff --git a/RESULTS.md b/RESULTS.md new file mode 100644 index 0000000..a4a88f1 --- /dev/null +++ b/RESULTS.md @@ -0,0 +1,128 @@ +# WorldAlign MVP: first real-data result + +Run date: 2026-07-28 + +## Question + +Can independently pretrained, frozen vision and language models be aligned +without image-text pairs or a contrastive objective, and can the resulting +bridge drive a frozen causal language model? + +## Protocol + +- Data: Flickr30k. +- Unpaired training: 12,000 image-only examples and 12,000 caption-only + examples with disjoint Flickr image IDs. +- Held out: 1,000 validation and 1,000 test image-caption groups. +- Vision: frozen `facebook/dinov2-small`, 384-dimensional CLS feature. +- Language: frozen `Qwen/Qwen2.5-0.5B` or + `Qwen/Qwen2.5-1.5B`, mean-pooled final hidden state. +- Bridge: approximately 2–3M trainable parameters. +- Unpaired energy: prototype Gromov-Wasserstein correspondence, sliced + Wasserstein distribution matching, and within-modality isometry. +- Generation interface: a text-only-trained latent-to-prefix adapter followed + by a frozen Qwen causal LM. +- Paired bridge: same features and bridge family, trained on 12,000 pairs only + as an interface upper bound. + +No validation/test pair is used by the unpaired objective or checkpoint +selection. + +## Results + +### Shared relational structure exists + +On 1,000 held-out paired examples used only for diagnosis: + +| Text backbone | Linear CKA | Pairwise cosine Spearman | Shuffled mean | Shuffle z | +|---|---:|---:|---:|---:| +| Qwen2.5-0.5B | 0.2622 | 0.1835 | -0.0003 | 12.34 | +| Qwen2.5-1.5B | 0.2611 | 0.1837 | 0.0011 | 12.87 | + +The two independently pretrained representations therefore share a strong, +non-random relational signal. Increasing Qwen from 0.5B to 1.5B does not +materially change these global geometry metrics. + +### Static geometry does not uniquely recover the semantic gauge + +The 128-prototype GW coupling is sharp but mostly incorrect: + +| Text backbone | GW distance | Row entropy | Held-out cluster match | Chance | Oracle permutation | +|---|---:|---:|---:|---:|---:| +| Qwen2.5-0.5B | 0.1642 | 0.3875 | 1.5% | 0.78% | 20.5% | +| Qwen2.5-1.5B | 0.1218 | 0.3814 | 1.7% | 0.78% | 20.0% | + +For reference, uniform 128-way entropy is about 4.85. The low entropy says GW +finds a confident structural matching; the low true match says confidence is +not identifiability. + +### Frozen-backbone retrieval + +All values below are from the 1,000-example held-out test split. + +| Bridge | i2t R@1 | i2t R@5 | i2t R@10 | i2t median rank | t2i R@10 | +|---|---:|---:|---:|---:|---:| +| Random chance | 0.1% | 0.5% | 1.0% | about 500 | 1.0% | +| Unpaired SWD, Qwen-0.5B | 0.1% | 0.5% | 1.3% | 473 | 0.9% | +| Unpaired GW, Qwen-0.5B | 0.0% | 0.5% | 0.8% | 465 | 1.1% | +| Unpaired GW, Qwen-1.5B | 0.2% | 1.3% | 2.2% | 379 | 1.2% | +| Paired upper bound, Qwen-0.5B | 13.7% | 35.4% | 46.4% | 13 | 15.6% | + +The Qwen-1.5B unpaired result is weak but reproducible. Three additional bridge +initializations obtained i2t R@10 of 2.1%, 2.3%, and 2.2%, with median ranks +384, 371, and 379. Under an independent-query binomial chance approximation, +the corresponding one-sided p-values are between 0.00027 and 0.00150. This is +evidence for partial alignment, not usable multimodal ability. + +SWD-only and a linear GW bridge both remain at chance, so the Qwen-0.5B +failure is not explained just by an overly flexible MLP or by conflict between +the GW and distribution terms. + +### Frozen-LLM generation + +The semantic-latent-to-prefix adapter is trained only on captions and frozen +Qwen states. It never receives image features or image-text pairs. + +On the same first 100 held-out test examples: + +| Input to text-only prefix adapter | Best-reference unigram F1 | +|---|---:| +| True Qwen text latent | 0.5370 | +| Paired image bridge | 0.2933 | +| Unpaired image bridge | 0.2202 | +| Paired image bridge with image order shuffled | 0.2009 | + +The text-only interface can make frozen Qwen decode its semantic state, and a +correctly trained vision bridge can drive that interface. The current +unpaired bridge is only slightly above the shuffled lexical baseline and its +examples are generally fluent descriptions of the wrong scene. + +## Conclusion + +This MVP separates three claims that are easy to conflate: + +1. Independently pretrained vision and language models share non-random + relational structure: supported. +2. Static distribution and pairwise-geometry matching are sufficient to + identify the correct cross-modal coordinate system: not supported. +3. Once the coordinate system is correct, a small bridge can invoke a frozen + language model's generative capability: supported by the paired control. + +The next experiment should preserve this real-data, frozen-LLM endpoint but +add constraints that break semantic automorphisms: matched transition +operators, interventions, or another functional signal. Merely scaling the +same static GW/SWD objective is unlikely to close the roughly 44-point i2t +R@10 gap to the paired interface. + +## Main artifacts + +- `artifacts/diagnostics.json` +- `artifacts/diagnostics_qwen1p5b.json` +- `artifacts/gw.pt` +- `artifacts/gw_qwen1p5b.pt` +- `artifacts/eval_paired.json` +- `artifacts/eval_unpaired.json` +- `artifacts/eval_paired_shuffled.json` +- `artifacts/eval_unpaired_qwen1p5b.json` +- `artifacts/eval_unpaired_qwen1p5b_seed{11,22,33}.json` +- `artifacts/prefix_evaluation.json` |