From a62cf4d2a99b4a7985c61b2a7feb92a82a8218b7 Mon Sep 17 00:00:00 2001 From: Yuren Hao Date: Sat, 1 Aug 2026 14:10:03 -0500 Subject: World Alignment: unpaired cross-modal correspondence by relational identifiability Method: scene states are sets of part states; relation fields are built within each modality and are invariant to how each side labels its own features; the cross-modal bridge is a coupling searched under an energy that is a closed-form functional of one matrix; solving is spectral initialisation followed by exact local refinement. Evidence: in a procedurally generated closed world, blind recovery of a hidden image-caption correspondence reaches 95.3% at 256 scenes against 0.39% chance, and the recovered pairs transfer to 200 held-out scenes at 93.0% exact retrieval with random-pair and shuffled-image controls at or near chance. Cross-modal value correspondence is derived from disjoint corpora rather than declared. On Visual Genome the field correlation reaches 0.656 against the 0.9 that polynomial recovery needs, with the deficit attributed away from segmentation and discretisation. Protocol: no image-text pair enters any objective, optimiser, initialisation, or model selection; hidden pairs score orderings only. Co-Authored-By: Claude --- NATURAL_RESULTS.md | 111 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 111 insertions(+) create mode 100644 NATURAL_RESULTS.md (limited to 'NATURAL_RESULTS.md') diff --git a/NATURAL_RESULTS.md b/NATURAL_RESULTS.md new file mode 100644 index 0000000..ef7339c --- /dev/null +++ b/NATURAL_RESULTS.md @@ -0,0 +1,111 @@ +# Natural data under the Tier 0 recipe + +Run date: 2026-08-01 + +## Question + +The synthetic world showed the machinery works end to end when the +representations are good enough: 95% blind recovery, 93% transfer to +scenes the matcher never saw. It could not show that natural data admits +such representations. This ports the recipe to Visual Genome and measures +the one statistic that decides whether to attempt recovery -- the +cross-modal field correlation, which the phase diagram puts at roughly +0.9 for polynomial matching. + +## Protocol + +Vision: DINOv2 patch features, segments from spectral clustering of the +patch affinity graph with a spatial prior, one state per segment. No +boxes, no labels, no text. Language: word classes and word vectors from +positive-PMI context statistics of the 80,000 region descriptions, one +state per phrase. Scene states are the resulting sets; relations are +moment-kernel similarities. Annotated boxes appear only as a diagnostic +oracle, never in the reported pipeline. + +## Text-side factor discovery transfers, but not the heuristic + +The synthetic pipeline found factor families by mutual exclusivity, since +templated phrases carry exactly one colour word. Real language refutes +that: colour terms are independent or better, with observed-over-expected +co-occurrence 1.29 for black with white and 0.93 for red with blue, while +head nouns are the genuinely exclusive family at 0.14 for car with truck. +Run unchanged, the method returns clean noun classes and no attributes. + +Standard distributional word-class induction, of which exclusivity is a +degenerate case, does transfer. Positive PMI, truncated SVD, and k-means +over the 300 most frequent words give a colour family at 0.67 purity plus +coherent vehicle, animal, clothing, person, and water-scene classes. +Colour terms split into three clusters rather than one, structurally +rather than noisily: white and blue sit with sky and clouds, because what +a colour word modifies shapes its distribution. + +## What moves the field correlation + +| Configuration | rho | +|---|---:| +| Hard class codes, six segments | 0.166 | +| Hard class codes, sixteen segments | 0.136 | +| Continuous states, unsupervised segments | 0.489 | +| Continuous states, oracle region boxes | 0.550 | +| Continuous states plus content projection, unsupervised | **0.656** | + +The reported figure is what `worldalign.natural_pipeline` produces; an +earlier exploratory script reached 0.664 with a wider projection-fitting +range, and the reproducible module is cited in preference. +| Continuous states plus content projection, oracle boxes | 0.671 | + +Three findings, in order of how much they overturned expectations. + +**Discretisation was the dominant loss.** Quantising segments and phrases +into class codes cost more than half the signal (0.166 against 0.489). +Relation fields need only scene-to-scene similarity, so compressing +states into discrete classes first is pure attrition. The synthetic world +hid this because its states are discrete by construction. + +**Segmentation is not the bottleneck here.** Oracle region boxes buy +0.061 over unsupervised spectral segmentation, so the unsupervised +segmenter captures 89% of what perfect boxes would give -- the exact +inverse of the synthetic world, where segmentation was the entire gap. +Flat untextured scenes are adversarial for object discovery in a way that +photographs are not, so the synthetic measurement overstated this +problem. + +**Content projection is again the largest lever.** Whitening the +within-scene direction and keeping the eight leading between-scene +directions lifts 0.489 to 0.656, matching its role in the earlier +node-level battery where it took VG from 0.21 to 0.61. The protocol-legal +configuration and the oracle-box configuration land within 0.007 of each +other, which says the remaining deficit is not about where the objects +are. + +A control worth recording: frozen general-purpose encoders do worse than +corpus-fitted statistics. DINOv2-small CLS features of isolated region +crops against Qwen mean-pooled phrase states reach only 0.19 in the same +set-moment fields. In-context patch features beat isolated crops, and +PPMI vectors fitted to this corpus beat a general language model's +states, echoing the earlier finding that what the model layer shares is +the mixed state rather than the isolated encoding. + +## Verdict + +The go/no-go statistic stands at 0.656 against the 0.9 that polynomial +recovery needs, so no recovery run is warranted. The trajectory in one +session is 0.166 to 0.656, and the remaining gap is now attributable: +not discretisation, which is removed, and not segmentation, which costs +0.06. What is left is the shared structure between self-supervised visual +features and distributional language statistics at the level of single +regions. + +The levers that have not been tried on natural data are the ones the +synthetic world showed to matter most after segmentation: observation +orbits, which VG lacks because each scene has one photograph, and richer +per-region descriptors. Multi-view natural data -- video frames, product +photographs, or augmentation-derived orbits -- is the next protocol +requirement, and the synthetic result that two-sided multi-view +observation is necessary now has a natural-data counterpart to test. + +## Main artifacts + +- `artifacts/vg_5k/natural_families{,_dist}.json` +- `artifacts/vg_5k/natural_objects{,_16,_oracle}.pt` +- `artifacts/vg_5k/natural_fields{,_16}.pt` -- cgit v1.2.3