diff options
Diffstat (limited to 'NATURAL_RESULTS.md')
| -rw-r--r-- | NATURAL_RESULTS.md | 111 |
1 files changed, 111 insertions, 0 deletions
diff --git a/NATURAL_RESULTS.md b/NATURAL_RESULTS.md new file mode 100644 index 0000000..ef7339c --- /dev/null +++ b/NATURAL_RESULTS.md @@ -0,0 +1,111 @@ +# Natural data under the Tier 0 recipe + +Run date: 2026-08-01 + +## Question + +The synthetic world showed the machinery works end to end when the +representations are good enough: 95% blind recovery, 93% transfer to +scenes the matcher never saw. It could not show that natural data admits +such representations. This ports the recipe to Visual Genome and measures +the one statistic that decides whether to attempt recovery -- the +cross-modal field correlation, which the phase diagram puts at roughly +0.9 for polynomial matching. + +## Protocol + +Vision: DINOv2 patch features, segments from spectral clustering of the +patch affinity graph with a spatial prior, one state per segment. No +boxes, no labels, no text. Language: word classes and word vectors from +positive-PMI context statistics of the 80,000 region descriptions, one +state per phrase. Scene states are the resulting sets; relations are +moment-kernel similarities. Annotated boxes appear only as a diagnostic +oracle, never in the reported pipeline. + +## Text-side factor discovery transfers, but not the heuristic + +The synthetic pipeline found factor families by mutual exclusivity, since +templated phrases carry exactly one colour word. Real language refutes +that: colour terms are independent or better, with observed-over-expected +co-occurrence 1.29 for black with white and 0.93 for red with blue, while +head nouns are the genuinely exclusive family at 0.14 for car with truck. +Run unchanged, the method returns clean noun classes and no attributes. + +Standard distributional word-class induction, of which exclusivity is a +degenerate case, does transfer. Positive PMI, truncated SVD, and k-means +over the 300 most frequent words give a colour family at 0.67 purity plus +coherent vehicle, animal, clothing, person, and water-scene classes. +Colour terms split into three clusters rather than one, structurally +rather than noisily: white and blue sit with sky and clouds, because what +a colour word modifies shapes its distribution. + +## What moves the field correlation + +| Configuration | rho | +|---|---:| +| Hard class codes, six segments | 0.166 | +| Hard class codes, sixteen segments | 0.136 | +| Continuous states, unsupervised segments | 0.489 | +| Continuous states, oracle region boxes | 0.550 | +| Continuous states plus content projection, unsupervised | **0.656** | + +The reported figure is what `worldalign.natural_pipeline` produces; an +earlier exploratory script reached 0.664 with a wider projection-fitting +range, and the reproducible module is cited in preference. +| Continuous states plus content projection, oracle boxes | 0.671 | + +Three findings, in order of how much they overturned expectations. + +**Discretisation was the dominant loss.** Quantising segments and phrases +into class codes cost more than half the signal (0.166 against 0.489). +Relation fields need only scene-to-scene similarity, so compressing +states into discrete classes first is pure attrition. The synthetic world +hid this because its states are discrete by construction. + +**Segmentation is not the bottleneck here.** Oracle region boxes buy +0.061 over unsupervised spectral segmentation, so the unsupervised +segmenter captures 89% of what perfect boxes would give -- the exact +inverse of the synthetic world, where segmentation was the entire gap. +Flat untextured scenes are adversarial for object discovery in a way that +photographs are not, so the synthetic measurement overstated this +problem. + +**Content projection is again the largest lever.** Whitening the +within-scene direction and keeping the eight leading between-scene +directions lifts 0.489 to 0.656, matching its role in the earlier +node-level battery where it took VG from 0.21 to 0.61. The protocol-legal +configuration and the oracle-box configuration land within 0.007 of each +other, which says the remaining deficit is not about where the objects +are. + +A control worth recording: frozen general-purpose encoders do worse than +corpus-fitted statistics. DINOv2-small CLS features of isolated region +crops against Qwen mean-pooled phrase states reach only 0.19 in the same +set-moment fields. In-context patch features beat isolated crops, and +PPMI vectors fitted to this corpus beat a general language model's +states, echoing the earlier finding that what the model layer shares is +the mixed state rather than the isolated encoding. + +## Verdict + +The go/no-go statistic stands at 0.656 against the 0.9 that polynomial +recovery needs, so no recovery run is warranted. The trajectory in one +session is 0.166 to 0.656, and the remaining gap is now attributable: +not discretisation, which is removed, and not segmentation, which costs +0.06. What is left is the shared structure between self-supervised visual +features and distributional language statistics at the level of single +regions. + +The levers that have not been tried on natural data are the ones the +synthetic world showed to matter most after segmentation: observation +orbits, which VG lacks because each scene has one photograph, and richer +per-region descriptors. Multi-view natural data -- video frames, product +photographs, or augmentation-derived orbits -- is the next protocol +requirement, and the synthetic result that two-sided multi-view +observation is necessary now has a natural-data counterpart to test. + +## Main artifacts + +- `artifacts/vg_5k/natural_families{,_dist}.json` +- `artifacts/vg_5k/natural_objects{,_16,_oracle}.pt` +- `artifacts/vg_5k/natural_fields{,_16}.pt` |