1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
|
# Natural data under the Tier 0 recipe
Run date: 2026-08-01
## Question
The synthetic world showed the machinery works end to end when the
representations are good enough: 95% blind recovery, 93% transfer to
scenes the matcher never saw. It could not show that natural data admits
such representations. This ports the recipe to Visual Genome and measures
the one statistic that decides whether to attempt recovery -- the
cross-modal field correlation, which the phase diagram puts at roughly
0.9 for polynomial matching.
## Protocol
Vision: DINOv2 patch features, segments from spectral clustering of the
patch affinity graph with a spatial prior, one state per segment. No
boxes, no labels, no text. Language: word classes and word vectors from
positive-PMI context statistics of the 80,000 region descriptions, one
state per phrase. Scene states are the resulting sets; relations are
moment-kernel similarities. Annotated boxes appear only as a diagnostic
oracle, never in the reported pipeline.
## Text-side factor discovery transfers, but not the heuristic
The synthetic pipeline found factor families by mutual exclusivity, since
templated phrases carry exactly one colour word. Real language refutes
that: colour terms are independent or better, with observed-over-expected
co-occurrence 1.29 for black with white and 0.93 for red with blue, while
head nouns are the genuinely exclusive family at 0.14 for car with truck.
Run unchanged, the method returns clean noun classes and no attributes.
Standard distributional word-class induction, of which exclusivity is a
degenerate case, does transfer. Positive PMI, truncated SVD, and k-means
over the 300 most frequent words give a colour family at 0.67 purity plus
coherent vehicle, animal, clothing, person, and water-scene classes.
Colour terms split into three clusters rather than one, structurally
rather than noisily: white and blue sit with sky and clouds, because what
a colour word modifies shapes its distribution.
## What moves the field correlation
| Configuration | rho |
|---|---:|
| Hard class codes, six segments | 0.166 |
| Hard class codes, sixteen segments | 0.136 |
| Continuous states, unsupervised segments | 0.489 |
| Continuous states, oracle region boxes | 0.550 |
| Continuous states plus content projection, unsupervised | **0.656** |
The reported figure is what `worldalign.natural_pipeline` produces; an
earlier exploratory script reached 0.664 with a wider projection-fitting
range, and the reproducible module is cited in preference.
| Continuous states plus content projection, oracle boxes | 0.671 |
Three findings, in order of how much they overturned expectations.
**Discretisation was the dominant loss.** Quantising segments and phrases
into class codes cost more than half the signal (0.166 against 0.489).
Relation fields need only scene-to-scene similarity, so compressing
states into discrete classes first is pure attrition. The synthetic world
hid this because its states are discrete by construction.
**Segmentation is not the bottleneck here.** Oracle region boxes buy
0.061 over unsupervised spectral segmentation, so the unsupervised
segmenter captures 89% of what perfect boxes would give -- the exact
inverse of the synthetic world, where segmentation was the entire gap.
Flat untextured scenes are adversarial for object discovery in a way that
photographs are not, so the synthetic measurement overstated this
problem.
**Content projection is again the largest lever.** Whitening the
within-scene direction and keeping the eight leading between-scene
directions lifts 0.489 to 0.656, matching its role in the earlier
node-level battery where it took VG from 0.21 to 0.61. The protocol-legal
configuration and the oracle-box configuration land within 0.007 of each
other, which says the remaining deficit is not about where the objects
are.
A control worth recording: frozen general-purpose encoders do worse than
corpus-fitted statistics. DINOv2-small CLS features of isolated region
crops against Qwen mean-pooled phrase states reach only 0.19 in the same
set-moment fields. In-context patch features beat isolated crops, and
PPMI vectors fitted to this corpus beat a general language model's
states, echoing the earlier finding that what the model layer shares is
the mixed state rather than the isolated encoding.
## Augmentation orbits are not a substitute for multiple photographs
The closed world made two-sided multi-view observation a protocol
requirement, and Visual Genome carries one photograph per scene, so the
orbit was synthesised: four random resized crops per image, segmented
independently, their fields averaged. The correlation moves from 0.6559
to 0.6559 -- no gain at four views.
The mechanism is visible in what each orbit cancels. Re-renders in the
closed world resample layout, which is nuisance by construction, so
averaging over them removes exactly the modality-private variation.
Random crops perturb framing, and self-supervised patch features are
already close to invariant under it, so the average has nothing to
cancel. Multi-view observation, where it matters, means separate
photographs of the same scene rather than transformations of one.
## Verdict
**Superseded 2026-08-01 — see `RANK_RESULTS.md`.** The go/no-go statistic
used below was shown not to govern recovery: a synthetic field truncated
to rank 8 correlates at 0.906 and recovers 13%, so 0.9 is neither a
threshold nor a licence. The verdict on natural data survives the
correction — the search was subsequently run and returned 0.0000 against
0.0039 chance — but it is now a measurement rather than an inference, and
the deficit is diagnosed as much in the width of the shared spectrum
(15 directions against 26) as in the correlation. The paragraph below is
kept as written for the record.
The go/no-go statistic stands at 0.656 against the 0.9 that polynomial
recovery needs, so no recovery run is warranted. The trajectory in one
session is 0.166 to 0.656, and the remaining gap is now attributable:
not discretisation, which is removed, and not segmentation, which costs
0.06. What is left is the shared structure between self-supervised visual
features and distributional language statistics at the level of single
regions.
The levers that have not been tried on natural data are the ones the
synthetic world showed to matter most after segmentation: observation
orbits, which VG lacks because each scene has one photograph, and richer
per-region descriptors. Multi-view natural data -- video frames, product
photographs, or augmentation-derived orbits -- is the next protocol
requirement, and the synthetic result that two-sided multi-view
observation is necessary now has a natural-data counterpart to test.
## Main artifacts
- `artifacts/vg_5k/natural_families{,_dist}.json`
- `artifacts/vg_5k/natural_objects{,_16,_oracle}.pt`
- `artifacts/vg_5k/natural_fields{,_16}.pt`
|
