path: root/NATURAL_RESULTS.md

# Natural data under the Tier 0 recipe

Run date: 2026-08-01

## Question

The synthetic world showed the machinery works end to end when the
representations are good enough: 95% blind recovery, 93% transfer to
scenes the matcher never saw. It could not show that natural data admits
such representations. This ports the recipe to Visual Genome and measures
the one statistic that decides whether to attempt recovery -- the
cross-modal field correlation, which the phase diagram puts at roughly
0.9 for polynomial matching.

## Protocol

Vision: DINOv2 patch features, segments from spectral clustering of the
patch affinity graph with a spatial prior, one state per segment. No
boxes, no labels, no text. Language: word classes and word vectors from
positive-PMI context statistics of the 80,000 region descriptions, one
state per phrase. Scene states are the resulting sets; relations are
moment-kernel similarities. Annotated boxes appear only as a diagnostic
oracle, never in the reported pipeline.

## Text-side factor discovery transfers, but not the heuristic

The synthetic pipeline found factor families by mutual exclusivity, since
templated phrases carry exactly one colour word. Real language refutes
that: colour terms are independent or better, with observed-over-expected
co-occurrence 1.29 for black with white and 0.93 for red with blue, while
head nouns are the genuinely exclusive family at 0.14 for car with truck.
Run unchanged, the method returns clean noun classes and no attributes.

Standard distributional word-class induction, of which exclusivity is a
degenerate case, does transfer. Positive PMI, truncated SVD, and k-means
over the 300 most frequent words give a colour family at 0.67 purity plus
coherent vehicle, animal, clothing, person, and water-scene classes.
Colour terms split into three clusters rather than one, structurally
rather than noisily: white and blue sit with sky and clouds, because what
a colour word modifies shapes its distribution.

## What moves the field correlation

| Configuration | rho |
|---|---:|
| Hard class codes, six segments | 0.166 |
| Hard class codes, sixteen segments | 0.136 |
| Continuous states, unsupervised segments | 0.489 |
| Continuous states, oracle region boxes | 0.550 |
| Continuous states plus content projection, unsupervised | **0.656** |

The reported figure is what `worldalign.natural_pipeline` produces; an
earlier exploratory script reached 0.664 with a wider projection-fitting
range, and the reproducible module is cited in preference.
| Continuous states plus content projection, oracle boxes | 0.671 |

Three findings, in order of how much they overturned expectations.

**Discretisation was the dominant loss.** Quantising segments and phrases
into class codes cost more than half the signal (0.166 against 0.489).
Relation fields need only scene-to-scene similarity, so compressing
states into discrete classes first is pure attrition. The synthetic world
hid this because its states are discrete by construction.

**Segmentation is not the bottleneck here.** Oracle region boxes buy
0.061 over unsupervised spectral segmentation, so the unsupervised
segmenter captures 89% of what perfect boxes would give -- the exact
inverse of the synthetic world, where segmentation was the entire gap.
Flat untextured scenes are adversarial for object discovery in a way that
photographs are not, so the synthetic measurement overstated this
problem.

**Content projection is again the largest lever.** Whitening the
within-scene direction and keeping the eight leading between-scene
directions lifts 0.489 to 0.656, matching its role in the earlier
node-level battery where it took VG from 0.21 to 0.61. The protocol-legal
configuration and the oracle-box configuration land within 0.007 of each
other, which says the remaining deficit is not about where the objects
are.

A control worth recording: frozen general-purpose encoders do worse than
corpus-fitted statistics. DINOv2-small CLS features of isolated region
crops against Qwen mean-pooled phrase states reach only 0.19 in the same
set-moment fields. In-context patch features beat isolated crops, and
PPMI vectors fitted to this corpus beat a general language model's
states, echoing the earlier finding that what the model layer shares is
the mixed state rather than the isolated encoding.

## Augmentation orbits are not a substitute for multiple photographs

The closed world made two-sided multi-view observation a protocol
requirement, and Visual Genome carries one photograph per scene, so the
orbit was synthesised: four random resized crops per image, segmented
independently, their fields averaged. The correlation moves from 0.6559
to 0.6559 -- no gain at four views.

The mechanism is visible in what each orbit cancels. Re-renders in the
closed world resample layout, which is nuisance by construction, so
averaging over them removes exactly the modality-private variation.
Random crops perturb framing, and self-supervised patch features are
already close to invariant under it, so the average has nothing to
cancel. Multi-view observation, where it matters, means separate
photographs of the same scene rather than transformations of one.

## Verdict

**Superseded 2026-08-01 — see `RANK_RESULTS.md`.** The go/no-go statistic
used below was shown not to govern recovery: a synthetic field truncated
to rank 8 correlates at 0.906 and recovers 13%, so 0.9 is neither a
threshold nor a licence. The verdict on natural data survives the
correction — the search was subsequently run and returned 0.0000 against
0.0039 chance — but it is now a measurement rather than an inference, and
the deficit is diagnosed as much in the width of the shared spectrum
(15 directions against 26) as in the correlation. The paragraph below is
kept as written for the record.

The go/no-go statistic stands at 0.656 against the 0.9 that polynomial
recovery needs, so no recovery run is warranted. The trajectory in one
session is 0.166 to 0.656, and the remaining gap is now attributable:
not discretisation, which is removed, and not segmentation, which costs
0.06. What is left is the shared structure between self-supervised visual
features and distributional language statistics at the level of single
regions.

The levers that have not been tried on natural data are the ones the
synthetic world showed to matter most after segmentation: observation
orbits, which VG lacks because each scene has one photograph, and richer
per-region descriptors. Multi-view natural data -- video frames, product
photographs, or augmentation-derived orbits -- is the next protocol
requirement, and the synthetic result that two-sided multi-view
observation is necessary now has a natural-data counterpart to test.

## Main artifacts

- `artifacts/vg_5k/natural_families{,_dist}.json`
- `artifacts/vg_5k/natural_objects{,_16,_oracle}.pt`
- `artifacts/vg_5k/natural_fields{,_16}.pt`

---

# The corpus is capped, and the part correspondence was never the problem

*Appended 2026-08-01.*

Six representational interventions were tried on Visual Genome across one
session — a wider weighted projection, larger text vectors, structured phrase
encoding, more segments, annotated boxes instead of segmentation, third-order
moments in the set kernel, and a backbone at twice the depth. They moved the
field correlation from 0.656 to 0.731 and the cross-modal anchor bound from
0.195 to 0.336. None was chosen against a ceiling, because no ceiling had been
measured. That was the same error being made on the matching side at the same
time: **optimising without a bound.**

Three measurements now bound it, and they disagree with where the effort went.

| Measurement | Value |
|---|---|
| Vision identifies its own scenes (split-half) | **0.994** |
| Text identifies its own scenes (split-half, IDF bag-of-words) | **0.805** |
| Text identifies its own scenes (our encoding pipeline) | 0.491 |
| Cross-modal, unsupervised parts | 0.336 |
| **Cross-modal, annotator-supplied part correspondence** | **0.359** |

Each region box in Visual Genome is index-aligned with its own description, so
the crop inside the box can be encoded and paired with the phrase that
describes it — putting the two modalities' parts in correspondence by
annotation, which is cross-modal supervision a deployed system would never
have. **It buys 0.023.**

That settles the question the pipeline work could not. The two encoders do not
covary at the part level either: a DINOv2 descriptor of a region and an English
phrase describing the same region do not agree about what makes two regions
alike, and no segmentation, aggregation, kernel or backbone repairs that. The
0.805 within-text ceiling was never the cross-modal ceiling; the cross-modal
ceiling for this encoder pair on this corpus is about **0.36**, against the 0.9
that recovery needs.

**Visual Genome is closed.** Not because the method fails but because this
pairing of encoder and corpus does not carry the agreement the method consumes,
and that is now a measurement rather than a suspicion.

What it leaves is a screening protocol rather than a repair list. Three numbers,
all cheap and all computable before any pipeline is built: each modality's
split-half self-identification, the cross-modal anchor bound, and — where the
corpus offers aligned parts — the same bound with part correspondence given.
A corpus whose cross-modal bound sits near its within-modality ceilings is worth
building on; Visual Genome's sits at 45% of the lower of them.