1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
|
# Visual Genome hidden-permutation protocol
## Purpose
The Flickr30k experiment compared two independent samples from a common
distribution. It did not present the alignment method with two empirical
graphs over the same latent node set.
This protocol instead constructs two independently indexed observations of the
same set of real scenes:
\[
\mathcal D_V=\{V_i\}_{i=1}^N,\qquad
\mathcal D_T=\{T_{\pi(i)}\}_{i=1}^N,
\]
where `pi` is sampled once, kept out of all training inputs, and retained only
in a private evaluation file.
## Closure tiers
The experiment treats linguistic information outside the image-observable
sigma-algebra as a controlled nuisance variable.
1. `region_closed`: a fixed-size bag of localized image crops and their human
region descriptions. Region order and scene identity are independently
permuted on the two sides.
2. `visible_relations`: region descriptions plus object attributes and
subject-predicate-object statements that are explicitly visible.
3. `qa_expanded`: the visible bundle plus Visual Genome/GQA questions and
answers. This may introduce inference and annotation priors.
4. `open_context`: later experiments add page-level or encyclopedic text. This
tier is intentionally excluded from the initial identifiability test.
The core claim must work at tier 1 before higher tiers are useful.
## Leakage controls
- Every scene receives the same number of visual and textual views.
- Opaque visual and text node IDs are sampled independently.
- Text records do not contain image IDs, URLs, coordinates, or region IDs.
- Vision records do not contain phrases, object names, predicates, answers, or
WordNet identifiers.
- Text order is independently permuted within every bundle.
- Scaling subsets are nested but independently shuffled on the two sides.
- The ground-truth permutation is stored separately and is loaded only by
evaluation commands.
## Representations
The first pass uses frozen, unimodal backbones:
- DINOv2 features for a full image and a fixed bag of localized crops;
- multiple Qwen hidden-state views for the corresponding descriptions;
- no CLIP/SigLIP or VLM-pretrained vision tower.
Per-scene inputs remain sets rather than being averaged immediately. The
alignment objective can therefore use:
- the within-scene region relation graph;
- the between-scene graph;
- multi-scale diffusion and return-probability signatures;
- higher-order or operator constraints.
## Scaling axes
\[
N\in\{5k,20k,50k,100k\}
\]
\[
\text{views per scene}\in\{1,4,16,32\}
\]
\[
\text{closure tier}\in
\{\text{region-closed},\text{visible-relations},\text{QA-expanded}\}.
\]
The primary diagnostic is the rank of the true hidden match as these three
axes grow. Frozen-LLM bridge training begins only after the structural rank
shows a reproducible scaling trend.
|
