From a62cf4d2a99b4a7985c61b2a7feb92a82a8218b7 Mon Sep 17 00:00:00 2001
From: Yuren Hao <blackhao0426@gmail.com>
Date: Sat, 1 Aug 2026 14:10:03 -0500
Subject: World Alignment: unpaired cross-modal correspondence by relational
 identifiability

Method: scene states are sets of part states; relation fields are built
within each modality and are invariant to how each side labels its own
features; the cross-modal bridge is a coupling searched under an energy
that is a closed-form functional of one matrix; solving is spectral
initialisation followed by exact local refinement.

Evidence: in a procedurally generated closed world, blind recovery of a
hidden image-caption correspondence reaches 95.3% at 256 scenes against
0.39% chance, and the recovered pairs transfer to 200 held-out scenes at
93.0% exact retrieval with random-pair and shuffled-image controls at or
near chance. Cross-modal value correspondence is derived from disjoint
corpora rather than declared. On Visual Genome the field correlation
reaches 0.656 against the 0.9 that polynomial recovery needs, with the
deficit attributed away from segmentation and discretisation.

Protocol: no image-text pair enters any objective, optimiser,
initialisation, or model selection; hidden pairs score orderings only.

Co-Authored-By: Claude <noreply@anthropic.com>
---
 VG_PROTOCOL.md | 83 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 83 insertions(+)
 create mode 100644 VG_PROTOCOL.md

(limited to 'VG_PROTOCOL.md')

diff --git a/VG_PROTOCOL.md b/VG_PROTOCOL.md
new file mode 100644
index 0000000..aed6ced
--- /dev/null
+++ b/VG_PROTOCOL.md
@@ -0,0 +1,83 @@
+# Visual Genome hidden-permutation protocol
+
+## Purpose
+
+The Flickr30k experiment compared two independent samples from a common
+distribution. It did not present the alignment method with two empirical
+graphs over the same latent node set.
+
+This protocol instead constructs two independently indexed observations of the
+same set of real scenes:
+
+\[
+\mathcal D_V=\{V_i\}_{i=1}^N,\qquad
+\mathcal D_T=\{T_{\pi(i)}\}_{i=1}^N,
+\]
+
+where `pi` is sampled once, kept out of all training inputs, and retained only
+in a private evaluation file.
+
+## Closure tiers
+
+The experiment treats linguistic information outside the image-observable
+sigma-algebra as a controlled nuisance variable.
+
+1. `region_closed`: a fixed-size bag of localized image crops and their human
+   region descriptions. Region order and scene identity are independently
+   permuted on the two sides.
+2. `visible_relations`: region descriptions plus object attributes and
+   subject-predicate-object statements that are explicitly visible.
+3. `qa_expanded`: the visible bundle plus Visual Genome/GQA questions and
+   answers. This may introduce inference and annotation priors.
+4. `open_context`: later experiments add page-level or encyclopedic text. This
+   tier is intentionally excluded from the initial identifiability test.
+
+The core claim must work at tier 1 before higher tiers are useful.
+
+## Leakage controls
+
+- Every scene receives the same number of visual and textual views.
+- Opaque visual and text node IDs are sampled independently.
+- Text records do not contain image IDs, URLs, coordinates, or region IDs.
+- Vision records do not contain phrases, object names, predicates, answers, or
+  WordNet identifiers.
+- Text order is independently permuted within every bundle.
+- Scaling subsets are nested but independently shuffled on the two sides.
+- The ground-truth permutation is stored separately and is loaded only by
+  evaluation commands.
+
+## Representations
+
+The first pass uses frozen, unimodal backbones:
+
+- DINOv2 features for a full image and a fixed bag of localized crops;
+- multiple Qwen hidden-state views for the corresponding descriptions;
+- no CLIP/SigLIP or VLM-pretrained vision tower.
+
+Per-scene inputs remain sets rather than being averaged immediately. The
+alignment objective can therefore use:
+
+- the within-scene region relation graph;
+- the between-scene graph;
+- multi-scale diffusion and return-probability signatures;
+- higher-order or operator constraints.
+
+## Scaling axes
+
+\[
+N\in\{5k,20k,50k,100k\}
+\]
+
+\[
+\text{views per scene}\in\{1,4,16,32\}
+\]
+
+\[
+\text{closure tier}\in
+\{\text{region-closed},\text{visible-relations},\text{QA-expanded}\}.
+\]
+
+The primary diagnostic is the rank of the true hidden match as these three
+axes grow. Frozen-LLM bridge training begins only after the structural rank
+shows a reproducible scaling trend.
+
-- 
cgit v1.2.3