diff options
Diffstat (limited to 'MANIFOLD_RESULTS.md')
| -rw-r--r-- | MANIFOLD_RESULTS.md | 244 |
1 files changed, 244 insertions, 0 deletions
diff --git a/MANIFOLD_RESULTS.md b/MANIFOLD_RESULTS.md new file mode 100644 index 0000000..4ca0098 --- /dev/null +++ b/MANIFOLD_RESULTS.md @@ -0,0 +1,244 @@ +# On-manifold assignment gate + +Run date: 2026-07-29 + +## Question + +The free-particle energy was falsified off-manifold: optimization left the +decodable language region and produced counterfeit states below the true +configuration. This experiment removes that escape route by construction. +Configurations are permutations of real frozen text states, so the +language-only terms of the falsified energy (sliced distribution, prototype +manifold) are constants, and the only varying terms are the cross-modal +relation MSE and conditional KL. If the earlier failure was caused by +leaving the manifold, the true assignment should now be an energy minimum. + +It is not. The falsification survives the restriction, and the gate +localizes the cause in the states rather than the search space. + +## Protocol + +- Configuration space: permutations of held-out real text states. Hidden + pairs place the true assignment in the ranking and are never visible to + the energy. +- Gate A, global ordering: energy of the true assignment against 1,000 + uniform permutations and a similarity-ordered cyclic shift. +- Gate B, local ordering: the exact energy change of every transposition of + the true assignment. The quadratic text terms cancel, so one matrix + product evaluates all N(N-1)/2 swaps; the closed form is unit-tested + against brute-force recomputation. +- Gate C, basin audit: exact steepest 2-swap descent from the true + assignment and from random assignments, with a local-minimum certificate. + Random-start descent doubles as a search for on-manifold counterfeits. +- Conditions: Flickr30k test/val (DINOv2-small CLS against Qwen2.5-0.5B + mean-pooled states, single caption and five-caption orbit means) and + Visual Genome 512-node subsets and the full 5,000 nodes (16-view means, + `region_closed` and `visible_relations` tiers, scalar and + distribution-valued relation channels). + +## The ordering passes globally and fails locally + +Every condition separates the true assignment from random permutations; +no condition makes it a local minimum. + +| Condition | Random-perm z | Improving swaps | Kept by descent | E true | E counterfeit | +|---|---:|---:|---:|---:|---:| +| Flickr test 512, orbit | 18.7 | 7.4% | 13.9% | 4.17 | 3.26 | +| Flickr test 512, single | 10.8 | 15.6% | 6.6% | 4.67 | 3.44 | +| Flickr val 512, orbit | 18.0 | 6.8% | 11.9% | 4.10 | 3.26 | +| Flickr test 1,000, orbit | 25.3 | 7.3% | 11.0% | 4.30 | 3.42 | +| VG 512, region_closed | 4.9 | 23.5% | 2.7% | 4.58 | 2.73 | +| VG 512, visible_relations | 9.2 | 17.9% | 3.7% | 4.14 | 2.27 | +| VG 512, region bundle | 2.7 | 33.2% | 4.5% | 4.86 | 3.22 | +| VG 5,000, region_closed | 18.3 | 21.6% | (descent disabled) | 4.69 | -- | +| VG 5,000, visible_relations | 28.2 | 17.7% | (descent disabled) | 4.39 | -- | + +"Improving swaps" is the fraction of all transpositions of the true +assignment that lower the energy: 9,723 of 130,816 in the best condition, +2,697,600 of 12,497,500 at N=5,000. Scale does not repair local ordering. + +VG rows show subset seed 0; seeds 1 and 2 replicate every verdict. Across +the twelve VG subset runs (3 seeds x 2 tiers x 2 channel modes), improving +swaps span 15.7--33.2%, descent retention 2.7--22.5%, and a counterfeit +below the true energy by 1.6--2.2 is found in 12 of 12 runs. + +The derangement curve shows where the energy has resolution. On Flickr +orbit, permutations that displace 32 or more nodes are detected in 200 of +200 samples; transpositions are missed in 9% of samples. On VG the same +inversion holds (k=2: 18%, k=128: 0%). **The energy resolves coarse +rearrangements and is blind to fine ones.** + +## Descent manufactures counterfeits from every start + +Steepest descent from the true assignment walks away from it in every +condition, keeping 2.7--13.9% of nodes. Quenches from random assignments +reach energies 0.9--1.9 below the true configuration at 0.0--0.4% accuracy. +The counterfeit family is not an artifact of continuous relaxation; it is a +property of the assignment landscape itself: + +E(true) > E(wrong local minimum), for every observed minimum. + +Five-caption orbits halve the improving-swap fraction (15.6% to 7.4%) and +double descent retention (6.6% to 13.9%); explicitly visible relations cut +it by a quarter (23.5% to 17.9%). Both help, neither changes the verdict. + +## The mechanism is spectral: about sixteen shared dimensions + +Three measurements locate the failure in the node states. + +First, the pooled text states are extremely anisotropic: 99.5% of all +Flickr caption pairs have cosine above 0.8, and every VG +`visible_relations` pair is above 0.9. Raw cosine carries almost no +instance identity; the energy operates on standardized relation values, +whose cross-modal Spearman correlation is 0.19 (single caption), 0.31 +(orbit mean), 0.21 (`region_closed`), 0.30 (`visible_relations`). Roughly +90% of fine relational variance is modality-private. + +Second, the shared part is spectrally concentrated. Projecting both +modalities onto their top population principal components before computing +relations gives: + +| Kept dimensions | Cross-modal Spearman | +|---:|---:| +| 2 | 0.315 | +| 16 | 0.367 | +| 128 | 0.322 | +| 384 | 0.315 | +| untruncated | 0.306 | + +Two directions already carry more cross-modal relational signal than the +full representation; the optimum near sixteen dimensions gains 0.06 over +full; every dimension past that dilutes. Population whitening, which +equalizes the spectrum, collapses the correlation to 0.03. **The shared +world signal lives in a roughly sixteen-dimensional coarse subspace, and +the fine structure that identification needs is modality-private.** + +Third, the worst improving swaps are pairs whose text states are nearly +parallel while their scenes are unrelated: text cosine 0.88--0.94 against +visual cosine -0.13--0.15. Example pair at text cosine 0.897: "A baseball +player in red leaps into the air..." against "A woman in a blue shirt is +walking across the street...". A sixteen-dimensional topical code cannot +separate 512 instances, and no functional of these relation fields can. + +## Higher moments and view bundles buy nothing at node level + +The distribution-valued relation channels (std, q10, q90 of the 256 +view-pair cosines between two VG nodes) were a direct test of richer edge +structure. Their cross-modal Spearman is -0.02, 0.05, and 0.11--0.16 +respectively, against 0.21--0.30 for the mean channel, and they are +strongly entangled with it within each modality (rank correlation -0.82 to +0.63). Adding them raises the improving-swap fraction from 23.5% to 33.2%: +channels without shared signal act as noise. Model scale was already known +to be inert here (Qwen 0.5B to 1.5B left relational diagnostics unchanged), +and whitening is destructive, so the node-state toolbox is exhausted: +every cheap transform either denoises within the sixteen shared dimensions +or dilutes them. None expands them. + +## Region-level structure inside nodes is shared + +The constructive result. Both VG modalities observe the same sixteen +regions per scene; the preparation RNG that hid the view correspondence is +replayable from the seed and the private image IDs, and the replay +reconstructs all 5,000 released phrase bundles exactly, so view-level truth +is available for evaluation. + +On all 5,000 nodes: + +- Within-node relation fields align: Spearman between the visual and + truth-aligned text view-relation fields is 0.128 against -0.002 for + shuffled views, z = 46.6 across nodes. +- Views become matchable through the coarse frame: ranking each view's + relation profile against a 1,000-node population frame and solving the + 16-view assignment gives 14.5% accuracy against 6.25% chance (2.3x; + top-3 29.5% against 18.8%). Growing the frame from 200 to 1,000 nodes + raises accuracy from 12.9% to 14.5%. This uses the true node alignment + as the frame, so it is an upper bound on what a recovered coarse + alignment would enable. +- Within-node relations alone reach 7.0% on a 200-node sample: the + population context, not the 16-view geometry, carries the signal. + +The fine-grained shared structure the node-level energy lacks does exist +one level down, in the compositional structure the pooled states discard. + +## Ricci flow doubles the frame and fails the resolution + +A discrete Ollivier-Ricci flow was run as a control for the hypothesis +that smoothing each modality's relational geometry could repair local +ordering. Each modality's kNN graph evolved independently for ten +iterations; identical heat-kernel channels were gated with and without the +flow, so the flow's own effect is isolated. + +| Condition | Random-perm z | Improving swaps | Kept by descent | E true | E counterfeit | +|---|---:|---:|---:|---:|---:| +| Flickr orbit 512, diffusion only | 21.4 | 15.2% | 4.1% | 1.263 | 0.717 | +| Flickr orbit 512, Ricci flow | 47.8 | 9.1% | 5.1% | 0.818 | 0.427 | +| VG 512 region, diffusion only | 6.9 | 22.9% | 3.1% | 1.601 | 0.648 | +| VG 512 region, Ricci flow | 22.5 | 13.9% | 1.2% | 1.405 | 0.650 | + +The flow is a genuinely good coarse-channel denoiser: it more than +doubles the global separation in both datasets. It changes no verdict: +9--14% of transpositions still improve the flowed energy, descent keeps +1--5% of the truth, and every random quench finds a counterfeit at about +half the true energy. **Better geometry buys a better frame, not a better +resolver** -- the two roles dissociate exactly as the spectral audit +predicted, so geometric flows belong in the coarse-frame layer of the next +design and cannot substitute for finer states. + +## Static view-level energies fail the same gate + +The view-level continuation gates the structured states directly. Given +the true coarse frame (upper-bound scaffolding), the true view assignment +is a strict local minimum for none of 5,000 nodes: the within-node +relational energy has 32.0% improving view-transpositions, the +population-profile energy 28.3%, and their best combination 25.5%. +Frame-free matching of internal relation fields (signature-initialized +Hungarian plus exact batched 2-swap descent, with a value-scrambled +capacity null and column centering) is dead at 512 nodes across three +seeds: R@10 2.0--3.3% against 2.0% chance, Hungarian recovery at most +0.6%, top-margin-decile precision 0 of 51, and the true-pair cost +separation is z = 0.1--1.8. A small negative separation seen at the +96-node smoke scale did not replicate and is recorded as noise. + +Static within-node relation fields therefore carry shared signal (the +z = 46.6 alignment result) but not enough for identification even at +sixteen candidates -- the view-level gate needs model-internal relations +(joint-context attention) or true re-encoding interventions, which is the +R3/R8 battery of `STRUCTURE_DESIGN.md`. + +## Conclusion + +The manifold restriction was necessary and is now insufficient for a +different reason than the free-particle failure. The energy is not being +cheated by off-manifold vectors; it is blind at the resolution where +instance identity lives, because pooled node states expose only a coarse +shared subspace. No matcher, objective reweighting, or scale change fixes +an information ceiling in the states. + +The gate for the next experiment therefore moves down one level: + +1. a node state is a set of region views with its internal relation field, + not a pooled vector; +2. the candidate energy scores view-level assignments conditioned on a + coarse frame, and must first pass this same transposition/descent gate + at the view level, with the replayed view truth as hidden evaluation; +3. response operators under controlled interventions (view removal, + attribute edits) are the candidate source of shared fine structure + beyond static view relations, to be tested through the identical gate + before any descent run. + +## Main artifacts + +- `artifacts/manifold_gate/flickr_test512_{orbit,single}.json` +- `artifacts/manifold_gate/flickr_val512_orbit.json` +- `artifacts/manifold_gate/flickr_test1000_orbit.json` +- `artifacts/manifold_gate/vg512_{region_closed,visible_relations}_seed{0,1,2}_{scalar,bundle}.json` +- `artifacts/manifold_gate/vg5000_{region_closed,visible_relations}_scalar.json` +- `artifacts/manifold_gate/resolution_audit.json` +- `artifacts/manifold_gate/spectral_localization.json` +- `artifacts/manifold_gate/bundle_channel_audit.json` +- `artifacts/manifold_gate/vg_view_probe_smoke.json` +- `artifacts/manifold_gate/vg_view_probe_full.json` +- `artifacts/manifold_gate/ricci_flickr_test512_orbit.json` +- `artifacts/manifold_gate/ricci_vg512_region_seed0.json` +- `artifacts/manifold_gate/view_gate_full.json` +- `artifacts/manifold_gate/view_gate_cost_seed{0,1,2}.pt` |