diff options
Diffstat (limited to 'RANK_RESULTS.md')
| -rw-r--r-- | RANK_RESULTS.md | 175 |
1 files changed, 175 insertions, 0 deletions
diff --git a/RANK_RESULTS.md b/RANK_RESULTS.md new file mode 100644 index 0000000..e1c4343 --- /dev/null +++ b/RANK_RESULTS.md @@ -0,0 +1,175 @@ +# The gate was measuring the wrong thing + +*2026-08-01. Supersedes the correlation threshold used as the project's +go/no-go statistic since the synthetic world results.* + +## The correlation does not govern recovery + +The standing rule was that the field correlation at the true pairing has to +reach about 0.9, that nothing works below it, and that any new representation +can therefore be priced in minutes without running a search. Natural data sat +at 0.656 and no recovery run was made on it, on the strength of that rule. + +**A controlled truncation refutes the rule.** Take the synthetic fields that +recover at 95.3%, project both to rank *r*, and match blind from a spectral +start with exact refinement. The correlation barely moves across the ladder; +recovery moves across its entire range. + +| Field | ρ at truth | Shared directions | Recovery (256 scenes, chance 0.4%) | +|---|---|---|---| +| synthetic, rank 4 | 0.902 | 11 | 6.2% | +| synthetic, rank 8 | 0.906 | 13 | 12.9% | +| synthetic, rank 16 | 0.928 | 18 | **95.6%** | +| synthetic, rank 32 | 0.928 | 27 | 93.8% | +| synthetic, rank 64 | 0.928 | 27 | 95.8% | +| synthetic, rank 128 | 0.929 | 27 | 96.1% | +| synthetic, full | 0.929 | 27 | 95.8% | + +A rank-8 field correlating at 0.906 — comfortably past the supposed threshold +— recovers 13%. The same field at rank 16 recovers 96%. Correlation is held +almost fixed while recovery crosses from failure to success, so the variable +that moves recovery is the width of the shared spectrum, not the strength of +the agreement. + +The theory says the same thing in advance and we misread it. Correlated-matching +thresholds are derived for exchangeable full-rank noise, where each of the +N²/2 entries carries an independent constraint on the pairing. A rank-*r* +shared component supplies about *rN*. At 256 scenes with *r* near ten that is +an eighth of what the threshold calculation assumes, so quoting the 0.9 figure +against a low-rank field compares a number to a bound derived for a different +object. + +The correction cuts against us as well as for us. With the old gate retired, +the reason for never having run a search on natural data went with it, so the +search was run: five blind trials on the best natural field, spectral start +chosen by energy over five candidates, exact refinement to convergence. +**Recovery is 0.0000 against a chance rate of 0.0039** — the spectral start +lands exactly on chance and refinement moves away from the truth, which is what +descent does when the truth is not the optimum. Natural data does not recover, +and now that is a measurement rather than an inference from a statistic that +does not govern it. + +## What the shared spectrum measures + +Each field is eigendecomposed and the principal angles between the two leading +32-dimensional eigenspaces are computed; directions with cosine above 0.7 count +as shared. A scene-shuffled null puts the chance count at 1.0 in every +condition below, so the counts are not an artefact of subspace dimension. + +| Field | ρ | vision eff. rank | text eff. rank | shared (net of null) | +|---|---|---|---|---| +| synthetic watershed (recovers 95.3%) | 0.928 | 14.5 | 11.8 | **26** | +| synthetic v0 (fails, 5.1%) | 0.508 | 31.3 | 27.9 | 18 | +| Visual Genome, previous best | 0.656 | 18.7 | 13.9 | 10 | +| Visual Genome, best today | 0.716 | 39.7 | 48.0 | 15 | + +The failing synthetic field settles the sufficiency question in the other +direction: 18 shared directions with a correlation of 0.508 also fails. +**Recovery needs both a wide shared spectrum and a strong correlation, and +neither alone predicts it.** The gate becomes a joint condition, and the honest +statement of our position is that natural data is short on both — though far +closer on width than the correlation gap alone suggested. + +## Natural data is narrow in the intersection, not in either modality + +The diagnosis is sharper than "the features are not good enough". Each side is +individually rich — vision effective rank 40, text 48 — while the two agree on +15 directions. **Each modality is rich about something, and they are rich about +different things.** + +That intersection is close to invariant under everything tunable on the vision +side, all measured at matched width: + +| Vision configuration | ρ | shared | +|---|---|---| +| 6 spectral segments | 0.692 | 15 | +| 16 spectral segments | 0.679 | 14 | +| annotated region boxes (oracle) | 0.703 | 15 | + +Handing the pipeline ground-truth boxes buys one direction over unsupervised +segmentation and nothing over using fewer segments. This confirms from a new +angle what the earlier oracle-box comparison found: **segmentation is not the +constraint on photographs.** + +The text side does move it, and saturates: + +| Text vector dimension | text eff. rank | ρ | shared | +|---|---|---|---| +| 24 | 19.0 | 0.661 | 10 | +| 48 | 31.8 | 0.692 | 15 | +| 128 | 48.0 | **0.716** | 16 | +| 256 | 53.0 | 0.700 | 17 | + +Tripling the text representation's own rank from 19 to 53 buys seven shared +directions and then stops. Enlarging the scene population does not help either +— at matched width the shared count goes 23, 20, 19 for 256, 512 and 1024 +scenes, so the ceiling is a property of what the two corpora are about rather +than of how many scenes are sampled. + +## Two smaller results, one refuted hypothesis + +**The projection improved.** Scaling each discriminant direction by its own +eigenvalue to the power 0.5 lets a wide basis be kept without the weak +directions drowning the strong ones. It dominates the unweighted projection at +every setting tested and, unlike the unweighted version, improves with width — +which is the direction that raises rank. With text vectors at 128 dimensions +this takes the field correlation from 0.656 to **0.716** and the shared count +from 10 to 16, at no cost. + +**Hubness was the wrong suspect.** The hypothesis was that the correlation is +inflated by a shared "this scene resembles everything" component carrying no +matching information. It is not: the additive row-and-column model accounts for +0.9% of the visual field's variance and 4.9% of the text field's, and removing +it *raises* the correlation slightly, from 0.656 to 0.671. The headline +statistic was honest; it was simply not the statistic that governs recovery. + +## Shrinking the problem is dead, and now by measurement + +Small blocks were the last standing non-correlation route, on the reasoning +that the hard phase is joint in correlation and population size. The reasoning +had the sign backwards. The information-theoretic threshold is +ρ_IT ≈ √(4 log N / N), which **rises** as N falls: 0.29 at 256 scenes, 0.51 at +64, 0.83 at 16. Shrinking the population raises the bar. + +The project's own gate confirms it directly. At every size tested, the best +state a strong searcher reaches — spectral start plus 200 random restarts, each +run to a local optimum under exact steepest descent — is **deeper than the +truth**, in three replicates out of three: + +| N | ρ | E(truth) | E(best found) | accuracy of best | truth deepest? | +|---|---|---|---|---|---| +| 16 | 0.705 | 2.95 | 2.37 | 0.271 | 0/3 | +| 24 | 0.703 | 3.24 | 2.29 | 0.194 | 0/3 | +| 32 | 0.679 | 3.69 | 2.38 | 0.073 | 0/3 | +| 48 | 0.706 | 3.48 | 2.65 | 0.125 | 0/3 | +| 64 | 0.700 | 3.24 | 2.15 | 0.151 | 0/3 | +| 96 | 0.698 | 3.65 | 2.57 | 0.115 | 0/3 | + +When the truth is not the optimum, no searcher of any cost finds it, and the +question of algorithm class does not arise. Accepting exponential cost buys +nothing here — this is the same conclusion tempering reached by equilibrating +below the truth, now established at sizes small enough that search cannot be +blamed. + +Combined with the earlier rank-ladder result — at natural field quality every +truncation from rank 4 to full returns chance — three of the five routes in the +candidate register are closed: reshaping the landscape, searching harder, and +shrinking the problem. Raising the shared spectrum and adding orthogonal signal +are what remain. + +## What this changes + +The target moves from a scalar to a pair. Raising the correlation from 0.656 to +0.716 was worth having, but the measurement that matters is that it came with +the shared count going from 10 to 16, and 16 is inside the interval where the +synthetic ladder crosses from 13% to 96%. + +The corpus, not the encoder, is where the width comes from. The synthetic world +has 26 shared directions because its captions state exactly the world state; +Visual Genome has 15 because a region description and a patch descriptor +overlap on roughly that many aspects of a photograph, and no amount of +segmentation quality, backbone capacity, or scene count moves it. That reframes +the next step as a question about which corpora have naturally wide overlap — +dense descriptions, product listings with photographs and full specifications, +screenshots paired with their accessibility trees — rather than a question +about better features for this one. |