diff options
Diffstat (limited to 'RANK_RESULTS.md')
| -rw-r--r-- | RANK_RESULTS.md | 91 |
1 files changed, 78 insertions, 13 deletions
diff --git a/RANK_RESULTS.md b/RANK_RESULTS.md index adaee21..d520e5e 100644 --- a/RANK_RESULTS.md +++ b/RANK_RESULTS.md @@ -13,7 +13,10 @@ at 0.656 and no recovery run was made on it, on the strength of that rule. **A controlled truncation refutes the rule.** Take the synthetic fields that recover at 95.3%, project both to rank *r*, and match blind from a spectral start with exact refinement. The correlation barely moves across the ladder; -recovery moves across its entire range. +recovery moves across its entire range. (What replaces the correlation is +*not* the shared-direction count in the next column — see the correction that +follows this section. The column is reported because it is what motivated the +hypothesis, not because it survived it.) | Field | ρ at truth | Shared directions | Recovery (256 scenes, chance 0.4%) | |---|---|---|---| @@ -27,9 +30,10 @@ recovery moves across its entire range. A rank-8 field correlating at 0.906 — comfortably past the supposed threshold — recovers 13%. The same field at rank 16 recovers 96%. Correlation is held -almost fixed while recovery crosses from failure to success, so the variable -that moves recovery is the width of the shared spectrum, not the strength of -the agreement. +almost fixed while recovery crosses from failure to success, so **something +other than the strength of the agreement is doing the work.** Identifying that +something is a separate problem, and the next section records two attempts at +it that both failed. The theory says the same thing in advance and we misread it. Correlated-matching thresholds are derived for exchangeable full-rank noise, where each of the @@ -49,6 +53,55 @@ descent does when the truth is not the optimum. Natural data does not recover, and now that is a measurement rather than an inference from a statistic that does not govern it. +## Correction, same day: width is not the statistic either + +The section below proposed the width of the shared spectrum as the replacement +gate. **A control run later the same session refutes it**, and the refutation +is recorded here rather than folded away because it was my own claim and it +lasted four hours. + +Adding independent noise to both synthetic fields lowers the correlation +without narrowing the shared signal underneath, which is still full-rank. That +control was run to guard against overcorrecting, and it did more than that: + +| Field | ρ | shared width | shared dimension (held out) | recovery | +|---|---|---|---|---| +| synthetic, noise 0.35 | 0.828 | 15.7 | 16.8 | **94.7%** | +| synthetic, caption omits size | 0.830 | 15.0 | 16.6 | **5.6%** | +| Visual Genome, best | 0.731 | 15.0 | 17.2 | **0.0%** | +| synthetic, noise 0.50 | 0.743 | 13.3 | — | 70.7% | + +Three fields agree on the correlation to within 0.1, agree on both width +measures to within 1.5, and differ in recovery across the entire range. The +principal-angle count fails because noise rotates eigenvectors and lowers the +measured overlap without narrowing anything. A second instrument built +specifically to fix that — canonical correlations fitted on one third of the +scenes and scored on another, so a direction counts only if it generalises — +fails the same way, and rates Visual Genome *highest* of the three failures. + +So **two candidate statistics have now been proposed and refuted in one +session**, and the honest position is narrower than the one I wrote four hours +ago: + +- **Established.** The correlation does not govern recovery. Rank-8 truncation + holds it at 0.906 and drops recovery to 12.9%; independent noise drops it to + 0.743 and keeps recovery at 70.7%. Both directions are controlled, and + together they retire the 0.9 rule for good. +- **Established.** Genuine rank truncation below about 8 destroys recovery, and + suppressing a caption factor destroys it while the correlation stays at 0.83. +- **Not established.** That any single width or dimension statistic predicts + recovery. Neither of the two tried does. + +What separates the noise case from the omission case is a live question with a +specific candidate. Suppressing a factor makes scenes that differed only in +that factor *exactly* interchangeable in the text field, so the energy acquires +an exact symmetry and the truth stops being a unique minimum — it becomes one +member of a degenerate orbit. Independent noise creates no such ties. That +predicts a three-way split which the gate can read directly: the truth deeper +than everything found (recoverable), tied with what is found (symmetry), or +shallower than what is found (information deficit). The gate is running on both +fields; the sections below should be read against this correction. + ## What the shared spectrum measures Each field is eigendecomposed and the principal angles between the two leading @@ -66,9 +119,9 @@ condition below, so the counts are not an artefact of subspace dimension. The failing synthetic field settles the sufficiency question in the other direction: 18 shared directions with a correlation of 0.508 also fails. **Recovery needs both a wide shared spectrum and a strong correlation, and -neither alone predicts it.** The gate becomes a joint condition, and the honest -statement of our position is that natural data is short on both — though far -closer on width than the correlation gap alone suggested. +neither alone predicts it.** *(Superseded by the correction above: the pair +does not predict it either. The counts below remain as measured; what they do +not support is the inference that they govern recovery.)* ## Natural data is narrow in the intersection, not in either modality @@ -147,12 +200,24 @@ photographs — and recovery is at chance. **The synthetic world degraded to Visual Genome's shared width fails exactly the way Visual Genome fails**, at a correlation Visual Genome never reaches. -This closes the causal chain. How much of the same world the two corpora -describe sets the width of the shared spectrum; the width sets recovery; and -the correlation reports on neither reliably. It also reframes the natural-data -deficit as a statement about Visual Genome rather than about our pipeline: a -region description simply does not say as much about a photograph as the -synthetic captions say about their scenes. +The full ladder is monotone in how much the caption states: + +| Caption content | ρ | shared | recovery | +|---|---|---|---| +| states everything | 0.928 | 20 | 96.4% | +| never states size | 0.830 | 15 | 5.6% | +| never states count | 0.678 | 13 | 1.3% | +| states neither size nor count | 0.544 | 10 | 1.3% | +| never states colour | 0.736 | 9 | 0.0% | + +What this establishes is that **corpus overlap controls recovery**, which is +the claim that matters for what to do next. What it does not establish is the +intermediate step — that it does so *through* the width, since the noise +control reaches the same width with recovery intact. The mechanism is under +test; the corpus-level conclusion does not depend on which way that test goes, +and it reframes the natural-data deficit as a statement about Visual Genome +rather than about our pipeline: a region description simply does not say as +much about a photograph as the synthetic captions say about their scenes. (Widths in this table are computed on standardised fields and are not comparable entry-for-entry with the tables above, which use raw fields; within |