diff options
| -rw-r--r-- | CANDIDATE_REGISTER.md | 26 | ||||
| -rw-r--r-- | EXTENDED_ABSTRACT.md | 56 | ||||
| -rw-r--r-- | RANK_RESULTS.md | 91 | ||||
| -rw-r--r-- | artifacts/synth_v1/omit_ladder_rest.json | 28 | ||||
| -rw-r--r-- | artifacts/vg_5k/shared_dimension.json | 92 | ||||
| -rw-r--r-- | logs/gate_noisy.log | 0 | ||||
| -rw-r--r-- | logs/gate_omit.log | 0 | ||||
| -rw-r--r-- | logs/omit_rest.log | 5 | ||||
| -rw-r--r-- | logs/rho_full_width.log | 2 | ||||
| -rw-r--r-- | worldalign/shared_dimension.py | 174 |
10 files changed, 429 insertions, 45 deletions
diff --git a/CANDIDATE_REGISTER.md b/CANDIDATE_REGISTER.md index e4aa235..bcfa3a0 100644 --- a/CANDIDATE_REGISTER.md +++ b/CANDIDATE_REGISTER.md @@ -5,24 +5,28 @@ The register used to be ordered by how much each candidate was expected to raise the field correlation, because correlation was the go/no-go statistic. That statistic has been shown not to govern recovery: a synthetic field -truncated to rank 8 correlates at 0.906 and recovers 13%, while the same field -at rank 16 recovers 96%. **The target is now a pair — the correlation and the -width of the shared spectrum — and candidates are ordered by which one they -move.** +truncated to rank 8 correlates at 0.906 and recovers 13%, while independent +noise that drops the correlation to 0.743 leaves recovery at 71%. Two +replacement statistics were then proposed and both failed their own controls +(see `RANK_RESULTS.md`), so **there is currently no cheap predictor and +candidates have to be scored by running the search.** Rows below keep their +width measurements because they were taken, not because they predict anything. Status is one of: **done** (measured, number recorded), **running**, **queued** (specified, not started), **open** (idea, not specified), **closed** (measured and eliminated). -Current position: correlation 0.731, shared directions 15, against a -recovering synthetic reference at 0.928 and 26. +Current position: correlation 0.731 and recovery 0.0000 against 0.0039 +chance, versus a synthetic reference at 0.928 and 95.8%. -## A. Widen the shared spectrum +## A. Widen what the two corpora share -The binding constraint, and the one only identified today. Both modalities are -individually rich — vision effective rank 40, text 48 — and agree on 15 -directions. Candidates here are judged on the shared count, not the -correlation. +The live constraint, though the quantity that expresses it is not yet +identified. What is established by intervention: suppressing one factor from +the synthetic captions takes recovery from 96.4% to 5.6% while the correlation +stays at 0.830, and five separate encoder-side interventions move neither +recovery nor any width measure. Candidates here are judged by whether they +change what the two corpora are about. | Candidate | Cost | Status | |---|---|---| diff --git a/EXTENDED_ABSTRACT.md b/EXTENDED_ABSTRACT.md index 1e96cae..7dec8f8 100644 --- a/EXTENDED_ABSTRACT.md +++ b/EXTENDED_ABSTRACT.md @@ -86,11 +86,18 @@ The theory said so in advance and we misread it. Matching thresholds are derived for exchangeable full-rank noise, where each of the N²/2 field entries constrains the pairing independently; a rank-*r* shared component supplies about *rN*, an eighth as many at 256 scenes with *r* near ten. -The corrected gate is a joint condition on the correlation and on the -width of the shared spectrum — the number of directions along which the -two modalities agree, measured by principal angles against a -scene-shuffled null. Neither alone suffices: 18 shared directions at -correlation 0.508 fails, and 11 at 0.902 fails. + +What replaces the correlation is an open question, and we report two +failed answers rather than one plausible one. The obvious candidate is +the width of the shared spectrum, and it does not survive its own +control: independent noise that lowers the correlation to 0.828 without +narrowing the underlying signal reaches the same measured width as a +field whose captions omit one factor — 15.7 against 15.0 — with recovery +at 94.7% and 5.6% respectively. A second instrument built to fix that, +counting canonical directions that generalise to held-out scenes, rates +the three failing fields at 16.6 to 17.2 and ranks the worst of them +highest. **What is established is that the correlation is not the +governing statistic; what is not established is what is.** ## A closed world settles the mechanism @@ -132,9 +139,9 @@ frequency rankings diverge in the tail. Ported to Visual Genome under the same recipe — DINOv2 patch features, spectral segmentation, distributional word statistics — the field -correlation reaches 0.716 and the shared spectrum 15 directions, against -0.928 and 26 for the synthetic world that recovers. The trajectory that -got there is more informative than either number. +correlation reaches 0.731 against 0.928 for the synthetic world, and +blind search returns 0.0000 against a chance rate of 0.0039. The +trajectory that got there is more informative than the number. Discretisation cost more than half the signal: hard class codes give 0.166 where continuous states give 0.489, because fields need only @@ -149,19 +156,26 @@ underperform corpus-fitted statistics, 0.19 against 0.656, so in-context features beat isolated crops and corpus-specific vectors beat a language model's pooled states. -The width diagnosis localises the remaining gap somewhere the correlation -alone never pointed. Both modalities are individually rich on -photographs — vision effective rank 40, text 48 — and they agree on 15 -directions, so **each side is rich about something and they are rich -about different things.** That intersection barely moves under anything -tunable on the vision side: near-tripling the segments costs one shared -direction, and handing the pipeline annotated boxes instead of -unsupervised segmentation buys one. The text side moves it from 10 to 17 -and saturates. Enlarging the population does not help either, 23 to 19 -shared directions from 256 to 1024 scenes. The ceiling behaves like a -property of what the two corpora are about rather than of how either is -encoded, which makes corpus choice a first-class variable: the closed -world reaches 26 because its captions state exactly the world state. +What does localise the gap is an intervention rather than a statistic. +In the closed world the caption is encoded in separate factor blocks, so +one can be suppressed — the caption stops stating a property while vision +still sees it, and nothing else changes. Deleting a single word class +takes recovery from 96.4% to 5.6% at a correlation of 0.830, higher than +anything we reach on photographs, and the full ladder is monotone in how +much the caption states. **Corpus overlap controls recovery**, whatever +the mediating quantity turns out to be. + +Encoder-side changes do not. Five interventions on the vision side — +near-tripling the segments, substituting annotated boxes for unsupervised +segmentation, third-order moments in the set kernel, and doubling the +backbone to DINOv2-large — each raise the vision field's own effective +rank and leave both the correlation and every width measure where they +were. The gains that did arrive came from the projection, the text +vectors and the phrase encoding, and total 0.075 in correlation. This +makes corpus choice a first-class variable rather than a fallback: a +region description does not say as much about a photograph as the closed +world's captions say about their scenes, and no vision encoder repairs +that. ## What is not shown diff --git a/RANK_RESULTS.md b/RANK_RESULTS.md index adaee21..d520e5e 100644 --- a/RANK_RESULTS.md +++ b/RANK_RESULTS.md @@ -13,7 +13,10 @@ at 0.656 and no recovery run was made on it, on the strength of that rule. **A controlled truncation refutes the rule.** Take the synthetic fields that recover at 95.3%, project both to rank *r*, and match blind from a spectral start with exact refinement. The correlation barely moves across the ladder; -recovery moves across its entire range. +recovery moves across its entire range. (What replaces the correlation is +*not* the shared-direction count in the next column — see the correction that +follows this section. The column is reported because it is what motivated the +hypothesis, not because it survived it.) | Field | ρ at truth | Shared directions | Recovery (256 scenes, chance 0.4%) | |---|---|---|---| @@ -27,9 +30,10 @@ recovery moves across its entire range. A rank-8 field correlating at 0.906 — comfortably past the supposed threshold — recovers 13%. The same field at rank 16 recovers 96%. Correlation is held -almost fixed while recovery crosses from failure to success, so the variable -that moves recovery is the width of the shared spectrum, not the strength of -the agreement. +almost fixed while recovery crosses from failure to success, so **something +other than the strength of the agreement is doing the work.** Identifying that +something is a separate problem, and the next section records two attempts at +it that both failed. The theory says the same thing in advance and we misread it. Correlated-matching thresholds are derived for exchangeable full-rank noise, where each of the @@ -49,6 +53,55 @@ descent does when the truth is not the optimum. Natural data does not recover, and now that is a measurement rather than an inference from a statistic that does not govern it. +## Correction, same day: width is not the statistic either + +The section below proposed the width of the shared spectrum as the replacement +gate. **A control run later the same session refutes it**, and the refutation +is recorded here rather than folded away because it was my own claim and it +lasted four hours. + +Adding independent noise to both synthetic fields lowers the correlation +without narrowing the shared signal underneath, which is still full-rank. That +control was run to guard against overcorrecting, and it did more than that: + +| Field | ρ | shared width | shared dimension (held out) | recovery | +|---|---|---|---|---| +| synthetic, noise 0.35 | 0.828 | 15.7 | 16.8 | **94.7%** | +| synthetic, caption omits size | 0.830 | 15.0 | 16.6 | **5.6%** | +| Visual Genome, best | 0.731 | 15.0 | 17.2 | **0.0%** | +| synthetic, noise 0.50 | 0.743 | 13.3 | — | 70.7% | + +Three fields agree on the correlation to within 0.1, agree on both width +measures to within 1.5, and differ in recovery across the entire range. The +principal-angle count fails because noise rotates eigenvectors and lowers the +measured overlap without narrowing anything. A second instrument built +specifically to fix that — canonical correlations fitted on one third of the +scenes and scored on another, so a direction counts only if it generalises — +fails the same way, and rates Visual Genome *highest* of the three failures. + +So **two candidate statistics have now been proposed and refuted in one +session**, and the honest position is narrower than the one I wrote four hours +ago: + +- **Established.** The correlation does not govern recovery. Rank-8 truncation + holds it at 0.906 and drops recovery to 12.9%; independent noise drops it to + 0.743 and keeps recovery at 70.7%. Both directions are controlled, and + together they retire the 0.9 rule for good. +- **Established.** Genuine rank truncation below about 8 destroys recovery, and + suppressing a caption factor destroys it while the correlation stays at 0.83. +- **Not established.** That any single width or dimension statistic predicts + recovery. Neither of the two tried does. + +What separates the noise case from the omission case is a live question with a +specific candidate. Suppressing a factor makes scenes that differed only in +that factor *exactly* interchangeable in the text field, so the energy acquires +an exact symmetry and the truth stops being a unique minimum — it becomes one +member of a degenerate orbit. Independent noise creates no such ties. That +predicts a three-way split which the gate can read directly: the truth deeper +than everything found (recoverable), tied with what is found (symmetry), or +shallower than what is found (information deficit). The gate is running on both +fields; the sections below should be read against this correction. + ## What the shared spectrum measures Each field is eigendecomposed and the principal angles between the two leading @@ -66,9 +119,9 @@ condition below, so the counts are not an artefact of subspace dimension. The failing synthetic field settles the sufficiency question in the other direction: 18 shared directions with a correlation of 0.508 also fails. **Recovery needs both a wide shared spectrum and a strong correlation, and -neither alone predicts it.** The gate becomes a joint condition, and the honest -statement of our position is that natural data is short on both — though far -closer on width than the correlation gap alone suggested. +neither alone predicts it.** *(Superseded by the correction above: the pair +does not predict it either. The counts below remain as measured; what they do +not support is the inference that they govern recovery.)* ## Natural data is narrow in the intersection, not in either modality @@ -147,12 +200,24 @@ photographs — and recovery is at chance. **The synthetic world degraded to Visual Genome's shared width fails exactly the way Visual Genome fails**, at a correlation Visual Genome never reaches. -This closes the causal chain. How much of the same world the two corpora -describe sets the width of the shared spectrum; the width sets recovery; and -the correlation reports on neither reliably. It also reframes the natural-data -deficit as a statement about Visual Genome rather than about our pipeline: a -region description simply does not say as much about a photograph as the -synthetic captions say about their scenes. +The full ladder is monotone in how much the caption states: + +| Caption content | ρ | shared | recovery | +|---|---|---|---| +| states everything | 0.928 | 20 | 96.4% | +| never states size | 0.830 | 15 | 5.6% | +| never states count | 0.678 | 13 | 1.3% | +| states neither size nor count | 0.544 | 10 | 1.3% | +| never states colour | 0.736 | 9 | 0.0% | + +What this establishes is that **corpus overlap controls recovery**, which is +the claim that matters for what to do next. What it does not establish is the +intermediate step — that it does so *through* the width, since the noise +control reaches the same width with recovery intact. The mechanism is under +test; the corpus-level conclusion does not depend on which way that test goes, +and it reframes the natural-data deficit as a statement about Visual Genome +rather than about our pipeline: a region description simply does not say as +much about a photograph as the synthetic captions say about their scenes. (Widths in this table are computed on standardised fields and are not comparable entry-for-entry with the tables above, which use raw fields; within diff --git a/artifacts/synth_v1/omit_ladder_rest.json b/artifacts/synth_v1/omit_ladder_rest.json new file mode 100644 index 0000000..fc77598 --- /dev/null +++ b/artifacts/synth_v1/omit_ladder_rest.json @@ -0,0 +1,28 @@ +{ + "protocol": "Vision sees every factor in all conditions; only the captions are impoverished. Hidden pairing generated per trial and read only for scoring.", + "size": 256, + "chance": 0.00390625, + "rows": [ + { + "caption_content": "no count words", + "omitted": "count", + "correlation": 0.6784429189341008, + "shared_directions": 13.0, + "recovery": 0.013020833333333334 + }, + { + "caption_content": "no size or count", + "omitted": "size_count", + "correlation": 0.5442660429860932, + "shared_directions": 10.0, + "recovery": 0.013020833333333334 + }, + { + "caption_content": "no colour words", + "omitted": "colour", + "correlation": 0.7356797393477765, + "shared_directions": 9.0, + "recovery": 0.0 + } + ] +}
\ No newline at end of file diff --git a/artifacts/vg_5k/shared_dimension.json b/artifacts/vg_5k/shared_dimension.json new file mode 100644 index 0000000..8dc89ad --- /dev/null +++ b/artifacts/vg_5k/shared_dimension.json @@ -0,0 +1,92 @@ +{ + "protocol": "Three-way scene split. Anchors define the coordinate system, canonical correlation is fitted on the second third and scored on the last, so a shared direction counts only if it generalises to scenes that took no part in fitting it. Calibrated against a scene-shuffled null.", + "components": 24, + "threshold": 0.3, + "rows": [ + { + "label": "synth-full REC96", + "correlation": 0.928480988366389, + "shared_dimension": 22.0, + "shuffled_null": 0.0, + "net_shared_dimension": 22.0, + "held_out_canonical_top8": [ + 0.999, + 0.998, + 0.997, + 0.997, + 0.997, + 0.997, + 0.994, + 0.985 + ] + }, + { + "label": "synth+noise REC95", + "correlation": 0.8276655471168534, + "shared_dimension": 16.8, + "shuffled_null": 0.0, + "net_shared_dimension": 16.8, + "held_out_canonical_top8": [ + 0.981, + 0.976, + 0.97, + 0.962, + 0.926, + 0.932, + 0.896, + 0.855 + ] + }, + { + "label": "synth-omitsize REC6", + "correlation": 0.8301825436675786, + "shared_dimension": 16.6, + "shuffled_null": 0.2, + "net_shared_dimension": 16.400000000000002, + "held_out_canonical_top8": [ + 0.997, + 0.996, + 0.995, + 0.995, + 0.993, + 0.994, + 0.992, + 0.987 + ] + }, + { + "label": "synth-rank8 REC13", + "correlation": 0.9058447537972103, + "shared_dimension": 7.0, + "shuffled_null": 0.0, + "net_shared_dimension": 7.0, + "held_out_canonical_top8": [ + 0.999, + 0.998, + 0.998, + 0.997, + 0.991, + 0.958, + 0.819, + 0.145 + ] + }, + { + "label": "natural REC0", + "correlation": 0.7309253746041517, + "shared_dimension": 17.2, + "shuffled_null": 0.2, + "net_shared_dimension": 17.0, + "held_out_canonical_top8": [ + 0.895, + 0.868, + 0.866, + 0.806, + 0.803, + 0.768, + 0.723, + 0.665 + ] + } + ] +}
\ No newline at end of file diff --git a/logs/gate_noisy.log b/logs/gate_noisy.log new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/logs/gate_noisy.log diff --git a/logs/gate_omit.log b/logs/gate_omit.log new file mode 100644 index 0000000..e69de29 --- /dev/null +++ b/logs/gate_omit.log diff --git a/logs/omit_rest.log b/logs/omit_rest.log index e69de29..bd507bd 100644 --- a/logs/omit_rest.log +++ b/logs/omit_rest.log @@ -0,0 +1,5 @@ +caption content rho shared recovery +no count words 0.678 13.0 0.013 +no size or count 0.544 10.0 0.013 +no colour words 0.736 9.0 0.000 +DONE diff --git a/logs/rho_full_width.log b/logs/rho_full_width.log index 7a8a777..b81ce33 100644 --- a/logs/rho_full_width.log +++ b/logs/rho_full_width.log @@ -1,2 +1,4 @@ noise=0.0 rho=0.928 shared= 21.0 recovery=0.961 noise=0.2 rho=0.893 shared= 18.0 recovery=0.971 +noise=0.35 rho=0.828 shared= 15.7 recovery=0.947 +noise=0.5 rho=0.743 shared= 13.3 recovery=0.707 diff --git a/worldalign/shared_dimension.py b/worldalign/shared_dimension.py new file mode 100644 index 0000000..2729a6b --- /dev/null +++ b/worldalign/shared_dimension.py @@ -0,0 +1,174 @@ +"""How many independent dimensions do the two modalities actually share? + +The principal-angle count used earlier does not survive its own control. Adding +independent noise to both synthetic fields drives it from 21 to 15.7 while +recovery stays at 94.7%; suppressing one caption factor drives it to 15.0 and +recovery to 5.6%. Same correlation, same count, opposite outcome -- so the +count conflates two different things. Noise rotates eigenvectors and lowers the +measured overlap without narrowing the shared signal, which is still full-rank +underneath and still recoverable. + +This measures the shared dimension in a way noise cannot fake, by asking how +many directions of agreement **generalise to scenes that took no part in +fitting them**. Scenes are split three ways. Anchors define a coordinate +system: a scene is described by its field row against the anchors, separately +in each modality. Canonical correlation is fitted on the second third and +evaluated on the last, so a direction counts only if the agreement it claims +holds up on scenes it never saw. Noise cannot survive that; genuine shared +structure can. + +Nothing here uses the pairing beyond the alignment the two fields already +carry, and the count is calibrated against a scene-shuffled null. +""" + +from __future__ import annotations + +import argparse +import json + +import numpy as np +import torch + +from .common import write_json + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("--fields", nargs="+", required=True) + parser.add_argument("--labels", nargs="+", default=None) + parser.add_argument("--components", type=int, default=24) + parser.add_argument("--ridge", type=float, default=1e-2) + parser.add_argument("--threshold", type=float, default=0.3) + parser.add_argument("--repeats", type=int, default=5) + parser.add_argument("--output", default="artifacts/vg_5k/shared_dimension.json") + return parser.parse_args() + + +def standardise(matrix: np.ndarray) -> np.ndarray: + values = matrix[~np.eye(len(matrix), dtype=bool)] + out = (matrix - values.mean()) / values.std() + np.fill_diagonal(out, 0.0) + return out + + +def reduce_columns(block: np.ndarray, components: int, basis=None): + """PCA of anchor-similarity profiles, fitted once on the fit split.""" + if basis is None: + centre = block.mean(0, keepdims=True) + _, _, vectors = np.linalg.svd(block - centre, full_matrices=False) + basis = (centre, vectors[:components].T) + centre, directions = basis + return (block - centre) @ directions, basis + + +def canonical(fit_a, fit_b, test_a, test_b, ridge: float) -> np.ndarray: + """Held-out canonical correlations between two views of the same scenes.""" + fit_a = fit_a - fit_a.mean(0) + fit_b = fit_b - fit_b.mean(0) + n, dimension = fit_a.shape + cov_aa = fit_a.T @ fit_a / n + ridge * np.eye(dimension) + cov_bb = fit_b.T @ fit_b / n + ridge * np.eye(fit_b.shape[1]) + cov_ab = fit_a.T @ fit_b / n + + inv_a = np.linalg.inv(np.linalg.cholesky(cov_aa)) + inv_b = np.linalg.inv(np.linalg.cholesky(cov_bb)) + whitened = inv_a @ cov_ab @ inv_b.T + left, _, right = np.linalg.svd(whitened, full_matrices=False) + + projection_a = (test_a - test_a.mean(0)) @ inv_a.T @ left + projection_b = (test_b - test_b.mean(0)) @ inv_b.T @ right.T + correlations = [] + for column in range(projection_a.shape[1]): + first, second = projection_a[:, column], projection_b[:, column] + denominator = first.std() * second.std() + correlations.append( + 0.0 if denominator < 1e-12 + else float(np.mean((first - first.mean()) * (second - second.mean())) + / denominator) + ) + return np.abs(np.array(correlations)) + + +def analyse(path: str, label: str, args: argparse.Namespace) -> dict: + state = torch.load(path, map_location="cpu", weights_only=False) + visual = standardise(state["visual_field"].double().numpy()) + text = standardise(state["text_field"].double().numpy()) + size = len(visual) + mask = ~np.eye(size, dtype=bool) + rho = float(np.corrcoef(visual[mask], text[mask])[0, 1]) + + counts, spectra, nulls = [], [], [] + for repeat in range(args.repeats): + generator = np.random.default_rng(repeat) + order = generator.permutation(size) + third = size // 3 + anchors, fit, test = order[:third], order[third : 2 * third], order[2 * third :] + + def views(rows, basis_v=None, basis_t=None): + reduced_v, basis_v = reduce_columns( + visual[np.ix_(rows, anchors)], args.components, basis_v + ) + reduced_t, basis_t = reduce_columns( + text[np.ix_(rows, anchors)], args.components, basis_t + ) + return reduced_v, reduced_t, basis_v, basis_t + + fit_v, fit_t, basis_v, basis_t = views(fit) + test_v, test_t, _, _ = views(test, basis_v, basis_t) + correlations = canonical(fit_v, fit_t, test_v, test_t, args.ridge) + counts.append(int((correlations > args.threshold).sum())) + spectra.append(correlations) + + shuffled = text[np.ix_(generator.permutation(size), generator.permutation(size))] + shuffled = standardise(shuffled) + + def shuffled_views(rows, basis=None): + return reduce_columns(shuffled[np.ix_(rows, anchors)], args.components, basis) + + fit_s, basis_s = shuffled_views(fit) + test_s, _ = shuffled_views(test, basis_s) + nulls.append( + int((canonical(fit_v, fit_s, test_v, test_s, args.ridge) > args.threshold).sum()) + ) + + mean_spectrum = np.mean(np.stack(spectra), axis=0) + return { + "label": label, + "correlation": rho, + "shared_dimension": float(np.mean(counts)), + "shuffled_null": float(np.mean(nulls)), + "net_shared_dimension": float(np.mean(counts) - np.mean(nulls)), + "held_out_canonical_top8": [round(float(v), 3) for v in mean_spectrum[:8]], + } + + +def main() -> None: + args = parse_args() + labels = args.labels or [path.split("/")[-1] for path in args.fields] + rows = [] + for path, label in zip(args.fields, labels): + row = analyse(path, label, args) + rows.append(row) + print( + f"{row['label']:<26} rho={row['correlation']:.3f} " + f"shared_dim={row['shared_dimension']:5.1f} " + f"(null {row['shuffled_null']:4.1f}, net {row['net_shared_dimension']:5.1f}) " + f"top={row['held_out_canonical_top8'][:5]}", + flush=True, + ) + write_json(args.output, { + "protocol": ( + "Three-way scene split. Anchors define the coordinate system, " + "canonical correlation is fitted on the second third and scored on " + "the last, so a shared direction counts only if it generalises to " + "scenes that took no part in fitting it. Calibrated against a " + "scene-shuffled null." + ), + "components": args.components, + "threshold": args.threshold, + "rows": rows, + }) + + +if __name__ == "__main__": + main() |