diff options
| -rw-r--r-- | BIAS_STORY_REASSESSMENT.md | 232 |
1 files changed, 232 insertions, 0 deletions
diff --git a/BIAS_STORY_REASSESSMENT.md b/BIAS_STORY_REASSESSMENT.md new file mode 100644 index 0000000..e14f931 --- /dev/null +++ b/BIAS_STORY_REASSESSMENT.md @@ -0,0 +1,232 @@ +# Structured-bias paper: ICLR reassessment + +Date: 2026-08-06. This assessment deliberately gives no novelty credit to the +clean KP scaling results. It asks whether the Harnett-specific operation alone +can carry an ICLR paper: + +```text +r = a - P(h) +Delta W = eta * r * eligibility +``` + +## Verdict + +The topic is strong enough for ICLR, but the current evidence is not. With the +clean scaling story removed, the present package is approximately **5/10 +(borderline reject)**. A coherent structured-bias paper can reach **6/10** if +it establishes a natural source of bias, predicts its learning dynamics, and +shows that the unchanged two-line SDIL rule removes it more effectively than +bias-specific alternatives at matched observation cost. It can reach a solid +**7/10** with a measured physical bias source or a second substantially +different system and a useful-scale confirmation. + +This should not be sold as another general alternative to backpropagation. +The paper-level question is: + +> When a local teaching channel mixes task instruction with a systematic, +> state-predictable component, what failure does that component cause, and can +> a locally learned neutral prediction remove it without removing instruction? + +## What the repository currently proves + +Positive evidence: + +- In the controlled MNIST intervention, raw and norm-matched raw signals fall + to about 10.3% at the strongest soma-predictable traffic, while innovation + retains 97.35% over five seeds. Norm matching rules out a pure scale effect. +- In five paired CIFAR-10 ResNet-20 seeds, dynamic neutral projection remains + within 0.2 points of clean KP under a four-times-RMS traffic intervention. + The same mechanism remains finite through ResNet-56. +- The implementation records instruction-off observations, checks that no task + loss or clean teaching signal enters the neutral fit, and retains failed + gates and untouched endpoints. + +Evidence that limits the claim: + +- The strongest controlled bias is generated from the same somatic statistic + made available to the predictor. Recovery is therefore a clean necessity + test, but not evidence that important real bias normally has this form. +- The five-seed endogenous MNIST gate failed one of four traffic realizations. + The FashionMNIST recovery found large gains but failed the no-traffic-gap + requirement for top-down traffic. The necessary shared-context task produced + only a 0.195-point innovation gain. These failures show that high predictor + R-squared is not sufficient to identify instruction. +- A fixed predictor did not stabilize the standard ResNet intervention. The + successful variant needs an instruction-off observation on every example and + a fast affine projection. This is useful evidence, but it weakens the claim + that the original two-line rule alone solves the problem. +- The clean ResNet-20/32/56 scaling belongs to reciprocal Kolen--Pollack credit. + It establishes that the substrate scales, not that residualization causes + scaling. +- The current theory proves a standard conditional-expectation projection + identity and a generic smooth-loss descent condition. It does not yet predict + a bias-induced error floor, limit cycle, instability, or depth law. + +## Prior-work pressure + +The novelty boundary is narrow: + +- Laborieux et al. show that finite one-sided nudging gives EP a first-order + estimator bias. Randomly reversing beta cancels that term in expectation at + two-phase cost, although their CIFAR-10 runs have high variance; centered EP + is stable but adds a phase. +- Holomorphic EP obtains exact finite-amplitude gradients through an oscillating + complex nudge and reports ImageNet32 performance comparable to BP. +- Jacobian homeostasis separately analyzes finite-nudge and weight-asymmetry + bias and repairs the latter on ImageNet32. This is direct precedent that a + bias-mechanism paper can be accepted, but it sets a high bar for theory and + scale. +- Dillavou et al. measure deterministic update bias in a physical contrastive + resistor network, connect it to error floors, null-space drift and limit + cycles, and introduce overclamping. Any physical contrastive experiment must + compare to overclamping rather than presenting bias robustness as unclaimed + territory. +- Conditional-mean subtraction itself is regression/control-variate machinery. + The contribution cannot be the algebraic subtraction alone; it must be the + locally identifiable bias model, its learning consequence, and causal + evidence that the removed component is nuisance rather than instruction. + +Primary sources: + +- Harnett/Francioni et al., Nature 2026: + https://www.nature.com/articles/s41586-026-10190-7 +- Finite-nudge EP bias, Frontiers 2021: + https://www.frontiersin.org/journals/neuroscience/articles/10.3389/fnins.2021.633674/full +- Holomorphic EP, NeurIPS 2022: + https://proceedings.neurips.cc/paper_files/paper/2022/hash/545a114e655f9d25ba0d56ea9a01fc6e-Abstract-Conference.html +- Jacobian homeostasis, ICLR 2024: + https://proceedings.iclr.cc/paper_files/paper/2024/hash/6a55f024db3f771194bdadc8f3a35381-Abstract-Conference.html +- Physical imperfection and overclamping, arXiv v2 2026: + https://arxiv.org/html/2505.22887 + +## The minimum complete mechanism + +Use one explicit model throughout: + +```text +a_t = s_t + b(z_t) + epsilon_t +r_t = a_t - P(z_t) +``` + +Here `s_t` is task instruction, `b(z_t)` is deterministic or slowly drifting +bias predictable from locally available neutral state `z_t`, and `epsilon_t` +is unpredictable zero-mean noise. The identification assumptions must be +stated, tested and violated deliberately: + +1. the neutral observation contains no instruction; +2. the neutral and task periods share the same bias law; +3. the predictor uses information available before instruction; +4. the predictor class can represent enough of the bias; +5. predictor learning is slow or cross-fitted enough not to absorb instruction. + +The theory must go beyond `P` minimizing mean-squared prediction error. At +minimum it should derive, for a tractable linear or locally linear system: + +- the raw learner's displaced fixed point, error floor or limit cycle as a + function of bias, curvature and learning rate; +- how coherent bias accumulates with steps, depth or flat directions, compared + with zero-mean noise of the same RMS; +- an optimization/stability bound in terms of residual predictor error and + neutral-to-task distribution shift; +- a failure theorem or counterexample showing when subtraction removes part of + the instruction. + +These statements generate quantitative curves before the large experiment. +Fitting a line after observing the endpoint is not sufficient. + +## Best experimental systems + +### 1. Physical coupled-learning imperfection -- primary candidate + +The closest natural match is the resistor-network setting of Dillavou et al. +Its per-edge measurement/application bias persists when the correct +contrastive update shrinks, producing drift along flat directions and +multi-task limit cycles. An instruction-off or zero-error observation gives a +natural local baseline. The unchanged SDIL predictor should be tested against: +raw coupled learning, overclamping, a per-edge constant-offset calibration, +random zero-mean noise matched in RMS, and an oracle bias subtraction. + +The decisive result is not only lower final error. SDIL must correctly predict +and reduce cycle span, null-space drift and forgetting using the bias measured +before task training. Real experimental traces or a bias distribution measured +from hardware would make this much stronger than a synthetic injection. + +### 2. Deep contrastive network with hardware-shaped differential offsets + +Use an author DCHN or Dual Propagation implementation and introduce offsets at +the actual local free/nudged measurement boundary, with distributions fixed +from a physical system rather than chosen as `P(h)` by construction. The task +ladder should include FashionMNIST, SVHN and CIFAR-10, with depth or relaxation +length varied. Compare the same two-line SDIL rule to raw learning, ordinary +constant calibration, overclamping where applicable, and clean/oracle paths. + +This supplies useful-scale evidence, but by itself it remains simulation. A +five-seed artificial differential-bias result in Dual Propagation is supporting +evidence, not the central natural-bias claim. + +### 3. Finite-nudge EP bias -- diagnostic, not primary + +Random-sign beta is the correct same-cost baseline because it cancels the +first-order term in expectation. Centered EP is the lower-variance expensive +reference. Unless the original SDIL rule beats random-sign EP at matched phase +and observation cost without calibration stages or extra mechanisms, this +setting should be reported as a boundary where randomization already solves +the mean bias. It should not drive method design. + +Weight-asymmetry EP is also unattractive as the first target because Jacobian +homeostasis is a specialized, accepted and large-scale solution. + +## Causal experiment matrix + +One compact matrix can distinguish the claimed mechanism: + +| factor | required values | purpose | +|---|---|---| +| corruption | none, zero-mean noise, fixed bias, state-dependent bias, slow drift | separates noise robustness from bias removal | +| predictability | same RMS with low to high neutral R-squared | tests the proposed sufficient variable | +| signal rule | raw, norm-matched raw, static calibration, SDIL, oracle | isolates subtraction from scale and ordinary calibration | +| neutral validity | matched, shifted, instruction leakage | tests identification and failure boundaries | +| scale | task/depth/relaxation or number of alternating tasks | tests coherent accumulation | + +The primary plots should be predetermined theory-to-data plots: error floor or +cycle span versus bias; degradation versus scale at fixed bias; recovery versus +neutral predictability; and accuracy/error versus measured observations or +wall time. Endpoint accuracy alone is not mechanism evidence. + +## Score gates + +**Current: 5/10.** Strong auditing and a large controlled effect, but the +natural importance, unique mechanism and theory are incomplete. + +**Minimal accept, 6/10:** + +- one natural or independently measured bias source; +- a theorem predicting the observed failure and residual dependence; +- unchanged two-line SDIL, without a new calibration stage; +- matched static-calibration and bias-specific baselines; +- three or more seeds on one useful-scale task plus causal negative controls; +- explicit neutral-shift and instruction-leakage failures. + +**Solid accept, 7/10:** add either real physical-network evidence or a second +distinct learning system, and show the same frozen mechanism predicts both. + +**Stop the ICLR push** if any of the following occurs: + +- gains appear only when bias is constructed directly from the predictor's + input; +- a constant offset/EMA calibration, overclamping, or random-sign beta matches + SDIL at lower cost; +- useful-scale recovery requires per-example oracle-like neutral projection or + another compound algorithm rather than the stated two-line rule; +- measured bias does not worsen with any meaningful scale variable; +- neutral observations absorb instruction under realistic distribution shift. + +## Immediate decision + +Continue only as a short, gated reassessment. First reproduce the physical +bias dynamics in the coupled-learning system and test the unmodified SDIL +predictor against overclamping and constant calibration. In parallel, finish +the already-frozen Dual Propagation differential-bias confirmation as +supporting evidence. Do not spend full CIFAR-scale compute until the simple +rule wins this mechanism gate. A failed gate should close the ICLR structured- +bias paper rather than trigger another compound recovery method. |