diff options
| author | YurenHao0426 <Blackhao0426@gmail.com> | 2026-07-22 13:33:33 -0500 |
|---|---|---|
| committer | YurenHao0426 <Blackhao0426@gmail.com> | 2026-07-22 13:33:33 -0500 |
| commit | 0db651ac83608ea8f5b94950a391da22f076a142 (patch) | |
| tree | aae335ea16eb1b2f8981ba1217b2f91cdc2ae9b9 /THEORY.md | |
| parent | c27e0824a41e6c54dd9079f17c20c306c5f2e9c9 (diff) | |
baseline: add normalized local response mirroring
Diffstat (limited to 'THEORY.md')
| -rw-r--r-- | THEORY.md | 38 |
1 files changed, 38 insertions, 0 deletions
@@ -253,6 +253,44 @@ the stable rate left early alignment near zero, while larger rates caused a does not imply usable finite-budget signal in the 267,904-dimensional joint parameter space. +### Local response mirroring as an inherited strong baseline + +V4's failure motivates changing the observation, not merely its rate. The +weight-mirror identity of Akrout et al. starts from white local parent noise +`z` and its child response `u=Wz`: + +```text +E[z u^T] = sigma^2 W^T. +``` + +For convolutions, the corresponding local weight correlation divided by the +number of valid spatial probe pairs and `sigma^2` is an unbiased estimator of +the forward kernel. The normalized response-mirror baseline uses a +bias-blocked pre-BatchNorm observation phase and updates + +```text +What_e = corr(z_e, u_e) / (count_e sigma^2), +Q_e <- (1-eta_M) Q_e + eta_M What_e. +``` + +The readout is identical, with the sign reversed to match this repository's +negative-gradient teaching convention. Observation generation uses the +ordinary local forward synapses; the Q/R update API receives only probe and +response activities. An executable independence audit changes every forward +parameter after observation and verifies that the feedback update is bitwise +unchanged. + +This is a stabilized system-identification variant of +[weight mirroring](https://proceedings.neurips.cc/paper_files/paper/2019/file/f387624df552cea2f369918c5e1e12bc-Paper.pdf), +not an SDIL contribution. The original Hebbian-plus-decay rule can be highly +metaparameter-sensitive, as analyzed by +[Kunin et al.](https://proceedings.mlr.press/v119/kunin20a/kunin20a.pdf); +normalizing the local sufficient statistic makes the estimator's scale +explicit. At the deterministic smoke setting, one 16-probe observation has +mean/minimum feedback-forward cosine `0.9887/0.9661`, norm ratios +`[0.9537,1.0444]`, and zero forward-parameter access error in the update. +These are mechanics, not task evidence. A frozen comparator gate is required. + ## 2. What is sufficient for a descent step Flatten all forward parameters into `theta`, let `p = grad L(theta)`, and let |
