1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
|
# Current Results (Consolidated)
Authoritative snapshot of where the project stands. Supersedes the framing in
`00_project_overview.md` and `03_paper_outline.md` (both stale: they still say
"redundancy-exhaustion **phase transition**" and omit the estimator). The
current, correct framing lives here and in notes 18, 24, 26, 29, 34.
## Central claim
> Random feedback imposes a quantifiable, **soft** (non-threshold) operator
> burden relative to BP that is present at every capacity margin (including
> overparameterized) and grows smoothly as capacity tightens. The static
> alignment geometry is exactly characterized (Beta law, prior-free minimax);
> the initial operator erosion is exactly the hidden-layer BP speed share; a
> closed-form spectral law turns that scalar burden into the finite-time gap and
> explains the soft ramp; and an early operator-velocity estimator predicts the
> finite-time FA/BP gap with no fit (corr 0.999).
## The erosion ladder (organizing spine)
All "burden" objects, as one ladder (note 24 is the through-line: *random
feedback induces a distribution of residual-direction operator erosion `e(r)`;
capacity controls that distribution; the finite-time gap is the product of the
tangent operators*):
| level | object | status | role |
|---|---|---|---|
| 0 static geometry | `Q_l = cos^2(W^T,B) ~ Beta(1/2,(D-1)/2)` | exact | defines alignment |
| 1 null model | `e ~ Beta(k/2,(P-k)/2)`, `E[e]=k/P` | exact, **overpredicts** | proves nonzero cost for any `k>0` |
| 2 exact init | `e_0 = 1 - r^T S_FA r / r^T K_BP r`, `E_B[e_0]=hidden_share` | exact, validated | the real `t=0` burden; gives `rho=1-E_B[e_0]` |
| 3 finite time | closed form `gap_T(rho,{lam_i})` (note 34); estimator `K_t` (notes 15-17) | closed form: shape; estimator: magnitude | the actual gap |
## Three contributions
Refined to three pillars: 1 & 2 are foundational (static capacity geometry),
3 is an independent track (initialization optimality), and the rest of the
results build on 1 to form the main gap pillar, ordered as a progression
(exists & soft -> exact at init -> finite-time law -> validation).
### Contribution 1 — Distributional capacity of random feedback (static geometry)
*(old 1 + 2.)*
- Exact static-alignment law `Q_l = cos^2(W_{l+1}^T, B_l) ~ Beta(1/2,(D_l-1)/2)`,
`E[Q]=1/D`, with log-volume capacity cost `C_l(q) = -log P(Q_l >= q)`.
- Multilayer scaling: independent blocks => `C_all = sum_l C_l = Theta(L n^2)`,
raw feasible volume `exp[-Theta(L n^2)]`.
- Role: defines and measures the feedback-alignment burden before training.
- **Proven.** `scripts/static_alignment_beta.py`, `scripts/capacity_scaling.py`.
### Contribution 2 — Prior-free minimax optimality of random feedback
*(old 3, independent track.)*
- Without a prior on forward weights or task, no initialization beats isotropic
random feedback: `sup_mu inf_a E[(a^T b)^2] = 1/D`, achieved by isotropic.
- Role: self-contained -- the burden cannot be designed away at initialization.
- **Proven.** `scripts/minimax_initialization.py`.
### Contribution 3 — The FA/BP operator gap: always nonzero, soft, exactly anchored at init, quantified over training
*(old 4 + 5 + 6, as a four-level progression on Contribution 1.)*
- **(a) Existence & softness.** `rho = output_share < 1` => `gap > 0` at every
`depth >= 2` and every capacity margin; analytic in the spectrum => smooth,
**no redundancy-exhaustion threshold** (null model `E[e]=k/P`, `P(e=0)=0`).
- **(b) Exact at t=0.** `E_B[e_0 | W,r] = hidden_share = 1 - ||g_out^BP||^2 /
sum_l ||g_l^BP||^2`; one-hidden-layer Gaussian feedback => full conditional
Gaussian. Supplies the scalar burden `rho = 1 - E_B[e_0]`.
**Proven + validated** (0.4378 vs 0.4403). `scripts/actual_fa_initial_operator_moments.py`.
- **(c) Finite-time law.** closed-form soft-ramp gap
`gap_T = (1/2N) sum_i c_i^2 [(1-eta rho lam_i/N)^{2T} - (1-eta lam_i/N)^{2T}]`
(mechanism + shape, `log gap ~ const - c rho lam_min`; note 34) **plus** the
early operator-velocity estimator `K_hat_t = K_0 + t(K_s-K_0)/s`
(drift-corrected magnitude, corr 0.999). closed form = why soft; estimator = how much.
`scripts/closed_form_soft_ramp.py`, `scripts/compressed_operator_predictor.py`.
- **(d) Empirical validation (no fit).** matrix-alignment Beta; e0 Gaussian;
soft ramp (smooth, nonzero at positive margin -> refutes phase transition);
estimator trajectory overlap.
### Old numbering -> new
```text
old 1, old 2 -> C1
old 3 -> C2
old 4 -> C3(b)
old 6 (closed form + estimator) -> C3(c)
old 5 -> C3(d)
(new) existence & soft -> C3(a)
```
## Key numbers (the evidence)
- init moment: `E_B[e_0]=0.4378 +/- 0.008` vs theory `hidden_share=0.4403`.
**[retired -- see note 37 §4; not in current CSV. Use the 6x6 table in note
42 §6: max |error| <= 0.008, predicted span 0.15-0.45.]**
- estimator: linear-velocity gap MAE `0.00189`, corr `0.99885` (256 traj);
stress grid corr `0.994-0.99989` across depth 1/2/3, width 32/64/96,
horizon 25/50/100 (notes 15-17). fixed `K(0)` is ~10x worse.
- soft ramp (dense `T=30000`): gap `0.147 -> 0.00016` smooth across margin
`-190 -> +130`, **nonzero at positive margin, no kink** (note 22).
- closed-form law (corrected, matched data/init -- note 37): no-fit
`corr(log pred, log measured)=0.933` (matched init) / `0.982` (5-init geo
mean); measured `log(gap)` linear in margin `R^2=0.993`;
`corr(lam_min_mean,-log gap)=0.987` (note 34+37; 0.977/0.963 retired).
## Concise figure list (5 figures, one per contribution)
1. static Beta histogram + QQ (sanity).
2. soft ramp: gap vs margin, log-y, smooth, nonzero at positive margin
(refutes phase transition). `outputs/phase_transition_dense_T30000_352traj/`.
3. init moment: `E_B[e_0]` vs `hidden_share`, no-fit scatter.
`outputs/actual_fa_initial_operator_moments/`.
4. **estimator: predicted vs empirical finite-time gap, 256 traj + stress
grid, corr 0.999 (centerpiece).** `outputs/compressed_operator_s20_256traj_T50_width64_plots/`.
5. **closed form vs measured ramp + ramp law.**
`outputs/closed_form_soft_ramp/closed_form_vs_measured_ramp.png`.
## Proven vs estimated vs approximate
- proven, no fit: contributions 1, 3; the `E[e]=k/P` null model; the init
moment `E_B[e_0]=hidden_share`.
- closed-form gap law (note 34): proven under lazy (A1) + scalar-erosion (A2);
captures shape, not magnitude.
- estimator: no fitted scalar, but **conditional on early operators `(K_0,K_s)`**
-- a predictive finite-time estimator, not an architecture-only theorem.
## Open problem (the one real gap)
Derive the drift / effective burden from architecture + initialization so the
finite-time gap becomes architecture-only:
- predict `rho` (hence `hidden_share`) from infinite-width NTK recursions;
- predict the drift `K_s - K_0` (alignment gain) -- naive `dot_K_0` fails
(note 31), so use a bounded short-time alignment-gain path or a deep-linear /
two-layer order-parameter solution;
- a per-mode spectral erosion `rho_i` refining the scalar A2.
## Superseded / cleanup
- `00_project_overview.md`, `03_paper_outline.md`: stale framing; retire
"phase transition", add the estimator + closed-form law.
- notes 19-21 (transition-as-kink): refuted by the dense sweep (note 22).
- hard-`k` as a *gap predictor*: it overpredicts (note 25); keep only as the
null model that motivates the operator object.
- `scripts/actual_fa_initial_erosion_distribution.py` (e0 Gaussian, note 32):
the sampler reuses the analytic coefficient (partly circular) -- switch it to
the real FA backward (verified to reproduce the same numbers).
- `scripts/alignment_recovery_dynamics.py` (rho(t) recovery probe): superseded
by the estimator + closed-form law; keep only as an optional `t=0` sanity or
remove.
|
