1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
|
# Exact e0 Distribution Validation
This note records the clean distributional validation for the actual FA initial
operator erosion theorem.
## Theorem Being Validated
Use a one-hidden-layer MLP with Gaussian feedback:
```text
x -> W1 -> ReLU -> W2 -> output
```
For fixed forward weights, data, gates, and residual:
```text
speed_BP = ||g_output^BP||^2 + ||g_hidden^BP||^2
speed_FA = ||g_output^BP||^2 + <g_hidden^BP, g_hidden^FA(B)>
e0(B) = 1 - speed_FA / speed_BP
```
Because the hidden FA gradient is linear in Gaussian feedback `B`:
```text
<g_hidden^BP, g_hidden^FA(B)> = <C(W,r), B>
```
Therefore:
```text
<C(W,r), B> ~ Normal(0, sigma_B^2 ||C(W,r)||^2)
```
and:
```text
e0(B) ~ Normal(
1 - ||g_output^BP||^2 / speed_BP,
sigma_B^2 ||C(W,r)||^2 / speed_BP^2
)
```
This is an exact conditional distribution for one-hidden-layer Gaussian FA at
initialization. It does not involve training horizon `T`.
## Experiment
Script:
```text
scripts/actual_fa_initial_erosion_distribution.py
```
Run:
```text
python scripts/actual_fa_initial_erosion_distribution.py \
--widths 16 32 64 128 \
--init-seeds 3 \
--feedback-samples 4096 \
--torch-threads 8 \
--outdir outputs/actual_fa_initial_erosion_distribution
```
Outputs:
```text
outputs/actual_fa_initial_erosion_distribution/e0_distribution_theory_vs_empirical.png
outputs/actual_fa_initial_erosion_distribution/e0_distribution_moment_calibration.png
outputs/actual_fa_initial_erosion_distribution/e0_distribution_rows.csv
```
## Results
Representative rows:
| width | init | theory mean | empirical mean | theory std | empirical std | KS | p-value |
|---:|---:|---:|---:|---:|---:|---:|---:|
| 16 | 0 | 0.113830 | 0.113413 | 0.027228 | 0.027401 | 0.0150 | 0.314 |
| 16 | 1 | 0.113705 | 0.113859 | 0.027261 | 0.027278 | 0.0103 | 0.778 |
| 32 | 0 | 0.121730 | 0.121544 | 0.018952 | 0.018693 | 0.0114 | 0.654 |
| 32 | 2 | 0.227193 | 0.227691 | 0.031148 | 0.030779 | 0.0166 | 0.206 |
| 64 | 0 | 0.031155 | 0.031119 | 0.003652 | 0.003583 | 0.0118 | 0.609 |
| 128 | 0 | 0.023004 | 0.022989 | 0.001630 | 0.001610 | 0.0135 | 0.441 |
All tested panels show close agreement between theory density and empirical
random-feedback histograms.
## Paper Use
This is the clean validation figure for the actual FA local erosion theorem.
It should replace any tangent-hierarchy figure if the goal is:
```text
theory predicts a distribution
large random-feedback sampling produces the same distribution
```
Suggested main-text wording:
```text
For one-hidden-layer Gaussian FA, the conditional distribution of the initial
operator erosion is exactly Gaussian. Figure X compares this predicted density
against 4096 random feedback draws per initialization across widths; the
empirical histograms and theoretical densities coincide without fitted
parameters.
```
This figure belongs to:
```text
Contribution 4: actual FA initial operator erosion theorem
Contribution 5: distributional empirical validation
```
It does not require a training-time axis.
|
