diff options
| author | YurenHao0426 <Blackhao0426@gmail.com> | 2026-06-05 15:44:19 -0500 |
|---|---|---|
| committer | YurenHao0426 <Blackhao0426@gmail.com> | 2026-06-05 15:44:19 -0500 |
| commit | 2df52d3dd7b35053d4a357c3814d7cba6db602ce (patch) | |
| tree | 96c08971d94cd931eb2727e3479d0b63f7319c5a /notes/18_contribution_roadmap.md | |
| parent | 4dee090530588792b0e0bce9a36981415e4a99a8 (diff) | |
Derive actual FA initial operator moment
Diffstat (limited to 'notes/18_contribution_roadmap.md')
| -rw-r--r-- | notes/18_contribution_roadmap.md | 34 |
1 files changed, 33 insertions, 1 deletions
diff --git a/notes/18_contribution_roadmap.md b/notes/18_contribution_roadmap.md index be4d775..aa26caf 100644 --- a/notes/18_contribution_roadmap.md +++ b/notes/18_contribution_roadmap.md @@ -93,6 +93,33 @@ conditioning loss, or effective-dimension loss. This is the soft capacity-erosion contribution. +There is now a cleaner actual-FA initialization theorem that should be used +before the random-subspace null model. For fixed forward weights and residuals, +with independent zero-mean feedback matrices: + +```text +E_B[speed_FA | W, r] = ||g_output^BP||^2 +``` + +where: + +```text +speed_FA = sum_l <g_l^BP, g_l^FA> +speed_BP = sum_l ||g_l^BP||^2 +``` + +Therefore: + +```text +E_B[e_0 | W, r] + = 1 - ||g_output^BP||^2 / sum_l ||g_l^BP||^2 +``` + +So the true initial FA burden is the hidden-layer BP speed share, not the raw +hard feedback-matrix constraint count. The hard-k random-subspace model remains +a useful null model and capacity sanity check, but it should not be presented +as the exact FA gap theorem. + ## 3. Prior-Free Minimax Initialization Bound Let `b` be the normalized feedback direction in `D` dimensions and `a` be the @@ -169,11 +196,16 @@ Experiments should validate three levels: ``` 2. capacity and soft erosion: FA/BP gap distribution changes smoothly with hard FA capacity margin; -3. tangent-operator trajectory distributions: +3. actual-FA initial operator moments: + ```text + E_B[e_0 | W,r] = hidden BP speed share + ``` +4. tangent-operator trajectory distributions: predicted gap distributions overlap empirical trajectory distributions. Current strong evidence: +- initial actual-FA operator moment validation over widths and feedback seeds; - 256-trajectory finite-time overlap plot for `d=2,w=64,T=50,s=20`; - stress grid over depth, width, and horizon; - systematic low-bias analysis showing residual error is kernel-path curvature, |
