From 2df52d3dd7b35053d4a357c3814d7cba6db602ce Mon Sep 17 00:00:00 2001 From: YurenHao0426 Date: Fri, 5 Jun 2026 15:44:19 -0500 Subject: Derive actual FA initial operator moment --- notes/18_contribution_roadmap.md | 34 +++++++++++++++++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-) (limited to 'notes/18_contribution_roadmap.md') diff --git a/notes/18_contribution_roadmap.md b/notes/18_contribution_roadmap.md index be4d775..aa26caf 100644 --- a/notes/18_contribution_roadmap.md +++ b/notes/18_contribution_roadmap.md @@ -93,6 +93,33 @@ conditioning loss, or effective-dimension loss. This is the soft capacity-erosion contribution. +There is now a cleaner actual-FA initialization theorem that should be used +before the random-subspace null model. For fixed forward weights and residuals, +with independent zero-mean feedback matrices: + +```text +E_B[speed_FA | W, r] = ||g_output^BP||^2 +``` + +where: + +```text +speed_FA = sum_l +speed_BP = sum_l ||g_l^BP||^2 +``` + +Therefore: + +```text +E_B[e_0 | W, r] + = 1 - ||g_output^BP||^2 / sum_l ||g_l^BP||^2 +``` + +So the true initial FA burden is the hidden-layer BP speed share, not the raw +hard feedback-matrix constraint count. The hard-k random-subspace model remains +a useful null model and capacity sanity check, but it should not be presented +as the exact FA gap theorem. + ## 3. Prior-Free Minimax Initialization Bound Let `b` be the normalized feedback direction in `D` dimensions and `a` be the @@ -169,11 +196,16 @@ Experiments should validate three levels: ``` 2. capacity and soft erosion: FA/BP gap distribution changes smoothly with hard FA capacity margin; -3. tangent-operator trajectory distributions: +3. actual-FA initial operator moments: + ```text + E_B[e_0 | W,r] = hidden BP speed share + ``` +4. tangent-operator trajectory distributions: predicted gap distributions overlap empirical trajectory distributions. Current strong evidence: +- initial actual-FA operator moment validation over widths and feedback seeds; - 256-trajectory finite-time overlap plot for `d=2,w=64,T=50,s=20`; - stress grid over depth, width, and horizon; - systematic low-bias analysis showing residual error is kernel-path curvature, -- cgit v1.2.3