summaryrefslogtreecommitdiff
path: root/notes/18_contribution_roadmap.md
diff options
context:
space:
mode:
authorYurenHao0426 <Blackhao0426@gmail.com>2026-06-05 15:44:19 -0500
committerYurenHao0426 <Blackhao0426@gmail.com>2026-06-05 15:44:19 -0500
commit2df52d3dd7b35053d4a357c3814d7cba6db602ce (patch)
tree96c08971d94cd931eb2727e3479d0b63f7319c5a /notes/18_contribution_roadmap.md
parent4dee090530588792b0e0bce9a36981415e4a99a8 (diff)
Derive actual FA initial operator moment
Diffstat (limited to 'notes/18_contribution_roadmap.md')
-rw-r--r--notes/18_contribution_roadmap.md34
1 files changed, 33 insertions, 1 deletions
diff --git a/notes/18_contribution_roadmap.md b/notes/18_contribution_roadmap.md
index be4d775..aa26caf 100644
--- a/notes/18_contribution_roadmap.md
+++ b/notes/18_contribution_roadmap.md
@@ -93,6 +93,33 @@ conditioning loss, or effective-dimension loss.
This is the soft capacity-erosion contribution.
+There is now a cleaner actual-FA initialization theorem that should be used
+before the random-subspace null model. For fixed forward weights and residuals,
+with independent zero-mean feedback matrices:
+
+```text
+E_B[speed_FA | W, r] = ||g_output^BP||^2
+```
+
+where:
+
+```text
+speed_FA = sum_l <g_l^BP, g_l^FA>
+speed_BP = sum_l ||g_l^BP||^2
+```
+
+Therefore:
+
+```text
+E_B[e_0 | W, r]
+ = 1 - ||g_output^BP||^2 / sum_l ||g_l^BP||^2
+```
+
+So the true initial FA burden is the hidden-layer BP speed share, not the raw
+hard feedback-matrix constraint count. The hard-k random-subspace model remains
+a useful null model and capacity sanity check, but it should not be presented
+as the exact FA gap theorem.
+
## 3. Prior-Free Minimax Initialization Bound
Let `b` be the normalized feedback direction in `D` dimensions and `a` be the
@@ -169,11 +196,16 @@ Experiments should validate three levels:
```
2. capacity and soft erosion:
FA/BP gap distribution changes smoothly with hard FA capacity margin;
-3. tangent-operator trajectory distributions:
+3. actual-FA initial operator moments:
+ ```text
+ E_B[e_0 | W,r] = hidden BP speed share
+ ```
+4. tangent-operator trajectory distributions:
predicted gap distributions overlap empirical trajectory distributions.
Current strong evidence:
+- initial actual-FA operator moment validation over widths and feedback seeds;
- 256-trajectory finite-time overlap plot for `d=2,w=64,T=50,s=20`;
- stress grid over depth, width, and horizon;
- systematic low-bias analysis showing residual error is kernel-path curvature,