summaryrefslogtreecommitdiff
path: root/notes/18_contribution_roadmap.md
diff options
context:
space:
mode:
Diffstat (limited to 'notes/18_contribution_roadmap.md')
-rw-r--r--notes/18_contribution_roadmap.md47
1 files changed, 35 insertions, 12 deletions
diff --git a/notes/18_contribution_roadmap.md b/notes/18_contribution_roadmap.md
index 179bd5d..4805a19 100644
--- a/notes/18_contribution_roadmap.md
+++ b/notes/18_contribution_roadmap.md
@@ -33,7 +33,7 @@ C_l(q_l) = -log P(Q_l >= q_l)
This formalizes what it means for feedback alignment to consume parameter
direction volume.
-## 2. Scaling Law and Redundancy Exhaustion
+## 2. Scaling Law and Soft Capacity Erosion
Across independent layers, raw feasible volume multiplies while log-capacity
cost adds:
@@ -50,17 +50,37 @@ C_all = Θ(L n^2)
p_all = exp[-Θ(L n^2)]
```
-Functional loss does not need to appear immediately. If total parameter
-dimension is `P`, task dimension is `d`, and alignment imposes `k` generic
-constraints, then hard functional rank loss is:
+Functional loss should be modeled as soft erosion, not a hard step. If total
+parameter dimension is `P`, task-sensitive dimension is `d`, and alignment
+imposes a generic `k`-dimensional constraint subspace, then its expected overlap
+with the task subspace is:
+
+```text
+E[tr(P_E P_S)] = k d / P
+```
+
+Equivalently, if `J` is the training-set output Jacobian and a random
+`k`-dimensional constraint leaves projection `Q`, then:
+
+```text
+K_k = J Q J^T
+E[K_k | J] = (1 - k/P) K_BP
+```
+
+So the FA/BP gap need not wait until redundant directions are fully exhausted.
+Redundancy suppresses the expected task-relevant alignment burden, but does not
+make it exactly zero.
+
+Hard functional rank loss remains a limiting sanity check:
```text
Δd_hard = max(0, k - (P - d))
```
-So the FA/BP gap should open when redundant directions are exhausted.
+This threshold describes exact rank collapse, not finite-time loss,
+conditioning loss, or effective-dimension loss.
-This is the phase-transition contribution.
+This is the soft capacity-erosion contribution.
## 3. Prior-Free Minimax Initialization Bound
@@ -136,8 +156,8 @@ Experiments should validate three levels:
```text
Q_l ~ Beta(1/2, (D_l - 1)/2)
```
-2. capacity and redundancy transition:
- FA/BP gap opens near the hard FA capacity margin crossing;
+2. capacity and soft erosion:
+ FA/BP gap distribution changes smoothly with hard FA capacity margin;
3. tangent-operator trajectory distributions:
predicted gap distributions overlap empirical trajectory distributions.
@@ -167,16 +187,19 @@ This separation avoids overclaiming while still giving a coherent theory chain.
## Next Priority
-The weakest current visual is the phase-transition contribution.
+The weakest current visual was the phase-transition contribution; the dense
+long-training sweep now shows that the correct phenomenon is a soft ramp, not a
+hard kink.
The desired figure should show:
```text
-capacity margin decreases -> redundant directions exhausted -> FA/BP gap opens
-and then grows
+capacity margin decreases -> task-relevant alignment overlap increases ->
+FA/BP gap distribution grows smoothly
```
The plot should use hard FA capacity margin on the x-axis, place the zero
-margin as a vertical reference, and show BP/FA train loss or FA-BP train gap.
+margin as a vertical reference, and show the FA-BP train-gap distribution on a
+log scale.
The most direct target is a width sweep on a random-label task, because random
labels make the task dimension controllable and force memorization capacity.