Separate estimator from nonzero cost theorem

1 files changed, 210 insertions, 0 deletions

diff --git a/notes/26_estimator_vs_nonzero_cost_theorem.md b/notes/26_estimator_vs_nonzero_cost_theorem.md
new file mode 100644
index 0000000..75559b1
--- /dev/null
+++ b/notes/26_estimator_vs_nonzero_cost_theorem.md
@@ -0,0 +1,210 @@
+# Estimator vs Nonzero-Cost Theorem
+
+We do not need the estimator to be the main theory.
+
+The estimator answers:
+
+```text
+given an initialization, feedback matrix, data, and early operator state,
+how large is the finite-time FA/BP train-gap?
+```
+
+The theorem we want for the paper can be weaker and cleaner:
+
+```text
+redundant parameters do not make random feedback alignment cost-free.
+```
+
+These are different claims.
+
+## Avoid the Wrong Claim
+
+Do not claim:
+
+```text
+overparameterized FA cannot reach zero training error.
+```
+
+That is too strong and likely false in important settings. Existing
+overparameterized FA results prove convergence to zero training error in
+specific two-layer settings.
+
+The right claim is:
+
+```text
+overparameterization does not imply zero FA/BP operator gap or zero alignment
+cost.
+```
+
+FA may eventually fit the task while still having a nonzero finite-time tangent
+capacity loss relative to BP.
+
+## Clean Random-Subspace Theorem
+
+Let:
+
+```text
+P = parameter dimension
+k = effective alignment burden dimension
+J = training-set output Jacobian
+K_BP = J J^T
+r = current residual
+E = random k-dimensional constrained subspace, independent of J and r
+P_E = projection onto E
+Q = I - P_E
+K_Q = J Q J^T
+```
+
+Define the one-step operator erosion:
+
+```text
+e(r) = 1 - (r^T K_Q r) / (r^T K_BP r)
+```
+
+Assume:
+
+```text
+r^T K_BP r > 0
+```
+
+Then by rotational invariance:
+
+```text
+E[P_E] = (k/P) I
+E[Q] = (1 - k/P) I
+```
+
+Therefore:
+
+```text
+E[e(r) | J, r] = k/P
+```
+
+Moreover, if:
+
+```text
+v = J^T r
+```
+
+then:
+
+```text
+(r^T K_Q r) / (r^T K_BP r)
+= ||Q v||^2 / ||v||^2
+```
+
+and for Haar-random `E`:
+
+```text
+||P_E v||^2 / ||v||^2 ~ Beta(k/2, (P-k)/2)
+```
+
+So:
+
+```text
+e(r) ~ Beta(k/2, (P-k)/2)
+```
+
+under the random-subspace model.
+
+This proves:
+
+```text
+P(e(r) = 0) = 0
+```
+
+for `k > 0`, and:
+
+```text
+E[e(r)] = k/P
+```
+
+Thus redundant capacity reduces the expected erosion only through `k/P`; it
+does not make the erosion exactly zero unless:
+
+```text
+k = 0
+```
+
+or in a scaling limit:
+
+```text
+k/P -> 0
+```
+
+## Scaling Consequence
+
+If:
+
+```text
+k is fixed and P -> infinity
+```
+
+then:
+
+```text
+E[e(r)] -> 0
+```
+
+so the alignment burden vanishes asymptotically.
+
+If:
+
+```text
+k/P -> alpha > 0
+```
+
+then:
+
+```text
+E[e(r)] -> alpha
+```
+
+and the relative operator erosion persists.
+
+This is the theorem that directly supports the sell:
+
+```text
+capacity can dilute random feedback alignment burden, but it cannot make it
+free under proportional scaling.
+```
+
+## Role of the Estimator
+
+The estimator is not needed to prove nonzero cost.
+
+It is needed for:
+
+1. checking whether the random-subspace model is too pessimistic;
+2. measuring `k_eff` for real FA rather than hard `k`;
+3. predicting finite-time train-gap distributions after training dynamics,
+   alignment, and feature drift.
+
+So the paper can separate:
+
+```text
+Theorem: random alignment burden has nonzero expected operator erosion.
+Estimator: finite-time dynamics convert operator erosion into observed train-gap.
+```
+
+This separation prevents overclaiming.
+
+## Revised Contribution Logic
+
+Contribution 2 should not promise exact gap prediction.
+
+It should prove:
+
+```text
+no free redundant capacity:
+E[e(r)] = k_eff/P, hence nonzero when k_eff/P is nonzero.
+```
+
+Contribution 4 can then say:
+
+```text
+to quantify actual finite-time gap, use tangent-operator estimators.
+```
+
+This makes the theorem clean and the estimator useful without forcing one to do
+the other's job.