Clarify proof and estimator status

1 files changed, 313 insertions, 0 deletions

diff --git a/notes/28_contribution_status_proof_vs_fit.md b/notes/28_contribution_status_proof_vs_fit.md
new file mode 100644
index 0000000..40c1118
--- /dev/null
+++ b/notes/28_contribution_status_proof_vs_fit.md
@@ -0,0 +1,313 @@
+# Contribution Status: Proof vs Estimator vs Fit
+
+This note separates the five current contributions by epistemic status.
+
+The key distinction:
+
+```text
+proof      = follows from stated assumptions, no empirical tuning
+estimator  = derived from a model/linearization and uses observed quantities
+fit        = post-hoc parameter tuning to match data
+```
+
+Our current project should avoid calling every component a theorem.
+
+## Summary Table
+
+| contribution | current status | fitted? | safe main-paper phrasing |
+|---|---|---:|---|
+| 1. capacity formalization | definition + exact random-geometry theorem | no | exact under isotropic feedback assumptions |
+| 2. scaling and soft erosion | exact scaling + exact random-subspace null model; real FA extension not yet proved | no, but partly only a null model | capacity cost scales exactly; random-subspace erosion gives a baseline, while real FA needs `k_eff` |
+| 3. prior-free minimax bound | exact theorem | no | minimax optimality for prior-free angular alignment |
+| 4. tangent-operator estimator | first-order tangent derivation + conditional predictor | no scalar fit, but uses early observed operators | estimator for finite-time gap, not architecture-only theorem |
+| 5. empirical distribution validation | experiment | no | validates which theoretical distributions and estimators match trajectories |
+
+## 1. Capacity Formalization
+
+What is proof-level:
+
+```text
+Q_l = cos^2(W_{l+1}^T, B_l)
+```
+
+If the normalized matrix directions are independent isotropic directions in
+dimension `D_l`, then:
+
+```text
+Q_l ~ Beta(1/2, (D_l - 1)/2)
+```
+
+This is an exact spherical-geometry theorem, not a fit.
+
+The log-volume capacity cost:
+
+```text
+C_l(q_l) = -log P(Q_l >= q_l)
+```
+
+is exact once this definition of capacity is chosen.
+
+What is not yet covered:
+
+```text
+training-time learned alignment
+ReLU gate-dependent effective dimension beyond simple assumptions
+final task loss
+```
+
+Those require later operator dynamics or experiments.
+
+## 2. Scaling and Soft Capacity Erosion
+
+What is proof-level:
+
+For independent layer blocks and fixed thresholds:
+
+```text
+p_all = product_l p_l
+C_all = sum_l C_l
+```
+
+So raw feasible volume multiplies and log-capacity cost adds. For equal-width
+MLPs:
+
+```text
+C_all = Theta(L n^2)
+p_all = exp[-Theta(L n^2)]
+```
+
+This is a direct consequence of the capacity definition and architecture
+dimensions. It is not a fit.
+
+The random-subspace erosion theorem is also proof-level under its own null
+model:
+
+```text
+P = parameter dimension
+k = random constrained-subspace dimension
+J = task Jacobian
+K_BP = J J^T
+Q = retained projection after removing a Haar-random k-dimensional subspace
+K_Q = J Q J^T
+```
+
+Then:
+
+```text
+E[K_Q | J] = (1 - k/P) K_BP
+```
+
+and for residual direction `r`:
+
+```text
+e(r) = 1 - (r^T K_Q r) / (r^T K_BP r)
+e(r) ~ Beta(k/2, (P-k)/2)
+E[e(r)] = k/P
+```
+
+This is theory, not empirical fitting.
+
+Important caveat:
+
+```text
+actual FA is not literally Haar-random deletion of k parameter dimensions
+```
+
+The hard `k` version overpredicts empirical FA gaps. Our experiments already
+showed that raw hard constraint count is too pessimistic.
+
+Therefore the safe claim is:
+
+```text
+random-subspace theory proves the no-free-capacity mechanism in a clean null
+model; real FA should be described by an effective burden k_eff or direct
+operator erosion e_t, not by raw hard k.
+```
+
+Unsafe claim:
+
+```text
+hard-k random-subspace formula exactly predicts real FA/BP gap
+```
+
+This is false in our current experiments.
+
+## 3. Prior-Free Minimax Initialization Bound
+
+This is proof-level.
+
+Let:
+
+```text
+b = normalized feedback direction
+a = unknown normalized target direction
+M_mu = E[b b^T]
+trace(M_mu) = 1
+```
+
+Then:
+
+```text
+inf_a E_mu[(a^T b)^2] = lambda_min(M_mu) <= 1/D
+```
+
+So:
+
+```text
+sup_mu inf_a E_mu[(a^T b)^2] = 1/D
+```
+
+Isotropic feedback reaches the bound.
+
+This is not fitted.
+
+Scope:
+
+```text
+prior-free angular squared alignment
+```
+
+It does not say isotropic feedback is best for:
+
+```text
+conditioning
+hardware cost
+sparsity
+low-rank efficiency
+task-aware priors
+```
+
+## 4. Tangent-Operator Loss/Gap Estimator
+
+This is not a fitted curve, but it is also not an architecture-only theorem.
+
+The local derivation is standard first-order residual dynamics:
+
+```text
+r_next_BP ≈ r - eta K_BP r / N
+K_BP = J J^T
+
+r_next_FA ≈ r - eta K_FA r / N
+K_FA = J J_tilde^T
+```
+
+Since FA gives a non-symmetric operator, the one-step loss decrease depends on:
+
+```text
+S_FA = (K_FA + K_FA^T) / 2
+```
+
+The finite-time estimator propagates these operators:
+
+```text
+r_T_BP_hat = product_t (I - eta K_BP,t / N) r_0
+r_T_FA_hat = product_t (I - eta K_FA,t / N) r_0
+gap_T_hat = (||r_T_FA_hat||^2 - ||r_T_BP_hat||^2) / (2N)
+```
+
+In our best current version, `K_t` is predicted from early operator motion,
+for example by using `K_0` and `K_s`.
+
+This has no post-hoc scalar hyperparameter. But it is conditional on early
+observed operator information:
+
+```text
+input includes data, initialization, feedback matrices, K_0, and early K_s
+```
+
+So the safe claim is:
+
+```text
+we provide a tangent-operator estimator for finite-time FA/BP gap
+```
+
+not:
+
+```text
+architecture alone predicts final FA/BP loss
+```
+
+To make it fully theorem-level, we would need a lazy-training error bound:
+
+```text
+linearized predicted residual - true residual
+```
+
+under assumptions on step size, operator drift, and smoothness.
+
+## 5. Distributional Empirical Validation
+
+This is experiment, not proof.
+
+What is currently strong:
+
+```text
+static matrix alignment follows the Beta law
+time-varying tangent-operator estimator matches empirical trajectory-gap
+distributions well across tested settings
+```
+
+What failed and should not be hidden:
+
+```text
+hard-k random-subspace gap prediction overpredicts real FA gaps
+```
+
+That failure is useful because it tells us:
+
+```text
+raw matrix-constraint count is not the effective FA burden
+```
+
+Current safe empirical story:
+
+```text
+the exact random-geometry theory validates the capacity baseline;
+the tangent-operator estimator validates the finite-time gap quantification;
+the hard-k null model is a baseline that real FA does not equal.
+```
+
+## What Is Actually Fitted?
+
+In the current clean version:
+
+```text
+no post-hoc scalar scale like 0.85 should appear
+```
+
+The strong estimator plots are not fitted in that sense. They are conditional
+predictions using early operator observations. That is acceptable if labeled
+clearly as an estimator.
+
+The thing we must avoid is presenting a conditional estimator as:
+
+```text
+architecture-only theory
+```
+
+or presenting the random-subspace null model as:
+
+```text
+actual FA theorem
+```
+
+## Current Main-Paper Structure
+
+The paper can honestly claim:
+
+1. exact distributional capacity formalization for random feedback alignment;
+2. exact log-volume scaling and a clean random-subspace theorem showing soft
+   nonzero capacity erosion;
+3. exact prior-free minimax initialization bound;
+4. a tangent-operator estimator for finite-time FA/BP gap;
+5. empirical distributional validation of the exact laws and estimator.
+
+The current unresolved theoretical gap:
+
+```text
+derive k_eff or e_t distribution for actual FA directly from architecture and
+feedback initialization, rather than measuring early tangent operators.
+```
+
+Until that is solved, contribution 2 should not overpromise exact FA gap
+prediction, and contribution 4 should be the quantitative gap tool.