path: root/notes/28_contribution_status_proof_vs_fit.md

# Contribution Status: Proof vs Estimator vs Fit

This note separates the five current contributions by epistemic status.

The key distinction:

```text
proof      = follows from stated assumptions, no empirical tuning
estimator  = derived from a model/linearization and uses observed quantities
fit        = post-hoc parameter tuning to match data
```

Our current project should avoid calling every component a theorem.

## Summary Table

| contribution | current status | fitted? | safe main-paper phrasing |
|---|---|---:|---|
| 1. capacity formalization | definition + exact random-geometry theorem | no | exact under isotropic feedback assumptions |
| 2. scaling and soft erosion | exact scaling + exact random-subspace null model; real FA extension not yet proved | no, but partly only a null model | capacity cost scales exactly; random-subspace erosion gives a baseline, while real FA needs `k_eff` |
| 3. prior-free minimax bound | exact theorem | no | minimax optimality for prior-free angular alignment |
| 4. tangent-operator estimator | first-order tangent derivation + conditional predictor | no scalar fit, but uses early observed operators | estimator for finite-time gap, not architecture-only theorem |
| 5. empirical distribution validation | experiment | no | validates which theoretical distributions and estimators match trajectories |

## 1. Capacity Formalization

What is proof-level:

```text
Q_l = cos^2(W_{l+1}^T, B_l)
```

If the normalized matrix directions are independent isotropic directions in
dimension `D_l`, then:

```text
Q_l ~ Beta(1/2, (D_l - 1)/2)
```

This is an exact spherical-geometry theorem, not a fit.

The log-volume capacity cost:

```text
C_l(q_l) = -log P(Q_l >= q_l)
```

is exact once this definition of capacity is chosen.

What is not yet covered:

```text
training-time learned alignment
ReLU gate-dependent effective dimension beyond simple assumptions
final task loss
```

Those require later operator dynamics or experiments.

## 2. Scaling and Soft Capacity Erosion

What is proof-level:

For independent layer blocks and fixed thresholds:

```text
p_all = product_l p_l
C_all = sum_l C_l
```

So raw feasible volume multiplies and log-capacity cost adds. For equal-width
MLPs:

```text
C_all = Theta(L n^2)
p_all = exp[-Theta(L n^2)]
```

This is a direct consequence of the capacity definition and architecture
dimensions. It is not a fit.

The random-subspace erosion theorem is also proof-level under its own null
model:

```text
P = parameter dimension
k = random constrained-subspace dimension
J = task Jacobian
K_BP = J J^T
Q = retained projection after removing a Haar-random k-dimensional subspace
K_Q = J Q J^T
```

Then:

```text
E[K_Q | J] = (1 - k/P) K_BP
```

and for residual direction `r`:

```text
e(r) = 1 - (r^T K_Q r) / (r^T K_BP r)
e(r) ~ Beta(k/2, (P-k)/2)
E[e(r)] = k/P
```

This is theory, not empirical fitting.

Important caveat:

```text
actual FA is not literally Haar-random deletion of k parameter dimensions
```

The hard `k` version overpredicts empirical FA gaps. Our experiments already
showed that raw hard constraint count is too pessimistic.

Therefore the safe claim is:

```text
random-subspace theory proves the no-free-capacity mechanism in a clean null
model; real FA should be described by an effective burden k_eff or direct
operator erosion e_t, not by raw hard k.
```

Unsafe claim:

```text
hard-k random-subspace formula exactly predicts real FA/BP gap
```

This is false in our current experiments.

## 3. Prior-Free Minimax Initialization Bound

This is proof-level.

Let:

```text
b = normalized feedback direction
a = unknown normalized target direction
M_mu = E[b b^T]
trace(M_mu) = 1
```

Then:

```text
inf_a E_mu[(a^T b)^2] = lambda_min(M_mu) <= 1/D
```

So:

```text
sup_mu inf_a E_mu[(a^T b)^2] = 1/D
```

Isotropic feedback reaches the bound.

This is not fitted.

Scope:

```text
prior-free angular squared alignment
```

It does not say isotropic feedback is best for:

```text
conditioning
hardware cost
sparsity
low-rank efficiency
task-aware priors
```

## 4. Tangent-Operator Loss/Gap Estimator

This is not a fitted curve, but it is also not an architecture-only theorem.

The local derivation is standard first-order residual dynamics:

```text
r_next_BP ≈ r - eta K_BP r / N
K_BP = J J^T

r_next_FA ≈ r - eta K_FA r / N
K_FA = J J_tilde^T
```

Since FA gives a non-symmetric operator, the one-step loss decrease depends on:

```text
S_FA = (K_FA + K_FA^T) / 2
```

The finite-time estimator propagates these operators:

```text
r_T_BP_hat = product_t (I - eta K_BP,t / N) r_0
r_T_FA_hat = product_t (I - eta K_FA,t / N) r_0
gap_T_hat = (||r_T_FA_hat||^2 - ||r_T_BP_hat||^2) / (2N)
```

In our best current version, `K_t` is predicted from early operator motion,
for example by using `K_0` and `K_s`.

This has no post-hoc scalar hyperparameter. But it is conditional on early
observed operator information:

```text
input includes data, initialization, feedback matrices, K_0, and early K_s
```

So the safe claim is:

```text
we provide a tangent-operator estimator for finite-time FA/BP gap
```

not:

```text
architecture alone predicts final FA/BP loss
```

To make it fully theorem-level, we would need a lazy-training error bound:

```text
linearized predicted residual - true residual
```

under assumptions on step size, operator drift, and smoothness.

## 5. Distributional Empirical Validation

This is experiment, not proof.

What is currently strong:

```text
static matrix alignment follows the Beta law
time-varying tangent-operator estimator matches empirical trajectory-gap
distributions well across tested settings
```

What failed and should not be hidden:

```text
hard-k random-subspace gap prediction overpredicts real FA gaps
```

That failure is useful because it tells us:

```text
raw matrix-constraint count is not the effective FA burden
```

Current safe empirical story:

```text
the exact random-geometry theory validates the capacity baseline;
the tangent-operator estimator validates the finite-time gap quantification;
the hard-k null model is a baseline that real FA does not equal.
```

## What Is Actually Fitted?

In the current clean version:

```text
no post-hoc scalar scale like 0.85 should appear
```

The strong estimator plots are not fitted in that sense. They are conditional
predictions using early operator observations. That is acceptable if labeled
clearly as an estimator.

The thing we must avoid is presenting a conditional estimator as:

```text
architecture-only theory
```

or presenting the random-subspace null model as:

```text
actual FA theorem
```

## Current Main-Paper Structure

The paper can honestly claim:

1. exact distributional capacity formalization for random feedback alignment;
2. exact log-volume scaling and a clean random-subspace theorem showing soft
   nonzero capacity erosion;
3. exact prior-free minimax initialization bound;
4. a tangent-operator estimator for finite-time FA/BP gap;
5. empirical distributional validation of the exact laws and estimator.

The current unresolved theoretical gap:

```text
derive k_eff or e_t distribution for actual FA directly from architecture and
feedback initialization, rather than measuring early tangent operators.
```

Until that is solved, contribution 2 should not overpromise exact FA gap
prediction, and contribution 4 should be the quantitative gap tool.