diff options
Diffstat (limited to 'THEORY.md')
| -rw-r--r-- | THEORY.md | 56 |
1 files changed, 56 insertions, 0 deletions
@@ -132,6 +132,62 @@ unrepresentable spatial components. It remains a variance reduction, not a guarantee of useful feedback or stable end-to-end optimization; those require a separately frozen empirical gate. +### Perturbing the vectorizer parameter subspace + +The channel-moment estimator still samples one independent base/gate +coefficient for every example and channel and only afterward regresses those +noisy coefficients on the output context. The quantity the local delta rule +actually needs is much smaller. For + +```text +p_ics = (A c_i)_c + t_ics (G c_i)_c +``` + +define the normalized prediction objective + +```text +J(A,G) = (1 / 2B) sum_i mean_s ||p_i - q_i||^2, +q_i = -d ell_i / dh_i. +``` + +Its causal target terms are matrices, not per-example coefficient fields: + +```text +U_A = (1/B) sum_i mean_s(q_ics) c_i^T, +U_G = (1/B) sum_i mean_s(q_ics t_ics) c_i^T. +``` + +Draw independent Rademacher matrices `Xi_A,Xi_G` with the same shapes as +`A,G`, independently across layers, and intervene along + +```text +d_ics = ((Xi_A c_i)_c + t_ics (Xi_G c_i)_c) / sqrt(2). +``` + +Let `D` be the centered derivative of the summed minibatch loss under all +layer interventions. Since `D=-sum_ics q_ics d_ics`, independence gives + +```text +Uhat_A = -sqrt(2) D Xi_A / (B S), E[Uhat_A] = U_A, +Uhat_G = -sqrt(2) D Xi_G / (B S), E[Uhat_G] = U_G. +``` + +Cross-layer and A/G cross terms vanish in expectation. The prediction terms +`(1/B) sum mean(p)c^T` and `(1/B) sum mean(pt)c^T` are locally available +exactly, so subtracting them from `Uhat_A,Uhat_G` gives an unbiased stochastic +gradient of `J`. BatchNorm changes neither identity: `D` is explicitly the +summed minibatch derivative and therefore includes its true cross-example +Jacobian. + +This estimator perturbs only the shared mapping that will survive +amortization. It uses the same two loss queries per direction as the earlier +estimators. It does not uniformly dominate them for every context covariance +or batch size, but at the frozen batch-128 synthetic audit its one-direction +MSE is `0.03381x` the coefficient-then-regress estimator's MSE. The executable +mean has cosine `0.96928` and norm ratio `1.04888` to the exact A/G target. +Those mechanics do not establish task utility; a separately frozen gate is +still required. + ## 2. What is sufficient for a descent step Flatten all forward parameters into `theta`, let `p = grad L(theta)`, and let |