From d841aa67aea64ef735c29e063829f36dc3c5d7de Mon Sep 17 00:00:00 2001 From: YurenHao0426 Date: Wed, 22 Jul 2026 12:52:05 -0500 Subject: algorithm: estimate causal credit in vectorizer space --- THEORY.md | 56 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 56 insertions(+) (limited to 'THEORY.md') diff --git a/THEORY.md b/THEORY.md index 8d20659..676d28e 100644 --- a/THEORY.md +++ b/THEORY.md @@ -132,6 +132,62 @@ unrepresentable spatial components. It remains a variance reduction, not a guarantee of useful feedback or stable end-to-end optimization; those require a separately frozen empirical gate. +### Perturbing the vectorizer parameter subspace + +The channel-moment estimator still samples one independent base/gate +coefficient for every example and channel and only afterward regresses those +noisy coefficients on the output context. The quantity the local delta rule +actually needs is much smaller. For + +```text +p_ics = (A c_i)_c + t_ics (G c_i)_c +``` + +define the normalized prediction objective + +```text +J(A,G) = (1 / 2B) sum_i mean_s ||p_i - q_i||^2, +q_i = -d ell_i / dh_i. +``` + +Its causal target terms are matrices, not per-example coefficient fields: + +```text +U_A = (1/B) sum_i mean_s(q_ics) c_i^T, +U_G = (1/B) sum_i mean_s(q_ics t_ics) c_i^T. +``` + +Draw independent Rademacher matrices `Xi_A,Xi_G` with the same shapes as +`A,G`, independently across layers, and intervene along + +```text +d_ics = ((Xi_A c_i)_c + t_ics (Xi_G c_i)_c) / sqrt(2). +``` + +Let `D` be the centered derivative of the summed minibatch loss under all +layer interventions. Since `D=-sum_ics q_ics d_ics`, independence gives + +```text +Uhat_A = -sqrt(2) D Xi_A / (B S), E[Uhat_A] = U_A, +Uhat_G = -sqrt(2) D Xi_G / (B S), E[Uhat_G] = U_G. +``` + +Cross-layer and A/G cross terms vanish in expectation. The prediction terms +`(1/B) sum mean(p)c^T` and `(1/B) sum mean(pt)c^T` are locally available +exactly, so subtracting them from `Uhat_A,Uhat_G` gives an unbiased stochastic +gradient of `J`. BatchNorm changes neither identity: `D` is explicitly the +summed minibatch derivative and therefore includes its true cross-example +Jacobian. + +This estimator perturbs only the shared mapping that will survive +amortization. It uses the same two loss queries per direction as the earlier +estimators. It does not uniformly dominate them for every context covariance +or batch size, but at the frozen batch-128 synthetic audit its one-direction +MSE is `0.03381x` the coefficient-then-regress estimator's MSE. The executable +mean has cosine `0.96928` and norm ratio `1.04888` to the exact A/G target. +Those mechanics do not establish task utility; a separately frozen gate is +still required. + ## 2. What is sufficient for a descent step Flatten all forward parameters into `theta`, let `p = grad L(theta)`, and let -- cgit v1.2.3