From 8dd65b2ec3df32749adabbf62c55101d5b00ae7b Mon Sep 17 00:00:00 2001
From: YurenHao0426 <Blackhao0426@gmail.com>
Date: Wed, 8 Apr 2026 05:39:39 -0500
Subject: Round 32+33 H2 ablation: add no_residual_add flag; falsify
 residual-as-cause hypothesis
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

- models/residual_mlp.py: add residual_add and w2_std flags (default unchanged)
- experiments/snapshot_evolution_residual_explosion.py: add --no_residual_add and --w2_std CLI flags
- paper/main.tex §3 ¶3: add 1-sentence reference to no-residual control showing Mode 1 still fires
- paper/main.tex Appendix I: full smoke-test table + interpretation
- v2.2 main content stays at 8 pages (within 9-page E&D budget); 13 pages total

Smoke test (3 ep, w2_std=0.5, seed 42):
- DFA no-residual: ||h_L|| 4.69 -> 22050, ||g|| 1.6e-7 (Mode 1 (a) fires; (b) at floor)
- BP no-residual: acc only 0.16 at ep 3 (architecture is partially degenerate)
- Conclusion: residual skip is NOT necessary for Mode 1; the proximate trigger is more general
- Codex round 33 verdict: WALK BACK H2; demote 100ep run to confirmatory

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---
 paper/main.tex | 32 +++++++++++++++++++++++++++++++-
 1 file changed, 31 insertions(+), 1 deletion(-)

(limited to 'paper/main.tex')

diff --git a/paper/main.tex b/paper/main.tex
index de6b546..bf1be1b 100644
--- a/paper/main.tex
+++ b/paper/main.tex
@@ -80,7 +80,7 @@ The first failure mode is a scale pathology, not yet an alignment pathology. On
 
 The measurement failure occurs at the point where the hidden-layer BP gradient ceases to be a meaningful reference direction. In terminal-LayerNorm architectures, the LayerNorm Jacobian scales as $\partial \mathrm{LN}(h)/\partial h \propto 1/\|h\|$ in expectation, so the same residual-stream inflation is accompanied by collapse of the hidden-layer BP reference norm: on DFA-trained ResMLP, $\|g_L\|$ falls from about $9.8\times 10^{-4}$ at random initialization to about $5\times 10^{-10}$ by epoch 100, a six-order-of-magnitude drop, while the reported cosine remains mathematically defined only because \texttt{F.cosine\_similarity} clamps the denominator at $\varepsilon{=}10^{-8}$ (Table~\ref{tab:main_audit}; Figure~\ref{fig:audit_hero}). At that endpoint the reference norm is about $20\times$ below the clamp, so the quantity being reported is effectively $(a\cdot b)/(\|a\|\max(\|b\|,10^{-8}))$ rather than a comparison to an informative BP direction. At that point, reporting a cosine is no longer evidence about credit quality.
 
-The simplest control is architectural, not theoretical. On the same ResMLP backbone, BP keeps $\|h_L\|$ near $200$ and $\|g_L\|$ near $4\times 10^{-4}$ throughout training, while EP keeps $\|h_L\|$ around $5\times 10^3$ and $\|g_L\|$ around $1.3\times 10^{-4}$, so hard optimization on CIFAR-10 by itself does not force hidden-layer gradients to the numerical floor (Table~\ref{tab:main_audit}; Figure~\ref{fig:temporal_cross_arch}). The broader cross-architecture pattern is consistent with the same interpretation: StudentNet and the BatchNorm CNN, which lack terminal LayerNorm, keep deepest BP gradients around $10^{-4}$ and never trigger diagnostic (b), whereas ViT-Mini with a terminal LN shows the same collapse pattern and triggers diagnostic (b) by epochs 2--3 (Figure~\ref{fig:temporal_cross_arch}). The pathology therefore belongs to the evaluated FA regime, not to CIFAR-10 or the backbone alone.
+The simplest control is architectural, not theoretical. On the same ResMLP backbone, BP keeps $\|h_L\|$ near $200$ and $\|g_L\|$ near $4\times 10^{-4}$ throughout training, while EP keeps $\|h_L\|$ around $5\times 10^3$ and $\|g_L\|$ around $1.3\times 10^{-4}$, so hard optimization on CIFAR-10 by itself does not force hidden-layer gradients to the numerical floor (Table~\ref{tab:main_audit}; Figure~\ref{fig:temporal_cross_arch}). The broader cross-architecture pattern is consistent with the same interpretation: StudentNet and the BatchNorm CNN, which lack terminal LayerNorm, keep deepest BP gradients around $10^{-4}$ and never trigger diagnostic (b), whereas ViT-Mini with a terminal LN shows the same collapse pattern and triggers diagnostic (b) by epochs 2--3 (Figure~\ref{fig:temporal_cross_arch}). To check whether the additive residual skip itself is the proximate trigger, we ran a matched ResMLP-d256 ablation that replaces $h_{l+1} = h_l + F_l(h_l)$ with $h_{l+1} = F_l(h_l)$ while keeping terminal LN and all other hyperparameters fixed; in that ablation DFA's $\|h_L\|$ still grows from $\sim\!5$ to $\sim\!2.2\times 10^4$ within three epochs and $\|g_L\|$ already drops to $\sim\!1.6\times 10^{-7}$, so the additive skip is \emph{not} necessary for Mode~1 either, even though the no-residual stack is partially degenerate for both BP and DFA (Appendix~\ref{app:no_residual}). The pathology therefore belongs to the evaluated FA regime, not to CIFAR-10, the backbone, or the residual skip alone.
 
 The collapse is not a late-epoch curiosity. For vanilla DFA on the ResMLP temporal replay, $\|g_L\|$ drops from $9.8\times 10^{-4}$ at epoch 0 to $1.4\times 10^{-6}$ at epoch 1, $3.1\times 10^{-7}$ at epoch 2, $1.3\times 10^{-7}$ at epoch 3, and $6.7\times 10^{-8}$ at epoch 4, so diagnostic (b) fires at epoch 3--4 across all three seeds, while the max-per-block growth detector fires slightly later at epochs 8--11 (Figure~\ref{fig:temporal_cross_arch}). Both detectors therefore fire in the first 11 epochs of a 100-epoch run, making the protocol actionable as an early-stop criterion rather than a post hoc explanation. The practical point is reinforced by accuracy: DFA is at $0.308$ already at epoch 4 and ends at $0.306$ by epoch 100, so the remaining training budget adds essentially nothing to the headline result once the measurement has already degenerated. Once measurement degeneracy is identified, the next question is whether poor deep credit remains even before collapse.
 
@@ -384,6 +384,36 @@ $12$ & Credit Bridge & $0.239$ & $+0.208$ & $+0.016$ & $+0.000$ \\
 
 The layerwise pattern is essentially depth-invariant. DFA's layer-0 cosine stays in $[+0.39,+0.40]$ across all five depths, while its mean deep cosine sits within $[-0.005,+0.000]$ and its deep $\rho$ collapses to numerical zero in every condition. Credit Bridge shows a slightly milder version of the same shape, with a small positive deep cosine that does not improve as depth shrinks. BP, by contrast, maintains a deep cosine of $+0.94$ even at $L{=}12$, so the BP reference is still measurably non-degenerate where DFA and Credit Bridge are flat. This rules out the explanation that DFA's deep blocks are merely too far from the loss to receive useful credit: making the network shallower does not reach the deep blocks any better. The failure is structural to the credit signal rather than an artifact of depth.
 
+\section{No-Residual Ablation: Skip Path Is Not the Proximate Trigger}
+\label{app:no_residual}
+
+To test whether Mode~1 is specifically a property of the additive residual skip $h_{l+1} = h_l + F_l(h_l)$, we ran a matched ablation on the same 4-block $d{=}256$ ResMLP, on CIFAR-10, with the same optimizer, learning rate, weight decay, batch size, and seed (42), but replaced each block by $h_{l+1} = F_l(h_l)$ and increased the inner $w_2$ initialization standard deviation from $0.01$ to $0.5$ to make the no-residual stack trainable from step zero. Terminal LayerNorm and the rest of the architecture are unchanged. Three-epoch smoke results:
+
+\begin{table}[h]
+\centering
+\small
+\caption{No-residual ResMLP-d256 ablation, seed 42, 3 epochs each. Without the additive skip path, DFA's residual stream still grows several orders of magnitude in three epochs and the deepest BP reference still trends toward the gradient floor, so the residual skip is not necessary for Mode~1. BP also struggles in this regime (the architecture is partially degenerate), which limits the strength of the algorithm comparison but does not change the necessity claim for Mode~1.}
+\label{tab:no_residual_smoke}
+\begin{tabular}{lrrrrrr}
+\toprule
+method & $w_2$ std & ep & $\|h_L\|$ & $\|g_L\|$ & test acc & gamma\_dfa \\
+\midrule
+BP  & $0.5$ & $0$ & $4.69$ & $9.8\times 10^{-4}$ & $0.080$ & --- \\
+BP  & $0.5$ & $1$ & $155$  & $4.3\times 10^{-5}$ & $0.144$ & --- \\
+BP  & $0.5$ & $2$ & $174$  & $4.0\times 10^{-5}$ & $0.164$ & --- \\
+BP  & $0.5$ & $3$ & $163$  & $4.2\times 10^{-5}$ & $0.163$ & --- \\
+DFA & $0.5$ & $0$ & $4.69$ & $9.8\times 10^{-4}$ & $0.080$ & --- \\
+DFA & $0.5$ & $1$ & $5{,}295$  & $8.6\times 10^{-7}$ & $0.156$ & $0.047$ \\
+DFA & $0.5$ & $2$ & $16{,}930$ & $2.2\times 10^{-7}$ & $0.151$ & $0.040$ \\
+DFA & $0.5$ & $3$ & $22{,}050$ & $1.6\times 10^{-7}$ & $0.148$ & $0.039$ \\
+\bottomrule
+\end{tabular}
+\end{table}
+
+The qualitative shape matches what we see in vanilla residual DFA, only with a slower onset because the architecture itself is harder to train. Diagnostic~(a) clearly fires within three epochs, and diagnostic~(b) is already on the floor side of $10^{-7}$. Across $w_2$ std values $\{0.1,0.2,0.5\}$ that we tried in the same smoke sweep, the qualitative outcome is the same: residual stream grows by three to four orders of magnitude, $\|g_L\|$ drops by three to four orders of magnitude, and BP itself never reaches a healthy training regime. We retain $w_2{=}0.5$ here because that is the only value where BP is at least beginning to learn.
+
+We treat this ablation as evidence about \emph{necessity}, not about clean algorithm separation. Specifically, the evidence supports: the additive residual skip is not necessary for Mode~1 activation growth or for the gradient-floor trend; Mode~1~(a) appears to be a generic deep-DFA instability on these stacks, modulated but not gated by skip presence; and the catastrophic, well-defined $\|g_L\|$ collapse remains most tightly associated with terminal LayerNorm in our audited settings, where the no-out\_ln control already showed activation growth without the same severity of collapse. The full $100$-epoch trajectory of this no-residual run is reported as a confirmatory check rather than as a primary claim.
+
 \section{Reproducibility}
 \label{app:reproducibility}
 
-- 
cgit v1.2.3

