| Age | Commit message (Collapse) | Author |
|---|
|
The paper had 3-seed penalized DFA deep cos at +0.155 ± 0.025, but
re-measuring on the saved checkpoints gives 0.1506 (mean of per-seed
deep means is 0.1507; pooled mean over 12 deep-layer values is 0.1506).
The std of 0.025 matches pooled ddof=1 ✓.
Same paragraph also had inconsistent values: "+0.155 ± 0.025" 3-seed
above, then "+0.165" single-seed s42 in the lambda sweep. Unified to
3-seed throughout.
§5 ¶2 lambda sweep updates:
lam=1e-4 ||h_L|| 2.4e4 (s42 only) → 2.2e4 (3-seed mean)
lam=1e-4 ||g_L|| 6.3e-7 (s42) → 7.0e-7 (3-seed)
lam=1e-4 deep cos -0.022 (s42) → -0.020 (3-seed)
lam=1e-2 deep cos +0.165 (s42) → +0.151 (3-seed, same as the
three-seed value used elsewhere)
Other places updated: §4 ¶4 prose, Table 2, Appendix J Table 9
DFA+pen mean row (deep cos +0.155 → +0.151 and ||h_L||/||g_L||
columns updated to 30-ep dfa_pen_short means rather than the round-19
single-seed numbers), Appendix L paragraph.
Page layout preserved: 9 main pages, refs p10, 18 total, 0 overfull.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
The §5 ¶3 BP-no-penalty value of 0.609 ± 0.004 and DFA-no-penalty value
of 0.308 ± 0.014 turned out to be unsourced — they were carried over
from a hardcoded comment in experiments/bp_with_penalty_control.py
("BP-trainable (3-seed mean): 0.609") that nobody had actually measured
with a matched 30-epoch run.
Ran the missing matched controls under the same recipe as BP+pen
(lam=0, 30 epochs, AdamW 1e-3, wd 0.01, cosine schedule, batch 128,
3 seeds 42/123/456):
BP no-pen 30ep: per-seed 0.5851, 0.5845, 0.5863 → 0.585 ± 0.001
(paper said 0.609 ± 0.004, off by 0.024)
DFA no-pen 30ep: per-seed 0.3070, 0.2985, 0.2966 → 0.301 ± 0.005
(paper said 0.308 ± 0.014)
Also re-grounded DFA+penalty 30ep using the dfa_pen_short 3-seed run
(0.3593, 0.3610, 0.3604 → 0.360 ± 0.001), which is what the deep-cosine
+0.155 figure was computed on. The paper had 0.363 ± 0.001 — that came
from the 100-epoch run, not the 30-epoch run, so it was an apples-to-
oranges comparison with BP+pen 30-ep.
Paper changes (§5 ¶3):
BP penalty cost: -8 pp → -5.5 pp
DFA pen rescue: +5.5 → +5.9 pp
DFA+pen margin vs frozen: +1.4 → +1.1 pp
BP-to-DFA gap: 17 → 17.0 pp (unchanged)
BP-to-SB gap: 7.7 → 7.7 pp (unchanged)
BP-to-DFA gap is still the lower-bound credit-quality cost claim;
17 pp gap is unchanged in magnitude.
Also updated:
- §5 ¶1 prose: 0.363 → 0.360, 0.308 → 0.301
- §4 ¶4 prose: DFA+pen 0.363 → 0.360
- Appendix J Table 9 caption: 0.363 → 0.360, +9.0 → +9.3 pp gap to SB
- Appendix L paragraph: +5.5 → +5.9 pp DFA penalty rescue
- Figure 3 panel C bar values + title pen-cost annotation
- New results/matched_30ep_control_summary.json as auditable record
Page layout preserved: 9 main pages + refs p10, 18 total, 0 overfull.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
Paragraph framing says "3 seeds" but the endpoint values were
actually s42-specific. Re-aggregated from
results/snapshot_evolution_v2/snapshot_evolution_s{42,123,456}.json:
||h_L|| final per seed: 4.39e8, 3.86e8, 7.30e8 → mean 5.18e8 (paper now: 5e8, was 4e8)
||g_L|| final per seed: 4.86e-10, 3.76e-10, 2.67e-10 → mean 3.76e-10 (paper now: 4e-10, was 5e-10)
The §5 ¶1 intervention paragraph still says "‖h_L‖ ~4.4e8" — that
one is explicitly the s42 vanilla DFA endpoint that Figure 3 panel (a)
plots, so it stays single-seed. §3 is the 3-seed version.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
Fresh re-measurement on the saved early checkpoints
(per_layer_cos_3seed.json) gives ||g_2|| at ep 1:
s42: 6.79e-7 → paper rounds 6.8e-7
s123: 6.57e-7 → paper rounds 6.6e-7
s456: 3.85e-7 → paper rounds 3.8e-7
The previous prose values (6.7, 6.5, 3.9) were carried over from
ad-hoc measurements with inconsistent rounding (3.9 was an error;
3.85 rounds to 3.8). All three values are still well above the
1e-7 diagnostic-(b) threshold, so the §4 ¶1 mode-2-without-mode-1
claim is unchanged.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
Found a numerical error in §4 ¶3: the layer-0 vanilla DFA cosines were
listed as +0.42, +0.45, +0.39 across seeds 42/123/456 but the actual
re-measurement on the saved early-epoch checkpoints gives +0.421, +0.436,
+0.418 (the s456 value was off by 0.03). The deep-mean numbers in
Table 2 (-0.008 ± 0.013) were already correct.
Changes:
- §4 ¶3: layer-0 trio updated to +0.42, +0.44, +0.42 across seeds and
cite now points to a new per-seed appendix.
- New Appendix M (Layer-0 Dominance): 6-row table of per-seed per-layer
cosines on vanilla DFA early checkpoints (3 seeds × ep 1, 2), with
per-layer ||g||. Documents the layer-0 dominance pattern that drives
the headline aggregate Γ on these checkpoints.
- results/vanilla_dfa_early_ckpts/per_layer_cos_3seed.json: machine-
readable dump of all 6 measurements for future audit.
- §7 compressed (~30 words trimmed across the closing paragraph) and
Figure 3 width 0.92 → 0.82 to keep main content at exactly 9 pages
after the appendix addition.
Verified: 9 pages main + refs on p10, 18 total, 0 overfull boxes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
§1 ¶1 referenced "equilibrium propagation" without a bibitem despite EP
being the trustworthy non-BP control throughout the paper. Added the
canonical Scellier & Bengio 2017 Frontiers in Computational Neuroscience
reference and cited it where EP is first named in the FA-first intro.
Main content stays at 9 pages (§7 closes mid-p9, refs start p10);
0 overfull boxes; 18 pages total.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
After moving Figure 5 from §6 to Appendix D in v2.23, §6 ¶2 still said
'Figure~\ref{fig:decision_utility} makes the decision value explicit'
which would render as 'Figure 5 makes...' but Figure 5 is now in the
appendix. Reader on p8 looking for Figure 5 nearby would not find it.
Added explicit '(Appendix~\ref{app:all_validations})' parenthetical
right after the figure ref so the reader knows where to look.
Audit of all other figure refs (Figures 1-4 in main text):
- fig:audit_hero (Figure 1, §2) → refs in §1/§2 main text ✓
- fig:temporal_cross_arch (Figure 2, §5) → refs in §3/§5 ✓
- fig:penalty_rescue (Figure 3, §5) → refs in §4/§5/§7 ✓
- fig:cross_arch_summary (Figure 4, §5) → refs in §5/§6 ✓
All clean. Main content still 9 pages.
|
|
formula + threshold pointer + hyperparameter fairness clause
Addressed 4 secondary reviewer concerns from the user's earlier list, all
small inline additions:
1. §3 ¶1 LN Jacobian: extended the 1-line claim into a 2-line derivation.
For y = LN(h) = (h-μ)/σ with σ ∝ ||h||/√d, ||∂y/∂h|| = Θ(1/σ),
so ||g_L|| = Θ(1/||h_L||). Connects the (a) growth and (b) collapse
formally.
2. §4 ¶2 ρ formal definition: added the inline formula
ρ_l = Pearson(<a_l, εv>, ℓ(h_l + εv) - ℓ(h_l)) over M=32 random
unit-norm directions v with ε=1e-3, evaluated per sample on a fixed
eval batch and averaged. Previously this was narrative-only.
3. §6 ¶3 diagnostic (c) cross-batch stability: added inline definition as
the mean pairwise cosine of per-batch-averaged BP-grad direction at
the chosen layer across K≥8 disjoint 128-sample minibatches, with
the empirical separation (drift 0.5-0.99 vs healthy 0.05-0.18).
4. §6 ¶3 threshold sensitivity pointer: added (Appendix~\ref{app:threshold_sweep})
pointer next to the (a)/(b) calibration claim.
5. §2 ¶1 hyperparameter fairness: changed 'against the same architecture,
optimizer, and training budget' to 'against the identical architecture,
optimizer, schedule, and training budget without method-specific tuning'
and added 'batch size 128'. Closes the 'fairness asserted but not
evidenced' reviewer concern.
Page budget: each addition ate ~1-2 lines. Net push was ~9 lines, which
spilled §7 onto p10. Recovered by:
- Shrinking Figure 3 (penalty rescue) from \linewidth to 0.92\linewidth
- Shrinking Figure 4 (cross_arch_summary) from 0.78\linewidth to 0.7\linewidth
- Compressing diagnostic (c) clause (kept the formula intent without all
the LaTeX math symbols inline)
- Trimming §7 closing sentence: dropped 'main lesson is to decompose'
preamble; merged 'a reporting rule' phrase into the same sentence as
the methodology-line citations
Result: §1-§7 + all figures fit strictly in 9 pages (verified via pdftotext;
p9 ends with §7 closing sentence + page number '9'; p10 starts with
References). Total 18 pages, 0 overfull hbox.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
User flagged that intro entered the critique too fast without first
explaining what feedback alignment is. Based on Bartunov et al. 2018's
intro structure (their Section 1 opens with the weight-transport problem,
introduces FA as the response, and only then motivates the evaluation
question), rewrote §1 paragraph 1.
New §1 ¶1:
- BP is standard but biologically implausible (weight-transport problem)
- FA (Lillicrap 2016) side-steps via random feedback
- DFA (Nokland 2016) simplifies by direct projection per layer
- Parallel lines: target propagation (Lee 2015), equilibrium propagation
- Modern scaling: Launay 2020 (transformers), Akrout 2019
- Evaluation converged on accuracy + Gamma cosine summary
§1 ¶2 (old ¶1) then starts the audit critique against this backdrop, so
a reader who arrived without any FA context now has one paragraph of
set-up before the critique begins.
Page-budget side effect: the ~110-word addition pushed main content to
10 pages briefly. Recovered by shrinking Figure 4 (cross_arch_summary)
from width=\linewidth to width=0.78\linewidth, which freed enough
p9 vertical space for §7 to fit entirely on p9.
Result: main content strictly 9 pages (§1-§7 on p1-p9, references and
appendices on p10+). Total 18 pages. 0 overfull hbox.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
Semantic Scholar check)
|
|
While user runs Semantic Scholar verification, I WebSearch-verified the 4
citations I flagged as 'never independently checked' and found one real bug
plus opportunity to expand others:
- Xiong 2020: second author was 'Yunchang Yu' in my bibitem, but the actual
author is 'Yunchang YANG'. Fixed. Also expanded to the full 10-author list.
- Paleka 2026: expanded 'Daniel Paleka et al.' -> 'Daniel Paleka, Shashwat
Goel, Jonas Geiping, and Florian Tramèr'. Title/venue confirmed correct.
- O'Bray 2022: expanded 'Leslie O'Bray et al.' -> 'Leslie O'Bray, Max Horn,
Bastian Rieck, and Karsten M. Borgwardt'. Title/venue confirmed correct.
- Jordan 2020: expanded 'Scott M. Jordan et al.' -> 'Scott Jordan, Yash
Chandak, Daniel Cohen, Mengxue Zhang, and Philip Thomas'. Also dropped
incorrect middle initial 'M.' Title/venue confirmed correct.
All 4 citations now have full verified author lists. The Yang/Yu typo was
a real factual error that Semantic Scholar would have caught.
Main content still 9 pages. Task list unchanged.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
User pushed back on SB/CB being treated as 'audited FA methods' because
they're our own constructions. Reframe them as diagnostic probes built on
two prior-literature assumptions (state=credit and credit=performance).
§1 intro: add 1 sentence clarifying BP/EP/DFA are established baselines and
SB/CB are probes constructed in this paper.
§2 ¶2 new opening paragraph (before 'By the field's usual criteria'):
- SB/CB are probes, not prior FA variants
- Each directly learns a target from a prior-literature view
- SB: target-propagation view (Bengio 2014, Lee 2015) — auxiliary G_ψ(h_l,t_l,s)
predicts h_L via MSE; a_l^SB = ∇_{h_l} CE(W_out LN(G_ψ(h_l,t_l,s)), y)
- CB: synthetic-gradient view (Jaderberg 2017) — auxiliary V_φ(h_l,t_l,s)
trained via bridge residual; a_l^CB = ∇_{h_l} V_φ(h_l,t_l,s)
- Both auxiliaries trained on detached hidden states
- Role: populate different points in the (angular alignment, functional
usefulness) plane, making the §4 cos-vs-acc dissociation visible
Bibliography: added Bengio 2014 (arXiv 1407.7906), Lee et al. 2015 (ECML
PKDD), Jaderberg et al. 2017 (ICML) — all verified via WebSearch.
Page budget: the ~180-word §2 addition pushed §7 onto p10. Recovered
space by:
(a) compressing §2 ¶1 opening
(b) compressing §3 ¶2 falsification chain (tighter number formatting)
(c) compressing §6 ¶3 asymmetry paragraph
(d) merging §7 into a single paragraph (was 3)
(e) moving Figure 5 (decision_utility) from §6 main text to a floated
appendix figure in Appendix D (the 'all seven validations' appendix,
which is conceptually related). The decision-utility ablation's
headline ('accuracy+Γ walks back 0/5, full protocol walks back 3/5')
is already in §6 prose so the figure functions as supporting backup.
Result: main content is strictly 9 pages (§1-§7 on p1-p9). References and
appendices on p10+. Total 18 pages, 0 overfull hbox.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
Ran DFA+pen via cifar_resmlp.py on s123, s456. 3-seed:
- acc 0.3601 ± 0.0014 (matches existing 0.363 from round 11)
- deep cos +0.1518 ± 0.0110 (matches existing +0.155 ± 0.025)
- deep nudge -5e-5 (was single-seed -6e-5)
- train Δloss 0.095 ± 0.007 (was single-seed 0.104)
- w2 drift 18.6 ± 0.5 (matches single-seed 18.8)
- embed drift 94.6 ± 1.4 (matches single-seed 92.7)
Single-seed s42 was within noise of the 3-seed mean across all 4 functional
metrics, so the cos-vs-acc dissociation story is unchanged — Appendix L
now just reports 3-seed for all 3 methods consistently.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
label overlap (fig4)
Per user feedback:
- fig4_penalty_rescue.pdf (Figure 3 in paper): was figsize=(13, 3.5), aspect 3.7:1,
which rendered as a thin strip with squeezed subplot content. Increased height
to figsize=(13, 6.0), aspect 2.2:1. Much taller panels that actually show axis
labels and legends readably.
- fig5_cross_arch_summary.pdf (Figure 4 in paper): the 'Key finding' italic text
annotation at y=-1.0 in axes transform was overlapping with the multiline
architecture y-tick labels at the bottom of the second subplot. Moved to
y=-1.55 and increased figsize height from 3.5 to 4.2 so the lower annotation
still fits in bbox_inches='tight' crop.
- Also bumped includegraphics width from 0.92\linewidth to \linewidth for both
figures so they use the full text width.
Main content still exactly 9 pages within E&D budget.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
Previous tabularx conversion caused visual problems:
- Table 1: 'lcccc' inside tabularx{\linewidth} did not stretch, leaving a
blank column on the right side of the page
- Table 2: 'p{0.18}LLL' forced originally-single-line cells to wrap into
multiple lines (e.g., 'Vanilla DFA, early epoch' and 'cos_deep=... rho_deep=...'
split across 2 lines each)
- Table 3: 'p{0.06}L p{0.16} p{0.22}' similarly compressed single-line rows
Fix: revert all 3 tables to original plain 'tabular' (Table 1 lcccc,
Table 2 lccc, Table 3 llll) and wrap each in \resizebox{\linewidth}{!}{...}.
This:
- Stretches Table 1 to full text width (no blank right column)
- Shrinks Table 2's originally-wide content uniformly so all rows stay
single-line
- Shrinks Table 3 similarly so (a)/(b)/(c)/(d) rows are single-line
Tables 4-9 (appendix) keep their tabularx treatment since they fit cleanly.
Result: 0 overfull hbox, main content still exactly 9 pages, first 4
references now fit on p9 as well.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
lines to p10)
The two final sentences both started with 'Once...' and were redundant.
Cut the last sentence entirely and tightened the preceding phrase from
'Once the field enforces that separation...' to just stating the claim.
Before: 'That is the sense in which this paper fits the evaluation-methodology line...: the contribution is not a new benchmark artifact, but a reporting rule for preventing a repeatable interpretive error. Once the field enforces that separation between measurement validity and substantive credit quality, positive results will become more trustworthy, negative results more precise, and the apparent evidence for successful deep credit assignment much harder to overstate.'
After: 'That is the sense in which this paper fits the evaluation-methodology line...: the contribution is a reporting rule for preventing a repeatable interpretive error, not a new benchmark artifact.'
§7 now fully fits on p9. Main content exactly 9 pages within E&D budget.
|
|
User flagged Table 2 (mode validation) as overflowing. Root cause: the 'Deep-layer
alignment signal' column had long multi-term cosine+rho expressions under
plain 'lccc' column spec with no width constraint.
Fix:
- Added \usepackage{tabularx} and a raggedright L column type
- Converted all 9 tables from tabular to tabularx{\linewidth}{...}
- Table 1 (main audit): plain lcccc inside tabularx, fits width
- Table 2 (mode validation): first column p{0.18\linewidth}, three wrapping L columns
- Table 3 (protocol def): tight left p{}, wrapping L measurement column, two right p{} columns
- Table 4 (all validations): p{0.18\linewidth} + three L columns
- Tables 5-9 (appendices, numeric): @{\extracolsep{\fill}} with existing lrrr... specs
Also shortened 'DFA+pen mean (3 seeds)' label to 'DFA+pen mean' in Appendix L
table to eliminate a 19.5pt overfull on that row.
Result: 0 overfull hbox warnings (was several), main content still 9 pages
exactly within E&D budget, total 17 pages.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
DFA+penalty single seed s42, 30ep via cifar_resmlp.py (not the earlier
dfa_residual_penalty_test.py which doesn't save nudging):
- test acc: 0.3607 (matches existing 3-seed 0.363±0.001)
- deep cos: +0.166 (matches existing 3-seed 0.155±0.025)
- deep nudge Δloss (eta=0.01): -6e-5 (smallest)
- trajectory loss decrease: 0.104 (smallest)
Full 3-method comparison at 30 epochs:
DFA+pen SB+pen CB+pen
test acc 0.361 0.453 0.360
deep cos +0.166 +0.322 +0.684
deep nudge -6e-5 -1.78e-3 -0.45e-3
traj Δloss 0.104 0.458 0.122
KEY INSIGHT: Deep cosine ranks methods CB > SB > DFA, but ALL functional
metrics (nudge, trajectory loss decrease, accuracy) rank them SB >> CB ≈ DFA.
Cos is the ONLY ordering that does not predict accuracy correctly.
This is the strongest form of the cos-vs-acc dissociation: the ordering
implied by angular agreement is contradicted by three independent functional
measurements, all of which do predict accuracy.
Appendix L ¶2 updated to report all 3 methods and note the ranking
contradiction. Main content still 9 pages.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
loss decrease
SB+penalty train loss: 2.047 -> 1.589 (Δ=0.458 over 30 epochs)
CB+penalty train loss: 1.996 -> 1.874 (Δ=0.122 over 30 epochs)
Ratio: 3.8x (matching the 4x nudging ratio)
This is the third independent functional measurement, from per-epoch logs
in the same round 38 JSONs (log['train_loss']). The cos-vs-acc
dissociation now has four independent pieces of evidence, all agreeing:
1. Test accuracy: CB 0.360 vs SB 0.453 (9.3pp gap)
2. Parameter drift: CB w2 19.3x vs SB 14.3x (CB larger updates)
3. Single-step nudging Δloss: CB -0.45e-3 vs SB -1.78e-3 (4x gap, eta=0.01)
4. Trajectory loss decrease: CB 0.122 vs SB 0.458 (3.8x gap over 30 epochs)
All four inversely correlated with deep cosine (CB has higher deep cos).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
dissociation
The nudging test values were already in the round 38 JSON under
diag['nudging']['0.01'] but never used. Extracted and added to Appendix L:
SB+penalty: deep nudge delta = -1.78e-3 (per-layer, eta=0.01)
CB+penalty: deep nudge delta = -0.45e-3 (per-layer, eta=0.01)
A single step of size eta=0.01 in each method's credit direction decreases
the test loss by 1.78e-3 (SB) vs 0.45e-3 (CB) — a 4x gap in functional
loss decrease that EXACTLY INVERTS the 4x deep-cosine gap between the
methods. This is the direct functional measurement for the
'angular agreement is not sufficient' claim.
Combined with the drift diagnostic (larger CB updates), the cos-vs-acc
mechanism hypothesis now has THREE independent pieces of support:
1. Test accuracy (headline: CB same as DFA, SB higher)
2. Parameter drift (CB larger updates than SB)
3. Nudging functional loss decrease (CB 4x smaller than SB)
Zero new compute — all from existing round 38 JSON data.
Main content still 9 pages exactly within E&D budget.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
hypothesis
Extracted from existing round 38 JSON data without running new compute.
The drift field (||W_final - W_init||_F / ||W_init||_F) is produced by
cifar_resmlp.py's feature_drift() and was already saved but not used in
the paper.
Key finding: CB+penalty has LARGER block updates than SB+penalty
(per-block w2 drift 19.3x vs 14.3x; embed drift 44.6x vs 7.1x) yet
9.3 pp LOWER accuracy. This rules out 'CB just has smaller updates'
as an alternative explanation for the cos-vs-acc dissociation.
Added 2 sentences to Appendix L paragraph 2 noting this supporting
evidence for the 'angular agreement does not certify functional
forward-state content' mechanism hypothesis in §4.
Main content still 9 pages exactly within E&D budget.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
The paper uses 'EP' throughout but never spelled it out in §1 or §2. Added
first-mention spell-out with a brief 1-line description ('a contrastive
energy-based alternative to BP that updates weights from the difference
between a free-phase and a nudged-phase hidden trajectory') so the reader
has context before EP is used as a key internal comparison.
Main content still 9 pages.
|
|
Appendix L title was 'State Bridge Penalty Rescue: 3-Seed Cross-Method Test'
but the table has both SB and CB rows. Updated to:
'State Bridge and Credit Bridge Penalty Rescue: 3-Seed Cross-Method Test'.
Intro sentence updated to mention re-running both SB and CB, and to note
both baselines were matched.
|
|
All 5 figures are rendered and current; the TODO comments were leftover
notes from when figures were being generated. No functional change — just
cleanup of invisible LaTeX source comments.
|
|
WebSearch verification
Three more bibitems had hallucinated or wrong fields; verified correct versions
against openreview/arxiv/proceedings URLs.
- Moskovitz et al. 2018: was 'In NeurIPS, 2018' -> arXiv preprint 1812.06488
(paper is arxiv-only, never published at NeurIPS). First name Ted -> Theodore.
- Crafton et al. 2019: title was 'Backpropagation through feedback alignment
for deep learning in analog hardware' -> 'Direct feedback alignment with
sparse connections for local learning'. Venue was 'ICASSP' -> 'Frontiers
in Neuroscience, 13:525'. Third author Eric -> Evan Gebhardt.
- Refinetti et al.: year was 2023 -> 2021, 4th author Krzakala -> Goldt,
title was 'Aligning residual pathways: normalization, scale, and feedback
in deep networks' -> 'Align, then memorise: the dynamics of learning with
feedback alignment'. Venue ICML (correct year 2021, not 2023).
All 12 bibitems now verified. Running total:
- Lillicrap 2016, Nokland 2016, Bartunov 2018, Launay 2020 verified via
WebSearch this round (were already correct).
- Xiong 2020 bib sort key cleaned up earlier.
- Akrout 2019 title fixed earlier (Deep feedback control -> Deep learning
without weight transport).
- Paleka 2026, O'Bray 2022, Jordan 2020 titles fixed earlier.
- Moskovitz + Crafton + Refinetti fixed this commit.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
weight transport'
|
|
Previous bibitems had paraphrased/invented titles for the 3 E&D-methodology
exemplar papers cited in §1 and §7. The correct titles are:
- Paleka et al. ICLR 2026: 'Pitfalls in Evaluating Language Model Forecasters'
(not 'Pitfalls in evaluating model behavior: measurement, reporting, and
interpretability failures')
- O'Bray et al. ICLR 2022: 'Evaluation Metrics for Graph Generative Models:
Problems, Pitfalls, and Practical Solutions' (not 'Evaluation beyond
leaderboard metrics: methodology matters')
- Jordan et al. ICML 2020: 'Evaluating the Performance of Reinforcement
Learning Algorithms' (not 'Evaluating machine learning: tests, cases, and
expectations'). Also corrected first author 'Matt' -> 'Scott M.'
Verified against codex round 23 memory which recorded the correct titles
from the OpenReview/ICML URLs. Previous bibitems were hallucinated titles
from earlier rounds and would have been a factual bug in the bibliography.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
|
|
Previous §3 ¶1 wrote the local loss as -<f_l, B_l^T e_T> as if it applied
to DFA, SB, and CB all three. But that's only DFA's form. SB and CB use
learned bridge networks to derive credit:
- DFA: a_l = B_l^T e_T (fixed random projection)
- State Bridge: a_l = gradient of CE(head(LN(G_psi(h_l, t_l, s))), y)
where G_psi is a learned state predictor of h_L
- Credit Bridge: a_l = gradient of learned value net V(h_l, t_l, s)
The fix correctly writes the shared local loss form -<f_l, a_l> and
defines a_l for each method in-line. This also serves as the first
definition of SB and CB in the paper (previously they were named in
Table 1 without being defined).
Main content still ends at p9 (just slightly before the bottom margin
now); references span p9-p10 but are not counted against the 9-page
content budget. Total 17 pages.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
|
|
38-40 findings
Codex round 40 sequencing: A then F. OPTION F (polish) pass found 4 real issues:
1. Abstract: added narrow conditional-positive finding sentence about
SB+penalty beating the shallow baseline + cos-vs-acc dissociation
2. §1 ¶2: replaced 'deep cosine can improve to about +0.16' (DFA-specific)
with a fuller cross-method statement mentioning SB +0.32, CB +0.68, and
the cos-vs-acc dissociation
3. §6 ¶3 (protocol asymmetry): added one sentence noting that the
cross-method cos-vs-acc dissociation REINFORCES the necessity of keeping
all four diagnostics separate
4. §7 ¶2 (limits): upgraded 'terminal-LN interpretation is observational
rather than causal identification' to 'established causally on the
audited residual ResMLP via the matched same-backbone no-terminal-LN
control but not proven to extend beyond that architecture family' —
reflects round 36 wording upgrade based on existing April 7 no_outln data
All four changes are prose-level updates driven by data that was already in
the paper. Main content still fits at 9 pages exactly (E&D limit).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
/ Mechanism hypothesis)
Codex round 40 recommended turning the Mode 2 dissociation from empirical
curiosity into a methodological claim via a 3-part proposition:
1. Observation: CB+pen and DFA+pen reach same acc despite 4x deep-cos gap;
SB+pen best acc with intermediate cos
2. Inference: layerwise BP-cosine is NECESSARY to rule out grossly wrong
credit signals but NOT SUFFICIENT to certify usable credit for depth
3. Mechanism hypothesis: usefulness depends on whether local updates induce
coordinated forward-state change across blocks, not just angular agreement
with BP
Method framing (codex-approved safer versions):
- CB = 'gradient-direction surrogate' (high angular agreement, low functional
credit)
- SB = 'state-level downstream teaching signal' (lower angular agreement,
higher functional credit)
- Explicitly framed as HYPOTHESIS not theorem
Main content still 9 pages exactly (within E&D limit).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
3 fixed-feedback methods
CB+penalty 3-seed (4-block d=256, 30ep, lam=1e-2):
- acc 0.360±0.003 (same as DFA+pen, 9pp below SB+pen)
- ||h_L||=5680±178, ||g_L||=1.9e-5 (HEALTHY)
- layer-0 cos +0.652±0.005
- deep cos +0.679±0.008 (4x DFA+pen, 2x SB+pen)
- deep rho +0.464±0.025 (6x DFA+pen)
Final 3-method rescue comparison:
DFA+pen: acc 0.363, deep cos 0.155, deep rho 0.080
SB+pen: acc 0.453, deep cos 0.322, deep rho 0.402
CB+pen: acc 0.360, deep cos 0.679, deep rho 0.464
Clean cos-vs-acc dissociation:
- CB has 4x higher deep cos than DFA but SAME accuracy
- SB has intermediate deep cos but HIGHEST accuracy
- Alignment to BP gradient is NECESSARY but NOT SUFFICIENT for usable credit
Paper updates:
- §4 ¶4: now includes all 3 methods with numbers, adds 'cos is necessary but not
sufficient' framing based on 3-method dissociation
- Appendix K: adds CB+pen 3-seed rows + vanilla CB baseline for comparison
- Main content still 9 pages exactly (within E&D limit)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
100ep convergence
- Appendix G: depth scan L=4 now 3-seed (s42, s123, s456). 3-seed DFA layer-0 cos
+0.412±0.011, deep cos -0.0004±0.0008, CB deep cos +0.039±0.010. Indistinguishable
from single-seed row shown in table.
- Appendix J: full 100-epoch random_targets trajectory for SB and CB:
SB: ||h_L||=3.6e5, ||g_L||=4e-8 (at floor), acc 0.100 (chance)
CB: ||h_L||=1.38e8, ||g_L||=0 (collapsed), acc 0.085 (chance)
Both reach Mode 1 (a)+(b) at 100ep, consistent with DFA's 1.67e8 / 8e-12.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
|
|
CODEX ROUND 39 VERDICT: PAPER-CHANGING for SB; wait for CB multi-seed for CB claims.
Round 38 3-seed SB+penalty (4-block d=256, 30ep, lam=1e-2):
- acc 0.453±0.003 (BEATS shallow baseline 0.349 by +10.4pp -- FIRST non-BP method)
- ||h_L||=302±8 (contained, not silenced)
- ||g_L||=1.8e-4 (HEALTHY)
- deep cos +0.322±0.007 (2x DFA+pen +0.155)
- deep rho +0.402±0.015 (5x DFA+pen +0.080)
Penalty rescue magnitudes (method-dependent):
- DFA: +5.5 pp (0.306 -> 0.363)
- SB: +24 pp (0.213 -> 0.453)
- CB: +15 pp (single seed, multi-seed in flight)
- BP: -8 pp (capacity cost, 0.609 -> 0.530)
Paper updates:
- §4 ¶4 NEW: Mode 2 has method-dependent severity within fixed-feedback family;
SB+penalty is the first audited non-BP method to substantively use deep blocks
via intervention; deep cos doesn't predict acc across methods (methodological obs)
- §5 ¶3 EXTENDED: BP+penalty -> 3x penalty control (BP, DFA, SB) with all margins
vs frozen-blocks baseline; BP-to-SB gap only 7.7 pp vs BP-to-DFA gap 17 pp
- Appendix K NEW: full SB+penalty 3-seed table with vanilla SB and DFA+pen
comparison
Main content stays at 9 pages exactly (within E&D limit). Total 16 pages.
CB multi-seed (s123, s456) launched in parallel (PIDs 576938, 576939) — claims
deferred until those land.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
validates falsification
3-seed mean: ||h_L||=8.2e7, ||g_L||=1.9e-10
Per-seed: ||h_L|| in {1.06e8, 3.15e7, 1.09e8}, ||g_L|| in {1.08, 2.94, 1.77}e-10
All deeply below the (b) floor, all confirm Mode 1 (a)+(b) fire on no-residual ResMLP+terminal-LN.
Multi-seed H2 falsification of 'residual skip causes Mode 1' is now robust.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
Patches:
- main(): add --penalty_lam (separate from CB's bridge temperature args.lam)
- train_dfa block update (line 195): add penalty_lam * (f_l**2).sum(-1).mean()
- train_state_bridge block update (line 326): same penalty
- train_credit_bridge block update (line 533): same penalty
Codex round 38 GO STAGE: keep penalty separate from CB lam, blocks-only,
sanity-check that hidden_norms remain nontrivial (not silencing the blocks).
2-epoch smoke (results/round38_smoke_sbcb_pen) passes the silencing check:
SB ||h_L||=229, CB ||h_L||=1258, both nontrivial. Deep cosines positive across
all layers for SB ([0.28, 0.25, 0.23]) and rising for CB ([0.04, 0.08, 0.13, 0.15]).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
(matches bibitem)
|
|
table), not 30ep 0.308±0.014
|
|
|
|
per-round narration
Codex round 37 verdict: 'Page budget is the bottleneck, not mechanism uncertainty.'
Mode 1 is mechanism-complete after rounds 32-36. Strict compression rule:
'one claim sentence per falsified alternative, one for the positive mechanism,
everything numeric goes to appendix tables.'
§3 rewrite (4 -> 3 paragraphs):
1. Phenomenon class — 6-line geometric argument inlined as one sentence;
LN Jacobian derivation for (b); empirical anchors for vanilla DFA.
2. Falsification chain — 4 alternative attributions each in one sentence:
not residual-skip-driven (App H), not task-signal-driven (App I),
not DFA-specific (App I), not shared by EP.
3. Positive necessity for (b) — same-backbone no_outln control with full
numbers; cross-architecture support; temporal early-fire result.
Result: main content 9 -> 8 pages (1 page of slack restored).
Total 15 -> 14 pages.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
Two changes from round 36:
1. §3 paragraph 3: replace 'observational association' with full causal claim
based on existing April 7 no-out_ln data (3 seeds, ResMLP-d256+terminal-LN
removed, residual skip kept): ||h_L||=1.21e7 (Mode 1 (a) still fires) but
||g_L||=7.4e-4 (HEALTHY, ~10000x above floor — (b) eliminated). Final acc
0.327±0.013 indistinguishable from vanilla DFA's 0.308±0.014. Wording
upgraded to 'terminal LayerNorm is necessary for Mode 1(b) in the audited
residual ResMLP and ViT-Mini setting'.
2. §3 paragraph after random-target ablation: add EP under random targets
smoke result (||h_L||=586 at ep 5 vs DFA's 14510 at ep 3, 25x gap).
Random-target assay now cleanly separates fixed-feedback methods (explode)
from EP (bounded). Cross-method negative control complete.
- experiments/ep_baseline.py: add --random_targets flag + train_ep parameter
- v2.5 paper compiles to 15 pages, main content 1-9 (right at E&D limit)
Combined picture (rounds 32-36):
- Mode 1 (a) localized to 'fixed-feedback local-credit objectives without
scale control on architectures absorbing scale at output'. Falsified:
residual skip (round 33), task signal (round 34), DFA-specific (round 35).
EP is the working negative control (round 36).
- Mode 1 (b) localized to terminal LayerNorm via the 1/||h|| Jacobian.
Causally established by April 7 no_outln 3-seed data.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
targets)
H2 100ep: ||h_L||=1.06e8, ||g_L||=1.09e-10 (below (b) floor)
Random-target DFA 100ep: ||h_L||=1.67e8, ||g_L||=8e-12 (worse than vanilla)
Both fully confirm the smoke-test trends at converged training horizons.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
- experiments/cifar_resmlp.py: add --methods filter and --random_targets flag;
extend compute_diagnostics to log hidden_norms_per_layer and bp_grad_norms_per_layer
- paper/main.tex §3 ¶1: broaden random-target finding to all 3 fixed-feedback methods
(DFA: ||h_L||=14510, SB: ||h_L||=6225, CB: ||h_L||=19974 at ep 3, all at chance acc)
- paper/main.tex Appendix J: extended with cross-method smoke-test table
This generalizes the §3 mechanism story from 'DFA-specific' to 'all 3 audited
fixed-feedback local-credit methods'. Combined with rounds 32-34, the proximate
cause of Mode 1 (a) is now well-localized:
- Not requires residual skip (round 33 H2 walkback)
- Not requires task signal (round 34 random targets, DFA)
- Not DFA-specific (round 35 random targets, SB+CB)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
Codex round 34 picked OPTION A (i.i.d. random class targets per minibatch) over the
analytic-only OPTION D as the most discriminating test of 'is (a) intrinsic to DFA
update geometry or task-driven?'. Smoke test result is unambiguous:
ep 0: ||h_L||=8.9 ||g_L||=9.8e-4
ep 1: ||h_L||=1616 ||g_L||=5.1e-6
ep 2: ||h_L||=9768 ||g_L||=8.5e-7
ep 3: ||h_L||=14510 ||g_L||=5.6e-7 (test acc still at chance ~0.07)
Three orders of magnitude growth in ||h_L|| in 3 epochs, three orders of magnitude
collapse in ||g_L|| in the same 3 epochs, with NO task signal whatsoever — DFA's
local-loss geometry is the proximate driver, not data adaptation.
- experiments/snapshot_evolution_residual_explosion.py: add --random_targets and
--skip_bp flags
- paper/main.tex §3 ¶1: replace 'no explicit scale constraint' framing with codex
round 34's 6-line geometric argument and the random-target empirical falsifier
- paper/main.tex Appendix J: full smoke-test table + interpretation
- v2.3: 14 pages total, main content still 8 pages
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
hypothesis
- models/residual_mlp.py: add residual_add and w2_std flags (default unchanged)
- experiments/snapshot_evolution_residual_explosion.py: add --no_residual_add and --w2_std CLI flags
- paper/main.tex §3 ¶3: add 1-sentence reference to no-residual control showing Mode 1 still fires
- paper/main.tex Appendix I: full smoke-test table + interpretation
- v2.2 main content stays at 8 pages (within 9-page E&D budget); 13 pages total
Smoke test (3 ep, w2_std=0.5, seed 42):
- DFA no-residual: ||h_L|| 4.69 -> 22050, ||g|| 1.6e-7 (Mode 1 (a) fires; (b) at floor)
- BP no-residual: acc only 0.16 at ep 3 (architecture is partially degenerate)
- Conclusion: residual skip is NOT necessary for Mode 1; the proximate trigger is more general
- Codex round 33 verdict: WALK BACK H2; demote 100ep run to confirmatory
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
L=2,4,6,8,12 data
The cifar_depth_scan_s42 results were already on disk but not surfaced in
the paper. Across L in {2,4,6,8,12} on the d=512 ResMLP, DFA's layer-0
cosine stays in [+0.39,+0.40] and its mean deep cosine stays within
[-0.005,+0.000], while BP retains a deep cosine of +0.94 even at L=12.
This rules out the 'too deep to receive useful credit' explanation:
making the network shallower does not reach the deep blocks any better.
- §5 paragraph 4: one-sentence depth-invariance summary citing the new appendix
- New Appendix H: Depth-Sweep Layerwise Profiles, with full table
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
content-complete
Six paragraphs total via codex round 31:
- §6 §6.1: measurement-validity-first ordering with 63x and 24338x calibration gaps
- §6 §6.2: minimal four-check protocol; decision-utility 0/5 vs 3/5 walk-back
- §6 §6.3: conservative asymmetry (BP/EP preserved, DFA/SB/CB walked back)
- §7 §7.1: scope claim — evaluation failure not algorithmic impossibility
- §7 §7.2: limits — CIFAR-10 only, observational LN interpretation, lower-bound BP+penalty control
- §7 §7.3: lesson — decompose evaluation question, position vs Jordan/O'Bray/Paleka
Compiles to 12 pages (main content 1-8, refs+appendices 8-12), within E&D 9-page main budget.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
|
|
