<feed xmlns='http://www.w3.org/2005/Atom'>
<title>faeval.git/paper/figures/fig4_penalty_rescue.pdf, branch master</title>
<subtitle>Unnamed repository; edit this file 'description' to name the repository.
</subtitle>
<link rel='alternate' type='text/html' href='https://git.blackhao.com/faeval.git/'/>
<entry>
<title>Add new experiment scripts, figures, and paper assets; untrack pyc/build artifacts</title>
<updated>2026-06-14T09:06:32+00:00</updated>
<author>
<name>YurenHao0426</name>
<email>Blackhao0426@gmail.com</email>
</author>
<published>2026-06-14T09:06:32+00:00</published>
<link rel='alternate' type='text/html' href='https://git.blackhao.com/faeval.git/commit/?id=aa73718eb6427d7da3b9cb416275802d90c4b2ed'/>
<id>aa73718eb6427d7da3b9cb416275802d90c4b2ed</id>
<content type='text'>
Co-Authored-By: Claude Opus 4.8 (1M context) &lt;noreply@anthropic.com&gt;
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Co-Authored-By: Claude Opus 4.8 (1M context) &lt;noreply@anthropic.com&gt;
</pre>
</div>
</content>
</entry>
<entry>
<title>paper v2.32: BP+penalty multi-seeded (was single-seed s42)</title>
<updated>2026-04-09T00:11:40+00:00</updated>
<author>
<name>YurenHao0426</name>
<email>Blackhao0426@gmail.com</email>
</author>
<published>2026-04-09T00:11:40+00:00</published>
<link rel='alternate' type='text/html' href='https://git.blackhao.com/faeval.git/commit/?id=d022688ea9fcfcb81f900751ee92e35597ef19b8'/>
<id>d022688ea9fcfcb81f900751ee92e35597ef19b8</id>
<content type='text'>
The §5 ¶3 BP+penalty value (0.530, +18.1 pp margin) was single-seed s42.
Ran s123 and s456 to multi-seed it, matching the BP-no-pen 3-seed control.

3-seed BP+pen 30ep results (lam=0.01, AdamW lr=1e-3 wd=0.01, cosine, batch 128):
  s42:  0.5303, +18.13 pp vs frozen
  s123: 0.5262, +17.72 pp
  s456: 0.5397, +19.07 pp
  3-seed mean: 0.5321 ± 0.0057, +18.31 pp

Updates:
- §5 ¶3: BP+pen "0.530 (single seed)" → "0.532 ± 0.006" (3-seed)
- §5 ¶3: BP penalty cost -5.5 pp → -5.3 pp
- §5 ¶3: BP+pen margin +18.1 → +18.3 pp
- §5 ¶3: BP-to-DFA gap 17.0 → 17.2 pp
- §4 ¶4: BP+pen +18.1 → +18.3 pp comparison
- Figure 3 panel C bar values: BP with_pen 0.530 → 0.532
- Figure 3 panel C title: BP-pen-cost -5.5pp → -5.3pp

The +18.3 pp 3-seed mean is essentially the same as the s42 single-seed
+18.13 pp, so the headline conclusion (BP+pen far above frozen baseline,
huge gap vs DFA+pen) is unchanged. This commit removes the last
single-seed value labeled as a key control.

New auditable file: results/bp_with_penalty_3seed_summary.json

Page layout preserved: 9 pages main, refs p10, 0 overfull boxes.

Co-Authored-By: Claude Opus 4.6 (1M context) &lt;noreply@anthropic.com&gt;
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
The §5 ¶3 BP+penalty value (0.530, +18.1 pp margin) was single-seed s42.
Ran s123 and s456 to multi-seed it, matching the BP-no-pen 3-seed control.

3-seed BP+pen 30ep results (lam=0.01, AdamW lr=1e-3 wd=0.01, cosine, batch 128):
  s42:  0.5303, +18.13 pp vs frozen
  s123: 0.5262, +17.72 pp
  s456: 0.5397, +19.07 pp
  3-seed mean: 0.5321 ± 0.0057, +18.31 pp

Updates:
- §5 ¶3: BP+pen "0.530 (single seed)" → "0.532 ± 0.006" (3-seed)
- §5 ¶3: BP penalty cost -5.5 pp → -5.3 pp
- §5 ¶3: BP+pen margin +18.1 → +18.3 pp
- §5 ¶3: BP-to-DFA gap 17.0 → 17.2 pp
- §4 ¶4: BP+pen +18.1 → +18.3 pp comparison
- Figure 3 panel C bar values: BP with_pen 0.530 → 0.532
- Figure 3 panel C title: BP-pen-cost -5.5pp → -5.3pp

The +18.3 pp 3-seed mean is essentially the same as the s42 single-seed
+18.13 pp, so the headline conclusion (BP+pen far above frozen baseline,
huge gap vs DFA+pen) is unchanged. This commit removes the last
single-seed value labeled as a key control.

New auditable file: results/bp_with_penalty_3seed_summary.json

Page layout preserved: 9 pages main, refs p10, 0 overfull boxes.

Co-Authored-By: Claude Opus 4.6 (1M context) &lt;noreply@anthropic.com&gt;
</pre>
</div>
</content>
</entry>
<entry>
<title>paper v2.31: matched 30-epoch BP/DFA controls (was unsourced 0.609/0.308)</title>
<updated>2026-04-08T23:03:16+00:00</updated>
<author>
<name>YurenHao0426</name>
<email>Blackhao0426@gmail.com</email>
</author>
<published>2026-04-08T23:03:16+00:00</published>
<link rel='alternate' type='text/html' href='https://git.blackhao.com/faeval.git/commit/?id=752dfb833b06a6fb974df892de560caf328ed1dd'/>
<id>752dfb833b06a6fb974df892de560caf328ed1dd</id>
<content type='text'>
The §5 ¶3 BP-no-penalty value of 0.609 ± 0.004 and DFA-no-penalty value
of 0.308 ± 0.014 turned out to be unsourced — they were carried over
from a hardcoded comment in experiments/bp_with_penalty_control.py
("BP-trainable (3-seed mean): 0.609") that nobody had actually measured
with a matched 30-epoch run.

Ran the missing matched controls under the same recipe as BP+pen
(lam=0, 30 epochs, AdamW 1e-3, wd 0.01, cosine schedule, batch 128,
3 seeds 42/123/456):

  BP no-pen 30ep: per-seed 0.5851, 0.5845, 0.5863  →  0.585 ± 0.001
                  (paper said 0.609 ± 0.004, off by 0.024)
  DFA no-pen 30ep: per-seed 0.3070, 0.2985, 0.2966 →  0.301 ± 0.005
                  (paper said 0.308 ± 0.014)

Also re-grounded DFA+penalty 30ep using the dfa_pen_short 3-seed run
(0.3593, 0.3610, 0.3604 → 0.360 ± 0.001), which is what the deep-cosine
+0.155 figure was computed on. The paper had 0.363 ± 0.001 — that came
from the 100-epoch run, not the 30-epoch run, so it was an apples-to-
oranges comparison with BP+pen 30-ep.

Paper changes (§5 ¶3):
  BP penalty cost:  -8 pp  →  -5.5 pp
  DFA pen rescue:   +5.5 → +5.9 pp
  DFA+pen margin vs frozen: +1.4 → +1.1 pp
  BP-to-DFA gap:     17 → 17.0 pp (unchanged)
  BP-to-SB gap:      7.7 → 7.7 pp (unchanged)
  BP-to-DFA gap is still the lower-bound credit-quality cost claim;
  17 pp gap is unchanged in magnitude.

Also updated:
- §5 ¶1 prose: 0.363 → 0.360, 0.308 → 0.301
- §4 ¶4 prose: DFA+pen 0.363 → 0.360
- Appendix J Table 9 caption: 0.363 → 0.360, +9.0 → +9.3 pp gap to SB
- Appendix L paragraph: +5.5 → +5.9 pp DFA penalty rescue
- Figure 3 panel C bar values + title pen-cost annotation
- New results/matched_30ep_control_summary.json as auditable record

Page layout preserved: 9 main pages + refs p10, 18 total, 0 overfull.

Co-Authored-By: Claude Opus 4.6 (1M context) &lt;noreply@anthropic.com&gt;
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
The §5 ¶3 BP-no-penalty value of 0.609 ± 0.004 and DFA-no-penalty value
of 0.308 ± 0.014 turned out to be unsourced — they were carried over
from a hardcoded comment in experiments/bp_with_penalty_control.py
("BP-trainable (3-seed mean): 0.609") that nobody had actually measured
with a matched 30-epoch run.

Ran the missing matched controls under the same recipe as BP+pen
(lam=0, 30 epochs, AdamW 1e-3, wd 0.01, cosine schedule, batch 128,
3 seeds 42/123/456):

  BP no-pen 30ep: per-seed 0.5851, 0.5845, 0.5863  →  0.585 ± 0.001
                  (paper said 0.609 ± 0.004, off by 0.024)
  DFA no-pen 30ep: per-seed 0.3070, 0.2985, 0.2966 →  0.301 ± 0.005
                  (paper said 0.308 ± 0.014)

Also re-grounded DFA+penalty 30ep using the dfa_pen_short 3-seed run
(0.3593, 0.3610, 0.3604 → 0.360 ± 0.001), which is what the deep-cosine
+0.155 figure was computed on. The paper had 0.363 ± 0.001 — that came
from the 100-epoch run, not the 30-epoch run, so it was an apples-to-
oranges comparison with BP+pen 30-ep.

Paper changes (§5 ¶3):
  BP penalty cost:  -8 pp  →  -5.5 pp
  DFA pen rescue:   +5.5 → +5.9 pp
  DFA+pen margin vs frozen: +1.4 → +1.1 pp
  BP-to-DFA gap:     17 → 17.0 pp (unchanged)
  BP-to-SB gap:      7.7 → 7.7 pp (unchanged)
  BP-to-DFA gap is still the lower-bound credit-quality cost claim;
  17 pp gap is unchanged in magnitude.

Also updated:
- §5 ¶1 prose: 0.363 → 0.360, 0.308 → 0.301
- §4 ¶4 prose: DFA+pen 0.363 → 0.360
- Appendix J Table 9 caption: 0.363 → 0.360, +9.0 → +9.3 pp gap to SB
- Appendix L paragraph: +5.5 → +5.9 pp DFA penalty rescue
- Figure 3 panel C bar values + title pen-cost annotation
- New results/matched_30ep_control_summary.json as auditable record

Page layout preserved: 9 main pages + refs p10, 18 total, 0 overfull.

Co-Authored-By: Claude Opus 4.6 (1M context) &lt;noreply@anthropic.com&gt;
</pre>
</div>
</content>
</entry>
<entry>
<title>Figures 3 and 4: fix aspect ratio (fig3 was squeezed strip) and key-finding label overlap (fig4)</title>
<updated>2026-04-08T19:35:20+00:00</updated>
<author>
<name>YurenHao0426</name>
<email>Blackhao0426@gmail.com</email>
</author>
<published>2026-04-08T19:35:20+00:00</published>
<link rel='alternate' type='text/html' href='https://git.blackhao.com/faeval.git/commit/?id=1c4672e465b9145d71cfddeae4e2ebaeeef2964f'/>
<id>1c4672e465b9145d71cfddeae4e2ebaeeef2964f</id>
<content type='text'>
Per user feedback:
- fig4_penalty_rescue.pdf (Figure 3 in paper): was figsize=(13, 3.5), aspect 3.7:1,
  which rendered as a thin strip with squeezed subplot content. Increased height
  to figsize=(13, 6.0), aspect 2.2:1. Much taller panels that actually show axis
  labels and legends readably.
- fig5_cross_arch_summary.pdf (Figure 4 in paper): the 'Key finding' italic text
  annotation at y=-1.0 in axes transform was overlapping with the multiline
  architecture y-tick labels at the bottom of the second subplot. Moved to
  y=-1.55 and increased figsize height from 3.5 to 4.2 so the lower annotation
  still fits in bbox_inches='tight' crop.
- Also bumped includegraphics width from 0.92\linewidth to \linewidth for both
  figures so they use the full text width.

Main content still exactly 9 pages within E&amp;D budget.

Co-Authored-By: Claude Opus 4.6 (1M context) &lt;noreply@anthropic.com&gt;
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Per user feedback:
- fig4_penalty_rescue.pdf (Figure 3 in paper): was figsize=(13, 3.5), aspect 3.7:1,
  which rendered as a thin strip with squeezed subplot content. Increased height
  to figsize=(13, 6.0), aspect 2.2:1. Much taller panels that actually show axis
  labels and legends readably.
- fig5_cross_arch_summary.pdf (Figure 4 in paper): the 'Key finding' italic text
  annotation at y=-1.0 in axes transform was overlapping with the multiline
  architecture y-tick labels at the bottom of the second subplot. Moved to
  y=-1.55 and increased figsize height from 3.5 to 4.2 so the lower annotation
  still fits in bbox_inches='tight' crop.
- Also bumped includegraphics width from 0.92\linewidth to \linewidth for both
  figures so they use the full text width.

Main content still exactly 9 pages within E&amp;D budget.

Co-Authored-By: Claude Opus 4.6 (1M context) &lt;noreply@anthropic.com&gt;
</pre>
</div>
</content>
</entry>
<entry>
<title>Fill in tables 1-3 + generate figures 2/4/5 from existing data</title>
<updated>2026-04-08T09:46:59+00:00</updated>
<author>
<name>YurenHao0426</name>
<email>Blackhao0426@gmail.com</email>
</author>
<published>2026-04-08T09:46:59+00:00</published>
<link rel='alternate' type='text/html' href='https://git.blackhao.com/faeval.git/commit/?id=07b10f06478514bbe9d9c77461a90f9d3254218b'/>
<id>07b10f06478514bbe9d9c77461a90f9d3254218b</id>
<content type='text'>
Tables filled with real values:
  Table 1: 5-method audit (3-seed mean ± std for acc, headline Γ, verdict)
  Table 2: 4-condition mode 2 validation (cos and ρ values from existing
           checkpoint measurements)
  Table 3: protocol thresholds (50×, 1e-7, 0.30, 2pp)

Figures generated from existing data:
  fig2_decision_utility.pdf: 5×7 verdict heatmap from
    results/protocol_audit/ablation_decision_utility.json
  fig4_penalty_rescue.pdf: 3-panel — trajectory + cos/ρ bars + 2×2 acc
    from snapshot_evolution_v2 + dfa_residual_penalty + bp_with_penalty
  fig5_cross_arch_summary.pdf: 5×4 BP/DFA verdict matrix across
    architectures

Compiles to 8 pages with all tables/figures rendered. §1-§7 main body
still has only paragraph topic sentences (TODO: per-section prose
filling via codex). Figure numbering is wrong (codex put figures in
section order not numerical order — need fixing).
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Tables filled with real values:
  Table 1: 5-method audit (3-seed mean ± std for acc, headline Γ, verdict)
  Table 2: 4-condition mode 2 validation (cos and ρ values from existing
           checkpoint measurements)
  Table 3: protocol thresholds (50×, 1e-7, 0.30, 2pp)

Figures generated from existing data:
  fig2_decision_utility.pdf: 5×7 verdict heatmap from
    results/protocol_audit/ablation_decision_utility.json
  fig4_penalty_rescue.pdf: 3-panel — trajectory + cos/ρ bars + 2×2 acc
    from snapshot_evolution_v2 + dfa_residual_penalty + bp_with_penalty
  fig5_cross_arch_summary.pdf: 5×4 BP/DFA verdict matrix across
    architectures

Compiles to 8 pages with all tables/figures rendered. §1-§7 main body
still has only paragraph topic sentences (TODO: per-section prose
filling via codex). Figure numbering is wrong (codex put figures in
section order not numerical order — need fixing).
</pre>
</div>
</content>
</entry>
<entry>
<title>v2 skeleton from round 25: section structure now matches round 23</title>
<updated>2026-04-08T09:43:39+00:00</updated>
<author>
<name>YurenHao0426</name>
<email>Blackhao0426@gmail.com</email>
</author>
<published>2026-04-08T09:43:39+00:00</published>
<link rel='alternate' type='text/html' href='https://git.blackhao.com/faeval.git/commit/?id=58259e151858a545e359c2134b1db84bee3a4be6'/>
<id>58259e151858a545e359c2134b1db84bee3a4be6</id>
<content type='text'>
Round 24's skeleton had 3 deviations from round 23 redo:
  - Made §3 'Diagnostic Protocol' instead of 'Failure Mode 1'
  - Collapsed Mode 1 + Mode 2 into one §4
  - Added §6 'Reference Implementation' (was supposed to be dropped)

Round 25 fixed all three. New §3-§7 match round 23 redo exactly:
  §3 Failure Mode 1: Measurement Degeneracy
  §4 Failure Mode 2: Low Intrinsic Credit-Direction Quality
  §5 Intervention and Cross-Architecture Evidence
  §6 Recommended FA Evaluation Protocol
  §7 Discussion, Limits, Conclusion

Also added:
  - In-line bibliography with 12 \bibitem entries (Paleka, O'Bray, Jordan
    + FA literature) — citations resolve correctly now
  - Appendices A-G with actual prose content (not just headers)
  - 7-pitfall catalog with descriptions
  - Walk-back chain methodology paragraph
  - 7-validation summary table

Compiles to 9 pages with figures 1+3 inline (existing PNGs) and figures
2/4/5 as placeholder text PDFs (TODO: regenerate). Tables 1/2/3 still
have TODO placeholders for numerical values.

Next: fill in tables 1-3 with existing JSON data, generate figures 2/4/5
from existing data, then consult codex per-section for prose filling.
</content>
<content type='xhtml'>
<div xmlns='http://www.w3.org/1999/xhtml'>
<pre>
Round 24's skeleton had 3 deviations from round 23 redo:
  - Made §3 'Diagnostic Protocol' instead of 'Failure Mode 1'
  - Collapsed Mode 1 + Mode 2 into one §4
  - Added §6 'Reference Implementation' (was supposed to be dropped)

Round 25 fixed all three. New §3-§7 match round 23 redo exactly:
  §3 Failure Mode 1: Measurement Degeneracy
  §4 Failure Mode 2: Low Intrinsic Credit-Direction Quality
  §5 Intervention and Cross-Architecture Evidence
  §6 Recommended FA Evaluation Protocol
  §7 Discussion, Limits, Conclusion

Also added:
  - In-line bibliography with 12 \bibitem entries (Paleka, O'Bray, Jordan
    + FA literature) — citations resolve correctly now
  - Appendices A-G with actual prose content (not just headers)
  - 7-pitfall catalog with descriptions
  - Walk-back chain methodology paragraph
  - 7-validation summary table

Compiles to 9 pages with figures 1+3 inline (existing PNGs) and figures
2/4/5 as placeholder text PDFs (TODO: regenerate). Tables 1/2/3 still
have TODO placeholders for numerical values.

Next: fill in tables 1-3 with existing JSON data, generate figures 2/4/5
from existing data, then consult codex per-section for prose filling.
</pre>
</div>
</content>
</entry>
</feed>
