summaryrefslogtreecommitdiff
path: root/BASELINES.md
diff options
context:
space:
mode:
authorYurenHao0426 <Blackhao0426@gmail.com>2026-07-22 12:19:44 -0500
committerYurenHao0426 <Blackhao0426@gmail.com>2026-07-22 12:19:44 -0500
commit9763a8ecf4d614370cfde3ff926c5070ab766433 (patch)
tree4ed0b067bfcf4a16f811f37b5723d728ffb5db74 /BASELINES.md
parent04de9326e60bb9e3592c0ddb5474265d228cbce0 (diff)
baselines: complete native author-code audit
Diffstat (limited to 'BASELINES.md')
-rw-r--r--BASELINES.md26
1 files changed, 16 insertions, 10 deletions
diff --git a/BASELINES.md b/BASELINES.md
index 5051fca..9c3a320 100644
--- a/BASELINES.md
+++ b/BASELINES.md
@@ -4,7 +4,8 @@ This ledger separates three comparison types:
1. matched architectures/budgets inside this repository;
2. native author protocols reproduced without tracked source edits;
-3. published author numbers used for context while reproduction is pending.
+3. published author numbers retained as multi-seed context beside the completed
+ one-seed executable-fidelity reproductions.
Native protocols are not equal-compute comparisons. Their value is to prevent
weak in-house reimplementations from defining the state of the art.
@@ -80,10 +81,11 @@ WANDB_DIR=/scratch/yurenh2/burstccn-wandb CUDA_VISIBLE_DEVICES=5 \
```
The native trainer evaluates test every epoch and records both latest and best
-test error. It has a validation loader but model saving is disabled in this
-configuration. We will report final test accuracy as the primary reproduction,
-native best-test accuracy separately, and explicitly label the latter as
-test-selected. No run will be removed for a poor endpoint.
+test error. The completed seed-0 record reports `80.07%` final test accuracy,
+`80.10%` at the validation-selected epoch, and `80.25%` test-selected, versus
+the published `82.97 +/- 0.21%`. Audited W&B wall time is `15451.3 s`. The lower
+reproduced endpoint is retained rather than removed; the best-test value is
+explicitly labelled test-selected.
## Dual Propagation native CIFAR-10
@@ -132,9 +134,13 @@ TF_FORCE_GPU_ALLOW_GROWTH=true \
The actual command also prepends the environment's NVIDIA wheel `lib`
directories to `LD_LIBRARY_PATH`; this changes runtime linkage, not author
source. A one-epoch full-VGG smoke completed without OOM/NaN in 203.4 training
-seconds, reaching 20.48% validation and 20.04% test accuracy. The 130-epoch
-seed is therefore scheduled as a long native reproduction rather than treated
-as a cheap matched baseline.
+seconds, reaching 20.48% validation and 20.04% test accuracy. The completed
+130-epoch seed-1988 reproduction reaches `92.46%` test accuracy from the
+best-validation checkpoint, versus the published `92.41 +/- 0.07%`. Test was
+evaluated exactly once. Author-measured training, validation, and test time
+sums to `23119.8 s`; this excludes setup and checkpoint I/O. The near-exact
+accuracy reproduction establishes executable fidelity, not equal compute with
+SDIL.
## Reporting rules for external runs
@@ -155,5 +161,5 @@ freeze, the wrong physical GPU, and missing dataset hashes. Completed records
live under `results/native/<protocol-id>.json`. The separate strict aggregator
`experiments/audit_native_baselines.py` distinguishes final,
validation-selected, and test-selected metrics and generates
-`results/native_baseline_audit.md`. `experiments/finalize_accept.sh` remains
-red until both author runs have passed this C4 audit.
+`results/native_baseline_audit.md`. Both author runs pass this C4 audit, and
+`experiments/finalize_accept.sh` is green.