diff options
| -rw-r--r-- | BASELINES.md | 26 | ||||
| -rw-r--r-- | PAPER_PLAN.md | 12 | ||||
| -rw-r--r-- | README.md | 12 | ||||
| -rw-r--r-- | RESULTS.md | 19 | ||||
| -rw-r--r-- | REVIEW_SCORECARD.md | 7 | ||||
| -rw-r--r-- | ROADMAP.md | 20 | ||||
| -rw-r--r-- | results/native/dualprop_cifar10_vgg16_a0_b0p1_seed1988.json | 287 | ||||
| -rw-r--r-- | results/native_baseline_audit.md | 24 |
8 files changed, 377 insertions, 30 deletions
diff --git a/BASELINES.md b/BASELINES.md index 5051fca..9c3a320 100644 --- a/BASELINES.md +++ b/BASELINES.md @@ -4,7 +4,8 @@ This ledger separates three comparison types: 1. matched architectures/budgets inside this repository; 2. native author protocols reproduced without tracked source edits; -3. published author numbers used for context while reproduction is pending. +3. published author numbers retained as multi-seed context beside the completed + one-seed executable-fidelity reproductions. Native protocols are not equal-compute comparisons. Their value is to prevent weak in-house reimplementations from defining the state of the art. @@ -80,10 +81,11 @@ WANDB_DIR=/scratch/yurenh2/burstccn-wandb CUDA_VISIBLE_DEVICES=5 \ ``` The native trainer evaluates test every epoch and records both latest and best -test error. It has a validation loader but model saving is disabled in this -configuration. We will report final test accuracy as the primary reproduction, -native best-test accuracy separately, and explicitly label the latter as -test-selected. No run will be removed for a poor endpoint. +test error. The completed seed-0 record reports `80.07%` final test accuracy, +`80.10%` at the validation-selected epoch, and `80.25%` test-selected, versus +the published `82.97 +/- 0.21%`. Audited W&B wall time is `15451.3 s`. The lower +reproduced endpoint is retained rather than removed; the best-test value is +explicitly labelled test-selected. ## Dual Propagation native CIFAR-10 @@ -132,9 +134,13 @@ TF_FORCE_GPU_ALLOW_GROWTH=true \ The actual command also prepends the environment's NVIDIA wheel `lib` directories to `LD_LIBRARY_PATH`; this changes runtime linkage, not author source. A one-epoch full-VGG smoke completed without OOM/NaN in 203.4 training -seconds, reaching 20.48% validation and 20.04% test accuracy. The 130-epoch -seed is therefore scheduled as a long native reproduction rather than treated -as a cheap matched baseline. +seconds, reaching 20.48% validation and 20.04% test accuracy. The completed +130-epoch seed-1988 reproduction reaches `92.46%` test accuracy from the +best-validation checkpoint, versus the published `92.41 +/- 0.07%`. Test was +evaluated exactly once. Author-measured training, validation, and test time +sums to `23119.8 s`; this excludes setup and checkpoint I/O. The near-exact +accuracy reproduction establishes executable fidelity, not equal compute with +SDIL. ## Reporting rules for external runs @@ -155,5 +161,5 @@ freeze, the wrong physical GPU, and missing dataset hashes. Completed records live under `results/native/<protocol-id>.json`. The separate strict aggregator `experiments/audit_native_baselines.py` distinguishes final, validation-selected, and test-selected metrics and generates -`results/native_baseline_audit.md`. `experiments/finalize_accept.sh` remains -red until both author runs have passed this C4 audit. +`results/native_baseline_audit.md`. Both author runs pass this C4 audit, and +`experiments/finalize_accept.sh` is green. diff --git a/PAPER_PLAN.md b/PAPER_PLAN.md index 1c958ff..1055738 100644 --- a/PAPER_PLAN.md +++ b/PAPER_PLAN.md @@ -43,11 +43,13 @@ as a limitation rather than a title claim. 5. Report the frozen depth panel: SDIL changes by only `-0.214 +/- 0.349` points over 12x hidden depth while DFA alignment falls to `0.047`; qualify that flattened CIFAR is depth-flat. -6. Report the failed standard-ResNet A3 gate as a limitation: short-run SDIL - beat tuned DFA, but the full recipe became nonfinite and did not open A4. - End with the supported scope: innovation is useful for soma-predictable - traffic, while learned feedback amortization and long-horizon stability - remain the useful-depth bottlenecks. +6. Report native BurstCCN and Dual Prop as method-native context, including + their unmatched protocols and audited walls. Report the failed + standard-ResNet A3 gate as a limitation: short-run SDIL beat tuned DFA, but + the full recipe became nonfinite and did not open A4. End with the supported + scope: innovation is useful for soma-predictable traffic, while learned + feedback amortization and long-horizon stability remain the useful-depth + bottlenecks. The abstract should not mention the failed desired-velocity hypothesis unless the paper explicitly positions that falsification as a contribution. It must @@ -45,6 +45,11 @@ and scaling behavior. See `NOVELTY.md` for the exact prior-art boundary. became nonfinite at epoch 90 and ended at chance, while DFA remained finite at `33.06%`. A4 was therefore not opened and no confirmation test seed was touched. +- Native author-code fidelity is complete. BurstCCN reaches `80.10%` at its + validation-selected epoch versus published `82.97 +/- 0.21%`; Dual Prop + reaches `92.46%` versus published `92.41 +/- 0.07%`. Their audited walls are + `15451.3 s` and `23119.8 s`, respectively, and they are not presented as + matched-compute points. The broad endogenous-traffic gate and the Harnett desired-velocity/online-control screen failed. Those results are retained and explicitly constrain the paper: @@ -96,9 +101,10 @@ eligibilities, the BatchNorm-coupled perturbation objective, predictor fitting, translation-shared feedback, and parameter/cost accounting. `experiments/finalize_accept.sh` additionally requires strict imports of the -frozen BurstCCN and Dual Propagation author-code runs. It is intentionally red -until both long native jobs are complete. The standard ResNet branch stopped -at its failed A3 validation gate; the untouched A4 test panel remains sealed. +frozen BurstCCN and Dual Propagation author-code runs. Both records now pass; +the accept-bar mechanical audit is green. The standard ResNet branch stopped +at its failed A3 validation gate, and the untouched A4 test panel remains +sealed. ## Result discipline @@ -586,6 +586,25 @@ strong modern EP variant remains necessary. DFA is also strong on easy MNIST, so comes from the five-depth CIFAR panel; this comparison establishes that SDIL need not trade away accuracy to obtain a large cost advantage over canonical EP. +### Native author-code CIFAR-10 baselines + +These are executable-fidelity reproductions under the authors' architectures +and schedules, not equal-compute comparisons with SDIL: + +| method | reproduced seed | primary test (%) | validation-selected test (%) | test-selected test (%) | published (%) | audited wall (s) | +|:--|--:|--:|--:|--:|--:|--:| +| BurstCCN | 0 | 80.070 | 80.100 | 80.250 | 82.97 +/- 0.21 | 15451.3 | +| Dual Prop | 1988 | 92.460 | 92.460 | n/a | 92.41 +/- 0.07 | 23119.8 | + +BurstCCN's primary value is the final epoch and its best-test value is +test-selected because native code evaluates test every epoch. Dual Prop +restores the best-validation checkpoint and evaluates test once, so primary +and validation-selected values coincide. Both strict records include source, +environment, dataset, artifact, selection, and wall-definition audits. Dual +Prop reproduces its published accuracy almost exactly, but uses VGG16 with 16 +nudged inference passes and about 6.42 hours of author-measured compute on a +GTX 1080. + ### Frozen standard-ResNet development funnel The prospective Oral-A funnel used a 45k/5k training-only CIFAR-10 split; no diff --git a/REVIEW_SCORECARD.md b/REVIEW_SCORECARD.md index 25d1247..861ba0e 100644 --- a/REVIEW_SCORECARD.md +++ b/REVIEW_SCORECARD.md @@ -47,15 +47,15 @@ Every formal result report records: 3. Innovation was not uniformly beneficial for arbitrary endogenous top-down traffic, so the supported mechanism is narrower than the initial claim. 4. The frozen oral-B screen falsified the desired-velocity and Harnett error-derivative claims. -5. The frozen ResNet-20 A3 run became nonfinite and ended at chance; native Dual Prop is the only - remaining incomplete baseline endpoint. The failed gate forbids the A4 test panel. +5. The frozen ResNet-20 A3 run became nonfinite and ended at chance. The failed gate forbids the + A4 test panel; completed native baselines improve fairness but do not supply SDIL scale evidence. ## Score trajectory and prospective gates | Checkpoint | Overall | What changed | Remaining ceiling | |:--|--:|:--|:--| | Current audited package | 5 | Strong depth-preservation and residual-necessity evidence; negative gates retained | Standard useful scale is absent | -| Native baselines complete | pending | Can close fairness/completeness objections, but cannot by itself establish the main claim | Usually no automatic score increase | +| Native baselines complete | 5 | BurstCCN is below its published endpoint; Dual Prop reproduces 92.46% versus 92.41%, with strict provenance and cost semantics | Fairness objection narrows, but SDIL gains no standard-scale evidence | | Oral-A A1/A2 | 5 | BP reached 91.62%; short channel-gated SDIL reached 41.98% versus tuned DFA at 37.16% | Development screening alone cannot raise the score | | Oral-A A3 fails | 5 | Full ResNet-20 SDIL became nonfinite at epoch 90 and ended at 10%; DFA ended finite at 33.06% | Standard-scale and oral-A claims are closed; A4 remains untouched | | Oral-A A4 | not opened | The prerequisite A3 gate failed | No oral-A confirmation claim is available | @@ -71,6 +71,7 @@ retroactively reopened by success on standard vision benchmarks. | 2026-07-22 / `2304e83` | Audit of all completed frozen branches | baseline → 5 | Strong preservation/residualization core, but no standard useful-scale result | | 2026-07-22 / `c753f51`, `1b24c87`, `6d19078` | Existing 60-run innovation panel promoted to a strict main figure; conditional-projection and norm-direction identities made executable | 5 → 5 | Closes a presentation/theory objection and makes the narrow novelty legible, but adds no new held-out evidence and therefore earns no score inflation | | 2026-07-22 / frozen Oral-A A1--A3 | A1 and A2 pass; full A3 SDIL becomes nonfinite and fails four of six checks; A4 untouched | 5 → 5 | Closes the standard-scale question negatively. The narrow mechanism paper survives, while any standard-ResNet or oral claim does not | +| 2026-07-22 / native C4 | BurstCCN and Dual Prop author-code records pass the strict audit; Dual Prop reproduces 92.46% test in 23119.8 s | 5 → 5 | Closes a baseline-fidelity objection and confirms a strong expensive comparator, but does not repair SDIL's failed A3 evidence | Future rows are appended only after an audited frozen stage. A score staying flat is informative: engineering, theory exposition, or visualization may make the paper more defensible without @@ -272,15 +272,17 @@ competitive under the frozen protocol. Each method receives a documented validation budget. Native-protocol and exact-architecture comparisons are labelled separately; neither substitutes for a matched compute/query comparison. -**Status: in progress.** In-repository BP/FA/DFA, direct NP, learned NP, PEPITA, -Forward-Forward, and canonical EP are complete. `BASELINES.md` freezes the author revisions, -commands, data provenance, and selection semantics for BurstCCN and Dual Prop. Paper-faithful -BurstCCN seed 0 is running on GPU5; author-code Dual Prop VGG16 seed 1988 is running on GPU7 after -a complete one-epoch smoke. C4 remains incomplete until both long runs finish and their artifacts -are imported and audited. The native protocol ledger, strict importer, result aggregator, and -`finalize_accept.sh` gate are frozen before either endpoint is observed. C4 does not impose an -accuracy threshold: poor faithful reproductions remain reportable and cannot be discarded, while -published multi-seed values remain explicitly separate from the one-seed executable-fidelity runs. +**Status: passed on 2026-07-22.** In-repository BP/FA/DFA, direct NP, learned NP, +PEPITA, Forward-Forward, and canonical EP are complete. Paper-faithful BurstCCN +seed 0 reaches `80.07%` final / `80.10%` validation-selected / `80.25%` +test-selected accuracy in `15451.3 s`, versus published `82.97 +/- 0.21%`. +Author-code Dual Prop VGG16 seed 1988 reaches `92.46%` from the best-validation +checkpoint in `23119.8 s`, versus published `92.41 +/- 0.07%`, with one test +evaluation. Both records pass artifact, source, environment, dataset, +completeness, finiteness, selection, and cost-definition checks in +`finalize_accept.sh`. These one-seed method-native reproductions remain +separate from matched-compute comparisons and from published multi-seed +uncertainty. ### C5. Theory predicts the observed regimes diff --git a/results/native/dualprop_cifar10_vgg16_a0_b0p1_seed1988.json b/results/native/dualprop_cifar10_vgg16_a0_b0p1_seed1988.json new file mode 100644 index 0000000..519c883 --- /dev/null +++ b/results/native/dualprop_cifar10_vgg16_a0_b0p1_seed1988.json @@ -0,0 +1,287 @@ +{ + "audit": { + "complete": true, + "expected_epochs": 130, + "finite": true, + "selection_semantics_frozen": true, + "source_revision_match": true, + "source_tracked_clean": true + }, + "dataset": "CIFAR-10", + "dataset_artifacts": [ + { + "bytes": 170498071, + "path": "/home/yurenh2/sdrn/data/cifar-10-python.tar.gz", + "sha256": "6d958be074577803d12ecdefd02955f39262c83c16fe9348329d7fe0b5c001ce" + }, + { + "bytes": 169566452, + "path": "/scratch/yurenh2/cifar10-binary-converted-v1.tar.gz", + "sha256": "0b97b0a786c64f4cd0996cace20e81e189669aba9f4b4e517eb529f9c7105cec" + } + ], + "gpu": { + "driver_version": "535.183.01", + "memory_total_mib": "8192", + "name": "NVIDIA GeForce GTX 1080", + "physical_index": "7", + "uuid": "GPU-df3d5933-c3e9-9ea1-d595-9d2a3b68c14b" + }, + "importer_provenance": { + "path": "/home/yurenh2/sdil", + "revision": "04de9326e60bb9e3592c0ddb5474265d228cbce0", + "tracked_dirty": false + }, + "method": "Dual Propagation", + "metrics": { + "epochs_observed": 130, + "final_epoch": 130, + "final_train_accuracy_percent": 99.96444702148438, + "final_validation_accuracy_percent": 92.76000213623047, + "primary_test_accuracy_percent": 92.45999908447266, + "primary_test_loss": 0.33965352177619934, + "primary_test_top5_accuracy_percent": 99.80000305175781, + "test_evaluations": 1, + "test_selected_test_accuracy_percent": null, + "test_wall_s": 3.2275850772857666, + "training_wall_s": 23052.372897148132, + "validation_selected_epoch": 118, + "validation_selected_test_accuracy_percent": 92.45999908447266, + "validation_selected_validation_accuracy_percent": 92.86000061035156, + "validation_wall_s": 64.21902823448181, + "wall_definition": "sum of author-measured train, validation, and final test runtimes; excludes setup, checkpoint I/O, and diagnostic plotting", + "wall_s": 23119.8195104599 + }, + "protocol": { + "command": [ + "python", + "train.py", + "--model", + "VGG16", + "--dataset", + "cifar10", + "--num-epochs", + "130", + "--batch-size", + "100", + "--alpha", + "0.0", + "--beta", + "0.1", + "--loss", + "sce", + "--inference-sequence", + "fwK", + "--inference-passes-nudged", + "16", + "--learning-rate", + "0.025", + "--learning-rate-final", + "0.000002", + "--warmup-learning-rate", + "0.001", + "--learning-algorithm", + "dualprop-lagr-ff", + "--activation", + "relu", + "--decay-epochs", + "120", + "--warmup-epochs", + "10", + "--momentum", + "0.9", + "--weight-decay", + "5e-4", + "--dtype", + "float32", + "--param-dtype", + "float32", + "--percent-train", + "90", + "--percent-val", + "10", + "--seeds", + "1988", + "--experiment-name", + "sdil-dp-full-a0-b0p1-seed1988-v1" + ], + "dataset": "CIFAR-10", + "expected_epochs": 130, + "expected_gpu": { + "name": "NVIDIA GeForce GTX 1080", + "physical_index": "7" + }, + "method": "Dual Propagation", + "paper_url": "https://arxiv.org/abs/2402.08573", + "published_accuracy_percent_mean": 92.41, + "published_accuracy_percent_sd": 0.07, + "repository_url": "https://github.com/Rasmuskh/dualprop_icml_2024", + "required_dataset_sha256": [ + "6d958be074577803d12ecdefd02955f39262c83c16fe9348329d7fe0b5c001ce", + "0b97b0a786c64f4cd0996cace20e81e189669aba9f4b4e517eb529f9c7105cec" + ], + "resolved_config": { + "alpha": 0.0, + "batch_size": 100, + "beta": 0.1, + "decay_epochs": 120, + "inference_passes_nudged": 16, + "inference_sequence": "fwK", + "learning_algorithm": "dualprop-lagr-ff", + "learning_rate": 0.025, + "model": "VGG16", + "momentum": 0.9, + "percent_train": 90, + "percent_val": 10, + "warmup_epochs": 10, + "weight_decay": 0.0005 + }, + "seed": 1988, + "selection_semantics": { + "primary": "single test evaluation from the best-validation checkpoint", + "test_selected": "not available; test is evaluated only once", + "validation_selected": "same as primary" + }, + "source_revision": "7b2595b34421e1483a721dbfdeff8cdabda3a1ff" + }, + "protocol_id": "dualprop_cifar10_vgg16_a0_b0p1_seed1988", + "resolved_config": { + "alpha": 0.0, + "batch_size": 100, + "beta": 0.1, + "decay_epochs": 120, + "inference_passes_nudged": 16, + "inference_sequence": "fwK", + "learning_algorithm": "dualprop-lagr-ff", + "learning_rate": 0.025, + "model": "VGG16", + "momentum": 0.9, + "percent_train": 90, + "percent_val": 10, + "warmup_epochs": 10, + "weight_decay": 0.0005 + }, + "run_artifacts": [ + { + "bytes": 7579669, + "path": "/scratch/yurenh2/sdil-reference/dualprop_icml_2024/runs/sdil-dp-full-a0-b0p1-seed1988-v1/2026_07_22_05_23_43/hist.npy", + "sha256": "401f24e1780b1ae1b57524694e9b9daaa68c5607ea7f68ef0c08698de87608dc" + } + ], + "runtime_environment": { + "pip_freeze": [ + "Markdown==3.10.2", + "MarkupSafe==3.0.3", + "PyYAML==6.0.3", + "Pygments==2.20.0", + "Werkzeug==3.1.8", + "absl-py==1.4.0", + "array_record==0.8.1", + "astunparse==1.6.3", + "attrs==26.1.0", + "certifi==2026.7.22", + "cffi==2.1.0", + "charset-normalizer==3.4.9", + "chex==0.1.7", + "click==8.4.2", + "contourpy==1.3.2", + "cryptography==49.0.0", + "cycler==0.12.1", + "dm-pix==0.4.1", + "dm-tree==0.1.10", + "einops==0.8.2", + "etils==1.13.0", + "flatbuffers==25.12.19", + "flax==0.7.0", + "fonttools==4.63.0", + "fsspec==2026.6.0", + "gast==0.7.0", + "google-auth-oauthlib==1.4.0", + "google-auth==2.56.2", + "google-pasta==0.2.0", + "grpcio==1.82.1", + "h5py==3.16.0", + "idna==3.18", + "importlib_resources==7.1.0", + "jax==0.4.14", + "jaxlib==0.4.14+cuda11.cudnn86", + "keras==2.15.0", + "kiwisolver==1.5.0", + "libclang==18.1.1", + "markdown-it-py==4.2.0", + "matplotlib==3.8.0", + "mdurl==0.1.2", + "ml-dtypes==0.3.2", + "msgpack==1.2.1", + "nest-asyncio==1.6.0", + "numpy==1.25.2", + "nvidia-cublas-cu117==11.10.1.25", + "nvidia-cublas-cu11==11.11.3.6", + "nvidia-cuda-cupti-cu117==11.7.50", + "nvidia-cuda-cupti-cu11==11.8.87", + "nvidia-cuda-nvcc-cu117==11.7.64", + "nvidia-cuda-nvcc-cu11==2022.5.4", + "nvidia-cuda-runtime-cu117==11.7.60", + "nvidia-cuda-runtime-cu11==11.8.89", + "nvidia-cudnn-cu116==8.4.0.27", + "nvidia-cudnn-cu11==8.6.0.163", + "nvidia-cufft-cu117==10.7.2.50", + "nvidia-cufft-cu11==10.9.0.58", + "nvidia-curand-cu11==10.3.0.86", + "nvidia-cusolver-cu117==11.3.5.50", + "nvidia-cusolver-cu11==11.4.1.48", + "nvidia-cusparse-cu117==11.7.3.50", + "nvidia-cusparse-cu11==11.7.5.86", + "nvidia-nccl-cu11==2.19.3", + "oauthlib==3.3.1", + "opt_einsum==3.4.0", + "optax==0.1.7", + "orbax-checkpoint==0.3.5", + "packaging==26.0", + "pandas==2.3.3", + "pillow==12.3.0", + "promise==2.3", + "protobuf==3.20.3", + "psutil==7.2.2", + "pyasn1==0.6.4", + "pyasn1_modules==0.4.2", + "pycparser==3.0", + "pyparsing==3.3.2", + "python-dateutil==2.9.0.post0", + "pytz==2026.2", + "requests-oauthlib==2.0.0", + "requests==2.34.2", + "rich==15.0.0", + "scipy==1.11.4", + "seaborn==0.13.2", + "six==1.17.0", + "tensorboard-data-server==0.7.2", + "tensorboard==2.15.2", + "tensorflow-datasets==4.9.4", + "tensorflow-estimator==2.15.0", + "tensorflow-io-gcs-filesystem==0.37.1", + "tensorflow-metadata==1.15.0", + "tensorflow==2.15.1", + "tensorstore==0.1.74", + "termcolor==3.3.0", + "toml==0.10.2", + "toolz==1.1.0", + "tqdm==4.69.0", + "typing_extensions==4.16.0", + "tzdata==2026.3", + "urllib3==2.7.0", + "wrapt==1.14.2", + "zipp==4.1.0" + ], + "pip_freeze_sha256": "ea5bd886c4dcc61041aef10f12cf1a21ac018ae3e13cd64c64e3d90d5b1daf59", + "python_executable": "/scratch/yurenh2/venvs/dualprop/bin/python", + "python_version": "3.10.20 (main, Jun 11 2026, 15:17:37) [GCC 14.3.0]" + }, + "schema_version": 1, + "seed": 1988, + "source_provenance": { + "path": "/scratch/yurenh2/sdil-reference/dualprop_icml_2024", + "revision": "7b2595b34421e1483a721dbfdeff8cdabda3a1ff", + "tracked_dirty": false + } +} diff --git a/results/native_baseline_audit.md b/results/native_baseline_audit.md new file mode 100644 index 0000000..73262db --- /dev/null +++ b/results/native_baseline_audit.md @@ -0,0 +1,24 @@ +# Native author-code baseline audit + +Both records passed the strict C4 artifact, source, protocol, environment, +dataset, completeness, finiteness, selection, and cost-definition checks. +These are method-native reproductions, not equal-compute comparisons with SDIL. + +| method | reproduced seed | primary test (%) | validation-selected test (%) | test-selected test (%) | published (%) | audited wall (s) | +|:--|--:|--:|--:|--:|--:|--:| +| BurstCCN | 0 | 80.070 | 80.100 | 80.250 | 82.97 ± 0.21 | 15451.3 | +| Dual Prop | 1988 | 92.460 | 92.460 | — | 92.41 ± 0.07 | 23119.8 | + +BurstCCN's primary metric is its final epoch. Its validation-selected metric is +the test accuracy from the minimum-validation-error epoch; its best test value +is explicitly test-selected because the native trainer evaluates test every +epoch. The `n_epochs=400` author configuration performs an epoch-0 evaluation +and 399 optimization epochs. Dual Prop evaluates test once after restoring the +best-validation checkpoint, so its primary and validation-selected values are +identical and no test-selected value exists. + +The BurstCCN wall value is W&B run time through the final epoch log. The Dual +Prop wall value sums author-measured train, validation, and final test time and +excludes setup, checkpoint I/O, and diagnostic plotting. One reproduced seed +establishes executable fidelity; published multi-seed uncertainty is retained +for context and is not replaced by a one-seed error bar. |