scaffold: model (OLMo2-ish + ZBP partition), trainer (DDP/config), data shards, bench

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GkgLsACEF6CCP7EUfA5fZe

1 files changed, 23 insertions, 0 deletions

diff --git a/README.md b/README.md
new file mode 100644
index 0000000..c9f5821
--- /dev/null
+++ b/README.md
@@ -0,0 +1,23 @@
+# zbp-scaling
+
+Scaling study for **ZBP (zeroth-order backpropagation)** vs BP on decoder-only language models
+(part 2 of the ZBP paper). Every nonlinear block is treated as a physical black box trained from
+forward queries only (score-space attention core + per-token SwiGLU FFN); linear maps and the score
+product are digital. See the main zobp repo for the method, theory (NSR ≈ c·d/n) and part-1/3 results.
+
+## Layout
+- `src/zbp_scaling/zbp/` — vendored ZBP package (probes, estimators, ZBPBlock autograd)
+- `src/zbp_scaling/model.py` — OLMo2-style transformer (RMSNorm, SwiGLU, untied embeddings; learned
+  positions in v1) with the ZBP physical/digital partition
+- `src/zbp_scaling/data.py` — FineWeb-Edu → GPT-2-BPE uint16 shards (WikiText-103 fallback for smoke)
+- `src/zbp_scaling/trainer.py` — DDP (torchrun, 4 or 8 GPUs), bf16 autocast with fp32 measurement
+  accumulation, gradient accumulation to a fixed global batch, cosine + warmup, resume, JSONL logs
+- `src/zbp_scaling/diagnostics.py` — per-block branch-gain rho_k profile and the NSR constant c(scale)
+- `configs/` — model sizes (60m/124m/350m/1b) x training arms (bp / zbp_n16 / zbp_n4)
+
+## Run
+```
+python scripts/prepare_data.py --dataset fineweb-edu --tokens 3e9 --out data/fineweb
+torchrun --nproc_per_node=8 scripts/train.py --model configs/model/m124.yaml --train configs/train/zbp_n16.yaml
+```
+Global batch is fixed in the train config; per-rank micro-batch and accumulation adapt to world size.