diff options
Diffstat (limited to 'README.md')
| -rw-r--r-- | README.md | 9 |
1 files changed, 9 insertions, 0 deletions
@@ -5,6 +5,11 @@ Scaling study for **ZBP (zeroth-order backpropagation)** vs BP on decoder-only l forward queries only (score-space attention core + per-token SwiGLU FFN); linear maps and the score product are digital. See the main zobp repo for the method, theory (NSR ≈ c·d/n) and part-1/3 results. +## Install +``` +pip install -e . # or: pip install -e .[dev] && pytest tests/ +``` + ## Layout - `src/zbp_scaling/zbp/` — vendored ZBP package (probes, estimators, ZBPBlock autograd) - `src/zbp_scaling/model.py` — OLMo2-style transformer (RMSNorm, SwiGLU, untied embeddings; learned @@ -28,3 +33,7 @@ BP 312 ms/step; ZBP n=16 **8.7x**, n=64 **29x** (FLOPs-bound: probe batching is probe_chunk 8; the score core uses its own chunk <= 4 since its memory goes as chunk*B*H*T^2). Ladder arms: **bp / zbp_n16 / zbp_n64** per size (n=4 optional). Headroom if needed: torch.compile on the query path; forward differences (n+1 instead of 2n queries) as a cheaper biased arm. + +## Citation +Paper in preparation ("Backpropagation Without Jacobians"); see the main zobp research repo for the +method, theory and part-1/part-3 results. License: Apache-2.0. |