working log updatexiang

1 files changed, 15 insertions, 3 deletions

diff --git a/working_log.md b/working_log.md
index 7405d69..2dbf8cf 100644
--- a/working_log.md
+++ b/working_log.md
@@ -14,7 +14,7 @@ This log is mainly written by an AI assistant. Each update must follow these rul
 
 ## Current Context
 
-As of 2026-07-21, Yuren's authoritative scaling contract is in `docs\BASELINE_SPEC.md`. The local `ept` branch `xiang` is fast-forwarded to current `origin/master` commit `ca209ee`. The first new rung is the specified width-only OLMo2-style `L12/C768/H12/T256` BP twin, approximately 135M parameters, not the assistant-selected 146M shape. Exact local count is 135,303,936 parameters. It keeps the 72M FineWeb-Edu 32k-BPE data, B24, 64-dimensional heads, Muon hybrid optimizer, weight decay 0.1, 500-step warmup, cosine schedule, BF16 autocast, seed protocol, validation cadence, and existing `casc_bp_train.py` code path. `--steps 440442` produces 440,443 complete updates and 2,706,081,792 tokens, approximately `20N`. The smallest new rung alone gets the `0.7×/1.0×/1.4×` BP LR sweep; the winner carries upward. At least two seeds are required before quoting a result.
+As of 2026-07-21, the 135M `L12/C768/H12/T256` OLMo2-style BP smoke test has passed on one RTX A6000 and synced to `eqprop-llm-training/ept-fineweb-135M`. FineWeb-Edu 32k bins are prepared locally under `ep_run/data/fineweb_edu/`. To twin the active `fw135m_bsign` EP run, the BP sweep uses B24 per rank, 4× NCCL data parallelism (effective batch 96), `--steps 440000` (440,001 inclusive updates), 1,000 warmup steps, Muon, BF16, weight decay 0.1, and cosine-to-0.1. The only intended scientific difference is BP versus EP's random-sign beta rule. The BP LR candidates are `7e-4`, `1e-3`, and `1.4e-3`; select the winner by validation metrics and run at least two BP seeds before reporting. Local handoff work is committed on `xiang` as `ad4622e`; the user reported pushing through SSH, but the assistant did not independently verify the remote branch because its local `origin` remained HTTPS-authenticated.
 
 ## Active Task List
 
@@ -27,15 +27,27 @@ As of 2026-07-21, Yuren's authoritative scaling contract is in `docs\BASELINE_SP
 2. **First width-only scaling rung**
    - Authoritative first rung: `L12/C768/H12/T256`, exactly 135,303,936 parameters, from `docs\BASELINE_SPEC.md`.
    - Data: existing FineWeb-Edu `sample/10BT` bins and local 32k tokenizer.
-   - Local exposure: 2,706,081,792 tokens over 440,443 B24/T256 updates, approximately `20N`.
+   - EP-matched exposure: 2,703,366,144 tokens over 440,001 B24/T256 updates, approximately `20N`.
    - Primary baseline: train a matched BP twin with a `7e-4/1e-3/1.4e-3` LR sweep at this smallest new rung only.
-   - Keep model/data/B/T/steps/optimizer/seeds/eval identical to EP; BP versus EP is the final scientific variable.
+   - Keep model/data/B/T/steps/optimizer/seeds/eval and four-rank DDP geometry identical to EP; BP versus EP is the final scientific variable.
    - Require at least two seeds before reporting results.
 
 ## Log Entries
 
 Note: entries are ordered newest-first (latest on top). Add new entries at the top of this section.
 
+### 2026-07-21 3:44 PM Pacific — Prepared and handed off the 135M BP twin sweep
+
+- FineWeb-Edu `sample/10BT` data preparation completed in Slurm job `16290543`: 14 raw shards, 32k tokenizer, 9,971,205,360 training tokens, and 20,000,000 validation tokens under `ep_run/data/fineweb_edu/`.
+- The one-A6000 smoke job `16291921` completed 400/400 steps in 11m32s. Run `fw135m_bp_smoke_s1` synced to `eqprop-llm-training/ept-fineweb-135M`; best validation CE was `5.71256`.
+- Made the data path portable with `EPT_DATA_ROOT` in `ep_run/casc_bp_train.py` and `ep_run/prepare_fineweb.py`.
+- Compared the active W&B `fw135m_bsign` EP run configuration and aligned BP shared settings to its `--steps 440000` and `--warmup 1000` arguments.
+- Added manual NCCL data parallel support to `casc_bp_train.py`: rank-specific data generators, parameter broadcast, gradient mean all-reduce, rank-0-only W&B/checkpoints/logging, and process-group teardown. This passed compilation/static checks but has not yet been exercised in a four-GPU smoke job.
+- Added `ep_run/runs/fw135m_bp_sweep.sh` for one LR candidate; with `GPUS=4`, it runs `torchrun --standalone --nproc_per_node=4`. Added `docs/campaign/FW135M_BP_HANDOFF.md` with the Yuren handoff commands.
+- Added local Orion Slurm scripts under `sbatch/` for smoke, LR sweep, and full seeds. They were useful for the one-GPU smoke but are not the intended four-GPU handoff path.
+- Committed the handoff set locally on `xiang` as `ad4622e` (`Add portable 135M BP sweep handoff`). GitHub HTTPS authentication failed from the assistant terminal. The user later reported an SSH push, but remote verification was not possible from that terminal because its `origin` URL remained HTTPS.
+- Next action: Yuren should fetch `xiang`, run a short four-A6000 NCCL BP smoke test, then run the three full LR candidates sequentially on all four GPUs if the smoke passes.
+
 ### 2026-07-21 10:50 AM Pacific — Moved the working log into the EPT repository
 
 - User request: move the project working log into `ept\` and make it git ignored.