1 files changed, 166 insertions, 0 deletions
diff --git a/working_log.md b/working_log.md
new file mode 100644
index 0000000..7405d69
--- /dev/null
+++ b/working_log.md
@@ -0,0 +1,166 @@
+# Training LLM Without BP Working Log
+
+## Rules For AI-Written Log Entries
+
+This log is mainly written by an AI assistant. Each update must follow these rules.
+
+1. Include a timestamp for what was done.
+2. Track file changes, folder changes, commands, and where results are stored.
+3. Use concise sentences with simple logic.
+4. Be specific. Include implementation details, paths, commands, numbers, and outputs when they matter.
+5. Start each write with an updated context paragraph. When adding new log entries, update the context paragraph at the same time.
+6. Separate confirmed facts from assumptions or blockers.
+7. Do not exaggerate progress. Record failed commands and unresolved issues.
+
+## Current Context
+
+As of 2026-07-21, Yuren's authoritative scaling contract is in `docs\BASELINE_SPEC.md`. The local `ept` branch `xiang` is fast-forwarded to current `origin/master` commit `ca209ee`. The first new rung is the specified width-only OLMo2-style `L12/C768/H12/T256` BP twin, approximately 135M parameters, not the assistant-selected 146M shape. Exact local count is 135,303,936 parameters. It keeps the 72M FineWeb-Edu 32k-BPE data, B24, 64-dimensional heads, Muon hybrid optimizer, weight decay 0.1, 500-step warmup, cosine schedule, BF16 autocast, seed protocol, validation cadence, and existing `casc_bp_train.py` code path. `--steps 440442` produces 440,443 complete updates and 2,706,081,792 tokens, approximately `20N`. The smallest new rung alone gets the `0.7×/1.0×/1.4×` BP LR sweep; the winner carries upward. At least two seeds are required before quoting a result.
+
+## Active Task List
+
+1. **Adaptive `β` tuning**
+   - Measure the time-varying `β` noise floor and relaxation-stability ceiling.
+   - Design a controller that keeps centered `±β` estimation inside the feasible window as the window narrows.
+   - Tune optimizer learning rates independently from `β`.
+   - Compare fixed schedules, online window estimation, conservative cap/floor control, and failure recovery.
+   - Report guard events, gradient-estimator SNR, stability margin, validation loss, and added compute.
+2. **First width-only scaling rung**
+   - Authoritative first rung: `L12/C768/H12/T256`, exactly 135,303,936 parameters, from `docs\BASELINE_SPEC.md`.
+   - Data: existing FineWeb-Edu `sample/10BT` bins and local 32k tokenizer.
+   - Local exposure: 2,706,081,792 tokens over 440,443 B24/T256 updates, approximately `20N`.
+   - Primary baseline: train a matched BP twin with a `7e-4/1e-3/1.4e-3` LR sweep at this smallest new rung only.
+   - Keep model/data/B/T/steps/optimizer/seeds/eval identical to EP; BP versus EP is the final scientific variable.
+   - Require at least two seeds before reporting results.
+
+## Log Entries
+
+Note: entries are ordered newest-first (latest on top). Add new entries at the top of this section.
+
+### 2026-07-21 10:50 AM Pacific — Moved the working log into the EPT repository
+
+- User request: move the project working log into `ept\` and make it git ignored.
+- Moved the log from `D:\stf\projects\Training LLM without bp\working_log.md` to `D:\stf\projects\Training LLM without bp\ept\working_log.md`.
+- Added `working_log.md` to `ept\.gitignore`.
+- The log remains local project memory and will not be staged by Git.
+- No training, cloud, W&B, data, checkpoint, or source-code experiment command was started.
+
+### 2026-07-21 9:39 AM Pacific — Adopted Yuren's authoritative baseline specification
+
+- User reported that Yuren had written the width/depth plan in `docs\BASELINE_SPEC.md`.
+- Confirmed that file was not present when the previous merge was performed; Yuren added it afterward in remote commit `ca209ee`.
+- Fetched and read `origin/master:docs/BASELINE_SPEC.md`.
+- Preserved local uncommitted setup, fast-forwarded `xiang` from `c43512c` to `ca209ee`, restored the setup, and dropped the temporary stash without conflicts.
+- Replaced the assistant-selected 146M shape with Yuren's specified first rung: `L12/C768/H12/T256`, width-only, head dimension 64.
+- Exact local parameter count is 135,303,936; exact `20N` target is 2,706,078,720 tokens.
+- Configured `--steps 440442`; the historical inclusive loop performs 440,443 B24/T256 updates and consumes 2,706,081,792 tokens.
+- Added `ep_run\fw135m_baseline.py`, `ep_run\baseline_configs\fw135m_matched_bp.json`, and `docs\campaign\FW135M_BP_BASELINE.md`.
+- Deleted the superseded uncommitted `fw146m` launcher, config, and run sheet.
+- Updated `experiments\baseline_research.md` to point to `docs\BASELINE_SPEC.md`.
+- Contract retained: use `casc_bp_train.py` as-is; keep model/data/B/T/steps/optimizer/seeds/eval identical to EP; sweep BP LR `7e-4/1e-3/1.4e-3` at this smallest new rung only; carry the winner upward; require at least two seeds; report best, final, and tail-median CE.
+- No training, cloud, W&B, or data command was started.
+
+### 2026-07-21 8:42 AM Pacific — Corrected the 2× baseline to preserve the 72M recipe
+
+- User questioned whether the new baseline actually matched the previous 72M runs.
+- Confirmed the first 146M setup still changed too many variables: context 256→1024, global batch, optimizer, warmup/schedule units, and sampling/evaluation code.
+- Replaced that setup with a strict continuation using the existing `casc_bp_train.py`.
+- Active architecture: `L=12`, `C=800`, `H=10`, `T=256`, `B=24`, SwiGLU width 2176, exactly 145,856,800 parameters.
+- Held fixed from 72M: L12 depth, FineWeb-Edu data, local 32k tokenizer, OLMo2-style block, context 256, B24, Muon LR 0.02, Adam-side LR center 1e-3, weight decay 0.1, 500-step warmup, cosine-to-0.1 schedule, BF16 autocast, and seed 1.
+- Changed only model width/head count and total training length.
+- Full BP run uses `--steps 474794`; because the historical trainer loops inclusively, this is 474,795 updates and 2,917,140,480 tokens, or `20.00003` tokens per parameter.
+- BP LR sweep follows the latest project audit: `7e-4`, `1e-3`, and `1.4e-3`, with all other settings fixed.
+- Removed the uncommitted standalone AdamW/DDP/token-scheduled trainer and its support files to prevent recipe confusion.
+- Added `ep_run\fw146m_baseline.py`, `ep_run\baseline_configs\fw146m_matched_bp.json`, and rewrote `docs\campaign\FW146M_BP_BASELINE.md`.
+- Rewrote `experiments\baseline_research.md` around the matched 72M→146M continuation.
+- No training, W&B, cloud, or data command was started.
+
+### 2026-07-21 8:22 AM Pacific — Merged latest master and retargeted baseline to approximately 2×
+
+- User update: Yuren said the approximately 70–72M training is complete and requested the next baseline at approximately `72–74M × 2`, not 200M or the prior 269.55M proposal.
+- Verified the remote default branch is named `master`, not `main`.
+- Preserved all uncommitted baseline files with a temporary stash, fetched `origin`, fast-forwarded `xiang` from `e983a80` to `c43512c`, restored the stash, and dropped the temporary stash.
+- The merge brought in Results 53–56 and the latest `casc_eq_train.py`/probe changes without conflicts.
+- Retargeted the local BP baseline to `L=11`, `C=832`, `H=13`, `T=1024`, SwiGLU width 2240, and exactly 146,522,688 parameters.
+- The new model is `2.032×` the completed 72,114,688-parameter crown.
+- Set the full token endpoint to 2,930,453,760 (`20N`), the stage-1 LR pilot to 293,045,376 (`2N`), and warmup to 203,843,024 tokens.
+- Renamed the locked config to `ep_run\baseline_configs\fw146m_bp.json` and the operator guide to `docs\campaign\FW146M_BP_BASELINE.md`.
+- Updated trainer defaults, LR-sweep tags, parameter assertions, tests, and `experiments\baseline_research.md`.
+- Improved final-update masking so the trainer can stop at the exact `20N` token endpoint even when that count is not divisible by context length 1024.
+- Verification: configuration prints 146,522,688 parameters, 5,590 updates, and approximately 389 warmup updates; six unit tests pass; Python compilation passes; the LR sweep dry-run prints all five `fw146m` commands.
+- No cloud job, W&B run, data freeze, or training run was started. Baseline setup files remain uncommitted on `xiang`.
+
+### 2026-07-19 8:51 PM Pacific — Set up the 269.55M BP baseline and researched cloud GPUs
+
+- Verified `D:\stf\projects\Training LLM without bp\ept` is on the clean local branch `xiang` at commit `e983a80`.
+- Added `ep_run\casc_bp_baseline.py`, a standalone reviewer-facing BP trainer. Historical `casc_bp_train.py` and active EP β machinery were not changed.
+- Added DDP, gradient accumulation, exact final-token masking, token-based warmup/cosine scheduling, deterministic stateless training offsets, fixed validation offsets, AdamW betas `(0.9,0.95)`, epsilon `1e-8`, grouped weight decay, gradient clipping, checkpoint resume, optional W&B, and parameter-count assertions.
+- Added `ep_run\baseline_common.py`, `freeze_fineweb.py`, `baseline_lr_sweep.py`, `baseline_configs\fw269m_bp.json`, and `test_bp_baseline.py`.
+- Added `docs\campaign\FW269M_BP_BASELINE.md` with data-freeze, smoke, LR-sweep, full-run, resume, and later W&B instructions.
+- Added `ep_run\baseline_runs\` to `ept\.gitignore`.
+- Updated `experiments\baseline_research.md` with implementation status.
+- Added `experiments\cloud_gpu_research.md`.
+- Cloud recommendation: benchmark Runpod Community/Secure A100 80GB and H100 80GB first; use Runpod for long runs if dollars per billion tokens wins. Modal is the easiest backup and currently advertises an academic grant application up to $10,000.
+- Current advertised base rates recorded in the report: Runpod Community A100 PCIe approximately `$1.19/hour`, H100 PCIe approximately `$1.99/hour`; Modal A100 80GB approximately `$2.50/hour`, H100 approximately `$3.95/hour`.
+- Verification: `python -m py_compile` passed; LR sweep dry-run printed five commands; six unit tests passed in the `cs224n_dfp` environment, including a tiny forward/backward optimizer step.
+- Initial tests with base Anaconda failed because PyTorch is not installed there. Two direct environment commands then failed because PowerShell requires the call operator for a quoted executable path. Retried with `& "...\python.exe"` and all tests passed.
+- No cloud account was modified, no credits were purchased, no W&B run was created, and no training/data-freeze command was started.
+
+### 2026-07-19 6:15 PM Pacific — Rewrote the next-model baseline plan after collaborator guidance
+
+- User provided Yuren Hao's guidance: either use released weights from a genuine open 20-times checkpoint or train a BP copy of a modern known architecture with settings strong enough to prevent reviewer objections about BP tuning.
+- Re-researched Cerebras-GPT, OLMo2, FineWeb-Edu, learning-rate schedules, and the current local trainers.
+- Rewrote `D:\stf\projects\Training LLM without bp\experiments\baseline_research.md`.
+- Replaced the vague approximately 300M proposal with an exact local configuration: `L=16`, `C=1024`, `H=16`, SwiGLU width 2752, and 269,550,592 total parameters.
+- Set the proposed local budget to 5,391,011,840 FineWeb-Edu tokens and context length 1024.
+- Selected a newly trained, independently tuned same-architecture BP twin as the primary causal baseline.
+- Selected released Cerebras-GPT-256M weights as the external schedule-complete 20-times anchor.
+- Defined the BP AdamW recipe, LR grid, token-based warmup and decay, staged LR selection, paired-seed protocol, evaluation metrics, and claim limits.
+- Found implementation gaps before production: the current local BP AdamW path uses default `β₂=0.999`; EP and BP do not prove identical sampled token order; and the trainers need gradient accumulation, exact token accounting, fixed validation indices, and BP DDP for the proposed global batch.
+- Ran a Python arithmetic check. It returned `N=269550592`, `D=5391011840`, 10,283 updates at 524,288 tokens/update, and 716 updates for a 375M-token warmup.
+- No training run was started, and no model checkpoint or prepared dataset was changed.
+
+### 2026-07-17 9:48 AM Pacific — Clarified the proposed 300M control and documented prior baselines
+
+- User clarification: job 1 concerns EP nudging strength `β`, not the optimizer learning rate.
+- Clarified that the matched approximately 300M OLMo2-style BP twin is a proposed new control, not an existing public OLMo2 model.
+- Confirmed that no official Ai2 OLMo2 checkpoint near 300M is available for this role.
+- Created the main-folder directory `D:\stf\projects\Training LLM without bp\experiments`.
+- Created `D:\stf\projects\Training LLM without bp\experiments\baseline_research.md`.
+- Documented the internal BP baselines used for the Shakespeare, looped TinyStories, approximately 19M cascade, short approximately 45M, 42.75M full-epoch, 72.11M FineWeb, and width-ladder experiments.
+- Documented the distinction between matched internal BP twins and published external checkpoints.
+- Added the external candidate list, exclusion reasons, and recommended next-run protocol.
+- No training command was run, and no experiment output or model checkpoint was changed.
+
+### 2026-07-17 9:35 AM Pacific — Added research jobs and baseline shortlist
+
+- User update: the 72.11M-parameter run is now the completed current scale.
+- Added an `Active Task List` with two jobs: adaptive `β`/learning-rate tuning and selection of the next hundreds-of-millions baseline.
+- Clarified that the squeezed variable shown in the supplied figure is EP nudging strength `β`, not the optimizer learning rate.
+- Researched open language-model families using official papers, repositories, and model cards.
+- Primary recommendation: train a matched approximately 300M OLMo2-style BP twin on the same FineWeb-Edu pipeline for approximately 6B tokens. This isolates the training rule.
+- Best published external baseline: Cerebras-GPT-256M, trained on 5.12B Pile tokens at approximately 20 tokens per parameter.
+- Larger published external anchor: Cerebras-GPT-590M, trained on 11.8B Pile tokens at approximately 20 tokens per parameter.
+- Pythia-410M `step4000` is a secondary 8.39B-token snapshot, but its learning-rate schedule was designed for the full approximately 300B-token run and did not terminate at the Chinchilla point.
+- Confirmed unsuitable final checkpoints for this protocol: SmolLM-360M used 600B tokens, SmolLM2-360M used 4T, OPT-350M used 180B, and final Pythia-410M used approximately 300B.
+- Stored the detailed baseline comparison in the Cursor canvas `open-LM-baselines-100M-900M.canvas.tsx`.
+- No training command was run, and no experiment output or model checkpoint was changed.
+
+### 2026-07-17 6:19 AM Pacific — Added the initial project context
+
+- User context: the project is a collaboration involving the user, Yuren Hao, and Alexi Gladstone.
+- User goal: train an 8B language model without backpropagation with the project's training method and ultimately run the method on analog hardware.
+- Read the `ept\` repository structure and its current project, experiment, method, and hardware documentation.
+- Updated `D:\stf\projects\Training LLM without bp\working_log.md` with the confirmed training approach, active code lines, demonstrated scales, analog-hardware plan, long-term goal, and unresolved blockers.
+- Recorded 8B analog training as an ultimate goal, not a completed result.
+- Recorded the strongest documented scale results as 42.75M parameters for a full 361M-token epoch and 72.11M parameters for a 1.44B-token FineWeb run.
+- No project source code, experiment outputs, data, or checkpoints were changed.
+- No commands were run.
+
+### 2026-07-17 3:19 AM Pacific — Created the project working log
+
+- User request: create a working log in the main `Training LLM without bp` folder using the format of `D:\stf\projects\Curis\working_log.md`.
+- Created `D:\stf\projects\Training LLM without bp\working_log.md`.
+- Added the seven required rules for future AI-written updates.
+- Added the `Current Context` and newest-first `Log Entries` sections.
+- No commands were run.
+- No project assumptions or blockers are recorded yet.