# Training LLM Without BP Working Log

## Rules For AI-Written Log Entries

This log is mainly written by an AI assistant. Each update must follow these rules.

1. Include a timestamp for what was done.
2. Track file changes, folder changes, commands, and where results are stored.
3. Use concise sentences with simple logic.
4. Be specific. Include implementation details, paths, commands, numbers, and outputs when they matter.
5. Start each write with an updated context paragraph. When adding new log entries, update the context paragraph at the same time.
6. Separate confirmed facts from assumptions or blockers.
7. Do not exaggerate progress. Record failed commands and unresolved issues.

## Current Context

As of 2026-07-21, Yuren's authoritative scaling contract is in `docs\BASELINE_SPEC.md`. The local `ept` branch `xiang` is fast-forwarded to current `origin/master` commit `ca209ee`. The first new rung is the specified width-only OLMo2-style `L12/C768/H12/T256` BP twin, approximately 135M parameters, not the assistant-selected 146M shape. Exact local count is 135,303,936 parameters. It keeps the 72M FineWeb-Edu 32k-BPE data, B24, 64-dimensional heads, Muon hybrid optimizer, weight decay 0.1, 500-step warmup, cosine schedule, BF16 autocast, seed protocol, validation cadence, and existing `casc_bp_train.py` code path. `--steps 440442` produces 440,443 complete updates and 2,706,081,792 tokens, approximately `20N`. The smallest new rung alone gets the `0.7×/1.0×/1.4×` BP LR sweep; the winner carries upward. At least two seeds are required before quoting a result.

## Active Task List

1. **Adaptive `β` tuning**
   - Measure the time-varying `β` noise floor and relaxation-stability ceiling.
   - Design a controller that keeps centered `±β` estimation inside the feasible window as the window narrows.
   - Tune optimizer learning rates independently from `β`.
   - Compare fixed schedules, online window estimation, conservative cap/floor control, and failure recovery.
   - Report guard events, gradient-estimator SNR, stability margin, validation loss, and added compute.
2. **First width-only scaling rung**
   - Authoritative first rung: `L12/C768/H12/T256`, exactly 135,303,936 parameters, from `docs\BASELINE_SPEC.md`.
   - Data: existing FineWeb-Edu `sample/10BT` bins and local 32k tokenizer.
   - Local exposure: 2,706,081,792 tokens over 440,443 B24/T256 updates, approximately `20N`.
   - Primary baseline: train a matched BP twin with a `7e-4/1e-3/1.4e-3` LR sweep at this smallest new rung only.
   - Keep model/data/B/T/steps/optimizer/seeds/eval identical to EP; BP versus EP is the final scientific variable.
   - Require at least two seeds before reporting results.

## Log Entries

Note: entries are ordered newest-first (latest on top). Add new entries at the top of this section.

### 2026-07-21 10:50 AM Pacific — Moved the working log into the EPT repository

- User request: move the project working log into `ept\` and make it git ignored.
- Moved the log from `D:\stf\projects\Training LLM without bp\working_log.md` to `D:\stf\projects\Training LLM without bp\ept\working_log.md`.
- Added `working_log.md` to `ept\.gitignore`.
- The log remains local project memory and will not be staged by Git.
- No training, cloud, W&B, data, checkpoint, or source-code experiment command was started.

### 2026-07-21 9:39 AM Pacific — Adopted Yuren's authoritative baseline specification

- User reported that Yuren had written the width/depth plan in `docs\BASELINE_SPEC.md`.
- Confirmed that file was not present when the previous merge was performed; Yuren added it afterward in remote commit `ca209ee`.
- Fetched and read `origin/master:docs/BASELINE_SPEC.md`.
- Preserved local uncommitted setup, fast-forwarded `xiang` from `c43512c` to `ca209ee`, restored the setup, and dropped the temporary stash without conflicts.
- Replaced the assistant-selected 146M shape with Yuren's specified first rung: `L12/C768/H12/T256`, width-only, head dimension 64.
- Exact local parameter count is 135,303,936; exact `20N` target is 2,706,078,720 tokens.
- Configured `--steps 440442`; the historical inclusive loop performs 440,443 B24/T256 updates and consumes 2,706,081,792 tokens.
- Added `ep_run\fw135m_baseline.py`, `ep_run\baseline_configs\fw135m_matched_bp.json`, and `docs\campaign\FW135M_BP_BASELINE.md`.
- Deleted the superseded uncommitted `fw146m` launcher, config, and run sheet.
- Updated `experiments\baseline_research.md` to point to `docs\BASELINE_SPEC.md`.
- Contract retained: use `casc_bp_train.py` as-is; keep model/data/B/T/steps/optimizer/seeds/eval identical to EP; sweep BP LR `7e-4/1e-3/1.4e-3` at this smallest new rung only; carry the winner upward; require at least two seeds; report best, final, and tail-median CE.
- No training, cloud, W&B, or data command was started.

### 2026-07-21 8:42 AM Pacific — Corrected the 2× baseline to preserve the 72M recipe

- User questioned whether the new baseline actually matched the previous 72M runs.
- Confirmed the first 146M setup still changed too many variables: context 256→1024, global batch, optimizer, warmup/schedule units, and sampling/evaluation code.
- Replaced that setup with a strict continuation using the existing `casc_bp_train.py`.
- Active architecture: `L=12`, `C=800`, `H=10`, `T=256`, `B=24`, SwiGLU width 2176, exactly 145,856,800 parameters.
- Held fixed from 72M: L12 depth, FineWeb-Edu data, local 32k tokenizer, OLMo2-style block, context 256, B24, Muon LR 0.02, Adam-side LR center 1e-3, weight decay 0.1, 500-step warmup, cosine-to-0.1 schedule, BF16 autocast, and seed 1.
- Changed only model width/head count and total training length.
- Full BP run uses `--steps 474794`; because the historical trainer loops inclusively, this is 474,795 updates and 2,917,140,480 tokens, or `20.00003` tokens per parameter.
- BP LR sweep follows the latest project audit: `7e-4`, `1e-3`, and `1.4e-3`, with all other settings fixed.
- Removed the uncommitted standalone AdamW/DDP/token-scheduled trainer and its support files to prevent recipe confusion.
- Added `ep_run\fw146m_baseline.py`, `ep_run\baseline_configs\fw146m_matched_bp.json`, and rewrote `docs\campaign\FW146M_BP_BASELINE.md`.
- Rewrote `experiments\baseline_research.md` around the matched 72M→146M continuation.
- No training, W&B, cloud, or data command was started.

### 2026-07-21 8:22 AM Pacific — Merged latest master and retargeted baseline to approximately 2×

- User update: Yuren said the approximately 70–72M training is complete and requested the next baseline at approximately `72–74M × 2`, not 200M or the prior 269.55M proposal.
- Verified the remote default branch is named `master`, not `main`.
- Preserved all uncommitted baseline files with a temporary stash, fetched `origin`, fast-forwarded `xiang` from `e983a80` to `c43512c`, restored the stash, and dropped the temporary stash.
- The merge brought in Results 53–56 and the latest `casc_eq_train.py`/probe changes without conflicts.
- Retargeted the local BP baseline to `L=11`, `C=832`, `H=13`, `T=1024`, SwiGLU width 2240, and exactly 146,522,688 parameters.
- The new model is `2.032×` the completed 72,114,688-parameter crown.
- Set the full token endpoint to 2,930,453,760 (`20N`), the stage-1 LR pilot to 293,045,376 (`2N`), and warmup to 203,843,024 tokens.
- Renamed the locked config to `ep_run\baseline_configs\fw146m_bp.json` and the operator guide to `docs\campaign\FW146M_BP_BASELINE.md`.
- Updated trainer defaults, LR-sweep tags, parameter assertions, tests, and `experiments\baseline_research.md`.
- Improved final-update masking so the trainer can stop at the exact `20N` token endpoint even when that count is not divisible by context length 1024.
- Verification: configuration prints 146,522,688 parameters, 5,590 updates, and approximately 389 warmup updates; six unit tests pass; Python compilation passes; the LR sweep dry-run prints all five `fw146m` commands.
- No cloud job, W&B run, data freeze, or training run was started. Baseline setup files remain uncommitted on `xiang`.

### 2026-07-19 8:51 PM Pacific — Set up the 269.55M BP baseline and researched cloud GPUs

- Verified `D:\stf\projects\Training LLM without bp\ept` is on the clean local branch `xiang` at commit `e983a80`.
- Added `ep_run\casc_bp_baseline.py`, a standalone reviewer-facing BP trainer. Historical `casc_bp_train.py` and active EP β machinery were not changed.
- Added DDP, gradient accumulation, exact final-token masking, token-based warmup/cosine scheduling, deterministic stateless training offsets, fixed validation offsets, AdamW betas `(0.9,0.95)`, epsilon `1e-8`, grouped weight decay, gradient clipping, checkpoint resume, optional W&B, and parameter-count assertions.
- Added `ep_run\baseline_common.py`, `freeze_fineweb.py`, `baseline_lr_sweep.py`, `baseline_configs\fw269m_bp.json`, and `test_bp_baseline.py`.
- Added `docs\campaign\FW269M_BP_BASELINE.md` with data-freeze, smoke, LR-sweep, full-run, resume, and later W&B instructions.
- Added `ep_run\baseline_runs\` to `ept\.gitignore`.
- Updated `experiments\baseline_research.md` with implementation status.
- Added `experiments\cloud_gpu_research.md`.
- Cloud recommendation: benchmark Runpod Community/Secure A100 80GB and H100 80GB first; use Runpod for long runs if dollars per billion tokens wins. Modal is the easiest backup and currently advertises an academic grant application up to $10,000.
- Current advertised base rates recorded in the report: Runpod Community A100 PCIe approximately `$1.19/hour`, H100 PCIe approximately `$1.99/hour`; Modal A100 80GB approximately `$2.50/hour`, H100 approximately `$3.95/hour`.
- Verification: `python -m py_compile` passed; LR sweep dry-run printed five commands; six unit tests passed in the `cs224n_dfp` environment, including a tiny forward/backward optimizer step.
- Initial tests with base Anaconda failed because PyTorch is not installed there. Two direct environment commands then failed because PowerShell requires the call operator for a quoted executable path. Retried with `& "...\python.exe"` and all tests passed.
- No cloud account was modified, no credits were purchased, no W&B run was created, and no training/data-freeze command was started.

### 2026-07-19 6:15 PM Pacific — Rewrote the next-model baseline plan after collaborator guidance

- User provided Yuren Hao's guidance: either use released weights from a genuine open 20-times checkpoint or train a BP copy of a modern known architecture with settings strong enough to prevent reviewer objections about BP tuning.
- Re-researched Cerebras-GPT, OLMo2, FineWeb-Edu, learning-rate schedules, and the current local trainers.
- Rewrote `D:\stf\projects\Training LLM without bp\experiments\baseline_research.md`.
- Replaced the vague approximately 300M proposal with an exact local configuration: `L=16`, `C=1024`, `H=16`, SwiGLU width 2752, and 269,550,592 total parameters.
- Set the proposed local budget to 5,391,011,840 FineWeb-Edu tokens and context length 1024.
- Selected a newly trained, independently tuned same-architecture BP twin as the primary causal baseline.
- Selected released Cerebras-GPT-256M weights as the external schedule-complete 20-times anchor.
- Defined the BP AdamW recipe, LR grid, token-based warmup and decay, staged LR selection, paired-seed protocol, evaluation metrics, and claim limits.
- Found implementation gaps before production: the current local BP AdamW path uses default `β₂=0.999`; EP and BP do not prove identical sampled token order; and the trainers need gradient accumulation, exact token accounting, fixed validation indices, and BP DDP for the proposed global batch.
- Ran a Python arithmetic check. It returned `N=269550592`, `D=5391011840`, 10,283 updates at 524,288 tokens/update, and 716 updates for a 375M-token warmup.
- No training run was started, and no model checkpoint or prepared dataset was changed.

### 2026-07-17 9:48 AM Pacific — Clarified the proposed 300M control and documented prior baselines

- User clarification: job 1 concerns EP nudging strength `β`, not the optimizer learning rate.
- Clarified that the matched approximately 300M OLMo2-style BP twin is a proposed new control, not an existing public OLMo2 model.
- Confirmed that no official Ai2 OLMo2 checkpoint near 300M is available for this role.
- Created the main-folder directory `D:\stf\projects\Training LLM without bp\experiments`.
- Created `D:\stf\projects\Training LLM without bp\experiments\baseline_research.md`.
- Documented the internal BP baselines used for the Shakespeare, looped TinyStories, approximately 19M cascade, short approximately 45M, 42.75M full-epoch, 72.11M FineWeb, and width-ladder experiments.
- Documented the distinction between matched internal BP twins and published external checkpoints.
- Added the external candidate list, exclusion reasons, and recommended next-run protocol.
- No training command was run, and no experiment output or model checkpoint was changed.

### 2026-07-17 9:35 AM Pacific — Added research jobs and baseline shortlist

- User update: the 72.11M-parameter run is now the completed current scale.
- Added an `Active Task List` with two jobs: adaptive `β`/learning-rate tuning and selection of the next hundreds-of-millions baseline.
- Clarified that the squeezed variable shown in the supplied figure is EP nudging strength `β`, not the optimizer learning rate.
- Researched open language-model families using official papers, repositories, and model cards.
- Primary recommendation: train a matched approximately 300M OLMo2-style BP twin on the same FineWeb-Edu pipeline for approximately 6B tokens. This isolates the training rule.
- Best published external baseline: Cerebras-GPT-256M, trained on 5.12B Pile tokens at approximately 20 tokens per parameter.
- Larger published external anchor: Cerebras-GPT-590M, trained on 11.8B Pile tokens at approximately 20 tokens per parameter.
- Pythia-410M `step4000` is a secondary 8.39B-token snapshot, but its learning-rate schedule was designed for the full approximately 300B-token run and did not terminate at the Chinchilla point.
- Confirmed unsuitable final checkpoints for this protocol: SmolLM-360M used 600B tokens, SmolLM2-360M used 4T, OPT-350M used 180B, and final Pythia-410M used approximately 300B.
- Stored the detailed baseline comparison in the Cursor canvas `open-LM-baselines-100M-900M.canvas.tsx`.
- No training command was run, and no experiment output or model checkpoint was changed.

### 2026-07-17 6:19 AM Pacific — Added the initial project context

- User context: the project is a collaboration involving the user, Yuren Hao, and Alexi Gladstone.
- User goal: train an 8B language model without backpropagation with the project's training method and ultimately run the method on analog hardware.
- Read the `ept\` repository structure and its current project, experiment, method, and hardware documentation.
- Updated `D:\stf\projects\Training LLM without bp\working_log.md` with the confirmed training approach, active code lines, demonstrated scales, analog-hardware plan, long-term goal, and unresolved blockers.
- Recorded 8B analog training as an ultimate goal, not a completed result.
- Recorded the strongest documented scale results as 42.75M parameters for a full 361M-token epoch and 72.11M parameters for a 1.44B-token FineWeb run.
- No project source code, experiment outputs, data, or checkpoints were changed.
- No commands were run.

### 2026-07-17 3:19 AM Pacific — Created the project working log

- User request: create a working log in the main `Training LLM without bp` folder using the format of `D:\stf\projects\Curis\working_log.md`.
- Created `D:\stf\projects\Training LLM without bp\working_log.md`.
- Added the seven required rules for future AI-written updates.
- Added the `Current Context` and newest-first `Log Entries` sections.
- No commands were run.
- No project assumptions or blockers are recorded yet.