From 09c52fa97a946cf6dbdc500aef88da8522c2ce7d Mon Sep 17 00:00:00 2001 From: Oscar Wan Date: Tue, 21 Jul 2026 10:54:05 -0700 Subject: add working log --- working_log.md | 166 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 166 insertions(+) create mode 100644 working_log.md diff --git a/working_log.md b/working_log.md new file mode 100644 index 0000000..7405d69 --- /dev/null +++ b/working_log.md @@ -0,0 +1,166 @@ +# Training LLM Without BP Working Log + +## Rules For AI-Written Log Entries + +This log is mainly written by an AI assistant. Each update must follow these rules. + +1. Include a timestamp for what was done. +2. Track file changes, folder changes, commands, and where results are stored. +3. Use concise sentences with simple logic. +4. Be specific. Include implementation details, paths, commands, numbers, and outputs when they matter. +5. Start each write with an updated context paragraph. When adding new log entries, update the context paragraph at the same time. +6. Separate confirmed facts from assumptions or blockers. +7. Do not exaggerate progress. Record failed commands and unresolved issues. + +## Current Context + +As of 2026-07-21, Yuren's authoritative scaling contract is in `docs\BASELINE_SPEC.md`. The local `ept` branch `xiang` is fast-forwarded to current `origin/master` commit `ca209ee`. The first new rung is the specified width-only OLMo2-style `L12/C768/H12/T256` BP twin, approximately 135M parameters, not the assistant-selected 146M shape. Exact local count is 135,303,936 parameters. It keeps the 72M FineWeb-Edu 32k-BPE data, B24, 64-dimensional heads, Muon hybrid optimizer, weight decay 0.1, 500-step warmup, cosine schedule, BF16 autocast, seed protocol, validation cadence, and existing `casc_bp_train.py` code path. `--steps 440442` produces 440,443 complete updates and 2,706,081,792 tokens, approximately `20N`. The smallest new rung alone gets the `0.7×/1.0×/1.4×` BP LR sweep; the winner carries upward. At least two seeds are required before quoting a result. + +## Active Task List + +1. **Adaptive `β` tuning** + - Measure the time-varying `β` noise floor and relaxation-stability ceiling. + - Design a controller that keeps centered `±β` estimation inside the feasible window as the window narrows. + - Tune optimizer learning rates independently from `β`. + - Compare fixed schedules, online window estimation, conservative cap/floor control, and failure recovery. + - Report guard events, gradient-estimator SNR, stability margin, validation loss, and added compute. +2. **First width-only scaling rung** + - Authoritative first rung: `L12/C768/H12/T256`, exactly 135,303,936 parameters, from `docs\BASELINE_SPEC.md`. + - Data: existing FineWeb-Edu `sample/10BT` bins and local 32k tokenizer. + - Local exposure: 2,706,081,792 tokens over 440,443 B24/T256 updates, approximately `20N`. + - Primary baseline: train a matched BP twin with a `7e-4/1e-3/1.4e-3` LR sweep at this smallest new rung only. + - Keep model/data/B/T/steps/optimizer/seeds/eval identical to EP; BP versus EP is the final scientific variable. + - Require at least two seeds before reporting results. + +## Log Entries + +Note: entries are ordered newest-first (latest on top). Add new entries at the top of this section. + +### 2026-07-21 10:50 AM Pacific — Moved the working log into the EPT repository + +- User request: move the project working log into `ept\` and make it git ignored. +- Moved the log from `D:\stf\projects\Training LLM without bp\working_log.md` to `D:\stf\projects\Training LLM without bp\ept\working_log.md`. +- Added `working_log.md` to `ept\.gitignore`. +- The log remains local project memory and will not be staged by Git. +- No training, cloud, W&B, data, checkpoint, or source-code experiment command was started. + +### 2026-07-21 9:39 AM Pacific — Adopted Yuren's authoritative baseline specification + +- User reported that Yuren had written the width/depth plan in `docs\BASELINE_SPEC.md`. +- Confirmed that file was not present when the previous merge was performed; Yuren added it afterward in remote commit `ca209ee`. +- Fetched and read `origin/master:docs/BASELINE_SPEC.md`. +- Preserved local uncommitted setup, fast-forwarded `xiang` from `c43512c` to `ca209ee`, restored the setup, and dropped the temporary stash without conflicts. +- Replaced the assistant-selected 146M shape with Yuren's specified first rung: `L12/C768/H12/T256`, width-only, head dimension 64. +- Exact local parameter count is 135,303,936; exact `20N` target is 2,706,078,720 tokens. +- Configured `--steps 440442`; the historical inclusive loop performs 440,443 B24/T256 updates and consumes 2,706,081,792 tokens. +- Added `ep_run\fw135m_baseline.py`, `ep_run\baseline_configs\fw135m_matched_bp.json`, and `docs\campaign\FW135M_BP_BASELINE.md`. +- Deleted the superseded uncommitted `fw146m` launcher, config, and run sheet. +- Updated `experiments\baseline_research.md` to point to `docs\BASELINE_SPEC.md`. +- Contract retained: use `casc_bp_train.py` as-is; keep model/data/B/T/steps/optimizer/seeds/eval identical to EP; sweep BP LR `7e-4/1e-3/1.4e-3` at this smallest new rung only; carry the winner upward; require at least two seeds; report best, final, and tail-median CE. +- No training, cloud, W&B, or data command was started. + +### 2026-07-21 8:42 AM Pacific — Corrected the 2× baseline to preserve the 72M recipe + +- User questioned whether the new baseline actually matched the previous 72M runs. +- Confirmed the first 146M setup still changed too many variables: context 256→1024, global batch, optimizer, warmup/schedule units, and sampling/evaluation code. +- Replaced that setup with a strict continuation using the existing `casc_bp_train.py`. +- Active architecture: `L=12`, `C=800`, `H=10`, `T=256`, `B=24`, SwiGLU width 2176, exactly 145,856,800 parameters. +- Held fixed from 72M: L12 depth, FineWeb-Edu data, local 32k tokenizer, OLMo2-style block, context 256, B24, Muon LR 0.02, Adam-side LR center 1e-3, weight decay 0.1, 500-step warmup, cosine-to-0.1 schedule, BF16 autocast, and seed 1. +- Changed only model width/head count and total training length. +- Full BP run uses `--steps 474794`; because the historical trainer loops inclusively, this is 474,795 updates and 2,917,140,480 tokens, or `20.00003` tokens per parameter. +- BP LR sweep follows the latest project audit: `7e-4`, `1e-3`, and `1.4e-3`, with all other settings fixed. +- Removed the uncommitted standalone AdamW/DDP/token-scheduled trainer and its support files to prevent recipe confusion. +- Added `ep_run\fw146m_baseline.py`, `ep_run\baseline_configs\fw146m_matched_bp.json`, and rewrote `docs\campaign\FW146M_BP_BASELINE.md`. +- Rewrote `experiments\baseline_research.md` around the matched 72M→146M continuation. +- No training, W&B, cloud, or data command was started. + +### 2026-07-21 8:22 AM Pacific — Merged latest master and retargeted baseline to approximately 2× + +- User update: Yuren said the approximately 70–72M training is complete and requested the next baseline at approximately `72–74M × 2`, not 200M or the prior 269.55M proposal. +- Verified the remote default branch is named `master`, not `main`. +- Preserved all uncommitted baseline files with a temporary stash, fetched `origin`, fast-forwarded `xiang` from `e983a80` to `c43512c`, restored the stash, and dropped the temporary stash. +- The merge brought in Results 53–56 and the latest `casc_eq_train.py`/probe changes without conflicts. +- Retargeted the local BP baseline to `L=11`, `C=832`, `H=13`, `T=1024`, SwiGLU width 2240, and exactly 146,522,688 parameters. +- The new model is `2.032×` the completed 72,114,688-parameter crown. +- Set the full token endpoint to 2,930,453,760 (`20N`), the stage-1 LR pilot to 293,045,376 (`2N`), and warmup to 203,843,024 tokens. +- Renamed the locked config to `ep_run\baseline_configs\fw146m_bp.json` and the operator guide to `docs\campaign\FW146M_BP_BASELINE.md`. +- Updated trainer defaults, LR-sweep tags, parameter assertions, tests, and `experiments\baseline_research.md`. +- Improved final-update masking so the trainer can stop at the exact `20N` token endpoint even when that count is not divisible by context length 1024. +- Verification: configuration prints 146,522,688 parameters, 5,590 updates, and approximately 389 warmup updates; six unit tests pass; Python compilation passes; the LR sweep dry-run prints all five `fw146m` commands. +- No cloud job, W&B run, data freeze, or training run was started. Baseline setup files remain uncommitted on `xiang`. + +### 2026-07-19 8:51 PM Pacific — Set up the 269.55M BP baseline and researched cloud GPUs + +- Verified `D:\stf\projects\Training LLM without bp\ept` is on the clean local branch `xiang` at commit `e983a80`. +- Added `ep_run\casc_bp_baseline.py`, a standalone reviewer-facing BP trainer. Historical `casc_bp_train.py` and active EP β machinery were not changed. +- Added DDP, gradient accumulation, exact final-token masking, token-based warmup/cosine scheduling, deterministic stateless training offsets, fixed validation offsets, AdamW betas `(0.9,0.95)`, epsilon `1e-8`, grouped weight decay, gradient clipping, checkpoint resume, optional W&B, and parameter-count assertions. +- Added `ep_run\baseline_common.py`, `freeze_fineweb.py`, `baseline_lr_sweep.py`, `baseline_configs\fw269m_bp.json`, and `test_bp_baseline.py`. +- Added `docs\campaign\FW269M_BP_BASELINE.md` with data-freeze, smoke, LR-sweep, full-run, resume, and later W&B instructions. +- Added `ep_run\baseline_runs\` to `ept\.gitignore`. +- Updated `experiments\baseline_research.md` with implementation status. +- Added `experiments\cloud_gpu_research.md`. +- Cloud recommendation: benchmark Runpod Community/Secure A100 80GB and H100 80GB first; use Runpod for long runs if dollars per billion tokens wins. Modal is the easiest backup and currently advertises an academic grant application up to $10,000. +- Current advertised base rates recorded in the report: Runpod Community A100 PCIe approximately `$1.19/hour`, H100 PCIe approximately `$1.99/hour`; Modal A100 80GB approximately `$2.50/hour`, H100 approximately `$3.95/hour`. +- Verification: `python -m py_compile` passed; LR sweep dry-run printed five commands; six unit tests passed in the `cs224n_dfp` environment, including a tiny forward/backward optimizer step. +- Initial tests with base Anaconda failed because PyTorch is not installed there. Two direct environment commands then failed because PowerShell requires the call operator for a quoted executable path. Retried with `& "...\python.exe"` and all tests passed. +- No cloud account was modified, no credits were purchased, no W&B run was created, and no training/data-freeze command was started. + +### 2026-07-19 6:15 PM Pacific — Rewrote the next-model baseline plan after collaborator guidance + +- User provided Yuren Hao's guidance: either use released weights from a genuine open 20-times checkpoint or train a BP copy of a modern known architecture with settings strong enough to prevent reviewer objections about BP tuning. +- Re-researched Cerebras-GPT, OLMo2, FineWeb-Edu, learning-rate schedules, and the current local trainers. +- Rewrote `D:\stf\projects\Training LLM without bp\experiments\baseline_research.md`. +- Replaced the vague approximately 300M proposal with an exact local configuration: `L=16`, `C=1024`, `H=16`, SwiGLU width 2752, and 269,550,592 total parameters. +- Set the proposed local budget to 5,391,011,840 FineWeb-Edu tokens and context length 1024. +- Selected a newly trained, independently tuned same-architecture BP twin as the primary causal baseline. +- Selected released Cerebras-GPT-256M weights as the external schedule-complete 20-times anchor. +- Defined the BP AdamW recipe, LR grid, token-based warmup and decay, staged LR selection, paired-seed protocol, evaluation metrics, and claim limits. +- Found implementation gaps before production: the current local BP AdamW path uses default `β₂=0.999`; EP and BP do not prove identical sampled token order; and the trainers need gradient accumulation, exact token accounting, fixed validation indices, and BP DDP for the proposed global batch. +- Ran a Python arithmetic check. It returned `N=269550592`, `D=5391011840`, 10,283 updates at 524,288 tokens/update, and 716 updates for a 375M-token warmup. +- No training run was started, and no model checkpoint or prepared dataset was changed. + +### 2026-07-17 9:48 AM Pacific — Clarified the proposed 300M control and documented prior baselines + +- User clarification: job 1 concerns EP nudging strength `β`, not the optimizer learning rate. +- Clarified that the matched approximately 300M OLMo2-style BP twin is a proposed new control, not an existing public OLMo2 model. +- Confirmed that no official Ai2 OLMo2 checkpoint near 300M is available for this role. +- Created the main-folder directory `D:\stf\projects\Training LLM without bp\experiments`. +- Created `D:\stf\projects\Training LLM without bp\experiments\baseline_research.md`. +- Documented the internal BP baselines used for the Shakespeare, looped TinyStories, approximately 19M cascade, short approximately 45M, 42.75M full-epoch, 72.11M FineWeb, and width-ladder experiments. +- Documented the distinction between matched internal BP twins and published external checkpoints. +- Added the external candidate list, exclusion reasons, and recommended next-run protocol. +- No training command was run, and no experiment output or model checkpoint was changed. + +### 2026-07-17 9:35 AM Pacific — Added research jobs and baseline shortlist + +- User update: the 72.11M-parameter run is now the completed current scale. +- Added an `Active Task List` with two jobs: adaptive `β`/learning-rate tuning and selection of the next hundreds-of-millions baseline. +- Clarified that the squeezed variable shown in the supplied figure is EP nudging strength `β`, not the optimizer learning rate. +- Researched open language-model families using official papers, repositories, and model cards. +- Primary recommendation: train a matched approximately 300M OLMo2-style BP twin on the same FineWeb-Edu pipeline for approximately 6B tokens. This isolates the training rule. +- Best published external baseline: Cerebras-GPT-256M, trained on 5.12B Pile tokens at approximately 20 tokens per parameter. +- Larger published external anchor: Cerebras-GPT-590M, trained on 11.8B Pile tokens at approximately 20 tokens per parameter. +- Pythia-410M `step4000` is a secondary 8.39B-token snapshot, but its learning-rate schedule was designed for the full approximately 300B-token run and did not terminate at the Chinchilla point. +- Confirmed unsuitable final checkpoints for this protocol: SmolLM-360M used 600B tokens, SmolLM2-360M used 4T, OPT-350M used 180B, and final Pythia-410M used approximately 300B. +- Stored the detailed baseline comparison in the Cursor canvas `open-LM-baselines-100M-900M.canvas.tsx`. +- No training command was run, and no experiment output or model checkpoint was changed. + +### 2026-07-17 6:19 AM Pacific — Added the initial project context + +- User context: the project is a collaboration involving the user, Yuren Hao, and Alexi Gladstone. +- User goal: train an 8B language model without backpropagation with the project's training method and ultimately run the method on analog hardware. +- Read the `ept\` repository structure and its current project, experiment, method, and hardware documentation. +- Updated `D:\stf\projects\Training LLM without bp\working_log.md` with the confirmed training approach, active code lines, demonstrated scales, analog-hardware plan, long-term goal, and unresolved blockers. +- Recorded 8B analog training as an ultimate goal, not a completed result. +- Recorded the strongest documented scale results as 42.75M parameters for a full 361M-token epoch and 72.11M parameters for a 1.44B-token FineWeb run. +- No project source code, experiment outputs, data, or checkpoints were changed. +- No commands were run. + +### 2026-07-17 3:19 AM Pacific — Created the project working log + +- User request: create a working log in the main `Training LLM without bp` folder using the format of `D:\stf\projects\Curis\working_log.md`. +- Created `D:\stf\projects\Training LLM without bp\working_log.md`. +- Added the seven required rules for future AI-written updates. +- Added the `Current Context` and newest-first `Log Entries` sections. +- No commands were run. +- No project assumptions or blockers are recorded yet. -- cgit v1.2.3