diff options
| author | Yuren Hao <yurenh2@illinois.edu> | 2026-07-03 06:06:09 -0500 |
|---|---|---|
| committer | Yuren Hao <yurenh2@illinois.edu> | 2026-07-03 06:06:09 -0500 |
| commit | 8626ac5d6cf6f548157cb349ea99b8b603b268ce (patch) | |
| tree | 04afed3ed84652154944a735272939980d888f53 /ONBOARDING.md | |
| parent | b7fab6a524c4c5cd29aaf9933fb150e7b7902a3f (diff) | |
Add pull_assets.py one-click restore (HF ept-assets) + ONBOARDING restore note
git clone + python pull_assets.py = full working tree: pulls TinyStories-BPE
data (~697M) and the 5 key checkpoints from the private HF dataset repo
blackhao0426/ept-assets straight into ep_run/{data,runs}.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014FAPDWQ49M5Ye3NpTndTpn
Diffstat (limited to 'ONBOARDING.md')
| -rw-r--r-- | ONBOARDING.md | 15 |
1 files changed, 10 insertions, 5 deletions
diff --git a/ONBOARDING.md b/ONBOARDING.md index f8f7dee..1b36a72 100644 --- a/ONBOARDING.md +++ b/ONBOARDING.md @@ -80,11 +80,16 @@ Diagnostics: add `--diag_cos 500` (log cos-to-BPTT over training) · `--init_ckp operator's 4-D fingerprint) · `--eigreg 0.1 --eig_margin 1.0` (leading-abscissa control, alt to `--jacreg`). BP baseline (fair control): `--mode bptt`. **All experiment processes must use `nohup`.** -**Getting the data & checkpoints (git-ignored — not in this repo):** -- **Data** (`ep_run/data/tinystories_bpe/`, ~712 MB): regenerate from the BPE tokenizer pipeline in `ep_run/` (build - the tokenizer + tokenize TinyStories → `train.bin` / `val.bin` / `meta.pkl`), or copy from the shared location. -- **Checkpoints** (`ep_run/runs/*.pt`, e.g. `redx_traj/s2000.pt` for warm-starting): ask Yuren for a share link — - too large for git. `s2000.pt` is the stable warm-start operator (see §5). +**Getting the data & checkpoints (git-ignored — not in this repo):** one command. +``` +python pull_assets.py # run from the repo root, after `huggingface-cli login` +``` +This pulls the TinyStories-BPE data (~697 MB) and the key checkpoints from a **private HF dataset repo** +(`blackhao0426/ept-assets`) straight into their correct paths — so **`git clone` + `python pull_assets.py` = a full +working tree**. It restores `ep_run/data/tinystories_bpe/` (`train.bin`/`val.bin`/`meta.pkl`) and +`ep_run/runs/{redx_traj/s2000.pt, ep_rr_ajr, ep_resreg_scratch, ep_fast_adaptive, bptt_clean}.pt`. `s2000.pt` is the +stable warm-start operator (see §5). *Prereqs:* `pip install -U huggingface_hub` and ask Yuren for access to the +private repo. (The full `runs/` history is larger; `pull_assets.py` fetches the load-bearing subset — ask for more.) ## 8. Deeper docs (organized under `docs/`) - **`docs/method/`** — `METHODS.md`, `EP_DERIVATION.md` (the EP/AsymEP gradient derivation), `ARCHITECTURE.md` |