1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
|
2025-12-25 08:58:07,929 - INFO - Loaded dataset: math-500
2025-12-25 08:58:08,121 - INFO - Loaded 100 profiles from ../data/complex_profiles_v2/profiles_100.jsonl
2025-12-25 08:58:08,122 - INFO - Running method: vanilla
2025-12-25 08:58:08,122 - INFO - Profile 1/20
`torch_dtype` is deprecated! Use `dtype` instead!
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:06, 2.33s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:06<00:06, 3.19s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:09<00:03, 3.35s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:09<00:00, 2.10s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:09<00:00, 2.46s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:03<00:14, 3.68s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:10<00:16, 5.46s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:15<00:10, 5.08s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:17<00:04, 4.22s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:19<00:00, 3.30s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:19<00:00, 3.91s/it]
2025-12-25 08:58:47,629 - ERROR - Error in session: 'NoneType' object is not subscriptable
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:03<00:10, 3.44s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:08<00:08, 4.11s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:12<00:04, 4.27s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:12<00:00, 2.65s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:12<00:00, 3.16s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:03<00:13, 3.45s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:09<00:15, 5.12s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:13<00:09, 4.65s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:16<00:04, 4.00s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:18<00:00, 3.10s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:18<00:00, 3.67s/it]
2025-12-25 08:59:23,124 - ERROR - Error in session: 'NoneType' object is not subscriptable
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.95s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:07<00:07, 3.65s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:11<00:03, 3.88s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:11<00:00, 2.41s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:11<00:00, 2.85s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:09, 2.41s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:09<00:15, 5.30s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:14<00:09, 4.86s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:17<00:04, 4.12s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:18<00:00, 3.11s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:18<00:00, 3.68s/it]
2025-12-25 08:59:57,630 - ERROR - Error in session: 'NoneType' object is not subscriptable
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.88s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:06<00:07, 3.60s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:11<00:03, 3.84s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:11<00:00, 2.39s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:11<00:00, 2.82s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:01<00:07, 1.91s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:08<00:14, 4.87s/it][2025-12-25T09:00:23.259] error: *** JOB 14355885 ON gpua055 CANCELLED AT 2025-12-25T09:00:23 DUE to SIGNAL Terminated ***
Loading checkpoint shards: 60%|██████ | 3/5 [00:13<00:09, 4.74s/it]
|
