1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
|
2025-12-25 09:02:11,022 - INFO - Loaded dataset: math-500
2025-12-25 09:02:11,141 - INFO - Loaded 100 profiles from ../data/complex_profiles_v2/profiles_100.jsonl
2025-12-25 09:02:11,142 - INFO - Running method: vanilla
2025-12-25 09:02:11,142 - INFO - Profile 1/20
`torch_dtype` is deprecated! Use `dtype` instead!
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.63s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:06<00:07, 3.64s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:11<00:03, 3.88s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:11<00:00, 2.41s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:11<00:00, 2.83s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:01<00:07, 1.91s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:08<00:14, 4.71s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:13<00:10, 5.03s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:17<00:04, 4.38s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:18<00:00, 3.35s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:18<00:00, 3.78s/it]
2025-12-25 09:02:50,489 - ERROR - Error in session: 'NoneType' object is not subscriptable
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.35s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:06<00:07, 3.60s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:11<00:03, 3.88s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:11<00:00, 2.41s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:11<00:00, 2.80s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:09, 2.48s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:08<00:14, 4.72s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:13<00:09, 4.85s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:16<00:04, 4.05s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:18<00:00, 3.15s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:18<00:00, 3.63s/it]
2025-12-25 09:03:24,392 - ERROR - Error in session: 'NoneType' object is not subscriptable
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:06, 2.32s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:06<00:07, 3.66s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:10<00:03, 3.84s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:11<00:00, 2.39s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:11<00:00, 2.79s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:01<00:07, 1.91s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:08<00:13, 4.62s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:12<00:08, 4.45s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:15<00:03, 3.70s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:16<00:00, 2.90s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:16<00:00, 3.34s/it]
2025-12-25 09:03:57,453 - ERROR - Error in session: 'NoneType' object is not subscriptable
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:03<00:10, 3.46s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:07<00:07, 3.87s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:12<00:04, 4.51s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:13<00:00, 2.79s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:13<00:00, 3.26s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:01<00:07, 1.93s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:08<00:13, 4.61s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:12<00:08, 4.45s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:15<00:03, 3.67s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:16<00:00, 2.80s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:16<00:00, 3.28s/it]
2025-12-25 09:04:31,571 - ERROR - Error in session: 'NoneType' object is not subscriptable
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.83s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:06<00:06, 3.43s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:11<00:03, 3.86s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:11<00:00, 2.40s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:11<00:00, 2.81s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.85s/it][2025-12-25T09:04:49.962] error: *** JOB 14355888 ON gpua055 CANCELLED AT 2025-12-25T09:04:49 DUE to SIGNAL Terminated ***
|
