1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
|
2025-12-25 08:39:20,163 - INFO - Loaded dataset: math-500
2025-12-25 08:39:20,285 - INFO - Loaded 100 profiles from ../data/complex_profiles_v2/profiles_100.jsonl
2025-12-25 08:39:20,285 - INFO - Running method: vanilla
2025-12-25 08:39:20,285 - INFO - Profile 1/20
`torch_dtype` is deprecated! Use `dtype` instead!
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.97s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:09<00:10, 5.18s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:13<00:04, 4.72s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:14<00:00, 3.17s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:14<00:00, 3.66s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:01<00:07, 1.93s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:08<00:13, 4.62s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:14<00:10, 5.27s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:18<00:04, 4.62s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:19<00:00, 3.36s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:19<00:00, 3.85s/it]
2025-12-25 08:40:03,460 - ERROR - Error in session: 'NoneType' object is not subscriptable
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:03<00:09, 3.11s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:07<00:08, 4.15s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:12<00:04, 4.14s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:12<00:00, 2.57s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:12<00:00, 3.07s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:01<00:07, 1.92s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:08<00:14, 4.72s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:14<00:10, 5.21s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:17<00:04, 4.32s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:18<00:00, 3.11s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:18<00:00, 3.66s/it]
2025-12-25 08:40:38,434 - ERROR - Error in session: 'NoneType' object is not subscriptable
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.68s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:07<00:07, 3.98s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:12<00:04, 4.20s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:12<00:00, 2.61s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:12<00:00, 3.05s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:01<00:07, 1.92s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.59s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:12<00:09, 4.61s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:15<00:04, 4.02s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:16<00:00, 2.90s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:16<00:00, 3.30s/it]
2025-12-25 08:41:11,060 - ERROR - Error in session: 'NoneType' object is not subscriptable
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.94s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:07<00:08, 4.15s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:12<00:04, 4.14s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:12<00:00, 2.57s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:12<00:00, 3.06s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:01<00:07, 1.91s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:04<00:07, 2.58s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:08, 4.08s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.66s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:14<00:00, 2.64s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:14<00:00, 2.94s/it]
2025-12-25 08:41:42,373 - ERROR - Error in session: 'NoneType' object is not subscriptable
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:03<00:10, 3.36s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:08<00:08, 4.22s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:12<00:04, 4.10s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:12<00:00, 2.54s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:12<00:00, 3.07s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:09, 2.34s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:05<00:08, 2.91s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:11<00:08, 4.20s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:14<00:03, 3.82s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:15<00:00, 2.77s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:15<00:00, 3.11s/it]
2025-12-25 08:42:14,703 - ERROR - Error in session: 'NoneType' object is not subscriptable
2025-12-25 08:42:14,780 - INFO - Profile 2/20
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.99s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:07<00:08, 4.14s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:12<00:04, 4.25s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:12<00:00, 2.63s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:12<00:00, 3.12s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:01<00:07, 1.92s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:04<00:07, 2.54s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:08, 4.13s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:14<00:03, 3.73s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:14<00:00, 2.70s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:14<00:00, 2.98s/it]
2025-12-25 08:42:46,213 - ERROR - Error in session: 'NoneType' object is not subscriptable
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.86s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:07<00:08, 4.18s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:12<00:04, 4.22s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:12<00:00, 2.62s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:12<00:00, 3.10s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:08, 2.18s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:05<00:09, 3.01s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.75s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.53s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:15<00:00, 2.83s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:15<00:00, 3.04s/it]
2025-12-25 08:43:19,397 - ERROR - Error in session: 'NoneType' object is not subscriptable
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.68s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:06<00:07, 3.52s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:12<00:04, 4.30s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:12<00:00, 2.67s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:12<00:00, 3.05s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:01<00:07, 1.92s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:04<00:07, 2.45s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:08<00:05, 2.85s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:11<00:02, 2.92s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:12<00:00, 2.42s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:12<00:00, 2.52s/it]
2025-12-25 08:43:48,538 - ERROR - Error in session: 'NoneType' object is not subscriptable
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.35s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:06<00:06, 3.34s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:10<00:03, 3.81s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:10<00:00, 2.37s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:10<00:00, 2.73s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:01<00:07, 1.94s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:05<00:08, 2.98s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:09<00:07, 3.59s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:12<00:03, 3.34s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:14<00:00, 2.68s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:14<00:00, 2.89s/it]
2025-12-25 08:44:18,865 - ERROR - Error in session: 'NoneType' object is not subscriptable
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:03<00:09, 3.03s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:07<00:07, 3.63s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:11<00:03, 3.78s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:11<00:00, 2.35s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:11<00:00, 2.80s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:01<00:07, 1.93s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:09, 3.29s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:09<00:06, 3.44s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:12<00:03, 3.09s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.49s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:13<00:00, 2.76s/it]
2025-12-25 08:44:48,291 - ERROR - Error in session: 'NoneType' object is not subscriptable
2025-12-25 08:44:48,358 - INFO - Profile 3/20
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.77s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:06<00:07, 3.55s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:10<00:03, 3.66s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:10<00:00, 2.28s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:10<00:00, 2.70s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:09, 2.38s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.52s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.60s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.24s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:14<00:00, 2.58s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:14<00:00, 2.90s/it]
2025-12-25 08:45:18,280 - ERROR - Error in session: 'NoneType' object is not subscriptable
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.91s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:06<00:18, 6.09s/it]
2025-12-25 08:45:24,840 - ERROR - Error in session: CUDA out of memory. Tried to allocate 96.00 MiB. GPU 2 has a total capacity of 39.49 GiB of which 90.31 MiB is free. Including non-PyTorch memory, this process has 39.40 GiB memory in use. Of the allocated memory 38.90 GiB is allocated by PyTorch, and 99.47 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.61s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:05<00:17, 5.83s/it]
2025-12-25 08:45:31,127 - ERROR - Error in session: CUDA out of memory. Tried to allocate 96.00 MiB. GPU 1 has a total capacity of 39.49 GiB of which 8.31 MiB is free. Including non-PyTorch memory, this process has 39.48 GiB memory in use. Of the allocated memory 38.89 GiB is allocated by PyTorch, and 181.97 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.57s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:06<00:06, 3.38s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:08<00:08, 4.24s/it]
2025-12-25 08:45:40,072 - ERROR - Error in session: CUDA out of memory. Tried to allocate 96.00 MiB. GPU 1 has a total capacity of 39.49 GiB of which 8.31 MiB is free. Including non-PyTorch memory, this process has 39.48 GiB memory in use. Of the allocated memory 38.89 GiB is allocated by PyTorch, and 181.97 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.61s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:06<00:07, 3.51s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:10<00:03, 3.60s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:10<00:00, 2.24s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:10<00:00, 2.65s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:01<00:07, 1.90s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:10<00:17, 5.73s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:14<00:10, 5.17s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:17<00:04, 4.20s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:18<00:00, 3.19s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:18<00:00, 3.79s/it]
2025-12-25 08:46:14,167 - ERROR - Error in session: 'NoneType' object is not subscriptable
2025-12-25 08:46:14,242 - INFO - Profile 4/20
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.97s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:06<00:06, 3.46s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:10<00:03, 3.72s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:10<00:00, 2.32s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:10<00:00, 2.74s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:09, 2.28s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:10<00:17, 5.84s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:14<00:10, 5.13s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:17<00:04, 4.20s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:19<00:00, 3.31s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:19<00:00, 3.88s/it]
2025-12-25 08:46:48,967 - ERROR - Error in session: 'NoneType' object is not subscriptable
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.93s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:07<00:07, 3.73s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:11<00:03, 3.96s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:11<00:00, 2.46s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:11<00:00, 2.91s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:01<00:07, 1.91s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:09<00:14, 4.99s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:12<00:08, 4.47s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:15<00:03, 3.84s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:17<00:00, 2.92s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:17<00:00, 3.41s/it]
2025-12-25 08:47:21,876 - ERROR - Error in session: 'NoneType' object is not subscriptable
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.67s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:08<00:09, 4.71s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:12<00:04, 4.36s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:12<00:00, 2.70s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:12<00:00, 3.23s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:01<00:07, 1.90s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:09<00:16, 5.51s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:14<00:09, 4.85s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:16<00:03, 3.99s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:17<00:00, 3.00s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:17<00:00, 3.59s/it]
2025-12-25 08:47:56,567 - ERROR - Error in session: 'NoneType' object is not subscriptable
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.83s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:09<00:09, 4.83s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:13<00:04, 4.44s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:13<00:00, 2.75s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:13<00:00, 3.30s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.75s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:10<00:16, 5.56s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:13<00:09, 4.71s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:16<00:03, 3.99s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:18<00:00, 3.03s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:18<00:00, 3.64s/it]
2025-12-25 08:48:31,945 - ERROR - Error in session: 'NoneType' object is not subscriptable
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:03<00:09, 3.19s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:08<00:09, 4.60s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:12<00:04, 4.40s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:13<00:00, 2.72s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:13<00:00, 3.27s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:01<00:07, 1.91s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:08<00:14, 4.87s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:12<00:08, 4.29s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:15<00:03, 3.62s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:16<00:00, 2.83s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:16<00:00, 3.29s/it]
2025-12-25 08:49:05,843 - ERROR - Error in session: 'NoneType' object is not subscriptable
2025-12-25 08:49:05,907 - INFO - Profile 5/20
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:06, 2.33s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:08<00:09, 4.59s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:12<00:04, 4.37s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:12<00:00, 2.71s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:12<00:00, 3.19s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:01<00:07, 1.92s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:08<00:14, 4.77s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:12<00:08, 4.49s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:15<00:03, 3.74s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:17<00:00, 3.00s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:17<00:00, 3.42s/it]
2025-12-25 08:49:39,852 - ERROR - Error in session: 'NoneType' object is not subscriptable
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.79s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:07<00:07, 3.78s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:11<00:03, 3.93s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:11<00:00, 2.45s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:11<00:00, 2.89s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:08, 2.12s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:08<00:13, 4.37s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:13<00:09, 4.63s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:15<00:03, 3.97s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:18<00:00, 3.36s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:18<00:00, 3.65s/it]
2025-12-25 08:50:14,293 - ERROR - Error in session: 'NoneType' object is not subscriptable
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:03<00:09, 3.02s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:07<00:07, 3.70s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:11<00:04, 4.19s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:12<00:00, 2.61s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:12<00:00, 3.04s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:01<00:07, 1.92s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:08<00:13, 4.66s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:13<00:09, 4.82s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:16<00:03, 3.95s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:17<00:00, 3.03s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:17<00:00, 3.51s/it]
2025-12-25 08:50:48,033 - ERROR - Error in session: 'NoneType' object is not subscriptable
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:03<00:09, 3.00s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:07<00:07, 3.64s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:11<00:03, 3.95s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:11<00:00, 2.46s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:11<00:00, 2.89s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:01<00:07, 1.92s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:08<00:14, 4.73s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:13<00:09, 4.68s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:15<00:03, 3.88s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:17<00:00, 2.94s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:17<00:00, 3.43s/it]
2025-12-25 08:51:20,931 - ERROR - Error in session: 'NoneType' object is not subscriptable
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.69s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:07<00:07, 3.68s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:11<00:04, 4.03s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:11<00:00, 2.51s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:11<00:00, 2.92s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:01<00:07, 1.91s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:08<00:14, 4.73s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:13<00:09, 4.92s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:16<00:04, 4.04s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:17<00:00, 3.07s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:17<00:00, 3.56s/it]
2025-12-25 08:51:54,525 - ERROR - Error in session: 'NoneType' object is not subscriptable
2025-12-25 08:51:54,585 - INFO - Profile 6/20
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:06, 2.32s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:06<00:06, 3.24s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:10<00:03, 3.61s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:10<00:00, 2.25s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:10<00:00, 2.61s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:01<00:07, 1.91s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:09<00:15, 5.24s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:14<00:10, 5.13s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:17<00:04, 4.23s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:18<00:00, 3.25s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:18<00:00, 3.77s/it]
2025-12-25 08:52:27,892 - ERROR - Error in session: 'NoneType' object is not subscriptable
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:03<00:09, 3.12s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:07<00:08, 4.10s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:12<00:04, 4.25s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:12<00:00, 2.64s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:12<00:00, 3.12s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:01<00:07, 1.93s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:08<00:14, 4.80s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:13<00:09, 4.74s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:16<00:04, 4.03s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:17<00:00, 3.15s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:17<00:00, 3.59s/it]
2025-12-25 08:53:02,452 - ERROR - Error in session: 'NoneType' object is not subscriptable
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.93s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:07<00:07, 3.66s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:11<00:04, 4.02s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:11<00:00, 2.49s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:11<00:00, 2.93s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:01<00:07, 1.90s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:07<00:12, 4.17s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:12<00:08, 4.45s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:15<00:03, 3.77s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:16<00:00, 2.94s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:16<00:00, 3.33s/it]
2025-12-25 08:53:35,074 - ERROR - Error in session: 'NoneType' object is not subscriptable
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:03<00:09, 3.10s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:07<00:07, 3.82s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:11<00:04, 4.13s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:12<00:00, 2.56s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:12<00:00, 3.02s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:01<00:07, 1.93s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:08<00:14, 4.67s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:12<00:08, 4.44s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:15<00:03, 3.69s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:16<00:00, 2.87s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:16<00:00, 3.33s/it]
2025-12-25 08:54:08,083 - ERROR - Error in session: 'NoneType' object is not subscriptable
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:03<00:09, 3.08s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:07<00:08, 4.03s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:11<00:04, 4.04s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:11<00:00, 2.51s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:11<00:00, 3.00s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:09, 2.33s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:08<00:14, 4.69s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:13<00:09, 4.77s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:16<00:04, 4.18s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:18<00:00, 3.24s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:18<00:00, 3.68s/it]
2025-12-25 08:54:42,574 - ERROR - Error in session: 'NoneType' object is not subscriptable
2025-12-25 08:54:42,637 - INFO - Profile 7/20
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.87s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:08<00:08, 4.39s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:12<00:04, 4.44s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:12<00:00, 2.75s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:12<00:00, 3.25s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:01<00:07, 1.91s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:08<00:13, 4.57s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:12<00:09, 4.58s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:15<00:03, 3.88s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:17<00:00, 2.98s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:17<00:00, 3.43s/it]
2025-12-25 08:55:16,985 - ERROR - Error in session: 'NoneType' object is not subscriptable
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:05<00:15, 5.10s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:09<00:09, 4.86s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:15<00:05, 5.26s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:15<00:00, 3.25s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:15<00:00, 3.92s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:03<00:15, 3.79s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:10<00:16, 5.50s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:15<00:10, 5.13s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:17<00:04, 4.19s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:19<00:00, 3.16s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:19<00:00, 3.85s/it]
2025-12-25 08:55:56,547 - ERROR - Error in session: 'NoneType' object is not subscriptable
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.74s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:07<00:07, 3.80s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:11<00:04, 4.11s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:11<00:00, 2.55s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:11<00:00, 2.98s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:03<00:12, 3.24s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:10<00:16, 5.60s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:15<00:10, 5.32s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:18<00:04, 4.34s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:20<00:00, 3.70s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:20<00:00, 4.18s/it]
2025-12-25 08:56:33,918 - ERROR - Error in session: 'NoneType' object is not subscriptable
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.98s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:07<00:07, 3.82s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:11<00:04, 4.10s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:11<00:00, 2.55s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:11<00:00, 3.00s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:03<00:12, 3.24s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:10<00:16, 5.43s/it][2025-12-25T08:56:58.364] error: *** JOB 14355878 ON gpua055 CANCELLED AT 2025-12-25T08:56:58 DUE to SIGNAL Terminated ***
|
