1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
|
/u/yurenh2/miniforge3/envs/eval/lib/python3.11/site-packages/transformers/utils/hub.py:110: FutureWarning: Using `TRANSFORMERS_CACHE` is deprecated and will be removed in v5 of Transformers. Use `HF_HOME` instead.
warnings.warn(
2025-12-27 01:32:07,469 - INFO - Loaded dataset: mmlu
2025-12-27 01:32:07,469 - INFO - Loaded dataset: aime
2025-12-27 01:32:07,469 - INFO - Loaded dataset: math-hard
2025-12-27 01:32:07,469 - INFO - Loaded dataset: humaneval
2025-12-27 01:32:07,504 - INFO - Loaded 100 profiles from ../data/complex_profiles_v2/profiles_100.jsonl
2025-12-27 01:32:07,505 - INFO - Running method: vanilla
`torch_dtype` is deprecated! Use `dtype` instead!
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:05<00:17, 5.81s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:11<00:11, 5.63s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:18<00:06, 6.23s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:18<00:00, 4.05s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:18<00:00, 4.74s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:04<00:19, 5.00s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:09<00:14, 4.82s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:14<00:09, 4.76s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:18<00:04, 4.42s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:19<00:00, 3.44s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:19<00:00, 4.00s/it]
2025-12-27 01:32:57,620 - INFO - Profile 1/30
/u/yurenh2/miniforge3/envs/eval/lib/python3.11/site-packages/awq/__init__.py:21: DeprecationWarning:
I have left this message as the final dev message to help you transition.
Important Notice:
- AutoAWQ is officially deprecated and will no longer be maintained.
- The last tested configuration used Torch 2.6.0 and Transformers 4.51.3.
- If future versions of Transformers break AutoAWQ compatibility, please report the issue to the Transformers project.
Alternative:
- AutoAWQ has been adopted by the vLLM Project: https://github.com/vllm-project/llm-compressor
For further inquiries, feel free to reach out:
- X: https://x.com/casper_hansen_
- LinkedIn: https://www.linkedin.com/in/casper-hansen-804005170/
warnings.warn(_FINAL_DEV_MESSAGE, category=DeprecationWarning, stacklevel=1)
Loading checkpoint shards: 0%| | 0/9 [00:00<?, ?it/s]
Loading checkpoint shards: 11%|█ | 1/9 [00:11<01:28, 11.01s/it]
Loading checkpoint shards: 22%|██▏ | 2/9 [00:21<01:15, 10.75s/it]
Loading checkpoint shards: 33%|███▎ | 3/9 [00:33<01:07, 11.20s/it]
Loading checkpoint shards: 44%|████▍ | 4/9 [00:43<00:54, 10.89s/it]
Loading checkpoint shards: 56%|█████▌ | 5/9 [01:02<00:54, 13.70s/it]
Loading checkpoint shards: 67%|██████▋ | 6/9 [01:12<00:37, 12.53s/it]
Loading checkpoint shards: 78%|███████▊ | 7/9 [01:21<00:22, 11.22s/it]
Loading checkpoint shards: 89%|████████▉ | 8/9 [01:28<00:09, 9.92s/it]
Loading checkpoint shards: 100%|██████████| 9/9 [01:32<00:00, 8.13s/it]
Loading checkpoint shards: 100%|██████████| 9/9 [01:32<00:00, 10.28s/it]
2025-12-27 01:39:14,965 - WARNING - User agent failed to respond at turn 3
2025-12-27 01:45:16,597 - WARNING - User agent failed to respond at turn 2
2025-12-27 02:07:27,447 - WARNING - User agent failed to respond at turn 3
2025-12-27 02:10:51,780 - WARNING - User agent failed to respond at turn 4
2025-12-27 02:21:16,753 - INFO - Profile 2/30
2025-12-27 02:56:57,401 - WARNING - User agent failed to respond at turn 7
2025-12-27 03:08:20,574 - WARNING - User agent failed to respond at turn 5
2025-12-27 03:13:14,352 - INFO - Profile 3/30
2025-12-27 03:25:06,846 - WARNING - User agent failed to respond at turn 4
2025-12-27 03:31:15,270 - WARNING - User agent failed to respond at turn 3
2025-12-27 03:51:13,452 - WARNING - User agent failed to respond at turn 5
2025-12-27 03:53:58,854 - WARNING - User agent failed to respond at turn 3
2025-12-27 03:56:56,391 - WARNING - User agent failed to respond at turn 3
2025-12-27 04:04:16,140 - INFO - Profile 4/30
2025-12-27 04:11:00,592 - WARNING - User agent failed to respond at turn 5
2025-12-27 04:25:11,672 - WARNING - User agent failed to respond at turn 6
2025-12-27 04:28:01,010 - WARNING - User agent failed to respond at turn 3
2025-12-27 04:35:32,474 - WARNING - User agent failed to respond at turn 3
2025-12-27 04:58:18,335 - WARNING - User agent failed to respond at turn 4
2025-12-27 05:01:09,794 - WARNING - User agent failed to respond at turn 3
2025-12-27 05:10:18,666 - INFO - Profile 5/30
2025-12-27 05:49:33,423 - INFO - Profile 6/30
2025-12-27 06:18:07,136 - WARNING - User agent failed to respond at turn 3
2025-12-27 06:40:37,716 - INFO - Profile 7/30
2025-12-27 06:52:19,840 - WARNING - User agent failed to respond at turn 4
2025-12-27 07:37:53,426 - WARNING - User agent failed to respond at turn 3
2025-12-27 07:40:19,372 - INFO - Profile 8/30
2025-12-27 07:51:25,858 - WARNING - User agent failed to respond at turn 2
2025-12-27 08:07:46,046 - WARNING - User agent failed to respond at turn 3
2025-12-27 08:13:28,244 - WARNING - User agent failed to respond at turn 4
2025-12-27 08:16:50,128 - WARNING - User agent failed to respond at turn 5
2025-12-27 08:43:58,910 - WARNING - User agent failed to respond at turn 3
2025-12-27 08:43:58,912 - INFO - Profile 9/30
2025-12-27 08:52:45,466 - WARNING - User agent failed to respond at turn 2
2025-12-27 09:28:02,037 - INFO - Profile 10/30
2025-12-27 09:43:20,171 - WARNING - User agent failed to respond at turn 3
2025-12-27 10:15:12,819 - WARNING - User agent failed to respond at turn 3
2025-12-27 10:29:30,033 - INFO - Profile 11/30
2025-12-27 11:22:03,526 - INFO - Profile 12/30
2025-12-27 12:01:00,590 - WARNING - User agent failed to respond at turn 3
2025-12-27 12:13:47,494 - INFO - Profile 13/30
2025-12-27 12:39:42,451 - WARNING - User agent failed to respond at turn 5
2025-12-27 12:44:54,076 - WARNING - User agent failed to respond at turn 2
2025-12-27 12:56:38,326 - INFO - Profile 14/30
2025-12-27 13:04:45,544 - WARNING - User agent failed to respond at turn 2
2025-12-27 13:23:13,051 - WARNING - User agent failed to respond at turn 2
2025-12-27 13:36:48,889 - INFO - Profile 15/30
2025-12-27 14:18:21,034 - INFO - Profile 16/30
2025-12-27 15:03:24,448 - INFO - Profile 17/30
2025-12-27 15:05:16,405 - WARNING - User agent failed to respond at turn 2
2025-12-27 15:09:02,780 - WARNING - User agent failed to respond at turn 4
2025-12-27 15:22:15,600 - WARNING - User agent failed to respond at turn 5
2025-12-27 15:32:10,486 - WARNING - User agent failed to respond at turn 3
2025-12-27 15:40:26,743 - WARNING - User agent failed to respond at turn 2
2025-12-27 15:43:30,526 - WARNING - User agent failed to respond at turn 3
2025-12-27 15:57:53,732 - INFO - Profile 18/30
2025-12-27 16:21:37,879 - WARNING - User agent failed to respond at turn 0
2025-12-27 16:32:11,066 - WARNING - User agent failed to respond at turn 5
2025-12-27 16:43:57,955 - WARNING - User agent failed to respond at turn 0
2025-12-27 16:43:57,957 - INFO - Profile 19/30
2025-12-27 17:09:46,077 - WARNING - User agent failed to respond at turn 3
2025-12-27 17:20:38,532 - WARNING - User agent failed to respond at turn 4
2025-12-27 17:36:07,824 - WARNING - User agent failed to respond at turn 4
2025-12-27 17:36:07,826 - INFO - Profile 20/30
2025-12-27 17:56:48,744 - WARNING - User agent failed to respond at turn 3
2025-12-27 18:16:09,433 - WARNING - User agent failed to respond at turn 4
2025-12-27 18:27:50,852 - WARNING - User agent failed to respond at turn 5
2025-12-27 18:33:04,262 - WARNING - User agent failed to respond at turn 2
2025-12-27 18:48:42,005 - INFO - Profile 21/30
2025-12-27 19:01:49,962 - WARNING - User agent failed to respond at turn 3
2025-12-27 19:04:32,307 - WARNING - User agent failed to respond at turn 4
2025-12-27 19:11:14,252 - WARNING - User agent failed to respond at turn 3
2025-12-27 19:49:58,335 - WARNING - User agent failed to respond at turn 5
2025-12-27 19:57:43,221 - WARNING - User agent failed to respond at turn 6
2025-12-27 20:00:05,573 - WARNING - User agent failed to respond at turn 3
2025-12-27 20:00:05,574 - INFO - Profile 22/30
2025-12-27 20:39:06,239 - WARNING - User agent failed to respond at turn 3
2025-12-27 20:47:11,767 - WARNING - User agent failed to respond at turn 9
2025-12-27 21:00:45,614 - INFO - Profile 23/30
2025-12-27 21:03:59,797 - WARNING - User agent failed to respond at turn 3
2025-12-27 22:02:44,465 - WARNING - User agent failed to respond at turn 4
2025-12-27 22:02:44,467 - INFO - Profile 24/30
2025-12-27 22:14:01,188 - WARNING - User agent failed to respond at turn 4
2025-12-27 22:44:39,432 - WARNING - User agent failed to respond at turn 3
2025-12-27 22:57:20,908 - WARNING - User agent failed to respond at turn 2
2025-12-27 22:58:16,593 - INFO - Profile 25/30
2025-12-27 23:18:08,405 - WARNING - User agent failed to respond at turn 13
2025-12-27 23:29:17,609 - WARNING - User agent failed to respond at turn 2
2025-12-27 23:50:46,712 - INFO - Profile 26/30
2025-12-28 00:02:31,997 - WARNING - User agent failed to respond at turn 4
2025-12-28 00:10:38,451 - WARNING - User agent failed to respond at turn 2
2025-12-28 00:37:35,648 - WARNING - User agent failed to respond at turn 3
2025-12-28 00:53:32,683 - INFO - Profile 27/30
2025-12-28 01:00:46,162 - WARNING - User agent failed to respond at turn 6
2025-12-28 01:27:50,250 - WARNING - User agent failed to respond at turn 3
2025-12-28 01:50:48,239 - WARNING - User agent failed to respond at turn 8
2025-12-28 01:58:29,456 - WARNING - User agent failed to respond at turn 3
2025-12-28 02:08:15,289 - WARNING - User agent failed to respond at turn 3
2025-12-28 02:08:15,290 - INFO - Profile 28/30
2025-12-28 02:26:56,690 - WARNING - User agent failed to respond at turn 2
2025-12-28 02:32:37,000 - WARNING - User agent failed to respond at turn 4
2025-12-28 03:11:21,975 - WARNING - User agent failed to respond at turn 4
2025-12-28 03:11:21,977 - INFO - Profile 29/30
2025-12-28 03:19:27,078 - WARNING - User agent failed to respond at turn 3
2025-12-28 04:05:06,853 - WARNING - User agent failed to respond at turn 5
2025-12-28 04:22:54,056 - INFO - Profile 30/30
2025-12-28 04:26:11,853 - WARNING - User agent failed to respond at turn 4
2025-12-28 04:59:46,155 - WARNING - User agent failed to respond at turn 5
2025-12-28 05:09:22,444 - INFO - Report saved to ../results/vanilla_20251227_013200/20251227_013207/report.md
|
