1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
|
/u/yurenh2/miniforge3/envs/eval/lib/python3.11/site-packages/transformers/utils/hub.py:110: FutureWarning: Using `TRANSFORMERS_CACHE` is deprecated and will be removed in v5 of Transformers. Use `HF_HOME` instead.
warnings.warn(
2025-12-27 01:56:00,092 - INFO - Loaded dataset: mmlu
2025-12-27 01:56:00,092 - INFO - Loaded dataset: aime
2025-12-27 01:56:00,092 - INFO - Loaded dataset: math-hard
2025-12-27 01:56:00,092 - INFO - Loaded dataset: humaneval
2025-12-27 01:56:00,108 - INFO - Loaded 100 profiles from ../data/complex_profiles_v2/profiles_100.jsonl
2025-12-27 01:56:00,108 - INFO - Running method: all_memory
`torch_dtype` is deprecated! Use `dtype` instead!
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:05<00:15, 5.29s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:12<00:12, 6.35s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:17<00:05, 6.00s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:18<00:00, 3.82s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:18<00:00, 4.61s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:03<00:14, 3.74s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:08<00:13, 4.53s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:14<00:10, 5.06s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:17<00:04, 4.22s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:19<00:00, 3.39s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:19<00:00, 3.87s/it]
2025-12-27 01:57:24,440 - INFO - Profile 1/30
/u/yurenh2/miniforge3/envs/eval/lib/python3.11/site-packages/awq/__init__.py:21: DeprecationWarning:
I have left this message as the final dev message to help you transition.
Important Notice:
- AutoAWQ is officially deprecated and will no longer be maintained.
- The last tested configuration used Torch 2.6.0 and Transformers 4.51.3.
- If future versions of Transformers break AutoAWQ compatibility, please report the issue to the Transformers project.
Alternative:
- AutoAWQ has been adopted by the vLLM Project: https://github.com/vllm-project/llm-compressor
For further inquiries, feel free to reach out:
- X: https://x.com/casper_hansen_
- LinkedIn: https://www.linkedin.com/in/casper-hansen-804005170/
warnings.warn(_FINAL_DEV_MESSAGE, category=DeprecationWarning, stacklevel=1)
Loading checkpoint shards: 0%| | 0/9 [00:00<?, ?it/s]
Loading checkpoint shards: 11%|█ | 1/9 [00:05<00:41, 5.24s/it]
Loading checkpoint shards: 22%|██▏ | 2/9 [00:11<00:41, 5.96s/it]
Loading checkpoint shards: 33%|███▎ | 3/9 [00:22<00:49, 8.20s/it]
Loading checkpoint shards: 44%|████▍ | 4/9 [00:33<00:45, 9.10s/it]
Loading checkpoint shards: 56%|█████▌ | 5/9 [00:51<00:49, 12.45s/it]
Loading checkpoint shards: 67%|██████▋ | 6/9 [01:01<00:35, 11.68s/it]
Loading checkpoint shards: 78%|███████▊ | 7/9 [01:08<00:20, 10.17s/it]
Loading checkpoint shards: 89%|████████▉ | 8/9 [01:15<00:08, 9.00s/it]
Loading checkpoint shards: 100%|██████████| 9/9 [01:17<00:00, 6.98s/it]
Loading checkpoint shards: 100%|██████████| 9/9 [01:17<00:00, 8.64s/it]
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
2025-12-27 02:02:45,722 - WARNING - User agent failed to respond at turn 4
2025-12-27 02:09:29,423 - WARNING - User agent failed to respond at turn 3
2025-12-27 02:38:42,128 - WARNING - User agent failed to respond at turn 2
2025-12-27 02:51:47,964 - INFO - Profile 2/30
2025-12-27 03:33:09,939 - INFO - Profile 3/30
2025-12-27 03:37:34,526 - WARNING - User agent failed to respond at turn 4
2025-12-27 04:17:59,068 - WARNING - User agent failed to respond at turn 6
2025-12-27 04:27:33,697 - INFO - Profile 4/30
2025-12-27 04:54:07,594 - WARNING - User agent failed to respond at turn 4
2025-12-27 05:15:55,995 - WARNING - User agent failed to respond at turn 3
2025-12-27 05:31:32,714 - INFO - Profile 5/30
2025-12-27 06:00:08,138 - INFO - Profile 6/30
2025-12-27 06:26:50,737 - WARNING - User agent failed to respond at turn 4
2025-12-27 06:48:12,610 - INFO - Profile 7/30
2025-12-27 06:50:45,060 - WARNING - User agent failed to respond at turn 3
2025-12-27 07:09:39,293 - WARNING - User agent failed to respond at turn 6
2025-12-27 07:21:31,085 - WARNING - User agent failed to respond at turn 4
2025-12-27 07:31:39,376 - WARNING - User agent failed to respond at turn 3
2025-12-27 07:50:16,030 - INFO - Profile 8/30
2025-12-27 08:08:45,945 - WARNING - User agent failed to respond at turn 5
2025-12-27 08:12:05,648 - WARNING - User agent failed to respond at turn 3
2025-12-27 08:35:55,463 - WARNING - User agent failed to respond at turn 2
2025-12-27 08:44:51,942 - WARNING - User agent failed to respond at turn 3
2025-12-27 08:54:22,336 - INFO - Profile 9/30
2025-12-27 09:27:31,666 - WARNING - User agent failed to respond at turn 7
2025-12-27 09:38:06,322 - INFO - Profile 10/30
2025-12-27 10:42:39,399 - INFO - Profile 11/30
2025-12-27 11:30:24,395 - WARNING - User agent failed to respond at turn 6
2025-12-27 11:47:24,803 - WARNING - User agent failed to respond at turn 4
2025-12-27 11:47:24,805 - INFO - Profile 12/30
2025-12-27 11:54:27,915 - WARNING - User agent failed to respond at turn 8
2025-12-27 12:45:25,244 - INFO - Profile 13/30
2025-12-27 13:25:06,083 - INFO - Profile 14/30
2025-12-27 13:59:04,923 - WARNING - User agent failed to respond at turn 4
2025-12-27 14:06:28,017 - WARNING - User agent failed to respond at turn 3
2025-12-27 14:08:43,353 - INFO - Profile 15/30
2025-12-27 14:19:46,215 - WARNING - User agent failed to respond at turn 3
2025-12-27 14:56:28,404 - WARNING - User agent failed to respond at turn 5
2025-12-27 15:02:31,026 - INFO - Profile 16/30
2025-12-27 15:41:30,611 - INFO - Profile 17/30
2025-12-27 16:22:32,360 - WARNING - User agent failed to respond at turn 3
2025-12-27 16:25:57,033 - WARNING - User agent failed to respond at turn 3
2025-12-27 16:28:31,989 - WARNING - User agent failed to respond at turn 0
2025-12-27 16:35:53,597 - WARNING - User agent failed to respond at turn 3
2025-12-27 16:35:53,598 - INFO - Profile 18/30
2025-12-27 16:53:53,194 - WARNING - User agent failed to respond at turn 2
2025-12-27 16:59:54,367 - WARNING - User agent failed to respond at turn 2
2025-12-27 17:26:40,246 - INFO - Profile 19/30
2025-12-27 17:29:49,305 - WARNING - User agent failed to respond at turn 3
2025-12-27 17:34:00,096 - WARNING - User agent failed to respond at turn 6
2025-12-27 18:29:51,532 - WARNING - User agent failed to respond at turn 3
2025-12-27 18:34:52,391 - INFO - Profile 20/30
2025-12-27 19:24:35,799 - WARNING - User agent failed to respond at turn 3
2025-12-27 19:40:55,850 - INFO - Profile 21/30
2025-12-27 19:45:01,108 - WARNING - User agent failed to respond at turn 4
2025-12-27 19:56:02,480 - WARNING - User agent failed to respond at turn 4
2025-12-27 19:59:59,405 - WARNING - User agent failed to respond at turn 3
2025-12-27 20:12:20,145 - WARNING - User agent failed to respond at turn 4
2025-12-27 20:16:09,511 - WARNING - User agent failed to respond at turn 5
2025-12-27 20:24:18,687 - WARNING - User agent failed to respond at turn 5
2025-12-27 20:41:10,014 - WARNING - User agent failed to respond at turn 3
2025-12-27 20:45:51,169 - WARNING - User agent failed to respond at turn 5
2025-12-27 20:54:31,036 - WARNING - User agent failed to respond at turn 5
2025-12-27 20:59:18,893 - WARNING - User agent failed to respond at turn 3
2025-12-27 21:09:55,958 - WARNING - User agent failed to respond at turn 3
2025-12-27 21:09:55,960 - INFO - Profile 22/30
2025-12-27 21:39:02,331 - WARNING - User agent failed to respond at turn 6
2025-12-27 22:07:27,301 - INFO - Profile 23/30
2025-12-27 23:18:34,663 - INFO - Profile 24/30
2025-12-27 23:36:19,489 - WARNING - User agent failed to respond at turn 4
2025-12-27 23:40:12,672 - WARNING - User agent failed to respond at turn 4
2025-12-27 23:53:47,032 - WARNING - User agent failed to respond at turn 3
2025-12-28 00:10:26,896 - INFO - Profile 25/30
2025-12-28 00:17:50,523 - WARNING - User agent failed to respond at turn 3
2025-12-28 00:34:10,018 - WARNING - User agent failed to respond at turn 5
2025-12-28 00:56:33,834 - WARNING - User agent failed to respond at turn 3
2025-12-28 01:10:01,462 - INFO - Profile 26/30
2025-12-28 01:26:26,996 - WARNING - User agent failed to respond at turn 5
2025-12-28 01:32:18,640 - WARNING - User agent failed to respond at turn 5
2025-12-28 01:43:28,926 - WARNING - User agent failed to respond at turn 2
2025-12-28 01:54:35,490 - WARNING - User agent failed to respond at turn 2
2025-12-28 01:58:33,822 - WARNING - User agent failed to respond at turn 5
2025-12-28 02:15:30,859 - INFO - Profile 27/30
2025-12-28 02:29:36,631 - WARNING - User agent failed to respond at turn 3
2025-12-28 02:33:19,067 - WARNING - User agent failed to respond at turn 3
2025-12-28 02:46:48,847 - WARNING - User agent failed to respond at turn 6
2025-12-28 02:55:44,372 - WARNING - User agent failed to respond at turn 3
2025-12-28 03:25:07,911 - INFO - Profile 28/30
2025-12-28 04:17:59,003 - WARNING - User agent failed to respond at turn 7
2025-12-28 04:28:02,080 - INFO - Profile 29/30
2025-12-28 04:33:42,783 - WARNING - User agent failed to respond at turn 2
2025-12-28 04:39:07,085 - WARNING - User agent failed to respond at turn 4
2025-12-28 04:42:42,813 - WARNING - User agent failed to respond at turn 4
2025-12-28 05:10:55,776 - WARNING - User agent failed to respond at turn 3
2025-12-28 05:30:48,797 - INFO - Profile 30/30
2025-12-28 06:06:54,294 - WARNING - User agent failed to respond at turn 3
2025-12-28 06:19:26,579 - INFO - Report saved to ../results/all_memory_20251227_015537/20251227_015600/report.md
|
