1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
|
/u/yurenh2/miniforge3/envs/eval/lib/python3.11/site-packages/transformers/utils/hub.py:110: FutureWarning: Using `TRANSFORMERS_CACHE` is deprecated and will be removed in v5 of Transformers. Use `HF_HOME` instead.
warnings.warn(
2025-12-27 01:59:04,500 - INFO - Loaded dataset: mmlu
2025-12-27 01:59:04,500 - INFO - Loaded dataset: aime
2025-12-27 01:59:04,500 - INFO - Loaded dataset: math-hard
2025-12-27 01:59:04,500 - INFO - Loaded dataset: humaneval
2025-12-27 01:59:04,516 - INFO - Loaded 100 profiles from ../data/complex_profiles_v2/profiles_100.jsonl
2025-12-27 01:59:04,517 - INFO - Running method: rag_vector
`torch_dtype` is deprecated! Use `dtype` instead!
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:03<00:11, 3.76s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:07<00:07, 3.65s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:10<00:03, 3.40s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:10<00:00, 2.15s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:10<00:00, 2.67s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:11, 2.97s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:05<00:08, 2.84s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:09<00:06, 3.09s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:11<00:02, 2.86s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:12<00:00, 2.26s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:12<00:00, 2.56s/it]
2025-12-27 01:59:47,035 - INFO - Profile 1/30
/u/yurenh2/miniforge3/envs/eval/lib/python3.11/site-packages/awq/__init__.py:21: DeprecationWarning:
I have left this message as the final dev message to help you transition.
Important Notice:
- AutoAWQ is officially deprecated and will no longer be maintained.
- The last tested configuration used Torch 2.6.0 and Transformers 4.51.3.
- If future versions of Transformers break AutoAWQ compatibility, please report the issue to the Transformers project.
Alternative:
- AutoAWQ has been adopted by the vLLM Project: https://github.com/vllm-project/llm-compressor
For further inquiries, feel free to reach out:
- X: https://x.com/casper_hansen_
- LinkedIn: https://www.linkedin.com/in/casper-hansen-804005170/
warnings.warn(_FINAL_DEV_MESSAGE, category=DeprecationWarning, stacklevel=1)
Loading checkpoint shards: 0%| | 0/9 [00:00<?, ?it/s]
Loading checkpoint shards: 11%|█ | 1/9 [00:08<01:04, 8.09s/it]
Loading checkpoint shards: 22%|██▏ | 2/9 [00:18<01:07, 9.62s/it]
Loading checkpoint shards: 33%|███▎ | 3/9 [00:29<01:00, 10.11s/it]
Loading checkpoint shards: 44%|████▍ | 4/9 [00:40<00:51, 10.29s/it]
Loading checkpoint shards: 56%|█████▌ | 5/9 [00:56<00:49, 12.41s/it]
Loading checkpoint shards: 67%|██████▋ | 6/9 [01:06<00:35, 11.72s/it]
Loading checkpoint shards: 78%|███████▊ | 7/9 [01:15<00:21, 10.95s/it]
Loading checkpoint shards: 89%|████████▉ | 8/9 [01:22<00:09, 9.70s/it]
Loading checkpoint shards: 100%|██████████| 9/9 [01:25<00:00, 7.58s/it]
Loading checkpoint shards: 100%|██████████| 9/9 [01:25<00:00, 9.55s/it]
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
2025-12-27 02:05:38,040 - WARNING - User agent failed to respond at turn 3
2025-12-27 02:26:36,031 - WARNING - User agent failed to respond at turn 2
2025-12-27 02:49:12,707 - WARNING - User agent failed to respond at turn 3
2025-12-27 02:49:16,867 - INFO - Profile 2/30
2025-12-27 03:15:00,486 - WARNING - User agent failed to respond at turn 3
2025-12-27 03:26:15,013 - INFO - Profile 3/30
2025-12-27 04:03:36,467 - WARNING - User agent failed to respond at turn 2
2025-12-27 04:08:15,791 - INFO - Profile 4/30
2025-12-27 04:22:36,794 - WARNING - User agent failed to respond at turn 2
2025-12-27 04:25:15,739 - WARNING - User agent failed to respond at turn 3
2025-12-27 05:08:15,379 - INFO - Profile 5/30
2025-12-27 05:45:53,211 - INFO - Profile 6/30
2025-12-27 06:37:29,994 - INFO - Profile 7/30
2025-12-27 06:44:34,982 - WARNING - User agent failed to respond at turn 5
2025-12-27 07:07:39,808 - WARNING - User agent failed to respond at turn 3
2025-12-27 07:24:40,817 - WARNING - User agent failed to respond at turn 2
2025-12-27 07:40:10,779 - WARNING - User agent failed to respond at turn 3
2025-12-27 07:40:10,781 - INFO - Profile 8/30
2025-12-27 07:42:53,487 - WARNING - User agent failed to respond at turn 3
2025-12-27 07:59:51,049 - WARNING - User agent failed to respond at turn 3
2025-12-27 08:20:46,161 - WARNING - User agent failed to respond at turn 2
2025-12-27 08:37:34,229 - WARNING - User agent failed to respond at turn 2
2025-12-27 08:37:34,231 - INFO - Profile 9/30
2025-12-27 09:21:28,395 - INFO - Profile 10/30
2025-12-27 09:47:44,540 - WARNING - User agent failed to respond at turn 4
2025-12-27 10:17:23,198 - WARNING - User agent failed to respond at turn 3
2025-12-27 10:24:45,981 - WARNING - User agent failed to respond at turn 4
2025-12-27 10:41:24,671 - INFO - Profile 11/30
2025-12-27 11:38:47,631 - INFO - Profile 12/30
2025-12-27 12:19:31,005 - WARNING - User agent failed to respond at turn 4
2025-12-27 12:31:03,630 - INFO - Profile 13/30
2025-12-27 12:54:50,025 - WARNING - User agent failed to respond at turn 6
2025-12-27 12:58:36,759 - WARNING - User agent failed to respond at turn 3
2025-12-27 13:02:04,376 - WARNING - User agent failed to respond at turn 5
2025-12-27 13:14:20,302 - INFO - Profile 14/30
2025-12-27 13:31:29,268 - WARNING - User agent failed to respond at turn 2
2025-12-27 13:34:19,996 - WARNING - User agent failed to respond at turn 3
2025-12-27 14:02:04,997 - INFO - Profile 15/30
2025-12-27 14:21:27,253 - WARNING - User agent failed to respond at turn 4
2025-12-27 14:41:02,004 - INFO - Profile 16/30
2025-12-27 15:11:47,767 - WARNING - User agent failed to respond at turn 3
2025-12-27 15:15:59,218 - WARNING - User agent failed to respond at turn 3
2025-12-27 15:23:18,670 - INFO - Profile 17/30
2025-12-27 15:26:29,631 - WARNING - User agent failed to respond at turn 3
2025-12-27 15:41:28,028 - WARNING - User agent failed to respond at turn 5
2025-12-27 16:16:11,075 - WARNING - User agent failed to respond at turn 2
2025-12-27 16:25:49,209 - INFO - Profile 18/30
2025-12-27 17:14:30,971 - INFO - Profile 19/30
2025-12-27 17:23:08,123 - WARNING - User agent failed to respond at turn 2
2025-12-27 18:20:12,384 - INFO - Profile 20/30
2025-12-27 18:24:15,177 - WARNING - User agent failed to respond at turn 3
2025-12-27 18:31:50,579 - WARNING - User agent failed to respond at turn 4
2025-12-27 18:39:24,278 - WARNING - User agent failed to respond at turn 4
2025-12-27 18:51:42,364 - WARNING - User agent failed to respond at turn 4
2025-12-27 18:59:25,266 - WARNING - User agent failed to respond at turn 3
2025-12-27 19:14:22,797 - WARNING - User agent failed to respond at turn 3
2025-12-27 19:34:17,612 - INFO - Profile 21/30
2025-12-27 19:56:12,348 - WARNING - User agent failed to respond at turn 2
2025-12-27 20:03:59,021 - WARNING - User agent failed to respond at turn 5
2025-12-27 20:10:11,631 - WARNING - User agent failed to respond at turn 6
2025-12-27 20:26:56,963 - WARNING - User agent failed to respond at turn 4
2025-12-27 20:30:31,496 - WARNING - User agent failed to respond at turn 3
2025-12-27 20:34:38,513 - WARNING - User agent failed to respond at turn 4
2025-12-27 20:53:09,810 - INFO - Profile 22/30
2025-12-27 21:48:05,766 - INFO - Profile 23/30
2025-12-27 22:28:29,304 - WARNING - User agent failed to respond at turn 4
2025-12-27 22:59:13,258 - WARNING - User agent failed to respond at turn 3
2025-12-27 22:59:13,261 - INFO - Profile 24/30
2025-12-27 23:06:39,301 - WARNING - User agent failed to respond at turn 3
2025-12-27 23:19:24,621 - WARNING - User agent failed to respond at turn 2
2025-12-27 23:38:18,263 - WARNING - User agent failed to respond at turn 3
2025-12-27 23:55:20,391 - WARNING - User agent failed to respond at turn 3
2025-12-27 23:55:20,393 - INFO - Profile 25/30
2025-12-28 00:02:06,164 - WARNING - User agent failed to respond at turn 3
2025-12-28 00:08:19,823 - WARNING - User agent failed to respond at turn 2
2025-12-28 00:23:05,450 - WARNING - User agent failed to respond at turn 2
2025-12-28 00:52:08,278 - INFO - Profile 26/30
2025-12-28 00:55:08,573 - WARNING - User agent failed to respond at turn 3
2025-12-28 01:00:58,753 - WARNING - User agent failed to respond at turn 2
2025-12-28 01:03:09,618 - WARNING - User agent failed to respond at turn 3
2025-12-28 01:19:09,162 - WARNING - User agent failed to respond at turn 2
2025-12-28 01:35:04,963 - WARNING - User agent failed to respond at turn 3
2025-12-28 01:37:11,449 - WARNING - User agent failed to respond at turn 2
2025-12-28 01:40:14,204 - WARNING - User agent failed to respond at turn 3
2025-12-28 01:50:50,908 - INFO - Profile 27/30
2025-12-28 01:53:17,203 - WARNING - User agent failed to respond at turn 3
2025-12-28 02:11:07,167 - WARNING - User agent failed to respond at turn 4
2025-12-28 02:24:30,855 - WARNING - User agent failed to respond at turn 3
2025-12-28 02:48:24,373 - WARNING - User agent failed to respond at turn 4
2025-12-28 02:48:24,376 - INFO - Profile 28/30
2025-12-28 03:09:58,744 - WARNING - User agent failed to respond at turn 4
2025-12-28 03:18:04,465 - WARNING - User agent failed to respond at turn 3
2025-12-28 03:31:51,389 - WARNING - User agent failed to respond at turn 7
2025-12-28 03:44:54,395 - INFO - Profile 29/30
2025-12-28 04:17:56,139 - WARNING - User agent failed to respond at turn 3
2025-12-28 04:51:52,370 - INFO - Profile 30/30
2025-12-28 05:22:33,090 - WARNING - User agent failed to respond at turn 4
2025-12-28 05:25:11,094 - WARNING - User agent failed to respond at turn 2
2025-12-28 05:35:41,114 - INFO - Report saved to ../results/rag_vector_20251227_015842/20251227_015904/report.md
|
