1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
|
/u/yurenh2/miniforge3/envs/eval/lib/python3.11/site-packages/transformers/utils/hub.py:110: FutureWarning: Using `TRANSFORMERS_CACHE` is deprecated and will be removed in v5 of Transformers. Use `HF_HOME` instead.
warnings.warn(
2025-12-27 02:02:07,264 - INFO - Loaded dataset: mmlu
2025-12-27 02:02:07,264 - INFO - Loaded dataset: aime
2025-12-27 02:02:07,264 - INFO - Loaded dataset: math-hard
2025-12-27 02:02:07,264 - INFO - Loaded dataset: humaneval
2025-12-27 02:02:07,278 - INFO - Loaded 100 profiles from ../data/complex_profiles_v2/profiles_100.jsonl
2025-12-27 02:02:07,278 - INFO - Running method: reflection
`torch_dtype` is deprecated! Use `dtype` instead!
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:06<00:19, 6.51s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:12<00:12, 6.24s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:19<00:06, 6.47s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:20<00:00, 4.57s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:20<00:00, 5.24s/it]
2025-12-27 02:02:32,181 - INFO - Profile 1/30
/u/yurenh2/miniforge3/envs/eval/lib/python3.11/site-packages/awq/__init__.py:21: DeprecationWarning:
I have left this message as the final dev message to help you transition.
Important Notice:
- AutoAWQ is officially deprecated and will no longer be maintained.
- The last tested configuration used Torch 2.6.0 and Transformers 4.51.3.
- If future versions of Transformers break AutoAWQ compatibility, please report the issue to the Transformers project.
Alternative:
- AutoAWQ has been adopted by the vLLM Project: https://github.com/vllm-project/llm-compressor
For further inquiries, feel free to reach out:
- X: https://x.com/casper_hansen_
- LinkedIn: https://www.linkedin.com/in/casper-hansen-804005170/
warnings.warn(_FINAL_DEV_MESSAGE, category=DeprecationWarning, stacklevel=1)
Loading checkpoint shards: 0%| | 0/9 [00:00<?, ?it/s]
Loading checkpoint shards: 11%|█ | 1/9 [00:04<00:33, 4.24s/it]
Loading checkpoint shards: 22%|██▏ | 2/9 [00:09<00:32, 4.71s/it]
Loading checkpoint shards: 33%|███▎ | 3/9 [00:14<00:29, 4.91s/it]
Loading checkpoint shards: 44%|████▍ | 4/9 [00:19<00:25, 5.00s/it]
Loading checkpoint shards: 56%|█████▌ | 5/9 [00:25<00:21, 5.32s/it]
Loading checkpoint shards: 67%|██████▋ | 6/9 [00:30<00:15, 5.23s/it]
Loading checkpoint shards: 78%|███████▊ | 7/9 [00:35<00:10, 5.14s/it]
Loading checkpoint shards: 89%|████████▉ | 8/9 [00:38<00:04, 4.62s/it]
Loading checkpoint shards: 100%|██████████| 9/9 [00:40<00:00, 3.75s/it]
Loading checkpoint shards: 100%|██████████| 9/9 [00:40<00:00, 4.54s/it]
2025-12-27 02:39:58,321 - WARNING - User agent failed to respond at turn 4
2025-12-27 02:51:18,723 - INFO - Profile 2/30
2025-12-27 03:42:18,671 - INFO - Profile 3/30
2025-12-27 04:06:36,514 - WARNING - User agent failed to respond at turn 5
2025-12-27 04:22:16,874 - WARNING - User agent failed to respond at turn 9
2025-12-27 04:25:25,486 - WARNING - User agent failed to respond at turn 3
2025-12-27 04:34:09,765 - WARNING - User agent failed to respond at turn 3
2025-12-27 04:47:50,266 - INFO - Profile 4/30
2025-12-27 05:08:18,870 - WARNING - User agent failed to respond at turn 2
2025-12-27 06:04:25,163 - INFO - Profile 5/30
2025-12-27 06:46:34,782 - INFO - Profile 6/30
2025-12-27 07:37:47,157 - INFO - Profile 7/30
2025-12-27 07:41:44,319 - WARNING - User agent failed to respond at turn 5
2025-12-27 07:44:57,141 - WARNING - User agent failed to respond at turn 3
2025-12-27 08:31:18,391 - WARNING - User agent failed to respond at turn 4
2025-12-27 08:53:24,527 - INFO - Profile 8/30
2025-12-27 09:02:17,811 - WARNING - User agent failed to respond at turn 2
2025-12-27 09:10:00,697 - WARNING - User agent failed to respond at turn 2
2025-12-27 10:06:01,427 - INFO - Profile 9/30
2025-12-27 10:10:10,794 - WARNING - User agent failed to respond at turn 2
2025-12-27 10:54:26,451 - INFO - Profile 10/30
2025-12-27 11:45:46,800 - INFO - Profile 11/30
2025-12-27 12:11:54,366 - WARNING - User agent failed to respond at turn 4
2025-12-27 12:37:49,938 - WARNING - User agent failed to respond at turn 5
2025-12-27 12:53:20,643 - INFO - Profile 12/30
2025-12-27 13:21:19,712 - WARNING - User agent failed to respond at turn 3
2025-12-27 13:52:22,200 - INFO - Profile 13/30
2025-12-27 14:32:11,397 - WARNING - User agent failed to respond at turn 1
2025-12-27 14:40:54,805 - INFO - Profile 14/30
2025-12-27 15:32:12,165 - INFO - Profile 15/30
2025-12-27 16:24:14,011 - INFO - Profile 16/30
2025-12-27 17:10:39,805 - INFO - Profile 17/30
2025-12-27 17:50:48,532 - WARNING - User agent failed to respond at turn 2
2025-12-27 18:17:20,986 - WARNING - User agent failed to respond at turn 4
2025-12-27 18:17:39,723 - INFO - Profile 18/30
2025-12-27 18:18:10,896 - WARNING - User agent failed to respond at turn 0
2025-12-27 18:50:45,817 - WARNING - User agent failed to respond at turn 2
2025-12-27 19:18:30,976 - INFO - Profile 19/30
2025-12-27 20:21:34,652 - INFO - Profile 20/30
2025-12-27 20:26:57,365 - WARNING - User agent failed to respond at turn 3
2025-12-27 20:30:35,756 - WARNING - User agent failed to respond at turn 5
2025-12-27 20:42:35,308 - WARNING - User agent failed to respond at turn 4
2025-12-27 20:48:30,267 - WARNING - User agent failed to respond at turn 4
2025-12-27 20:51:42,151 - WARNING - User agent failed to respond at turn 3
2025-12-27 20:55:16,916 - WARNING - User agent failed to respond at turn 4
2025-12-27 21:17:50,813 - WARNING - User agent failed to respond at turn 4
2025-12-27 21:26:06,307 - WARNING - User agent failed to respond at turn 7
2025-12-27 21:50:43,671 - INFO - Profile 21/30
2025-12-27 22:35:18,540 - WARNING - User agent failed to respond at turn 5
2025-12-27 22:53:01,391 - WARNING - User agent failed to respond at turn 3
2025-12-27 23:13:29,157 - INFO - Profile 22/30
2025-12-27 23:51:06,223 - WARNING - User agent failed to respond at turn 8
2025-12-28 00:17:23,488 - INFO - Profile 23/30
2025-12-28 01:19:07,558 - WARNING - User agent failed to respond at turn 10
2025-12-28 01:37:49,690 - WARNING - User agent failed to respond at turn 6
2025-12-28 01:38:08,367 - INFO - Profile 24/30
2025-12-28 02:41:38,285 - INFO - Profile 25/30
2025-12-28 03:52:11,789 - INFO - Profile 26/30
2025-12-28 04:23:39,557 - WARNING - User agent failed to respond at turn 3
2025-12-28 04:34:09,940 - WARNING - User agent failed to respond at turn 3
2025-12-28 04:46:57,468 - WARNING - User agent failed to respond at turn 3
2025-12-28 05:05:43,353 - INFO - Profile 27/30
2025-12-28 05:38:16,855 - WARNING - User agent failed to respond at turn 2
2025-12-28 06:17:59,240 - INFO - Profile 28/30
2025-12-28 06:23:46,429 - WARNING - User agent failed to respond at turn 3
2025-12-28 06:40:27,991 - WARNING - User agent failed to respond at turn 4
2025-12-28 07:06:02,435 - WARNING - User agent failed to respond at turn 3
2025-12-28 07:16:42,790 - WARNING - User agent failed to respond at turn 4
2025-12-28 07:34:31,553 - INFO - Profile 29/30
2025-12-28 08:28:07,646 - WARNING - User agent failed to respond at turn 4
2025-12-28 08:53:34,126 - INFO - Profile 30/30
2025-12-28 09:32:37,813 - WARNING - User agent failed to respond at turn 7
2025-12-28 09:40:54,513 - INFO - Report saved to ../results/reflection_20251227_020146/20251227_020207/report.md
|
