1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
|
2025-12-25 09:48:42,727 - INFO - Loaded dataset: math-500
2025-12-25 09:48:42,889 - INFO - Loaded 100 profiles from ../data/complex_profiles_v2/profiles_100.jsonl
2025-12-25 09:48:42,890 - INFO - Running method: vanilla
`torch_dtype` is deprecated! Use `dtype` instead!
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:04<00:13, 4.41s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:08<00:08, 4.19s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:17<00:06, 6.44s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:17<00:00, 3.96s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:17<00:00, 4.43s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:02<00:10, 2.75s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:10, 3.40s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:06, 3.41s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:14<00:03, 3.87s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:15<00:00, 2.84s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:15<00:00, 3.12s/it]
2025-12-25 09:49:25,411 - INFO - Profile 1/20
2025-12-25 09:50:33,069 - INFO - Profile 2/20
2025-12-25 09:51:32,211 - INFO - Profile 3/20
2025-12-25 09:52:38,264 - INFO - Profile 4/20
2025-12-25 09:53:45,537 - INFO - Profile 5/20
2025-12-25 09:54:50,435 - INFO - Profile 6/20
2025-12-25 09:55:54,476 - INFO - Profile 7/20
2025-12-25 09:56:57,290 - INFO - Profile 8/20
2025-12-25 09:57:59,489 - INFO - Profile 9/20
2025-12-25 09:59:06,837 - INFO - Profile 10/20
2025-12-25 10:00:16,031 - INFO - Profile 11/20
2025-12-25 10:01:22,406 - INFO - Profile 12/20
2025-12-25 10:02:29,213 - INFO - Profile 13/20
2025-12-25 10:03:33,203 - INFO - Profile 14/20
2025-12-25 10:04:37,973 - INFO - Profile 15/20
2025-12-25 10:05:48,613 - INFO - Profile 16/20
2025-12-25 10:06:55,828 - INFO - Profile 17/20
2025-12-25 10:08:00,583 - INFO - Profile 18/20
2025-12-25 10:09:08,100 - INFO - Profile 19/20
2025-12-25 10:10:14,307 - INFO - Profile 20/20
2025-12-25 10:11:19,207 - INFO - Running method: all_memory
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:06<00:18, 6.04s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:13<00:13, 6.63s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:19<00:06, 6.56s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:20<00:00, 4.27s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:20<00:00, 5.08s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:05<00:21, 5.37s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:10<00:15, 5.16s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:16<00:10, 5.40s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:20<00:04, 4.91s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:22<00:00, 3.86s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:22<00:00, 4.45s/it]
2025-12-25 10:12:53,238 - INFO - Profile 1/20
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
2025-12-25 10:13:59,767 - INFO - Profile 2/20
2025-12-25 10:15:08,683 - INFO - Profile 3/20
2025-12-25 10:16:16,674 - INFO - Profile 4/20
2025-12-25 10:17:27,036 - INFO - Profile 5/20
2025-12-25 10:18:36,939 - INFO - Profile 6/20
2025-12-25 10:19:50,147 - INFO - Profile 7/20
2025-12-25 10:20:58,377 - INFO - Profile 8/20
2025-12-25 10:22:04,174 - INFO - Profile 9/20
2025-12-25 10:23:04,466 - INFO - Profile 10/20
2025-12-25 10:23:59,551 - INFO - Profile 11/20
2025-12-25 10:25:00,817 - INFO - Profile 12/20
2025-12-25 10:26:12,175 - INFO - Profile 13/20
2025-12-25 10:27:07,257 - INFO - Profile 14/20
2025-12-25 10:28:21,198 - INFO - Profile 15/20
2025-12-25 10:29:25,304 - INFO - Profile 16/20
2025-12-25 10:30:33,063 - INFO - Profile 17/20
2025-12-25 10:31:41,527 - INFO - Profile 18/20
2025-12-25 10:32:46,068 - INFO - Profile 19/20
2025-12-25 10:33:55,833 - INFO - Profile 20/20
2025-12-25 10:34:52,399 - INFO - Running method: rag
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:05<00:17, 5.92s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:12<00:12, 6.25s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:19<00:06, 6.47s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:19<00:00, 4.24s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:19<00:00, 4.99s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:07<00:31, 7.92s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:13<00:19, 6.53s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:19<00:12, 6.32s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:24<00:05, 5.94s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:26<00:00, 4.46s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:26<00:00, 5.35s/it]
2025-12-25 10:36:26,619 - INFO - Profile 1/20
2025-12-25 10:37:35,693 - INFO - Profile 2/20
2025-12-25 10:38:41,748 - INFO - Profile 3/20
2025-12-25 10:39:47,539 - INFO - Profile 4/20
2025-12-25 10:40:52,966 - INFO - Profile 5/20
2025-12-25 10:41:55,106 - INFO - Profile 6/20
2025-12-25 10:43:03,681 - INFO - Profile 7/20
2025-12-25 10:44:11,999 - INFO - Profile 8/20
2025-12-25 10:45:27,468 - INFO - Profile 9/20
2025-12-25 10:46:29,111 - INFO - Profile 10/20
2025-12-25 10:47:36,526 - INFO - Profile 11/20
2025-12-25 10:48:40,778 - INFO - Profile 12/20
2025-12-25 10:49:56,738 - INFO - Profile 13/20
2025-12-25 10:51:03,796 - INFO - Profile 14/20
2025-12-25 10:52:10,335 - INFO - Profile 15/20
2025-12-25 10:53:16,174 - INFO - Profile 16/20
2025-12-25 10:54:21,228 - INFO - Profile 17/20
2025-12-25 10:55:26,718 - INFO - Profile 18/20
2025-12-25 10:56:33,006 - INFO - Profile 19/20
2025-12-25 10:57:36,404 - INFO - Profile 20/20
2025-12-25 10:58:45,324 - INFO - Running method: rag_vector
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:07<00:22, 7.43s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:13<00:13, 6.65s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:21<00:07, 7.12s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:21<00:00, 4.58s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:21<00:00, 5.48s/it]
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:07<00:30, 7.59s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:13<00:20, 6.82s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:19<00:12, 6.48s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:25<00:06, 6.23s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:28<00:00, 4.78s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:28<00:00, 5.60s/it]
2025-12-25 10:59:46,496 - INFO - Profile 1/20
2025-12-25 11:00:34,992 - INFO - Profile 2/20
2025-12-25 11:01:23,029 - INFO - Profile 3/20
2025-12-25 11:02:15,132 - INFO - Profile 4/20
2025-12-25 11:02:57,063 - INFO - Profile 5/20
2025-12-25 11:03:49,132 - INFO - Profile 6/20
2025-12-25 11:04:41,128 - INFO - Profile 7/20
2025-12-25 11:05:28,179 - INFO - Profile 8/20
2025-12-25 11:06:17,236 - INFO - Profile 9/20
2025-12-25 11:07:04,377 - INFO - Profile 10/20
2025-12-25 11:07:55,209 - INFO - Profile 11/20
2025-12-25 11:08:41,927 - INFO - Profile 12/20
2025-12-25 11:09:33,573 - INFO - Profile 13/20
2025-12-25 11:10:20,629 - INFO - Profile 14/20
2025-12-25 11:11:05,039 - INFO - Profile 15/20
2025-12-25 11:11:57,558 - INFO - Profile 16/20
2025-12-25 11:12:45,379 - INFO - Profile 17/20
2025-12-25 11:13:30,642 - INFO - Profile 18/20
2025-12-25 11:14:22,279 - INFO - Profile 19/20
2025-12-25 11:15:11,250 - INFO - Profile 20/20
2025-12-25 11:16:03,557 - INFO - Report saved to ../results/full_20251225_094835/20251225_094842/report.md
|
