2025-12-25 22:08:21,888 - INFO - Loaded dataset: math-500
2025-12-25 22:08:22,033 - INFO - Loaded 100 profiles from ../data/complex_profiles_v2/profiles_100.jsonl
2025-12-25 22:08:22,034 - INFO - Running method: vanilla
`torch_dtype` is deprecated! Use `dtype` instead!

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:04<00:14,  4.85s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:08<00:08,  4.08s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:13<00:04,  4.69s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:14<00:00,  3.07s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:14<00:00,  3.60s/it]

Loading checkpoint shards:   0%|          | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards:  20%|██        | 1/5 [00:03<00:15,  3.81s/it]
Loading checkpoint shards:  40%|████      | 2/5 [00:06<00:09,  3.25s/it]
Loading checkpoint shards:  60%|██████    | 3/5 [00:10<00:06,  3.45s/it]
Loading checkpoint shards:  80%|████████  | 4/5 [00:12<00:03,  3.07s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:14<00:00,  2.49s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:14<00:00,  2.86s/it]
2025-12-25 22:08:59,678 - INFO -   Profile 1/5

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:10<00:32, 10.97s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:15<00:14,  7.08s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:19<00:05,  5.68s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:20<00:00,  4.06s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:20<00:00,  5.23s/it]
2025-12-25 22:12:43,460 - WARNING - User agent failed to respond at turn 4
2025-12-25 22:14:39,792 - WARNING - User agent failed to respond at turn 3
2025-12-25 22:14:39,793 - INFO -   Profile 2/5
2025-12-25 22:17:30,565 - INFO -   Profile 3/5
2025-12-25 22:20:00,571 - INFO -   Profile 4/5
2025-12-25 22:23:05,146 - WARNING - User agent failed to respond at turn 4
2025-12-25 22:23:35,365 - INFO -   Profile 5/5
2025-12-25 22:26:59,994 - INFO - Running method: all_memory

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:08<00:24,  8.06s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:11<00:10,  5.48s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:20<00:07,  7.15s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:21<00:00,  4.51s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:21<00:00,  5.33s/it]

Loading checkpoint shards:   0%|          | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards:  20%|██        | 1/5 [00:05<00:22,  5.64s/it]
Loading checkpoint shards:  40%|████      | 2/5 [00:08<00:12,  4.14s/it]
Loading checkpoint shards:  60%|██████    | 3/5 [00:14<00:10,  5.08s/it]
Loading checkpoint shards:  80%|████████  | 4/5 [00:17<00:03,  3.98s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:18<00:00,  2.97s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:18<00:00,  3.68s/it]
2025-12-25 22:28:23,340 - INFO -   Profile 1/5
The following generation flags are not valid and may be ignored: ['temperature', 'top_p', 'top_k']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
2025-12-25 22:29:58,580 - WARNING - User agent failed to respond at turn 3
2025-12-25 22:30:39,359 - INFO -   Profile 2/5
2025-12-25 22:34:23,370 - INFO -   Profile 3/5
2025-12-25 22:35:04,289 - WARNING - User agent failed to respond at turn 2
2025-12-25 22:35:30,064 - WARNING - User agent failed to respond at turn 2
2025-12-25 22:36:33,412 - WARNING - User agent failed to respond at turn 6
2025-12-25 22:36:33,412 - INFO -   Profile 4/5
2025-12-25 22:38:38,658 - WARNING - User agent failed to respond at turn 3
2025-12-25 22:39:23,955 - INFO -   Profile 5/5
2025-12-25 22:42:19,402 - INFO - Running method: rag

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:06<00:20,  6.89s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:12<00:12,  6.09s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:19<00:06,  6.42s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:19<00:00,  4.16s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:19<00:00,  4.98s/it]

Loading checkpoint shards:   0%|          | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards:  20%|██        | 1/5 [00:06<00:24,  6.16s/it]
Loading checkpoint shards:  40%|████      | 2/5 [00:11<00:16,  5.55s/it]
Loading checkpoint shards:  60%|██████    | 3/5 [00:17<00:11,  5.90s/it]
Loading checkpoint shards:  80%|████████  | 4/5 [00:22<00:05,  5.37s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:23<00:00,  4.06s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:23<00:00,  4.78s/it]
2025-12-25 22:43:31,077 - INFO -   Profile 1/5
2025-12-25 22:46:19,125 - INFO -   Profile 2/5
2025-12-25 22:49:18,368 - INFO -   Profile 3/5
2025-12-25 22:53:00,495 - WARNING - User agent failed to respond at turn 3
2025-12-25 22:53:00,497 - INFO -   Profile 4/5
2025-12-25 22:54:01,784 - WARNING - User agent failed to respond at turn 3
2025-12-25 22:58:07,157 - INFO -   Profile 5/5
2025-12-25 22:58:54,351 - WARNING - User agent failed to respond at turn 3
2025-12-25 22:59:40,507 - WARNING - User agent failed to respond at turn 2
2025-12-25 23:00:10,569 - INFO - Running method: rag_vector

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:07<00:22,  7.37s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:10<00:10,  5.14s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:18<00:06,  6.24s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:18<00:00,  3.96s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:18<00:00,  4.74s/it]

Loading checkpoint shards:   0%|          | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards:  20%|██        | 1/5 [00:06<00:25,  6.35s/it]
Loading checkpoint shards:  40%|████      | 2/5 [00:09<00:12,  4.31s/it]
Loading checkpoint shards:  60%|██████    | 3/5 [00:15<00:10,  5.22s/it]
Loading checkpoint shards:  80%|████████  | 4/5 [00:18<00:04,  4.18s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:19<00:00,  3.16s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:19<00:00,  3.89s/it]
2025-12-25 23:01:19,774 - INFO -   Profile 1/5
2025-12-25 23:03:56,207 - INFO -   Profile 2/5
2025-12-25 23:06:30,341 - WARNING - User agent failed to respond at turn 2
2025-12-25 23:06:30,342 - INFO -   Profile 3/5
2025-12-25 23:09:50,352 - WARNING - User agent failed to respond at turn 7
2025-12-25 23:11:12,291 - WARNING - User agent failed to respond at turn 4
2025-12-25 23:11:12,293 - INFO -   Profile 4/5
2025-12-25 23:14:00,507 - WARNING - User agent failed to respond at turn 2
2025-12-25 23:15:21,185 - INFO -   Profile 5/5
2025-12-25 23:17:09,189 - WARNING - User agent failed to respond at turn 4
2025-12-25 23:17:38,489 - INFO - Running method: contextual

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:05<00:17,  5.83s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:11<00:11,  5.97s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:17<00:05,  5.85s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:19<00:00,  4.31s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:19<00:00,  4.89s/it]
2025-12-25 23:17:58,530 - INFO -   Profile 1/5
2025-12-25 23:22:15,614 - WARNING - User agent failed to respond at turn 5
2025-12-25 23:23:01,495 - INFO -   Profile 2/5
2025-12-25 23:26:21,325 - WARNING - User agent failed to respond at turn 12
2025-12-25 23:26:21,326 - INFO -   Profile 3/5
2025-12-25 23:29:17,191 - WARNING - User agent failed to respond at turn 7
2025-12-25 23:30:41,180 - INFO -   Profile 4/5
2025-12-25 23:31:24,578 - WARNING - User agent failed to respond at turn 2
2025-12-25 23:33:26,694 - WARNING - User agent failed to respond at turn 6
2025-12-25 23:35:25,025 - WARNING - User agent failed to respond at turn 6
2025-12-25 23:35:25,025 - INFO -   Profile 5/5
2025-12-25 23:36:14,963 - WARNING - User agent failed to respond at turn 3
2025-12-25 23:37:33,084 - WARNING - User agent failed to respond at turn 3
2025-12-25 23:38:11,089 - INFO - Running method: reflection

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:05<00:17,  5.99s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:09<00:09,  4.61s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:13<00:04,  4.32s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:15<00:00,  3.35s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:15<00:00,  3.87s/it]
2025-12-25 23:38:27,190 - INFO -   Profile 1/5
2025-12-25 23:41:28,520 - WARNING - User agent failed to respond at turn 3
2025-12-25 23:42:37,103 - INFO -   Profile 2/5
2025-12-25 23:46:33,054 - WARNING - User agent failed to respond at turn 7
2025-12-25 23:46:46,658 - INFO -   Profile 3/5
2025-12-25 23:49:40,906 - WARNING - User agent failed to respond at turn 4
2025-12-25 23:50:58,786 - WARNING - User agent failed to respond at turn 2
2025-12-25 23:51:12,246 - INFO -   Profile 4/5
2025-12-25 23:52:14,159 - WARNING - User agent failed to respond at turn 4
2025-12-25 23:55:01,535 - WARNING - User agent failed to respond at turn 4
2025-12-25 23:56:57,317 - INFO -   Profile 5/5
2025-12-25 23:58:27,891 - WARNING - User agent failed to respond at turn 2
2025-12-25 23:59:29,746 - INFO - Running method: reflection_grpo

Loading checkpoint shards:   0%|          | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards:  25%|██▌       | 1/4 [00:07<00:21,  7.21s/it]
Loading checkpoint shards:  50%|█████     | 2/4 [00:12<00:12,  6.18s/it]
Loading checkpoint shards:  75%|███████▌  | 3/4 [00:18<00:05,  5.82s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:19<00:00,  4.28s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:19<00:00,  4.99s/it]
2025-12-25 23:59:50,260 - INFO -   Profile 1/5
2025-12-26 00:00:31,696 - WARNING - User agent failed to respond at turn 3
2025-12-26 00:03:33,202 - INFO -   Profile 2/5
2025-12-26 00:06:53,817 - INFO -   Profile 3/5
2025-12-26 00:10:53,169 - WARNING - User agent failed to respond at turn 4
2025-12-26 00:12:53,034 - WARNING - User agent failed to respond at turn 4
2025-12-26 00:13:06,491 - INFO -   Profile 4/5
2025-12-26 00:13:59,355 - WARNING - User agent failed to respond at turn 3
2025-12-26 00:18:16,345 - INFO -   Profile 5/5
2025-12-26 00:18:53,569 - WARNING - User agent failed to respond at turn 3
2025-12-26 00:19:48,324 - WARNING - User agent failed to respond at turn 2
2025-12-26 00:20:53,392 - WARNING - User agent failed to respond at turn 3
2025-12-26 00:21:06,861 - INFO - Report saved to ../results/multiturn_test_20251225_220813/20251225_220821/report.md