diff options
Diffstat (limited to 'collaborativeagents/slurm/logs/run_expts_a100_14355901.err')
| -rw-r--r-- | collaborativeagents/slurm/logs/run_expts_a100_14355901.err | 47 |
1 files changed, 47 insertions, 0 deletions
diff --git a/collaborativeagents/slurm/logs/run_expts_a100_14355901.err b/collaborativeagents/slurm/logs/run_expts_a100_14355901.err new file mode 100644 index 0000000..4ce1958 --- /dev/null +++ b/collaborativeagents/slurm/logs/run_expts_a100_14355901.err @@ -0,0 +1,47 @@ +2025-12-25 09:12:44,146 - INFO - Loaded dataset: math-500 +2025-12-25 09:12:44,269 - INFO - Loaded 100 profiles from ../data/complex_profiles_v2/profiles_100.jsonl +2025-12-25 09:12:44,270 - INFO - Running method: vanilla +2025-12-25 09:12:44,270 - INFO - Profile 1/20 +`torch_dtype` is deprecated! Use `dtype` instead! +
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:07, 2.35s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:06<00:06, 3.16s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:09<00:03, 3.50s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:10<00:00, 2.19s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:10<00:00, 2.54s/it] +
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:01<00:07, 1.97s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:09, 3.27s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.61s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.42s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:14<00:00, 2.69s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:14<00:00, 2.94s/it] +2025-12-25 09:13:17,230 - ERROR - Error in session: 'NoneType' object is not subscriptable +2025-12-25 09:13:17,235 - ERROR - Full traceback: +Traceback (most recent call last): + File "/projects/bfqt/users/yurenh2/ml-projects/personalization-user-model/collaborativeagents/scripts/run_experiments.py", line 192, in run_single_session + agent_adapter.initialize() + File "/projects/bfqt/users/yurenh2/ml-projects/personalization-user-model/collaborativeagents/adapters/personalized_llm_adapter.py", line 87, in initialize + self._llm = PersonalizedLLM( + ^^^^^^^^^^^^^^^^ + File "/projects/bfqt/users/yurenh2/ml-projects/personalization-user-model/src/personalization/serving/personalized_llm.py", line 227, in __init__ + self._load_models() + File "/projects/bfqt/users/yurenh2/ml-projects/personalization-user-model/src/personalization/serving/personalized_llm.py", line 318, in _load_models + self._extractor = get_preference_extractor("rule") + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/projects/bfqt/users/yurenh2/ml-projects/personalization-user-model/src/personalization/config/registry.py", line 124, in get_preference_extractor + model_path=spec_dict["local_path"], + ~~~~~~~~~^^^^^^^^^^^^^^ +TypeError: 'NoneType' object is not subscriptable + +
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.97s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:07<00:07, 3.66s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:11<00:03, 3.78s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:11<00:00, 2.35s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:11<00:00, 2.80s/it] +
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:01<00:07, 1.93s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:09, 3.20s/it]
Loading checkpoint shards: 60%|██████ | 3/5 [00:10<00:07, 3.59s/it]
Loading checkpoint shards: 80%|████████ | 4/5 [00:13<00:03, 3.35s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:14<00:00, 2.64s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [00:14<00:00, 2.89s/it] +2025-12-25 09:13:47,069 - ERROR - Error in session: 'NoneType' object is not subscriptable +2025-12-25 09:13:47,070 - ERROR - Full traceback: +Traceback (most recent call last): + File "/projects/bfqt/users/yurenh2/ml-projects/personalization-user-model/collaborativeagents/scripts/run_experiments.py", line 192, in run_single_session + agent_adapter.initialize() + File "/projects/bfqt/users/yurenh2/ml-projects/personalization-user-model/collaborativeagents/adapters/personalized_llm_adapter.py", line 87, in initialize + self._llm = PersonalizedLLM( + ^^^^^^^^^^^^^^^^ + File "/projects/bfqt/users/yurenh2/ml-projects/personalization-user-model/src/personalization/serving/personalized_llm.py", line 227, in __init__ + self._load_models() + File "/projects/bfqt/users/yurenh2/ml-projects/personalization-user-model/src/personalization/serving/personalized_llm.py", line 318, in _load_models + self._extractor = get_preference_extractor("rule") + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/projects/bfqt/users/yurenh2/ml-projects/personalization-user-model/src/personalization/config/registry.py", line 124, in get_preference_extractor + model_path=spec_dict["local_path"], + ~~~~~~~~~^^^^^^^^^^^^^^ +TypeError: 'NoneType' object is not subscriptable + +
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:02<00:08, 2.87s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:06<00:07, 3.57s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:10<00:03, 3.53s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:10<00:00, 2.20s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:10<00:00, 2.64s/it] +
Loading checkpoint shards: 0%| | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards: 20%|██ | 1/5 [00:01<00:07, 1.93s/it]
Loading checkpoint shards: 40%|████ | 2/5 [00:06<00:09, 3.21s/it][2025-12-25T09:14:07.340] error: *** JOB 14355901 ON gpua036 CANCELLED AT 2025-12-25T09:14:07 DUE to SIGNAL Terminated *** |