File size: 68,177 Bytes
b1dea02 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 |
[2024-10-20 11:18:17,804] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
10/20/2024 11:19:54 - INFO - llamafactory.cli - Initializing distributed tasks at: 127.0.0.1:22328
W1020 11:20:22.856000 5294 torch/distributed/run.py:793]
W1020 11:20:22.856000 5294 torch/distributed/run.py:793] *****************************************
W1020 11:20:22.856000 5294 torch/distributed/run.py:793] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
W1020 11:20:22.856000 5294 torch/distributed/run.py:793] *****************************************
[2024-10-20 11:22:00,125] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-10-20 11:22:00,125] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-10-20 11:23:12,639] [INFO] [comm.py:652:init_distributed] cdb=None
[2024-10-20 11:23:12,639] [INFO] [comm.py:652:init_distributed] cdb=None
[2024-10-20 11:23:12,639] [INFO] [comm.py:683:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
10/20/2024 11:23:12 - INFO - llamafactory.hparams.parser - Process rank: 0, device: cuda:0, n_gpu: 1, distributed training: True, compute dtype: torch.bfloat16
10/20/2024 11:23:12 - INFO - llamafactory.hparams.parser - Process rank: 1, device: cuda:1, n_gpu: 1, distributed training: True, compute dtype: torch.bfloat16
[INFO|configuration_utils.py:673] 2024-10-20 11:23:12,820 >> loading configuration file /mnt/data/zifeng.cao/reasoning/arc-agi/LLaMA-Factory/saves/Qwen2.5-Coder-7B-Instruct/pt_output_plus_step_output/checkpoint-274/config.json
[INFO|configuration_utils.py:742] 2024-10-20 11:23:12,849 >> Model config Qwen2Config {
"_name_or_path": "/mnt/data/zifeng.cao/reasoning/arc-agi/LLaMA-Factory/saves/Qwen2.5-Coder-7B-Instruct/pt_output_plus_step_output/checkpoint-274",
"architectures": [
"Qwen2ForCausalLM"
],
"attention_dropout": 0.0,
"bos_token_id": 151643,
"eos_token_id": 151643,
"hidden_act": "silu",
"hidden_size": 3584,
"initializer_range": 0.02,
"intermediate_size": 18944,
"max_position_embeddings": 32768,
"max_window_layers": 28,
"model_type": "qwen2",
"num_attention_heads": 28,
"num_hidden_layers": 28,
"num_key_value_heads": 4,
"rms_norm_eps": 1e-06,
"rope_scaling": null,
"rope_theta": 1000000.0,
"sliding_window": null,
"tie_word_embeddings": false,
"torch_dtype": "bfloat16",
"transformers_version": "4.45.2",
"use_cache": false,
"use_sliding_window": false,
"vocab_size": 152064
}
[INFO|tokenization_utils_base.py:2204] 2024-10-20 11:23:12,926 >> loading file vocab.json
[INFO|tokenization_utils_base.py:2204] 2024-10-20 11:23:12,926 >> loading file merges.txt
[INFO|tokenization_utils_base.py:2204] 2024-10-20 11:23:12,926 >> loading file tokenizer.json
[INFO|tokenization_utils_base.py:2204] 2024-10-20 11:23:12,926 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:2204] 2024-10-20 11:23:12,926 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2204] 2024-10-20 11:23:12,926 >> loading file tokenizer_config.json
[INFO|tokenization_utils_base.py:2470] 2024-10-20 11:23:13,325 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
[INFO|configuration_utils.py:673] 2024-10-20 11:23:13,327 >> loading configuration file /mnt/data/zifeng.cao/reasoning/arc-agi/LLaMA-Factory/saves/Qwen2.5-Coder-7B-Instruct/pt_output_plus_step_output/checkpoint-274/config.json
[INFO|configuration_utils.py:742] 2024-10-20 11:23:13,328 >> Model config Qwen2Config {
"_name_or_path": "/mnt/data/zifeng.cao/reasoning/arc-agi/LLaMA-Factory/saves/Qwen2.5-Coder-7B-Instruct/pt_output_plus_step_output/checkpoint-274",
"architectures": [
"Qwen2ForCausalLM"
],
"attention_dropout": 0.0,
"bos_token_id": 151643,
"eos_token_id": 151643,
"hidden_act": "silu",
"hidden_size": 3584,
"initializer_range": 0.02,
"intermediate_size": 18944,
"max_position_embeddings": 32768,
"max_window_layers": 28,
"model_type": "qwen2",
"num_attention_heads": 28,
"num_hidden_layers": 28,
"num_key_value_heads": 4,
"rms_norm_eps": 1e-06,
"rope_scaling": null,
"rope_theta": 1000000.0,
"sliding_window": null,
"tie_word_embeddings": false,
"torch_dtype": "bfloat16",
"transformers_version": "4.45.2",
"use_cache": false,
"use_sliding_window": false,
"vocab_size": 152064
}
[INFO|tokenization_utils_base.py:2204] 2024-10-20 11:23:13,340 >> loading file vocab.json
[INFO|tokenization_utils_base.py:2204] 2024-10-20 11:23:13,341 >> loading file merges.txt
[INFO|tokenization_utils_base.py:2204] 2024-10-20 11:23:13,341 >> loading file tokenizer.json
[INFO|tokenization_utils_base.py:2204] 2024-10-20 11:23:13,341 >> loading file added_tokens.json
[INFO|tokenization_utils_base.py:2204] 2024-10-20 11:23:13,341 >> loading file special_tokens_map.json
[INFO|tokenization_utils_base.py:2204] 2024-10-20 11:23:13,341 >> loading file tokenizer_config.json
10/20/2024 11:23:13 - WARNING - llamafactory.model.loader - Processor was not found: 'Qwen2Config' object has no attribute 'vision_config'.
10/20/2024 11:23:13 - INFO - llamafactory.data.template - Replace eos token: <|im_end|>
[rank1]:[W1020 11:23:13.167659749 ProcessGroupNCCL.cpp:4115] [PG ID 0 PG GUID 0 Rank 1] using GPU 1 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device,or call init_process_group() with a device_id.
[INFO|tokenization_utils_base.py:2470] 2024-10-20 11:23:13,765 >> Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
10/20/2024 11:23:13 - WARNING - llamafactory.model.loader - Processor was not found: 'Qwen2Config' object has no attribute 'vision_config'.
10/20/2024 11:23:13 - INFO - llamafactory.data.template - Replace eos token: <|im_end|>
10/20/2024 11:23:13 - INFO - llamafactory.data.loader - Loading dataset re_arc_v3.json...
Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 15661 examples [00:01, 9407.36 examples/s]
Generating train split: 15661 examples [00:01, 9346.62 examples/s]
Converting format of dataset (num_proc=16): 0%| | 0/15661 [00:00<?, ? examples/s]
Converting format of dataset (num_proc=16): 5%|▍ | 709/15661 [00:00<00:02, 6827.04 examples/s]
Converting format of dataset (num_proc=16): 87%|████████▋ | 13688/15661 [00:00<00:00, 77938.57 examples/s]
Converting format of dataset (num_proc=16): 100%|██████████| 15661/15661 [00:00<00:00, 54667.82 examples/s]
[rank0]:[W1020 11:23:17.104395895 ProcessGroupNCCL.cpp:4115] [PG ID 0 PG GUID 0 Rank 0] using GPU 0 to perform barrier as devices used by this process are currently unknown. This can potentially cause a hang if this rank to GPU mapping is incorrect.Specify device_ids in barrier() to force use of a particular device,or call init_process_group() with a device_id.
10/20/2024 11:23:18 - INFO - llamafactory.data.loader - Loading dataset re_arc_v3.json...
Running tokenizer on dataset (num_proc=16): 0%| | 0/15661 [00:00<?, ? examples/s]
Running tokenizer on dataset (num_proc=16): 6%|▌ | 978/15661 [00:11<02:48, 87.40 examples/s]
Running tokenizer on dataset (num_proc=16): 12%|█▏ | 1957/15661 [00:13<01:26, 158.93 examples/s]
Running tokenizer on dataset (num_proc=16): 25%|██▍ | 3915/15661 [00:14<00:28, 414.72 examples/s]
Running tokenizer on dataset (num_proc=16): 31%|███ | 4894/15661 [00:14<00:19, 566.10 examples/s]
Running tokenizer on dataset (num_proc=16): 44%|████▍ | 6852/15661 [00:14<00:08, 1038.87 examples/s]
Running tokenizer on dataset (num_proc=16): 56%|█████▋ | 8810/15661 [00:14<00:04, 1674.62 examples/s]
Running tokenizer on dataset (num_proc=16): 69%|██████▉ | 10768/15661 [00:14<00:02, 2163.92 examples/s]
Running tokenizer on dataset (num_proc=16): 81%|████████▏ | 12726/15661 [00:15<00:01, 2598.01 examples/s]
Running tokenizer on dataset (num_proc=16): 88%|████████▊ | 13705/15661 [00:15<00:00, 2878.08 examples/s]
Running tokenizer on dataset (num_proc=16): 94%|█████████▍| 14683/15661 [00:16<00:00, 2606.21 examples/s]
Running tokenizer on dataset (num_proc=16): 100%|██████████| 15661/15661 [00:16<00:00, 2390.36 examples/s]
Running tokenizer on dataset (num_proc=16): 100%|██████████| 15661/15661 [00:16<00:00, 935.41 examples/s]
training example:
input_ids:
[151644, 8948, 198, 2610, 525, 264, 10950, 17847, 429, 646, 11625, 32711, 9079, 553, 1667, 264, 7199, 738, 315, 45558, 5746, 429, 525, 11537, 304, 13027, 13, 715, 12210, 5430, 4008, 75241, 12, 8886, 3383, 17167, 315, 2163, 264, 22955, 315, 4862, 10295, 11, 1380, 458, 4862, 3110, 17167, 315, 458, 1946, 5827, 323, 458, 2550, 5827, 13, 715, 12, 1752, 1817, 4862, 3110, 11, 279, 2550, 5827, 374, 279, 1102, 315, 18950, 279, 1852, 3383, 18906, 17991, 311, 279, 1946, 5827, 13, 715, 12, 576, 5795, 374, 311, 23583, 279, 17991, 504, 279, 2421, 4862, 10295, 624, 12, 576, 17991, 374, 264, 3383, 18906, 5827, 17991, 11, 892, 646, 387, 28502, 3865, 1119, 264, 8500, 315, 279, 45558, 5746, 624, 12210, 45558, 4008, 75241, 12, 20768, 323, 16605, 198, 220, 481, 3070, 4173, 95518, 18614, 5257, 821, 4494, 1075, 1565, 3543, 7808, 1565, 1190, 7808, 1565, 31941, 7808, 323, 803, 311, 27596, 5827, 7525, 624, 220, 481, 3070, 9386, 95518, 29734, 1894, 18021, 320, 68, 1302, 2572, 1565, 73956, 7808, 1565, 5225, 63, 701, 2710, 18021, 28654, 51, 7808, 1565, 37, 63, 701, 323, 72845, 22879, 320, 68, 1302, 2572, 1565, 3124, 7808, 1565, 22554, 63, 4292, 12, 2340, 35209, 198, 220, 481, 3070, 8815, 24883, 95518, 23550, 1075, 1565, 718, 7808, 1565, 59442, 7808, 1565, 64648, 7808, 323, 1565, 59394, 63, 2736, 6770, 34784, 389, 25780, 476, 45225, 624, 220, 481, 3070, 64312, 24883, 95518, 23550, 1741, 438, 1565, 16788, 7808, 1565, 39017, 7808, 323, 1565, 21028, 63, 3705, 19819, 55081, 624, 220, 481, 3070, 1043, 24883, 95518, 23550, 1075, 1565, 16912, 7808, 1565, 1358, 7808, 1565, 19052, 7808, 1565, 59251, 7808, 323, 1565, 9789, 85477, 63, 10091, 821, 23853, 624, 12, 10587, 323, 3002, 60911, 2914, 198, 220, 481, 3070, 3543, 34286, 95518, 1565, 19943, 63, 11450, 56349, 448, 5189, 15336, 323, 2750, 624, 220, 481, 3070, 3543, 53652, 95518, 23550, 1075, 1565, 4640, 24, 15, 7808, 1565, 71, 73225, 7808, 1565, 8602, 2246, 7808, 323, 1565, 2923, 12445, 63, 5165, 56349, 304, 5257, 5510, 624, 220, 481, 3070, 3136, 4203, 24883, 95518, 1565, 34147, 7808, 1565, 4997, 2292, 7808, 1565, 85, 6960, 7808, 323, 1565, 10666, 63, 8649, 476, 5602, 5479, 315, 56349, 624, 220, 481, 3070, 1190, 323, 30412, 55713, 95518, 23550, 1075, 1565, 19210, 7808, 1565, 30590, 7808, 1565, 13418, 7808, 1565, 983, 14987, 7808, 323, 1565, 2758, 789, 63, 3705, 5827, 28660, 323, 6171, 624, 12, 18320, 323, 81531, 198, 220, 481, 3070, 1636, 18320, 95518, 23550, 1741, 438, 1565, 3562, 3423, 7808, 1565, 55271, 3423, 7808, 1565, 3423, 1830, 7808, 323, 1565, 59674, 63, 23643, 1894, 42685, 624, 220, 481, 3070, 1190, 81531, 95518, 1565, 3423, 5315, 63, 323, 1565, 2141, 5315, 63, 4051, 6171, 553, 1894, 476, 1379, 624, 220, 481, 3070, 90618, 18320, 95518, 23550, 1075, 1565, 3057, 7808, 1565, 3487, 7808, 1565, 1515, 25337, 7808, 323, 1565, 24739, 18181, 63, 23643, 27979, 11871, 624, 12, 96054, 323, 425, 13586, 198, 220, 481, 3070, 14611, 1927, 95518, 1565, 6459, 7808, 1565, 79488, 7808, 1565, 67, 79488, 7808, 323, 1565, 482, 24101, 63, 8253, 13234, 1948, 5827, 14937, 624, 220, 481, 3070, 37909, 95518, 23550, 1075, 1565, 2011, 7808, 1565, 84699, 7808, 1565, 411, 2011, 7808, 323, 1565, 38630, 388, 63, 10091, 30618, 5671, 315, 28660, 624, 12, 17954, 198, 220, 481, 3070, 13999, 4440, 23470, 95518, 1565, 359, 333, 396, 63, 26885, 4194, 25780, 2878, 5189, 14262, 323, 16829, 5866, 624, 220, 481, 3070, 3543, 18954, 95518, 1565, 285, 15604, 63, 12341, 421, 458, 1946, 374, 264, 2697, 5827, 624, 220, 481, 3070, 3543, 89588, 95518, 1565, 2243, 15604, 63, 56033, 11469, 311, 279, 5827, 943, 624, 12210, 15042, 315, 279, 7907, 2038, 75241, 12, 576, 1172, 5420, 7525, 525, 27572, 279, 1102, 315, 264, 729, 1618, 304, 264, 3890, 11, 1380, 678, 5977, 1969, 2987, 387, 279, 1946, 5827, 11, 1045, 18021, 1741, 438, 25780, 476, 4185, 22879, 18860, 17961, 11, 476, 264, 3890, 8597, 24182, 2878, 279, 1852, 28961, 11, 323, 1817, 729, 429, 374, 1660, 2598, 1969, 2987, 387, 264, 45558, 729, 476, 264, 3890, 8597, 20346, 2878, 279, 1852, 28961, 13, 715, 12, 1096, 1083, 3363, 429, 1817, 1555, 315, 2038, 374, 44321, 311, 387, 264, 3175, 729, 1618, 624, 4416, 11, 498, 525, 2661, 264, 3383, 323, 264, 738, 315, 10295, 11, 498, 1184, 311, 6923, 264, 2038, 429, 646, 11625, 279, 3383, 624, 151645, 198, 151644, 872, 198, 334, 13383, 220, 16, 3070, 715, 1946, 25, 320, 24, 553, 220, 23, 8, 11631, 715, 19, 91, 17, 91, 17, 91, 19, 91, 15, 91, 17, 91, 19, 91, 17, 198, 17, 91, 19, 91, 19, 91, 15, 91, 17, 91, 17, 91, 17, 91, 17, 198, 17, 91, 17, 91, 19, 91, 17, 91, 19, 91, 17, 91, 17, 91, 17, 198, 17, 91, 17, 91, 17, 91, 17, 91, 19, 91, 17, 91, 19, 91, 17, 198, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 19, 91, 17, 198, 19, 91, 19, 91, 19, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 198, 17, 91, 17, 91, 17, 91, 15, 91, 17, 91, 17, 91, 19, 91, 17, 198, 17, 91, 17, 91, 17, 91, 17, 91, 19, 91, 17, 91, 19, 91, 17, 198, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 19, 91, 17, 198, 2550, 25, 320, 23, 553, 220, 24, 8, 11631, 715, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 198, 19, 91, 19, 91, 19, 91, 17, 91, 19, 91, 19, 91, 17, 91, 17, 91, 19, 198, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 198, 17, 91, 19, 91, 17, 91, 17, 91, 17, 91, 19, 91, 19, 91, 17, 91, 15, 198, 17, 91, 17, 91, 15, 91, 17, 91, 17, 91, 17, 91, 17, 91, 15, 91, 19, 198, 17, 91, 17, 91, 17, 91, 19, 91, 17, 91, 17, 91, 19, 91, 19, 91, 17, 198, 17, 91, 17, 91, 17, 91, 19, 91, 17, 91, 17, 91, 17, 91, 19, 91, 17, 198, 17, 91, 17, 91, 17, 91, 19, 91, 17, 91, 17, 91, 17, 91, 17, 91, 19, 271, 334, 13383, 220, 17, 3070, 715, 1946, 25, 320, 16, 17, 553, 220, 16, 22, 8, 11631, 715, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 23, 198, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 15, 91, 17, 91, 17, 91, 17, 91, 17, 91, 23, 91, 17, 91, 23, 91, 17, 91, 17, 91, 15, 91, 17, 198, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 198, 23, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 198, 17, 91, 23, 91, 17, 91, 23, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 23, 91, 17, 91, 17, 91, 17, 91, 17, 198, 15, 91, 17, 91, 17, 91, 17, 91, 17, 91, 23, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 15, 91, 17, 91, 17, 198, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 23, 91, 17, 91, 23, 198, 23, 91, 17, 91, 17, 91, 17, 91, 17, 91, 23, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 23, 91, 17, 91, 17, 91, 17, 198, 17, 91, 17, 91, 23, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 23, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 198, 17, 91, 17, 91, 23, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 23, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 198, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 23, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 198, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 23, 91, 17, 91, 23, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 198, 2550, 25, 320, 16, 22, 553, 220, 16, 17, 8, 11631, 715, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 23, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 23, 198, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 15, 91, 17, 198, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 23, 91, 15, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 198, 17, 91, 17, 91, 17, 91, 17, 91, 23, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 198, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 23, 91, 17, 91, 17, 91, 23, 91, 17, 198, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 198, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 23, 91, 17, 198, 23, 91, 23, 91, 23, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 198, 17, 91, 17, 91, 17, 91, 23, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 198, 23, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 198, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 198, 17, 91, 17, 91, 17, 91, 17, 91, 23, 91, 17, 91, 23, 91, 17, 91, 17, 91, 17, 91, 15, 91, 17, 198, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 198, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 23, 91, 17, 91, 17, 91, 17, 91, 17, 198, 17, 91, 17, 91, 23, 91, 23, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 198, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 17, 91, 23, 91, 17, 91, 17, 91, 17, 91, 17, 198, 17, 91, 17, 91, 17, 91, 17, 91, 23, 91, 17, 91, 15, 91, 17, 91, 23, 91, 17, 91, 17, 91, 17, 271, 334, 13383, 220, 18, 3070, 715, 1946, 25, 320, 23, 553, 220, 21, 8, 11631, 715, 22, 91, 16, 91, 22, 91, 16, 91, 22, 91, 22, 198, 22, 91, 22, 91, 22, 91, 16, 91, 16, 91, 22, 198, 22, 91, 22, 91, 22, 91, 16, 91, 22, 91, 22, 198, 22, 91, 22, 91, 22, 91, 24, 91, 22, 91, 22, 198, 16, 91, 16, 91, 22, 91, 16, 91, 22, 91, 16, 198, 22, 91, 22, 91, 22, 91, 22, 91, 22, 91, 22, 198, 16, 91, 16, 91, 22, 91, 24, 91, 22, 91, 24, 198, 22, 91, 16, 91, 16, 91, 22, 91, 22, 91, 22, 198, 2550, 25, 320, 21, 553, 220, 23, 8, 11631, 715, 22, 91, 24, 91, 22, 91, 16, 91, 22, 91, 22, 91, 22, 91, 22, 198, 22, 91, 22, 91, 22, 91, 22, 91, 22, 91, 22, 91, 16, 91, 22, 198, 22, 91, 24, 91, 22, 91, 16, 91, 24, 91, 16, 91, 16, 91, 16, 198, 16, 91, 22, 91, 22, 91, 22, 91, 22, 91, 22, 91, 22, 91, 22, 198, 16, 91, 16, 91, 22, 91, 16, 91, 22, 91, 22, 91, 22, 91, 16, 198, 22, 91, 16, 91, 22, 91, 16, 91, 22, 91, 22, 91, 22, 91, 22, 271, 151645, 198, 151644, 77091, 198, 750, 28961, 8972, 25, 10587, 8, 1464, 10587, 510, 262, 856, 15, 284, 5749, 24, 15, 8972, 340, 262, 856, 16, 284, 305, 73225, 2075, 15, 340, 262, 470, 856, 16, 151645]
inputs:
<|im_start|>system
You are a helpful assistant that can solve reasoning tasks by using a limited set of DSL functions that are implemented in Python.
*** Task description ***
- Each task consists of around a handful of training examples, where an training example consists of an input grid and an output grid.
- For each training example, the output grid is the result of applying the same task-specific transformation to the input grid.
- The goal is to infer the transformation from the few training examples.
- The transformation is a task-specific grid transformation, which can be decomposed into a sequence of the DSL functions.
*** DSL description ***
- Types and Constants
- **Types**: Define various data types like `Grid`, `Object`, `Indices`, and more to facilitate grid operations.
- **Constants**: Include color constants (e.g., `ZERO`, `ONE`), boolean constants (`T`, `F`), and directional vectors (e.g., `UP`, `DOWN`).
- Primitives
- **Math Operations**: Functions like `add`, `subtract`, `multiply`, and `divide` perform basic arithmetic on integers or tuples.
- **Logical Operations**: Functions such as `even`, `flip`, and `both` handle logical evaluations.
- **Data Operations**: Functions like `identity`, `order`, `merge`, `difference`, and `dedupe` manage data containers.
- Grid and Object Manipulation
- **Grid Creation**: `canvas` creates grids with specified dimensions and values.
- **Grid Transformation**: Functions like `rot90`, `hmirror`, `upscale`, and `downscale` transform grids in various ways.
- **Subgrid Operations**: `crop`, `hsplit`, `vsplit`, and `trim` extract or modify parts of grids.
- **Object and Patch Handling**: Functions like `objects`, `normalize`, `shift`, `toindices`, and `recolor` handle grid patches and objects.
- Analysis and Filtering
- **Color Analysis**: Functions such as `mostcolor`, `leastcolor`, `colorcount`, and `palette` analyze color distributions.
- **Object Filtering**: `colorfilter` and `sizefilter` filter objects by color or size.
- **Spatial Analysis**: Functions like `center`, `position`, `manhattan`, and `adjacent` analyze spatial relationships.
- Connectivity and Bounding
- **Connectivity**: `connect`, `neighbors`, `dneighbors`, and `ineighbors` determine connections between grid indices.
- **Bounding**: Functions like `box`, `inbox`, `outbox`, and `corners` manage bounding areas of patches.
- Utils
- **Random Integer Generation**: `unifint` generates random integers within specified bounds and difficulty levels.
- **Grid Validation**: `is_grid` checks if an input is a valid grid.
- **Grid Formatting**: `format_grid` casts lists to the grid type.
*** Format of the generated code ***
- The only allowed operations are storing the result of a function call in a variable, where all arguments must either be the input grid, some constants such as integers or common vectors indicating directions, or a variable previously computed within the same solver, and each function that is being called must either be a DSL function or a variable previously constructed within the same solver.
- This also means that each line of code is enforced to be a single function call.
So, you are given a task and a set of examples, you need to generate a code that can solve the task.
<|im_end|>
<|im_start|>user
** Example 1 **
input: (9 by 8) Matrix
4|2|2|4|0|2|4|2
2|4|4|0|2|2|2|2
2|2|4|2|4|2|2|2
2|2|2|2|4|2|4|2
2|2|2|2|2|2|4|2
4|4|4|2|2|2|2|2
2|2|2|0|2|2|4|2
2|2|2|2|4|2|4|2
2|2|2|2|2|2|4|2
output: (8 by 9) Matrix
2|2|2|2|2|2|2|2|2
4|4|4|2|4|4|2|2|4
2|2|2|2|2|2|2|2|2
2|4|2|2|2|4|4|2|0
2|2|0|2|2|2|2|0|4
2|2|2|4|2|2|4|4|2
2|2|2|4|2|2|2|4|2
2|2|2|4|2|2|2|2|4
** Example 2 **
input: (12 by 17) Matrix
2|2|2|2|2|2|2|2|2|2|2|2|2|2|2|2|8
2|2|2|2|2|0|2|2|2|2|8|2|8|2|2|0|2
2|2|2|2|2|2|2|2|2|2|2|2|2|2|2|2|2
8|2|2|2|2|2|2|2|2|2|2|2|2|2|2|2|2
2|8|2|8|2|2|2|2|2|2|2|2|8|2|2|2|2
0|2|2|2|2|8|2|2|2|2|2|2|2|2|0|2|2
2|2|2|2|2|2|2|2|2|2|2|2|2|2|8|2|8
8|2|2|2|2|8|2|2|2|2|2|2|2|8|2|2|2
2|2|8|2|2|2|2|2|8|2|2|2|2|2|2|2|2
2|2|8|2|2|2|2|2|2|8|2|2|2|2|2|2|2
2|2|2|2|2|2|2|2|2|8|2|2|2|2|2|2|2
2|2|2|2|2|2|2|8|2|8|2|2|2|2|2|2|2
output: (17 by 12) Matrix
2|2|2|2|2|8|2|2|2|2|2|8
2|2|2|2|2|2|2|2|2|2|0|2
2|2|2|2|2|8|0|2|2|2|2|2
2|2|2|2|8|2|2|2|2|2|2|2
2|2|2|2|2|2|2|8|2|2|8|2
2|2|2|2|2|2|2|2|2|2|2|2
2|2|2|2|2|2|2|2|2|2|8|2
8|8|8|2|2|2|2|2|2|2|2|2
2|2|2|8|2|2|2|2|2|2|2|2
8|2|2|2|2|2|2|2|2|2|2|2
2|2|2|2|2|2|2|2|2|2|2|2
2|2|2|2|8|2|8|2|2|2|0|2
2|2|2|2|2|2|2|2|2|2|2|2
2|2|2|2|2|2|2|8|2|2|2|2
2|2|8|8|2|2|2|2|2|2|2|2
2|2|2|2|2|2|2|8|2|2|2|2
2|2|2|2|8|2|0|2|8|2|2|2
** Example 3 **
input: (8 by 6) Matrix
7|1|7|1|7|7
7|7|7|1|1|7
7|7|7|1|7|7
7|7|7|9|7|7
1|1|7|1|7|1
7|7|7|7|7|7
1|1|7|9|7|9
7|1|1|7|7|7
output: (6 by 8) Matrix
7|9|7|1|7|7|7|7
7|7|7|7|7|7|1|7
7|9|7|1|9|1|1|1
1|7|7|7|7|7|7|7
1|1|7|1|7|7|7|1
7|1|7|1|7|7|7|7
<|im_end|>
<|im_start|>assistant
def solver(I: Grid) -> Grid:
x0 = rot90(I)
x1 = hmirror(x0)
return x1<|im_end|>
label_ids:
[-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 750, 28961, 8972, 25, 10587, 8, 1464, 10587, 510, 262, 856, 15, 284, 5749, 24, 15, 8972, 340, 262, 856, 16, 284, 305, 73225, 2075, 15, 340, 262, 470, 856, 16, 151645]
labels:
def solver(I: Grid) -> Grid:
x0 = rot90(I)
x1 = hmirror(x0)
return x1<|im_end|>
[INFO|configuration_utils.py:673] 2024-10-20 11:23:35,214 >> loading configuration file /mnt/data/zifeng.cao/reasoning/arc-agi/LLaMA-Factory/saves/Qwen2.5-Coder-7B-Instruct/pt_output_plus_step_output/checkpoint-274/config.json
[INFO|configuration_utils.py:742] 2024-10-20 11:23:35,215 >> Model config Qwen2Config {
"_name_or_path": "/mnt/data/zifeng.cao/reasoning/arc-agi/LLaMA-Factory/saves/Qwen2.5-Coder-7B-Instruct/pt_output_plus_step_output/checkpoint-274",
"architectures": [
"Qwen2ForCausalLM"
],
"attention_dropout": 0.0,
"bos_token_id": 151643,
"eos_token_id": 151643,
"hidden_act": "silu",
"hidden_size": 3584,
"initializer_range": 0.02,
"intermediate_size": 18944,
"max_position_embeddings": 32768,
"max_window_layers": 28,
"model_type": "qwen2",
"num_attention_heads": 28,
"num_hidden_layers": 28,
"num_key_value_heads": 4,
"rms_norm_eps": 1e-06,
"rope_scaling": null,
"rope_theta": 1000000.0,
"sliding_window": null,
"tie_word_embeddings": false,
"torch_dtype": "bfloat16",
"transformers_version": "4.45.2",
"use_cache": false,
"use_sliding_window": false,
"vocab_size": 152064
}
[INFO|modeling_utils.py:3729] 2024-10-20 11:23:41,683 >> loading weights file /mnt/data/zifeng.cao/reasoning/arc-agi/LLaMA-Factory/saves/Qwen2.5-Coder-7B-Instruct/pt_output_plus_step_output/checkpoint-274/model.safetensors.index.json
[INFO|modeling_utils.py:3874] 2024-10-20 11:23:41,685 >> Detected DeepSpeed ZeRO-3: activating zero.init() for this model
[2024-10-20 11:23:41,685] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 2
[2024-10-20 11:23:41,685] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 2
[INFO|configuration_utils.py:1099] 2024-10-20 11:23:41,692 >> Generate config GenerationConfig {
"bos_token_id": 151643,
"eos_token_id": 151643,
"use_cache": false
}
[2024-10-20 11:23:41,939] [INFO] [partition_parameters.py:348:__exit__] finished initializing model - num_params = 339, num_elems = 7.62B
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:03<00:10, 3.40s/it]
Loading checkpoint shards: 25%|██▌ | 1/4 [00:03<00:11, 3.76s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:06<00:06, 3.42s/it]
Loading checkpoint shards: 50%|█████ | 2/4 [00:07<00:07, 3.70s/it]
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:09<00:03, 3.28s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:10<00:00, 2.06s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:10<00:00, 2.53s/it]
10/20/2024 11:23:52 - INFO - llamafactory.model.model_utils.checkpointing - Gradient checkpointing enabled.
10/20/2024 11:23:52 - INFO - llamafactory.model.model_utils.attention - Using torch SDPA for faster training and inference.
10/20/2024 11:23:52 - INFO - llamafactory.model.adapter - ZeRO3 / FSDP detected, remaining trainable params in float32.
10/20/2024 11:23:52 - INFO - llamafactory.model.adapter - Fine-tuning method: Full
10/20/2024 11:23:52 - INFO - llamafactory.model.loader - trainable params: 7,615,616,512 || all params: 7,615,616,512 || trainable%: 100.0000
Loading checkpoint shards: 75%|███████▌ | 3/4 [00:10<00:03, 3.37s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:11<00:00, 2.28s/it]
Loading checkpoint shards: 100%|██████████| 4/4 [00:11<00:00, 2.75s/it]
[INFO|modeling_utils.py:4574] 2024-10-20 11:23:52,977 >> All model checkpoint weights were used when initializing Qwen2ForCausalLM.
[INFO|modeling_utils.py:4582] 2024-10-20 11:23:52,977 >> All the weights of Qwen2ForCausalLM were initialized from the model checkpoint at /mnt/data/zifeng.cao/reasoning/arc-agi/LLaMA-Factory/saves/Qwen2.5-Coder-7B-Instruct/pt_output_plus_step_output/checkpoint-274.
If your task is similar to the task the model of the checkpoint was trained on, you can already use Qwen2ForCausalLM for predictions without further training.
[INFO|configuration_utils.py:1052] 2024-10-20 11:23:53,013 >> loading configuration file /mnt/data/zifeng.cao/reasoning/arc-agi/LLaMA-Factory/saves/Qwen2.5-Coder-7B-Instruct/pt_output_plus_step_output/checkpoint-274/generation_config.json
[INFO|configuration_utils.py:1099] 2024-10-20 11:23:53,013 >> Generate config GenerationConfig {
"bos_token_id": 151643,
"do_sample": true,
"eos_token_id": [
151645,
151643
],
"pad_token_id": 151643,
"repetition_penalty": 1.1,
"temperature": 0.7,
"top_k": 20,
"top_p": 0.8
}
10/20/2024 11:23:53 - INFO - llamafactory.model.model_utils.checkpointing - Gradient checkpointing enabled.
10/20/2024 11:23:53 - INFO - llamafactory.model.model_utils.attention - Using torch SDPA for faster training and inference.
10/20/2024 11:23:53 - INFO - llamafactory.model.adapter - ZeRO3 / FSDP detected, remaining trainable params in float32.
10/20/2024 11:23:53 - INFO - llamafactory.model.adapter - Fine-tuning method: Full
10/20/2024 11:23:53 - INFO - llamafactory.model.loader - trainable params: 7,615,616,512 || all params: 7,615,616,512 || trainable%: 100.0000
Detected kernel version 4.19.91, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
[INFO|trainer.py:667] 2024-10-20 11:23:53,139 >> Using auto half precision backend
[2024-10-20 11:23:53,339] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed info: version=0.15.2, git-hash=unknown, git-branch=unknown
[2024-10-20 11:23:53,340] [INFO] [config.py:733:__init__] Config mesh_device None world_size = 2
[2024-10-20 11:23:53,349] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
[2024-10-20 11:23:53,350] [INFO] [logging.py:96:log_dist] [Rank 0] Using client Optimizer as basic optimizer
[2024-10-20 11:23:53,350] [INFO] [logging.py:96:log_dist] [Rank 0] Removing param_group that has no 'params' in the basic Optimizer
[2024-10-20 11:23:53,363] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Basic Optimizer = AdamW
[2024-10-20 11:23:53,363] [INFO] [utils.py:59:is_zero_supported_optimizer] Checking ZeRO support for optimizer=AdamW type=<class 'torch.optim.adamw.AdamW'>
[2024-10-20 11:23:53,363] [INFO] [logging.py:96:log_dist] [Rank 0] Creating fp16 ZeRO stage 3 optimizer, MiCS is enabled False, Hierarchical params gather False
[2024-10-20 11:23:53,363] [INFO] [logging.py:96:log_dist] [Rank 0] Creating torch.bfloat16 ZeRO stage 3 optimizer
[2024-10-20 11:23:53,537] [INFO] [utils.py:781:see_memory_usage] Stage 3 initialize beginning
[2024-10-20 11:23:53,538] [INFO] [utils.py:782:see_memory_usage] MA 7.11 GB Max_MA 9.65 GB CA 8.89 GB Max_CA 10 GB
[2024-10-20 11:23:53,538] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory: used = 3.75 GB, percent = 0.9%
[2024-10-20 11:23:53,540] [INFO] [stage3.py:165:__init__] Reduce bucket size 12845056
[2024-10-20 11:23:53,540] [INFO] [stage3.py:166:__init__] Prefetch bucket size 11560550
[2024-10-20 11:23:53,716] [INFO] [utils.py:781:see_memory_usage] DeepSpeedZeRoOffload initialize [begin]
[2024-10-20 11:23:53,717] [INFO] [utils.py:782:see_memory_usage] MA 7.11 GB Max_MA 7.11 GB CA 8.89 GB Max_CA 9 GB
[2024-10-20 11:23:53,717] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory: used = 3.75 GB, percent = 0.9%
Parameter Offload: Total persistent parameters: 333312 in 141 params
[2024-10-20 11:23:53,913] [INFO] [utils.py:781:see_memory_usage] DeepSpeedZeRoOffload initialize [end]
[2024-10-20 11:23:53,914] [INFO] [utils.py:782:see_memory_usage] MA 7.11 GB Max_MA 7.11 GB CA 8.89 GB Max_CA 9 GB
[2024-10-20 11:23:53,914] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory: used = 3.75 GB, percent = 0.9%
[2024-10-20 11:23:54,093] [INFO] [utils.py:781:see_memory_usage] Before creating fp16 partitions
[2024-10-20 11:23:54,094] [INFO] [utils.py:782:see_memory_usage] MA 7.11 GB Max_MA 7.11 GB CA 8.89 GB Max_CA 9 GB
[2024-10-20 11:23:54,094] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory: used = 3.75 GB, percent = 0.9%
[2024-10-20 11:23:57,802] [INFO] [utils.py:781:see_memory_usage] After creating fp16 partitions: 5
[2024-10-20 11:23:57,803] [INFO] [utils.py:782:see_memory_usage] MA 7.09 GB Max_MA 7.11 GB CA 7.1 GB Max_CA 9 GB
[2024-10-20 11:23:57,803] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory: used = 3.75 GB, percent = 0.9%
[2024-10-20 11:23:57,979] [INFO] [utils.py:781:see_memory_usage] Before creating fp32 partitions
[2024-10-20 11:23:57,980] [INFO] [utils.py:782:see_memory_usage] MA 7.09 GB Max_MA 7.09 GB CA 7.1 GB Max_CA 7 GB
[2024-10-20 11:23:57,980] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory: used = 3.75 GB, percent = 0.9%
[2024-10-20 11:23:58,175] [INFO] [utils.py:781:see_memory_usage] After creating fp32 partitions
[2024-10-20 11:23:58,176] [INFO] [utils.py:782:see_memory_usage] MA 21.28 GB Max_MA 22.72 GB CA 23.19 GB Max_CA 23 GB
[2024-10-20 11:23:58,176] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory: used = 3.75 GB, percent = 0.9%
[2024-10-20 11:23:58,350] [INFO] [utils.py:781:see_memory_usage] Before initializing optimizer states
[2024-10-20 11:23:58,351] [INFO] [utils.py:782:see_memory_usage] MA 21.28 GB Max_MA 21.28 GB CA 23.19 GB Max_CA 23 GB
[2024-10-20 11:23:58,351] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory: used = 3.75 GB, percent = 0.9%
[2024-10-20 11:23:58,527] [INFO] [utils.py:781:see_memory_usage] After initializing optimizer states
[2024-10-20 11:23:58,527] [INFO] [utils.py:782:see_memory_usage] MA 21.28 GB Max_MA 25.08 GB CA 26.99 GB Max_CA 27 GB
[2024-10-20 11:23:58,528] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory: used = 3.75 GB, percent = 0.9%
[2024-10-20 11:23:58,528] [INFO] [stage3.py:520:_setup_for_real_optimizer] optimizer state initialized
[2024-10-20 11:23:58,815] [INFO] [utils.py:781:see_memory_usage] After initializing ZeRO optimizer
[2024-10-20 11:23:58,816] [INFO] [utils.py:782:see_memory_usage] MA 28.4 GB Max_MA 30.43 GB CA 34.08 GB Max_CA 34 GB
[2024-10-20 11:23:58,816] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory: used = 3.75 GB, percent = 0.9%
[2024-10-20 11:23:58,816] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Final Optimizer = DeepSpeedZeroOptimizer_Stage3
[2024-10-20 11:23:58,816] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed using configured LR scheduler = None
[2024-10-20 11:23:58,816] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed LR Scheduler = None
[2024-10-20 11:23:58,817] [INFO] [logging.py:96:log_dist] [Rank 0] step=0, skipped=0, lr=[0.0, 0.0], mom=[(0.9, 0.999), (0.9, 0.999)]
[2024-10-20 11:23:58,818] [INFO] [config.py:999:print] DeepSpeedEngine configuration:
[2024-10-20 11:23:58,818] [INFO] [config.py:1003:print] activation_checkpointing_config {
"partition_activations": false,
"contiguous_memory_optimization": false,
"cpu_checkpointing": false,
"number_checkpoints": null,
"synchronize_checkpoint_boundary": false,
"profile": false
}
[2024-10-20 11:23:58,818] [INFO] [config.py:1003:print] aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True, 'use_gds': False}
[2024-10-20 11:23:58,818] [INFO] [config.py:1003:print] amp_enabled .................. False
[2024-10-20 11:23:58,819] [INFO] [config.py:1003:print] amp_params ................... False
[2024-10-20 11:23:58,819] [INFO] [config.py:1003:print] autotuning_config ............ {
"enabled": false,
"start_step": null,
"end_step": null,
"metric_path": null,
"arg_mappings": null,
"metric": "throughput",
"model_info": null,
"results_dir": "autotuning_results",
"exps_dir": "autotuning_exps",
"overwrite": true,
"fast": true,
"start_profile_step": 3,
"end_profile_step": 5,
"tuner_type": "gridsearch",
"tuner_early_stopping": 5,
"tuner_num_trials": 50,
"model_info_path": null,
"mp_size": 1,
"max_train_batch_size": null,
"min_train_batch_size": 1,
"max_train_micro_batch_size_per_gpu": 1.024000e+03,
"min_train_micro_batch_size_per_gpu": 1,
"num_tuning_micro_batch_sizes": 3
}
[2024-10-20 11:23:58,819] [INFO] [config.py:1003:print] bfloat16_enabled ............. True
[2024-10-20 11:23:58,819] [INFO] [config.py:1003:print] bfloat16_immediate_grad_update False
[2024-10-20 11:23:58,819] [INFO] [config.py:1003:print] checkpoint_parallel_write_pipeline False
[2024-10-20 11:23:58,819] [INFO] [config.py:1003:print] checkpoint_tag_validation_enabled True
[2024-10-20 11:23:58,819] [INFO] [config.py:1003:print] checkpoint_tag_validation_fail False
[2024-10-20 11:23:58,819] [INFO] [config.py:1003:print] comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x7f9ec0617850>
[2024-10-20 11:23:58,819] [INFO] [config.py:1003:print] communication_data_type ...... None
[2024-10-20 11:23:58,819] [INFO] [config.py:1003:print] compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}}
[2024-10-20 11:23:58,819] [INFO] [config.py:1003:print] curriculum_enabled_legacy .... False
[2024-10-20 11:23:58,819] [INFO] [config.py:1003:print] curriculum_params_legacy ..... False
[2024-10-20 11:23:58,819] [INFO] [config.py:1003:print] data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}}
[2024-10-20 11:23:58,819] [INFO] [config.py:1003:print] data_efficiency_enabled ...... False
[2024-10-20 11:23:58,820] [INFO] [config.py:1003:print] dataloader_drop_last ......... False
[2024-10-20 11:23:58,820] [INFO] [config.py:1003:print] disable_allgather ............ False
[2024-10-20 11:23:58,820] [INFO] [config.py:1003:print] dump_state ................... False
[2024-10-20 11:23:58,820] [INFO] [config.py:1003:print] dynamic_loss_scale_args ...... None
[2024-10-20 11:23:58,820] [INFO] [config.py:1003:print] eigenvalue_enabled ........... False
[2024-10-20 11:23:58,820] [INFO] [config.py:1003:print] eigenvalue_gas_boundary_resolution 1
[2024-10-20 11:23:58,820] [INFO] [config.py:1003:print] eigenvalue_layer_name ........ bert.encoder.layer
[2024-10-20 11:23:58,820] [INFO] [config.py:1003:print] eigenvalue_layer_num ......... 0
[2024-10-20 11:23:58,820] [INFO] [config.py:1003:print] eigenvalue_max_iter .......... 100
[2024-10-20 11:23:58,820] [INFO] [config.py:1003:print] eigenvalue_stability ......... 1e-06
[2024-10-20 11:23:58,820] [INFO] [config.py:1003:print] eigenvalue_tol ............... 0.01
[2024-10-20 11:23:58,820] [INFO] [config.py:1003:print] eigenvalue_verbose ........... False
[2024-10-20 11:23:58,820] [INFO] [config.py:1003:print] elasticity_enabled ........... False
[2024-10-20 11:23:58,820] [INFO] [config.py:1003:print] flops_profiler_config ........ {
"enabled": false,
"recompute_fwd_factor": 0.0,
"profile_step": 1,
"module_depth": -1,
"top_modules": 1,
"detailed": true,
"output_file": null
}
[2024-10-20 11:23:58,820] [INFO] [config.py:1003:print] fp16_auto_cast ............... None
[2024-10-20 11:23:58,820] [INFO] [config.py:1003:print] fp16_enabled ................. False
[2024-10-20 11:23:58,820] [INFO] [config.py:1003:print] fp16_master_weights_and_gradients False
[2024-10-20 11:23:58,820] [INFO] [config.py:1003:print] global_rank .................. 0
[2024-10-20 11:23:58,820] [INFO] [config.py:1003:print] grad_accum_dtype ............. None
[2024-10-20 11:23:58,821] [INFO] [config.py:1003:print] gradient_accumulation_steps .. 4
[2024-10-20 11:23:58,821] [INFO] [config.py:1003:print] gradient_clipping ............ 1.0
[2024-10-20 11:23:58,821] [INFO] [config.py:1003:print] gradient_predivide_factor .... 1.0
[2024-10-20 11:23:58,821] [INFO] [config.py:1003:print] graph_harvesting ............. False
[2024-10-20 11:23:58,821] [INFO] [config.py:1003:print] hybrid_engine ................ enabled=False max_out_tokens=512 inference_tp_size=1 release_inference_cache=False pin_parameters=True tp_gather_partition_size=8
[2024-10-20 11:23:58,821] [INFO] [config.py:1003:print] initial_dynamic_scale ........ 1
[2024-10-20 11:23:58,821] [INFO] [config.py:1003:print] load_universal_checkpoint .... False
[2024-10-20 11:23:58,821] [INFO] [config.py:1003:print] loss_scale ................... 1.0
[2024-10-20 11:23:58,821] [INFO] [config.py:1003:print] memory_breakdown ............. False
[2024-10-20 11:23:58,821] [INFO] [config.py:1003:print] mics_hierarchial_params_gather False
[2024-10-20 11:23:58,821] [INFO] [config.py:1003:print] mics_shard_size .............. -1
[2024-10-20 11:23:58,821] [INFO] [config.py:1003:print] monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') comet=CometConfig(enabled=False, samples_log_interval=100, project=None, workspace=None, api_key=None, experiment_name=None, experiment_key=None, online=None, mode=None) wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName')
[2024-10-20 11:23:58,821] [INFO] [config.py:1003:print] nebula_config ................ {
"enabled": false,
"persistent_storage_path": null,
"persistent_time_interval": 100,
"num_of_version_in_retention": 2,
"enable_nebula_load": true,
"load_path": null
}
[2024-10-20 11:23:58,821] [INFO] [config.py:1003:print] optimizer_legacy_fusion ...... False
[2024-10-20 11:23:58,821] [INFO] [config.py:1003:print] optimizer_name ............... None
[2024-10-20 11:23:58,821] [INFO] [config.py:1003:print] optimizer_params ............. None
[2024-10-20 11:23:58,821] [INFO] [config.py:1003:print] pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0, 'pipe_partitioned': True, 'grad_partitioned': True}
[2024-10-20 11:23:58,822] [INFO] [config.py:1003:print] pld_enabled .................. False
[2024-10-20 11:23:58,822] [INFO] [config.py:1003:print] pld_params ................... False
[2024-10-20 11:23:58,822] [INFO] [config.py:1003:print] prescale_gradients ........... False
[2024-10-20 11:23:58,822] [INFO] [config.py:1003:print] scheduler_name ............... None
[2024-10-20 11:23:58,822] [INFO] [config.py:1003:print] scheduler_params ............. None
[2024-10-20 11:23:58,822] [INFO] [config.py:1003:print] seq_parallel_communication_data_type torch.float32
[2024-10-20 11:23:58,822] [INFO] [config.py:1003:print] sparse_attention ............. None
[2024-10-20 11:23:58,822] [INFO] [config.py:1003:print] sparse_gradients_enabled ..... False
[2024-10-20 11:23:58,822] [INFO] [config.py:1003:print] steps_per_print .............. inf
[2024-10-20 11:23:58,822] [INFO] [config.py:1003:print] timers_config ................ enabled=True synchronized=True
[2024-10-20 11:23:58,822] [INFO] [config.py:1003:print] train_batch_size ............. 8
[2024-10-20 11:23:58,822] [INFO] [config.py:1003:print] train_micro_batch_size_per_gpu 1
[2024-10-20 11:23:58,822] [INFO] [config.py:1003:print] use_data_before_expert_parallel_ False
[2024-10-20 11:23:58,822] [INFO] [config.py:1003:print] use_node_local_storage ....... False
[2024-10-20 11:23:58,822] [INFO] [config.py:1003:print] wall_clock_breakdown ......... False
[2024-10-20 11:23:58,822] [INFO] [config.py:1003:print] weight_quantization_config ... None
[2024-10-20 11:23:58,822] [INFO] [config.py:1003:print] world_size ................... 2
[2024-10-20 11:23:58,822] [INFO] [config.py:1003:print] zero_allow_untested_optimizer True
[2024-10-20 11:23:58,822] [INFO] [config.py:1003:print] zero_config .................. stage=3 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=12845056 use_multi_rank_bucket_allreduce=True allgather_partitions=True allgather_bucket_size=500000000 overlap_comm=True load_from_fp32_weights=True elastic_checkpoint=False offload_param=None offload_optimizer=None sub_group_size=1000000000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=11560550 param_persistence_threshold=35840 model_persistence_threshold=9223372036854775807 max_live_parameters=1000000000 max_reuse_distance=1000000000 gather_16bit_weights_on_model_save=True use_all_reduce_for_fetch_params=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False zero_hpz_partition_size=1 zero_quantized_weights=False zero_quantized_nontrainable_weights=False zero_quantized_gradients=False mics_shard_size=-1 mics_hierarchical_params_gather=False memory_efficient_linear=True pipeline_loading_checkpoint=False override_module_apply=True
[2024-10-20 11:23:58,823] [INFO] [config.py:1003:print] zero_enabled ................. True
[2024-10-20 11:23:58,823] [INFO] [config.py:1003:print] zero_force_ds_cpu_optimizer .. True
[2024-10-20 11:23:58,823] [INFO] [config.py:1003:print] zero_optimization_stage ...... 3
[2024-10-20 11:23:58,823] [INFO] [config.py:989:print_user_config] json = {
"train_batch_size": 8,
"train_micro_batch_size_per_gpu": 1,
"gradient_accumulation_steps": 4,
"gradient_clipping": 1.0,
"zero_allow_untested_optimizer": true,
"fp16": {
"enabled": false,
"loss_scale": 0,
"loss_scale_window": 1000,
"initial_scale_power": 16,
"hysteresis": 2,
"min_loss_scale": 1
},
"bf16": {
"enabled": true
},
"zero_optimization": {
"stage": 3,
"overlap_comm": true,
"contiguous_gradients": true,
"sub_group_size": 1.000000e+09,
"reduce_bucket_size": 1.284506e+07,
"stage3_prefetch_bucket_size": 1.156055e+07,
"stage3_param_persistence_threshold": 3.584000e+04,
"stage3_max_live_parameters": 1.000000e+09,
"stage3_max_reuse_distance": 1.000000e+09,
"stage3_gather_16bit_weights_on_model_save": true
},
"steps_per_print": inf
}
[INFO|trainer.py:2243] 2024-10-20 11:23:58,823 >> ***** Running training *****
[INFO|trainer.py:2244] 2024-10-20 11:23:58,823 >> Num examples = 14,094
[INFO|trainer.py:2245] 2024-10-20 11:23:58,823 >> Num Epochs = 4
[INFO|trainer.py:2246] 2024-10-20 11:23:58,823 >> Instantaneous batch size per device = 1
[INFO|trainer.py:2249] 2024-10-20 11:23:58,823 >> Total train batch size (w. parallel, distributed & accumulation) = 8
[INFO|trainer.py:2250] 2024-10-20 11:23:58,823 >> Gradient Accumulation steps = 4
[INFO|trainer.py:2251] 2024-10-20 11:23:58,823 >> Total optimization steps = 7,044
[INFO|trainer.py:2252] 2024-10-20 11:23:58,824 >> Number of trainable parameters = 7,615,616,512
0%| | 0/7044 [00:00<?, ?it/s][2024-10-20 11:24:15,553] [WARNING] [stage3.py:2104:step] 1 pytorch allocator cache flushes since last step. this happens when there is high memory pressure and is detrimental to performance. if this is happening frequently consider adjusting settings to reduce memory consumption. If you are unable to make the cache flushes go away consider adding get_accelerator().empty_cache() calls in your training loop to ensure that all ranks flush their caches at the same time
0%| | 1/7044 [00:16<32:21:12, 16.54s/it][rank0]: Traceback (most recent call last):
[rank0]: File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/src/llamafactory/launcher.py", line 23, in <module>
[rank0]: launch()
[rank0]: File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/src/llamafactory/launcher.py", line 19, in launch
[rank0]: run_exp()
[rank0]: File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/src/llamafactory/train/tuner.py", line 50, in run_exp
[rank0]: run_sft(model_args, data_args, training_args, finetuning_args, generating_args, callbacks)
[rank0]: File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/src/llamafactory/train/sft/workflow.py", line 96, in run_sft
[rank0]: train_result = trainer.train(resume_from_checkpoint=training_args.resume_from_checkpoint)
[rank0]: File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/lib/python3.10/site-packages/transformers/trainer.py", line 2052, in train
[rank0]: return inner_training_loop(
[rank0]: File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/lib/python3.10/site-packages/transformers/trainer.py", line 2388, in _inner_training_loop
[rank0]: tr_loss_step = self.training_step(model, inputs)
[rank0]: File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/lib/python3.10/site-packages/transformers/trainer.py", line 3518, in training_step
[rank0]: self.accelerator.backward(loss, **kwargs)
[rank0]: File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/lib/python3.10/site-packages/accelerate/accelerator.py", line 2188, in backward
[rank0]: self.deepspeed_engine_wrapped.backward(loss, **kwargs)
[rank0]: File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/lib/python3.10/site-packages/accelerate/utils/deepspeed.py", line 166, in backward
[rank0]: self.engine.backward(loss, **kwargs)
[rank0]: File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/lib/python3.10/site-packages/deepspeed/utils/nvtx.py", line 18, in wrapped_fn
[rank0]: ret_val = func(*args, **kwargs)
[rank0]: File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 2020, in backward
[rank0]: self.optimizer.backward(loss, retain_graph=retain_graph)
[rank0]: File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/lib/python3.10/site-packages/deepspeed/utils/nvtx.py", line 18, in wrapped_fn
[rank0]: ret_val = func(*args, **kwargs)
[rank0]: File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/lib/python3.10/site-packages/deepspeed/runtime/zero/stage3.py", line 2249, in backward
[rank0]: self.loss_scaler.backward(loss.float(), retain_graph=retain_graph)
[rank0]: File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/lib/python3.10/site-packages/deepspeed/runtime/fp16/loss_scaler.py", line 63, in backward
[rank0]: scaled_loss.backward(retain_graph=retain_graph)
[rank0]: File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/lib/python3.10/site-packages/torch/_tensor.py", line 581, in backward
[rank0]: torch.autograd.backward(
[rank0]: File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/lib/python3.10/site-packages/torch/autograd/__init__.py", line 347, in backward
[rank0]: _engine_run_backward(
[rank0]: File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/lib/python3.10/site-packages/torch/autograd/graph.py", line 825, in _engine_run_backward
[rank0]: return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
[rank0]: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 3.34 GiB. GPU 0 has a total capacity of 79.35 GiB of which 2.95 GiB is free. Process 22118 has 76.39 GiB memory in use. Of the allocated memory 67.73 GiB is allocated by PyTorch, and 7.82 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
0%| | 1/7044 [00:20<40:23:26, 20.65s/it]
W1020 11:24:20.981000 5294 torch/distributed/elastic/multiprocessing/api.py:897] Sending process 5344 closing signal SIGTERM
E1020 11:24:21.272000 5294 torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: 1) local_rank: 0 (pid: 5343) of binary: /mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/bin/python
Traceback (most recent call last):
File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/bin/torchrun", line 8, in <module>
sys.exit(main())
File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
return f(*args, **kwargs)
File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/lib/python3.10/site-packages/torch/distributed/run.py", line 919, in main
run(args)
File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/lib/python3.10/site-packages/torch/distributed/run.py", line 910, in run
elastic_launch(
File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 138, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/mnt/data/guibin.chen/open-o1/LLaMA-Factory/venv/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 269, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
/mnt/data/guibin.chen/open-o1/LLaMA-Factory/src/llamafactory/launcher.py FAILED
------------------------------------------------------------
Failures:
<NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2024-10-20_11:24:20
host : dsw-116518-6475cdf9d-vpqks
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 5343)
error_file: <N/A>
traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html
============================================================
|