[2025-10-17 18:46:12,088] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Warning: The default cache directory for DeepSpeed Triton autotune, /home/jk100/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
[93m [WARNING] [0m async_io requires the dev libaio .so object and headers but these were not found.
[93m [WARNING] [0m async_io: please install the libaio-dev package with apt
[93m [WARNING] [0m If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
[93m [WARNING] [0m Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
[93m [WARNING] [0m sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
[93m [WARNING] [0m using untested triton version (2.3.1), only 1.0.0 is known to be compatible
[2025-10-17 18:46:14,413] [WARNING] [runner.py:202:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
[2025-10-17 18:46:14,414] [INFO] [runner.py:568:main] cmd = /shared/nas2/blume5/lib/conda/miniconda3/envs/segllm/bin/python3 -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMCwgMV19 --master_addr=127.0.0.1 --master_port=12345 --enable_each_rank_log=None llava/train/train_mem.py --deepspeed ./scripts/deepspeed_configs/zero2.json --model_name_or_path liuhaotian/llava-v1.5-7b --image_folder ./images_folder --annotation_folder ./annotations_folder --conversation_folder ./conversations_folder/all_data_mix_train --val_dataset reason_seg_test.json --segmentation_config ./scripts/annotation_configs/train/all_data_mix_train.yaml --conversation_config ./scripts/conversation_configs/train/all_data_mix_train.yaml --output_dir trained_checkpoints --eval_use_gt_mask_encode True --limit_rounds 6 --mm_use_bbox_encode True --lora_enable False --split_loading False --version plain --mm_use_seg True --segmentator hipie --vision_tower openai/clip-vit-large-patch14 --mm_projector_type mlp2x_gelu --tune_mm_mlp_adapter False --mm_vision_select_layer -2 --mm_use_im_start_end False --mm_vision_select_feature patch --mm_use_im_patch_token False --bf16 True --fp16 False --tf32 False --mm_use_gen True --num_train_epochs 2 --per_device_train_batch_size 4 --per_device_eval_batch_size 4 --gradient_accumulation_steps 1 --evaluation_strategy steps --save_strategy steps --save_steps 1000 --save_total_limit 2 --save_only_model True --learning_rate 2e-5 --weight_decay 0. --eval_steps 1000 --warmup_ratio 0.03 --lr_scheduler_type cosine --logging_steps 1 --model_max_length 2048 --gradient_checkpointing True --dataloader_num_workers 4 --lazy_preprocess True --report_to wandb --output_text --do_eval
[2025-10-17 18:46:15,825] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Warning: The default cache directory for DeepSpeed Triton autotune, /home/jk100/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
[93m [WARNING] [0m async_io requires the dev libaio .so object and headers but these were not found.
[93m [WARNING] [0m async_io: please install the libaio-dev package with apt
[93m [WARNING] [0m If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
[93m [WARNING] [0m Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
[93m [WARNING] [0m sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
[93m [WARNING] [0m using untested triton version (2.3.1), only 1.0.0 is known to be compatible
[2025-10-17 18:46:17,821] [INFO] [launch.py:146:main] WORLD INFO DICT: {'localhost': [0, 1]}
[2025-10-17 18:46:17,822] [INFO] [launch.py:152:main] nnodes=1, num_local_procs=2, node_rank=0
[2025-10-17 18:46:17,822] [INFO] [launch.py:163:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0, 1]})
[2025-10-17 18:46:17,822] [INFO] [launch.py:164:main] dist_world_size=2
[2025-10-17 18:46:17,822] [INFO] [launch.py:168:main] Setting CUDA_VISIBLE_DEVICES=0,1
[2025-10-17 18:46:17,822] [INFO] [launch.py:256:main] process 1986091 spawned with command: ['/shared/nas2/blume5/lib/conda/miniconda3/envs/segllm/bin/python3', '-u', 'llava/train/train_mem.py', '--local_rank=0', '--deepspeed', './scripts/deepspeed_configs/zero2.json', '--model_name_or_path', 'liuhaotian/llava-v1.5-7b', '--image_folder', './images_folder', '--annotation_folder', './annotations_folder', '--conversation_folder', './conversations_folder/all_data_mix_train', '--val_dataset', 'reason_seg_test.json', '--segmentation_config', './scripts/annotation_configs/train/all_data_mix_train.yaml', '--conversation_config', './scripts/conversation_configs/train/all_data_mix_train.yaml', '--output_dir', 'trained_checkpoints', '--eval_use_gt_mask_encode', 'True', '--limit_rounds', '6', '--mm_use_bbox_encode', 'True', '--lora_enable', 'False', '--split_loading', 'False', '--version', 'plain', '--mm_use_seg', 'True', '--segmentator', 'hipie', '--vision_tower', 'openai/clip-vit-large-patch14', '--mm_projector_type', 'mlp2x_gelu', '--tune_mm_mlp_adapter', 'False', '--mm_vision_select_layer', '-2', '--mm_use_im_start_end', 'False', '--mm_vision_select_feature', 'patch', '--mm_use_im_patch_token', 'False', '--bf16', 'True', '--fp16', 'False', '--tf32', 'False', '--mm_use_gen', 'True', '--num_train_epochs', '2', '--per_device_train_batch_size', '4', '--per_device_eval_batch_size', '4', '--gradient_accumulation_steps', '1', '--evaluation_strategy', 'steps', '--save_strategy', 'steps', '--save_steps', '1000', '--save_total_limit', '2', '--save_only_model', 'True', '--learning_rate', '2e-5', '--weight_decay', '0.', '--eval_steps', '1000', '--warmup_ratio', '0.03', '--lr_scheduler_type', 'cosine', '--logging_steps', '1', '--model_max_length', '2048', '--gradient_checkpointing', 'True', '--dataloader_num_workers', '4', '--lazy_preprocess', 'True', '--report_to', 'wandb', '--output_text', '--do_eval']
[2025-10-17 18:46:17,823] [INFO] [launch.py:256:main] process 1986092 spawned with command: ['/shared/nas2/blume5/lib/conda/miniconda3/envs/segllm/bin/python3', '-u', 'llava/train/train_mem.py', '--local_rank=1', '--deepspeed', './scripts/deepspeed_configs/zero2.json', '--model_name_or_path', 'liuhaotian/llava-v1.5-7b', '--image_folder', './images_folder', '--annotation_folder', './annotations_folder', '--conversation_folder', './conversations_folder/all_data_mix_train', '--val_dataset', 'reason_seg_test.json', '--segmentation_config', './scripts/annotation_configs/train/all_data_mix_train.yaml', '--conversation_config', './scripts/conversation_configs/train/all_data_mix_train.yaml', '--output_dir', 'trained_checkpoints', '--eval_use_gt_mask_encode', 'True', '--limit_rounds', '6', '--mm_use_bbox_encode', 'True', '--lora_enable', 'False', '--split_loading', 'False', '--version', 'plain', '--mm_use_seg', 'True', '--segmentator', 'hipie', '--vision_tower', 'openai/clip-vit-large-patch14', '--mm_projector_type', 'mlp2x_gelu', '--tune_mm_mlp_adapter', 'False', '--mm_vision_select_layer', '-2', '--mm_use_im_start_end', 'False', '--mm_vision_select_feature', 'patch', '--mm_use_im_patch_token', 'False', '--bf16', 'True', '--fp16', 'False', '--tf32', 'False', '--mm_use_gen', 'True', '--num_train_epochs', '2', '--per_device_train_batch_size', '4', '--per_device_eval_batch_size', '4', '--gradient_accumulation_steps', '1', '--evaluation_strategy', 'steps', '--save_strategy', 'steps', '--save_steps', '1000', '--save_total_limit', '2', '--save_only_model', 'True', '--learning_rate', '2e-5', '--weight_decay', '0.', '--eval_steps', '1000', '--warmup_ratio', '0.03', '--lr_scheduler_type', 'cosine', '--logging_steps', '1', '--model_max_length', '2048', '--gradient_checkpointing', 'True', '--dataloader_num_workers', '4', '--lazy_preprocess', 'True', '--report_to', 'wandb', '--output_text', '--do_eval']
/shared/nas2/blume5/lib/conda/miniconda3/envs/segllm/bin/python3: can't open file '/shared/nas2/jk100/partonomy_private/src/models/segllm/scripts/train/llava/train/train_mem.py': [Errno 2] No such file or directory
/shared/nas2/blume5/lib/conda/miniconda3/envs/segllm/bin/python3: can't open file '/shared/nas2/jk100/partonomy_private/src/models/segllm/scripts/train/llava/train/train_mem.py': [Errno 2] No such file or directory
[2025-10-17 18:46:18,824] [INFO] [launch.py:319:sigkill_handler] Killing subprocess 1986091
[2025-10-17 18:46:18,824] [INFO] [launch.py:319:sigkill_handler] Killing subprocess 1986092
[2025-10-17 18:46:18,851] [ERROR] [launch.py:325:sigkill_handler] ['/shared/nas2/blume5/lib/conda/miniconda3/envs/segllm/bin/python3', '-u', 'llava/train/train_mem.py', '--local_rank=1', '--deepspeed', './scripts/deepspeed_configs/zero2.json', '--model_name_or_path', 'liuhaotian/llava-v1.5-7b', '--image_folder', './images_folder', '--annotation_folder', './annotations_folder', '--conversation_folder', './conversations_folder/all_data_mix_train', '--val_dataset', 'reason_seg_test.json', '--segmentation_config', './scripts/annotation_configs/train/all_data_mix_train.yaml', '--conversation_config', './scripts/conversation_configs/train/all_data_mix_train.yaml', '--output_dir', 'trained_checkpoints', '--eval_use_gt_mask_encode', 'True', '--limit_rounds', '6', '--mm_use_bbox_encode', 'True', '--lora_enable', 'False', '--split_loading', 'False', '--version', 'plain', '--mm_use_seg', 'True', '--segmentator', 'hipie', '--vision_tower', 'openai/clip-vit-large-patch14', '--mm_projector_type', 'mlp2x_gelu', '--tune_mm_mlp_adapter', 'False', '--mm_vision_select_layer', '-2', '--mm_use_im_start_end', 'False', '--mm_vision_select_feature', 'patch', '--mm_use_im_patch_token', 'False', '--bf16', 'True', '--fp16', 'False', '--tf32', 'False', '--mm_use_gen', 'True', '--num_train_epochs', '2', '--per_device_train_batch_size', '4', '--per_device_eval_batch_size', '4', '--gradient_accumulation_steps', '1', '--evaluation_strategy', 'steps', '--save_strategy', 'steps', '--save_steps', '1000', '--save_total_limit', '2', '--save_only_model', 'True', '--learning_rate', '2e-5', '--weight_decay', '0.', '--eval_steps', '1000', '--warmup_ratio', '0.03', '--lr_scheduler_type', 'cosine', '--logging_steps', '1', '--model_max_length', '2048', '--gradient_checkpointing', 'True', '--dataloader_num_workers', '4', '--lazy_preprocess', 'True', '--report_to', 'wandb', '--output_text', '--do_eval'] exits with return code = 2
