# 코랩 GPU RAM이 계속 0 **URL:** \ **Category:** 묻고 답하기\ **Created:** [1월 7, 2025, 1:07오후 UTC](https://discuss.pytorch.kr/t/gpu-ram-0/5813 "2025-01-07T13:07:33Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![thdrlals135](https://discuss.pytorch.kr/user_avatar/discuss.pytorch.kr/thdrlals135/32/9450_2.png) [@thdrlals135](https://discuss.pytorch.kr/u/thdrlals135)\ **Post date:** [1월 7, 2025, 1:07오후 UTC](https://discuss.pytorch.kr/t/gpu-ram-0/5813/1 "2025-01-07T13:07:33Z")
![image](https://discuss.pytorch.kr/uploads/default/original/2X/5/5ebffaa42fa4159ef880a766fb5858b83af6f76f.png) ! bash ./scripts/TimeLLM\_ETTh1.sh TimeLLM\_ETTh1.sh 파일은 model\_name=TimeLLM train\_epochs=100 learning\_rate=0.01 llama\_layers=32 master\_port=00097 num\_process=1 batch\_size=24 d\_model=32 d\_ff=128 comment='TimeLLM-ETTh1' accelerate launch --mixed\_precision bf16 --num\_processes $num\_process run\_main.py --task\_name long\_term\_forecast --is\_training 1 --root\_path ./dataset/ETT-small/ --data\_path ETTh1.csv --model\_id ETTh1\_512\_96 --model $model\_name --data ETTh1 --features M --seq\_len 512 --label\_len 48 --pred\_len 96 --factor 3 --enc\_in 7 --dec\_in 7 --c\_out 7 --des 'Exp' --itr 1 --d\_model $d\_model --d\_ff $d\_ff --batch\_size $batch\_size --learning\_rate $learning\_rate --llm\_layers $llama\_layers --train\_epochs $train\_epochs --model\_comment $comment accelerate launch --mixed\_precision bf16 --num\_processes $num\_process run\_main.py --task\_name long\_term\_forecast --is\_training 1 --root\_path ./dataset/ETT-small/ --data\_path ETTh1.csv --model\_id ETTh1\_512\_192 --model $model\_name --data ETTh1 --features M --seq\_len 512 --label\_len 48 --pred\_len 192 --factor 3 --enc\_in 7 --dec\_in 7 --c\_out 7 --des 'Exp' --itr 1 --d\_model 32 --d\_ff 128 --batch\_size $batch\_size --learning\_rate 0.02 --llm\_layers $llama\_layers --train\_epochs $train\_epochs --model\_comment $comment accelerate launch --mixed\_precision bf16 --num\_processes $num\_process run\_main.py --task\_name long\_term\_forecast --is\_training 1 --root\_path ./dataset/ETT-small/ --data\_path ETTh1.csv --model\_id ETTh1\_512\_336 --model $model\_name --data ETTh1 --features M --seq\_len 512 --label\_len 48 --pred\_len 336 --factor 3 --enc\_in 7 --dec\_in 7 --c\_out 7 --des 'Exp' --itr 1 --d\_model $d\_model --d\_ff $d\_ff --batch\_size $batch\_size --lradj 'COS' --learning\_rate 0.001 --llm\_layers $llama\_layers --train\_epochs $train\_epochs --model\_comment $comment accelerate launch --mixed\_precision bf16 --num\_processes $num\_process run\_main.py --task\_name long\_term\_forecast --is\_training 1 --root\_path ./dataset/ETT-small/ --data\_path ETTh1.csv --model\_id ETTh1\_512\_720 --model $model\_name --data ETTh1 --features M --seq\_len 512 --label\_len 48 --pred\_len 720 --factor 3 --enc\_in 7 --dec\_in 7 --c\_out 7 --des 'Exp' --itr 1 --d\_model $d\_model --d\_ff $d\_ff --batch\_size $batch\_size --learning\_rate $learning\_rate --llm\_layers $llama\_layers --train\_epochs $train\_epochs --model\_comment $comment 입니다. 실행결과 Loading checkpoint shards: 100% 2/2 [00:05\<00:00, 2.72s/it] [2025-01-07 12:50:13,988] [INFO] [real\_accelerator.py:191:get\_accelerator] Setting ds\_accelerator to cuda (auto detect) [2025-01-07 12:50:14,267] [INFO] [logging.py:96:log\_dist] [Rank -1] DeepSpeed info: version=0.14.0, git-hash=unknown, git-branch=unknown [2025-01-07 12:50:14,267] [INFO] [comm.py:637:init\_distributed] cdb=None [2025-01-07 12:50:14,267] [INFO] [comm.py:652:init\_distributed] Not using the DeepSpeed or dist launchers, attempting to detect MPI environment... [2025-01-07 12:50:15,350] [INFO] [comm.py:702:mpi\_discovery] Discovered MPI settings of world\_rank=0, local\_rank=0, world\_size=1, master\_addr=172.28.0.12, master\_port=29500 [2025-01-07 12:50:15,351] [INFO] [comm.py:668:init\_distributed] Initializing TorchBackend in DeepSpeed with backend nccl [2025-01-07 12:50:30,478] [INFO] [logging.py:96:log\_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False [2025-01-07 12:50:30,480] [INFO] [logging.py:96:log\_dist] [Rank 0] Using client Optimizer as basic optimizer [2025-01-07 12:50:30,480] [INFO] [logging.py:96:log\_dist] [Rank 0] Removing param\_group that has no 'params' in the basic Optimizer [2025-01-07 12:50:30,481] [INFO] [logging.py:96:log\_dist] [Rank 0] DeepSpeed Basic Optimizer = Adam [2025-01-07 12:50:30,481] [INFO] [utils.py:56:is\_zero\_supported\_optimizer] Checking ZeRO support for optimizer=Adam type=\ [2025-01-07 12:50:30,482] [INFO] [logging.py:96:log\_dist] [Rank 0] Creating torch.bfloat16 ZeRO stage 2 optimizer [2025-01-07 12:50:30,482] [INFO] [stage\_1\_and\_2.py:149: **init**] Reduce bucket size 200000000 [2025-01-07 12:50:30,482] [INFO] [stage\_1\_and\_2.py:150: **init**] Allgather bucket size 200000000 [2025-01-07 12:50:30,482] [INFO] [stage\_1\_and\_2.py:151: **init**] CPU Offload: False [2025-01-07 12:50:30,482] [INFO] [stage\_1\_and\_2.py:152: **init**] Round robin gradient partitioning: False [2025-01-07 12:50:30,845] [INFO] [utils.py:800:see\_memory\_usage] Before initializing optimizer states [2025-01-07 12:50:30,846] [INFO] [utils.py:801:see\_memory\_usage] MA 12.59 GB Max\_MA 12.68 GB CA 12.68 GB Max\_CA 13 GB [2025-01-07 12:50:30,846] [INFO] [utils.py:808:see\_memory\_usage] CPU Virtual Memory: used = 3.43 GB, percent = 4.1% [2025-01-07 12:50:31,014] [INFO] [utils.py:800:see\_memory\_usage] After initializing optimizer states [2025-01-07 12:50:31,015] [INFO] [utils.py:801:see\_memory\_usage] MA 12.59 GB Max\_MA 12.76 GB CA 12.85 GB Max\_CA 13 GB [2025-01-07 12:50:31,015] [INFO] [utils.py:808:see\_memory\_usage] CPU Virtual Memory: used = 3.43 GB, percent = 4.1% [2025-01-07 12:50:31,015] [INFO] [stage\_1\_and\_2.py:539: **init**] optimizer state initialized [2025-01-07 12:50:31,181] [INFO] [utils.py:800:see\_memory\_usage] After initializing ZeRO optimizer [2025-01-07 12:50:31,182] [INFO] [utils.py:801:see\_memory\_usage] MA 12.59 GB Max\_MA 12.59 GB CA 12.85 GB Max\_CA 13 GB [2025-01-07 12:50:31,182] [INFO] [utils.py:808:see\_memory\_usage] CPU Virtual Memory: used = 3.43 GB, percent = 4.1% [2025-01-07 12:50:31,183] [INFO] [logging.py:96:log\_dist] [Rank 0] DeepSpeed Final Optimizer = Adam [2025-01-07 12:50:31,183] [INFO] [logging.py:96:log\_dist] [Rank 0] DeepSpeed using client LR scheduler [2025-01-07 12:50:31,183] [INFO] [logging.py:96:log\_dist] [Rank 0] DeepSpeed LR Scheduler = None [2025-01-07 12:50:31,183] [INFO] [logging.py:96:log\_dist] [Rank 0] step=0, skipped=0, lr=[0.0003999999999999993], mom=[(0.95, 0.999)] [2025-01-07 12:50:31,184] [INFO] [config.py:996:print] DeepSpeedEngine configuration: [2025-01-07 12:50:31,185] [INFO] [config.py:1000:print] activation\_checkpointing\_config { "partition\_activations": false, "contiguous\_memory\_optimization": false, "cpu\_checkpointing": false, "number\_checkpoints": null, "synchronize\_checkpoint\_boundary": false, "profile": false } [2025-01-07 12:50:31,185] [INFO] [config.py:1000:print] aio\_config ................... {'block\_size': 1048576, 'queue\_depth': 8, 'thread\_count': 1, 'single\_submit': False, 'overlap\_events': True} [2025-01-07 12:50:31,185] [INFO] [config.py:1000:print] amp\_enabled .................. False [2025-01-07 12:50:31,185] [INFO] [config.py:1000:print] amp\_params ................... False [2025-01-07 12:50:31,185] [INFO] [config.py:1000:print] autotuning\_config ............ { "enabled": false, "start\_step": null, "end\_step": null, "metric\_path": null, "arg\_mappings": null, "metric": "throughput", "model\_info": null, "results\_dir": "autotuning\_results", "exps\_dir": "autotuning\_exps", "overwrite": true, "fast": true, "start\_profile\_step": 3, "end\_profile\_step": 5, "tuner\_type": "gridsearch", "tuner\_early\_stopping": 5, "tuner\_num\_trials": 50, "model\_info\_path": null, "mp\_size": 1, "max\_train\_batch\_size": null, "min\_train\_batch\_size": 1, "max\_train\_micro\_batch\_size\_per\_gpu": 1.024000e+03, "min\_train\_micro\_batch\_size\_per\_gpu": 1, "num\_tuning\_micro\_batch\_sizes": 3 } [2025-01-07 12:50:31,185] [INFO] [config.py:1000:print] bfloat16\_enabled ............. True [2025-01-07 12:50:31,185] [INFO] [config.py:1000:print] bfloat16\_immediate\_grad\_update False [2025-01-07 12:50:31,185] [INFO] [config.py:1000:print] checkpoint\_parallel\_write\_pipeline False [2025-01-07 12:50:31,185] [INFO] [config.py:1000:print] checkpoint\_tag\_validation\_enabled True [2025-01-07 12:50:31,185] [INFO] [config.py:1000:print] checkpoint\_tag\_validation\_fail False [2025-01-07 12:50:31,185] [INFO] [config.py:1000:print] comms\_config ................. \ [2025-01-07 12:50:31,185] [INFO] [config.py:1000:print] communication\_data\_type ...... None [2025-01-07 12:50:31,185] [INFO] [config.py:1000:print] compile\_config ............... enabled=False backend='inductor' kwargs={} [2025-01-07 12:50:31,185] [INFO] [config.py:1000:print] compression\_config ........... {'weight\_quantization': {'shared\_parameters': {'enabled': False, 'quantizer\_kernel': False, 'schedule\_offset': 0, 'quantize\_groups': 1, 'quantize\_verbose': False, 'quantization\_type': 'symmetric', 'quantize\_weight\_in\_forward': False, 'rounding': 'nearest', 'fp16\_mixed\_quantize': False, 'quantize\_change\_ratio': 0.001}, 'different\_groups': {}}, 'activation\_quantization': {'shared\_parameters': {'enabled': False, 'quantization\_type': 'symmetric', 'range\_calibration': 'dynamic', 'schedule\_offset': 1000}, 'different\_groups': {}}, 'sparse\_pruning': {'shared\_parameters': {'enabled': False, 'method': 'l1', 'schedule\_offset': 1000}, 'different\_groups': {}}, 'row\_pruning': {'shared\_parameters': {'enabled': False, 'method': 'l1', 'schedule\_offset': 1000}, 'different\_groups': {}}, 'head\_pruning': {'shared\_parameters': {'enabled': False, 'method': 'topk', 'schedule\_offset': 1000}, 'different\_groups': {}}, 'channel\_pruning': {'shared\_parameters': {'enabled': False, 'method': 'l1', 'schedule\_offset': 1000}, 'different\_groups': {}}, 'layer\_reduction': {'enabled': False}} [2025-01-07 12:50:31,186] [INFO] [config.py:1000:print] curriculum\_enabled\_legacy .... False [2025-01-07 12:50:31,186] [INFO] [config.py:1000:print] curriculum\_params\_legacy ..... False [2025-01-07 12:50:31,186] [INFO] [config.py:1000:print] data\_efficiency\_config ....... {'enabled': False, 'seed': 1234, 'data\_sampling': {'enabled': False, 'num\_epochs': 1000, 'num\_workers': 0, 'curriculum\_learning': {'enabled': False}}, 'data\_routing': {'enabled': False, 'random\_ltd': {'enabled': False, 'layer\_token\_lr\_schedule': {'enabled': False}}}} [2025-01-07 12:50:31,186] [INFO] [config.py:1000:print] data\_efficiency\_enabled ...... False [2025-01-07 12:50:31,186] [INFO] [config.py:1000:print] dataloader\_drop\_last ......... False [2025-01-07 12:50:31,186] [INFO] [config.py:1000:print] disable\_allgather ............ False [2025-01-07 12:50:31,186] [INFO] [config.py:1000:print] dump\_state ................... False [2025-01-07 12:50:31,186] [INFO] [config.py:1000:print] dynamic\_loss\_scale\_args ...... None [2025-01-07 12:50:31,186] [INFO] [config.py:1000:print] eigenvalue\_enabled ........... False [2025-01-07 12:50:31,186] [INFO] [config.py:1000:print] eigenvalue\_gas\_boundary\_resolution 1 [2025-01-07 12:50:31,186] [INFO] [config.py:1000:print] eigenvalue\_layer\_name ........ bert.encoder.layer [2025-01-07 12:50:31,186] [INFO] [config.py:1000:print] eigenvalue\_layer\_num ......... 0 [2025-01-07 12:50:31,186] [INFO] [config.py:1000:print] eigenvalue\_max\_iter .......... 100 [2025-01-07 12:50:31,186] [INFO] [config.py:1000:print] eigenvalue\_stability ......... 1e-06 [2025-01-07 12:50:31,186] [INFO] [config.py:1000:print] eigenvalue\_tol ............... 0.01 [2025-01-07 12:50:31,186] [INFO] [config.py:1000:print] eigenvalue\_verbose ........... False [2025-01-07 12:50:31,186] [INFO] [config.py:1000:print] elasticity\_enabled ........... False [2025-01-07 12:50:31,186] [INFO] [config.py:1000:print] flops\_profiler\_config ........ { "enabled": false, "recompute\_fwd\_factor": 0.0, "profile\_step": 1, "module\_depth": -1, "top\_modules": 1, "detailed": true, "output\_file": null } [2025-01-07 12:50:31,186] [INFO] [config.py:1000:print] fp16\_auto\_cast ............... None [2025-01-07 12:50:31,186] [INFO] [config.py:1000:print] fp16\_enabled ................. False [2025-01-07 12:50:31,186] [INFO] [config.py:1000:print] fp16\_master\_weights\_and\_gradients False [2025-01-07 12:50:31,186] [INFO] [config.py:1000:print] global\_rank .................. 0 [2025-01-07 12:50:31,186] [INFO] [config.py:1000:print] grad\_accum\_dtype ............. None [2025-01-07 12:50:31,186] [INFO] [config.py:1000:print] gradient\_accumulation\_steps .. 1 [2025-01-07 12:50:31,186] [INFO] [config.py:1000:print] gradient\_clipping ............ 0.0 [2025-01-07 12:50:31,186] [INFO] [config.py:1000:print] gradient\_predivide\_factor .... 1.0 [2025-01-07 12:50:31,186] [INFO] [config.py:1000:print] graph\_harvesting ............. False [2025-01-07 12:50:31,186] [INFO] [config.py:1000:print] hybrid\_engine ................ enabled=False max\_out\_tokens=512 inference\_tp\_size=1 release\_inference\_cache=False pin\_parameters=True tp\_gather\_partition\_size=8 [2025-01-07 12:50:31,186] [INFO] [config.py:1000:print] initial\_dynamic\_scale ........ 1 [2025-01-07 12:50:31,186] [INFO] [config.py:1000:print] load\_universal\_checkpoint .... False [2025-01-07 12:50:31,186] [INFO] [config.py:1000:print] loss\_scale ................... 1.0 [2025-01-07 12:50:31,186] [INFO] [config.py:1000:print] memory\_breakdown ............. False [2025-01-07 12:50:31,186] [INFO] [config.py:1000:print] mics\_hierarchial\_params\_gather False [2025-01-07 12:50:31,186] [INFO] [config.py:1000:print] mics\_shard\_size .............. -1 [2025-01-07 12:50:31,187] [INFO] [config.py:1000:print] monitor\_config ............... tensorboard=TensorBoardConfig(enabled=False, output\_path='', job\_name='DeepSpeedJobName') wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv\_monitor=CSVConfig(enabled=False, output\_path='', job\_name='DeepSpeedJobName') enabled=False [2025-01-07 12:50:31,187] [INFO] [config.py:1000:print] nebula\_config ................ { "enabled": false, "persistent\_storage\_path": null, "persistent\_time\_interval": 100, "num\_of\_version\_in\_retention": 2, "enable\_nebula\_load": true, "load\_path": null } [2025-01-07 12:50:31,187] [INFO] [config.py:1000:print] optimizer\_legacy\_fusion ...... False [2025-01-07 12:50:31,187] [INFO] [config.py:1000:print] optimizer\_name ............... None [2025-01-07 12:50:31,187] [INFO] [config.py:1000:print] optimizer\_params ............. None [2025-01-07 12:50:31,187] [INFO] [config.py:1000:print] pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed\_layers': False, 'activation\_checkpoint\_interval': 0, 'pipe\_partitioned': True, 'grad\_partitioned': True} [2025-01-07 12:50:31,187] [INFO] [config.py:1000:print] pld\_enabled .................. False [2025-01-07 12:50:31,187] [INFO] [config.py:1000:print] pld\_params ................... False [2025-01-07 12:50:31,187] [INFO] [config.py:1000:print] prescale\_gradients ........... False [2025-01-07 12:50:31,187] [INFO] [config.py:1000:print] scheduler\_name ............... None [2025-01-07 12:50:31,187] [INFO] [config.py:1000:print] scheduler\_params ............. None [2025-01-07 12:50:31,187] [INFO] [config.py:1000:print] seq\_parallel\_communication\_data\_type torch.float32 [2025-01-07 12:50:31,187] [INFO] [config.py:1000:print] sparse\_attention ............. None [2025-01-07 12:50:31,187] [INFO] [config.py:1000:print] sparse\_gradients\_enabled ..... False [2025-01-07 12:50:31,187] [INFO] [config.py:1000:print] steps\_per\_print .............. inf [2025-01-07 12:50:31,187] [INFO] [config.py:1000:print] train\_batch\_size ............. 24 [2025-01-07 12:50:31,187] [INFO] [config.py:1000:print] train\_micro\_batch\_size\_per\_gpu 24 [2025-01-07 12:50:31,187] [INFO] [config.py:1000:print] use\_data\_before\_expert\_parallel\_ False [2025-01-07 12:50:31,187] [INFO] [config.py:1000:print] use\_node\_local\_storage ....... False [2025-01-07 12:50:31,187] [INFO] [config.py:1000:print] wall\_clock\_breakdown ......... False [2025-01-07 12:50:31,187] [INFO] [config.py:1000:print] weight\_quantization\_config ... None [2025-01-07 12:50:31,187] [INFO] [config.py:1000:print] world\_size ................... 1 [2025-01-07 12:50:31,187] [INFO] [config.py:1000:print] zero\_allow\_untested\_optimizer True [2025-01-07 12:50:31,187] [INFO] [config.py:1000:print] zero\_config .................. stage=2 contiguous\_gradients=True reduce\_scatter=True reduce\_bucket\_size=200000000 use\_multi\_rank\_bucket\_allreduce=True allgather\_partitions=True allgather\_bucket\_size=200000000 overlap\_comm=True load\_from\_fp32\_weights=True elastic\_checkpoint=False offload\_param=None offload\_optimizer=None sub\_group\_size=1000000000 cpu\_offload\_param=None cpu\_offload\_use\_pin\_memory=None cpu\_offload=None prefetch\_bucket\_size=50,000,000 param\_persistence\_threshold=100,000 model\_persistence\_threshold=sys.maxsize max\_live\_parameters=1,000,000,000 max\_reuse\_distance=1,000,000,000 gather\_16bit\_weights\_on\_model\_save=False stage3\_gather\_fp16\_weights\_on\_model\_save=False ignore\_unused\_parameters=True legacy\_stage1=False round\_robin\_gradients=False zero\_hpz\_partition\_size=1 zero\_quantized\_weights=False zero\_quantized\_nontrainable\_weights=False zero\_quantized\_gradients=False mics\_shard\_size=-1 mics\_hierarchical\_params\_gather=False memory\_efficient\_linear=True pipeline\_loading\_checkpoint=False override\_module\_apply=True [2025-01-07 12:50:31,187] [INFO] [config.py:1000:print] zero\_enabled ................. True [2025-01-07 12:50:31,187] [INFO] [config.py:1000:print] zero\_force\_ds\_cpu\_optimizer .. True [2025-01-07 12:50:31,187] [INFO] [config.py:1000:print] zero\_optimization\_stage ...... 2 [2025-01-07 12:50:31,187] [INFO] [config.py:986:print\_user\_config] json = { "bf16": { "enabled": true, "auto\_cast": true }, "zero\_optimization": { "stage": 2, "allgather\_partitions": true, "allgather\_bucket\_size": 2.000000e+08, "overlap\_comm": true, "reduce\_scatter": true, "reduce\_bucket\_size": 2.000000e+08, "contiguous\_gradients": true, "sub\_group\_size": 1.000000e+09 }, "gradient\_accumulation\_steps": 1, "train\_batch\_size": 24, "train\_micro\_batch\_size\_per\_gpu": 24, "steps\_per\_print": inf, "wall\_clock\_breakdown": false, "fp16": { "enabled": false }, "zero\_allow\_untested\_optimizer": true } 0it [00:01, ?it/s] Traceback (most recent call last): File "/content/drive/MyDrive/time-llm/Time-LLM/run\_main.py", line 211, in outputs = model(batch\_x, batch\_x\_mark, dec\_inp, batch\_y\_mark) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in \_wrapped\_call\_impl return self.\_call\_impl(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in \_call\_impl return forward\_call(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/deepspeed/utils/nvtx.py", line 15, in wrapped\_fn ret\_val = func(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/deepspeed/runtime/engine.py", line 1852, in forward loss = self.module(\*inputs, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in \_wrapped\_call\_impl return self.\_call\_impl(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in \_call\_impl return forward\_call(\*args, \*\*kwargs) File "/content/drive/MyDrive/time-llm/Time-LLM/models/TimeLLM.py", line 196, in forward dec\_out = self.forecast(x\_enc, x\_mark\_enc, x\_dec, x\_mark\_dec) File "/content/drive/MyDrive/time-llm/Time-LLM/models/TimeLLM.py", line 243, in forecast dec\_out = self.llm\_model(inputs\_embeds=llama\_enc\_out).last\_hidden\_state File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in \_wrapped\_call\_impl return self.\_call\_impl(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in \_call\_impl return forward\_call(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/transformers/models/llama/modeling\_llama.py", line 693, in forward layer\_outputs = decoder\_layer( File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in \_wrapped\_call\_impl return self.\_call\_impl(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in \_call\_impl return forward\_call(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/transformers/models/llama/modeling\_llama.py", line 408, in forward hidden\_states, self\_attn\_weights, present\_key\_value = self.self\_attn( File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in \_wrapped\_call\_impl return self.\_call\_impl(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call\_impl return forward\_call(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/transformers/models/llama/modeling\_llama.py", line 346, in forward attn\_weights = nn.functional.softmax(attn\_weights, dim=-1, dtype=torch.float32).to(query\_states.dtype) File "/usr/local/lib/python3.10/dist-packages/torch/nn/functional.py", line 1860, in softmax ret = input.softmax(dim, dtype=dtype) torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 170.00 MiB. GPU 0 has a total capacity of 39.56 GiB of which 144.81 MiB is free. Process 152763 has 39.41 GiB memory in use. Of the allocated memory 38.58 GiB is allocated by PyTorch, and 12.12 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH\_CUDA\_ALLOC\_CONF=expandable\_segments:True to avoid fragmentation. See documentation for Memory Management ([CUDA semantics — PyTorch 2.6 documentation](https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)) Traceback (most recent call last): File "/usr/local/bin/accelerate", line 8, in sys.exit(main()) File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/accelerate\_cli.py", line 46, in main args.func(args) File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py", line 1057, in launch\_command simple\_launcher(args) File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py", line 673, in simple\_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['/usr/bin/python3', 'run\_main.py', '--task\_name', 'long\_term\_forecast', '--is\_training', '1', '--root\_path', './dataset/ETT-small/', '--data\_path', 'ETTh1.csv', '--model\_id', 'ETTh1\_512\_96', '--model', 'TimeLLM', '--data', 'ETTh1', '--features', 'M', '--seq\_len', '512', '--label\_len', '48', '--pred\_len', '96', '--factor', '3', '--enc\_in', '7', '--dec\_in', '7', '--c\_out', '7', '--des', 'Exp', '--itr', '1', '--d\_model', '32', '--d\_ff', '128', '--batch\_size', '24', '--learning\_rate', '0.01', '--llm\_layers', '32', '--train\_epochs', '100', '--model\_comment', 'TimeLLM-ETTh1']' returned non-zero exit status 1. Loading checkpoint shards: 100% 2/2 [00:05\<00:00, 2.79s/it] [2025-01-07 12:51:42,380] [INFO] [real\_accelerator.py:191:get\_accelerator] Setting ds\_accelerator to cuda (auto detect) [2025-01-07 12:51:42,648] [INFO] [logging.py:96:log\_dist] [Rank -1] DeepSpeed info: version=0.14.0, git-hash=unknown, git-branch=unknown [2025-01-07 12:51:42,648] [INFO] [comm.py:637:init\_distributed] cdb=None [2025-01-07 12:51:42,648] [INFO] [comm.py:652:init\_distributed] Not using the DeepSpeed or dist launchers, attempting to detect MPI environment... [2025-01-07 12:51:43,524] [INFO] [comm.py:702:mpi\_discovery] Discovered MPI settings of world\_rank=0, local\_rank=0, world\_size=1, master\_addr=172.28.0.12, master\_port=29500 [2025-01-07 12:51:43,524] [INFO] [comm.py:668:init\_distributed] Initializing TorchBackend in DeepSpeed with backend nccl [2025-01-07 12:51:58,542] [INFO] [logging.py:96:log\_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False [2025-01-07 12:51:58,543] [INFO] [logging.py:96:log\_dist] [Rank 0] Using client Optimizer as basic optimizer [2025-01-07 12:51:58,543] [INFO] [logging.py:96:log\_dist] [Rank 0] Removing param\_group that has no 'params' in the basic Optimizer [2025-01-07 12:51:58,545] [INFO] [logging.py:96:log\_dist] [Rank 0] DeepSpeed Basic Optimizer = Adam [2025-01-07 12:51:58,545] [INFO] [utils.py:56:is\_zero\_supported\_optimizer] Checking ZeRO support for optimizer=Adam type=\ [2025-01-07 12:51:58,545] [INFO] [logging.py:96:log\_dist] [Rank 0] Creating torch.bfloat16 ZeRO stage 2 optimizer [2025-01-07 12:51:58,545] [INFO] [stage\_1\_and\_2.py:149: **init**] Reduce bucket size 200000000 [2025-01-07 12:51:58,545] [INFO] [stage\_1\_and\_2.py:150: **init**] Allgather bucket size 200000000 [2025-01-07 12:51:58,545] [INFO] [stage\_1\_and\_2.py:151: **init**] CPU Offload: False [2025-01-07 12:51:58,545] [INFO] [stage\_1\_and\_2.py:152: **init**] Round robin gradient partitioning: False [2025-01-07 12:51:58,902] [INFO] [utils.py:800:see\_memory\_usage] Before initializing optimizer states [2025-01-07 12:51:58,903] [INFO] [utils.py:801:see\_memory\_usage] MA 12.6 GB Max\_MA 12.69 GB CA 12.69 GB Max\_CA 13 GB [2025-01-07 12:51:58,903] [INFO] [utils.py:808:see\_memory\_usage] CPU Virtual Memory: used = 3.45 GB, percent = 4.1% [2025-01-07 12:51:59,064] [INFO] [utils.py:800:see\_memory\_usage] After initializing optimizer states [2025-01-07 12:51:59,065] [INFO] [utils.py:801:see\_memory\_usage] MA 12.6 GB Max\_MA 12.77 GB CA 12.86 GB Max\_CA 13 GB [2025-01-07 12:51:59,065] [INFO] [utils.py:808:see\_memory\_usage] CPU Virtual Memory: used = 3.45 GB, percent = 4.1% [2025-01-07 12:51:59,065] [INFO] [stage\_1\_and\_2.py:539: **init**] optimizer state initialized [2025-01-07 12:51:59,222] [INFO] [utils.py:800:see\_memory\_usage] After initializing ZeRO optimizer [2025-01-07 12:51:59,223] [INFO] [utils.py:801:see\_memory\_usage] MA 12.6 GB Max\_MA 12.6 GB CA 12.86 GB Max\_CA 13 GB [2025-01-07 12:51:59,223] [INFO] [utils.py:808:see\_memory\_usage] CPU Virtual Memory: used = 3.45 GB, percent = 4.1% [2025-01-07 12:51:59,224] [INFO] [logging.py:96:log\_dist] [Rank 0] DeepSpeed Final Optimizer = Adam [2025-01-07 12:51:59,224] [INFO] [logging.py:96:log\_dist] [Rank 0] DeepSpeed using client LR scheduler [2025-01-07 12:51:59,224] [INFO] [logging.py:96:log\_dist] [Rank 0] DeepSpeed LR Scheduler = None [2025-01-07 12:51:59,224] [INFO] [logging.py:96:log\_dist] [Rank 0] step=0, skipped=0, lr=[0.0007999999999999986], mom=[(0.95, 0.999)] [2025-01-07 12:51:59,225] [INFO] [config.py:996:print] DeepSpeedEngine configuration: [2025-01-07 12:51:59,225] [INFO] [config.py:1000:print] activation\_checkpointing\_config { "partition\_activations": false, "contiguous\_memory\_optimization": false, "cpu\_checkpointing": false, "number\_checkpoints": null, "synchronize\_checkpoint\_boundary": false, "profile": false } [2025-01-07 12:51:59,226] [INFO] [config.py:1000:print] aio\_config ................... {'block\_size': 1048576, 'queue\_depth': 8, 'thread\_count': 1, 'single\_submit': False, 'overlap\_events': True} [2025-01-07 12:51:59,226] [INFO] [config.py:1000:print] amp\_enabled .................. False [2025-01-07 12:51:59,226] [INFO] [config.py:1000:print] amp\_params ................... False [2025-01-07 12:51:59,226] [INFO] [config.py:1000:print] autotuning\_config ............ { "enabled": false, "start\_step": null, "end\_step": null, "metric\_path": null, "arg\_mappings": null, "metric": "throughput", "model\_info": null, "results\_dir": "autotuning\_results", "exps\_dir": "autotuning\_exps", "overwrite": true, "fast": true, "start\_profile\_step": 3, "end\_profile\_step": 5, "tuner\_type": "gridsearch", "tuner\_early\_stopping": 5, "tuner\_num\_trials": 50, "model\_info\_path": null, "mp\_size": 1, "max\_train\_batch\_size": null, "min\_train\_batch\_size": 1, "max\_train\_micro\_batch\_size\_per\_gpu": 1.024000e+03, "min\_train\_micro\_batch\_size\_per\_gpu": 1, "num\_tuning\_micro\_batch\_sizes": 3 } [2025-01-07 12:51:59,226] [INFO] [config.py:1000:print] bfloat16\_enabled ............. True [2025-01-07 12:51:59,226] [INFO] [config.py:1000:print] bfloat16\_immediate\_grad\_update False [2025-01-07 12:51:59,226] [INFO] [config.py:1000:print] checkpoint\_parallel\_write\_pipeline False [2025-01-07 12:51:59,226] [INFO] [config.py:1000:print] checkpoint\_tag\_validation\_enabled True [2025-01-07 12:51:59,226] [INFO] [config.py:1000:print] checkpoint\_tag\_validation\_fail False [2025-01-07 12:51:59,226] [INFO] [config.py:1000:print] comms\_config ................. \ [2025-01-07 12:51:59,226] [INFO] [config.py:1000:print] communication\_data\_type ...... None [2025-01-07 12:51:59,226] [INFO] [config.py:1000:print] compile\_config ............... enabled=False backend='inductor' kwargs={} [2025-01-07 12:51:59,226] [INFO] [config.py:1000:print] compression\_config ........... {'weight\_quantization': {'shared\_parameters': {'enabled': False, 'quantizer\_kernel': False, 'schedule\_offset': 0, 'quantize\_groups': 1, 'quantize\_verbose': False, 'quantization\_type': 'symmetric', 'quantize\_weight\_in\_forward': False, 'rounding': 'nearest', 'fp16\_mixed\_quantize': False, 'quantize\_change\_ratio': 0.001}, 'different\_groups': {}}, 'activation\_quantization': {'shared\_parameters': {'enabled': False, 'quantization\_type': 'symmetric', 'range\_calibration': 'dynamic', 'schedule\_offset': 1000}, 'different\_groups': {}}, 'sparse\_pruning': {'shared\_parameters': {'enabled': False, 'method': 'l1', 'schedule\_offset': 1000}, 'different\_groups': {}}, 'row\_pruning': {'shared\_parameters': {'enabled': False, 'method': 'l1', 'schedule\_offset': 1000}, 'different\_groups': {}}, 'head\_pruning': {'shared\_parameters': {'enabled': False, 'method': 'topk', 'schedule\_offset': 1000}, 'different\_groups': {}}, 'channel\_pruning': {'shared\_parameters': {'enabled': False, 'method': 'l1', 'schedule\_offset': 1000}, 'different\_groups': {}}, 'layer\_reduction': {'enabled': False}} [2025-01-07 12:51:59,226] [INFO] [config.py:1000:print] curriculum\_enabled\_legacy .... False [2025-01-07 12:51:59,226] [INFO] [config.py:1000:print] curriculum\_params\_legacy ..... False [2025-01-07 12:51:59,226] [INFO] [config.py:1000:print] data\_efficiency\_config ....... {'enabled': False, 'seed': 1234, 'data\_sampling': {'enabled': False, 'num\_epochs': 1000, 'num\_workers': 0, 'curriculum\_learning': {'enabled': False}}, 'data\_routing': {'enabled': False, 'random\_ltd': {'enabled': False, 'layer\_token\_lr\_schedule': {'enabled': False}}}} [2025-01-07 12:51:59,226] [INFO] [config.py:1000:print] data\_efficiency\_enabled ...... False [2025-01-07 12:51:59,226] [INFO] [config.py:1000:print] dataloader\_drop\_last ......... False [2025-01-07 12:51:59,226] [INFO] [config.py:1000:print] disable\_allgather ............ False [2025-01-07 12:51:59,226] [INFO] [config.py:1000:print] dump\_state ................... False [2025-01-07 12:51:59,226] [INFO] [config.py:1000:print] dynamic\_loss\_scale\_args ...... None [2025-01-07 12:51:59,226] [INFO] [config.py:1000:print] eigenvalue\_enabled ........... False [2025-01-07 12:51:59,226] [INFO] [config.py:1000:print] eigenvalue\_gas\_boundary\_resolution 1 [2025-01-07 12:51:59,226] [INFO] [config.py:1000:print] eigenvalue\_layer\_name ........ bert.encoder.layer [2025-01-07 12:51:59,226] [INFO] [config.py:1000:print] eigenvalue\_layer\_num ......... 0 [2025-01-07 12:51:59,226] [INFO] [config.py:1000:print] eigenvalue\_max\_iter .......... 100 [2025-01-07 12:51:59,227] [INFO] [config.py:1000:print] eigenvalue\_stability ......... 1e-06 [2025-01-07 12:51:59,227] [INFO] [config.py:1000:print] eigenvalue\_tol ............... 0.01 [2025-01-07 12:51:59,227] [INFO] [config.py:1000:print] eigenvalue\_verbose ........... False [2025-01-07 12:51:59,227] [INFO] [config.py:1000:print] elasticity\_enabled ........... False [2025-01-07 12:51:59,227] [INFO] [config.py:1000:print] flops\_profiler\_config ........ { "enabled": false, "recompute\_fwd\_factor": 0.0, "profile\_step": 1, "module\_depth": -1, "top\_modules": 1, "detailed": true, "output\_file": null } [2025-01-07 12:51:59,227] [INFO] [config.py:1000:print] fp16\_auto\_cast ............... None [2025-01-07 12:51:59,227] [INFO] [config.py:1000:print] fp16\_enabled ................. False [2025-01-07 12:51:59,227] [INFO] [config.py:1000:print] fp16\_master\_weights\_and\_gradients False [2025-01-07 12:51:59,227] [INFO] [config.py:1000:print] global\_rank .................. 0 [2025-01-07 12:51:59,227] [INFO] [config.py:1000:print] grad\_accum\_dtype ............. None [2025-01-07 12:51:59,227] [INFO] [config.py:1000:print] gradient\_accumulation\_steps .. 1 [2025-01-07 12:51:59,227] [INFO] [config.py:1000:print] gradient\_clipping ............ 0.0 [2025-01-07 12:51:59,227] [INFO] [config.py:1000:print] gradient\_predivide\_factor .... 1.0 [2025-01-07 12:51:59,227] [INFO] [config.py:1000:print] graph\_harvesting ............. False [2025-01-07 12:51:59,227] [INFO] [config.py:1000:print] hybrid\_engine ................ enabled=False max\_out\_tokens=512 inference\_tp\_size=1 release\_inference\_cache=False pin\_parameters=True tp\_gather\_partition\_size=8 [2025-01-07 12:51:59,227] [INFO] [config.py:1000:print] initial\_dynamic\_scale ........ 1 [2025-01-07 12:51:59,227] [INFO] [config.py:1000:print] load\_universal\_checkpoint .... False [2025-01-07 12:51:59,227] [INFO] [config.py:1000:print] loss\_scale ................... 1.0 [2025-01-07 12:51:59,227] [INFO] [config.py:1000:print] memory\_breakdown ............. False [2025-01-07 12:51:59,227] [INFO] [config.py:1000:print] mics\_hierarchial\_params\_gather False [2025-01-07 12:51:59,227] [INFO] [config.py:1000:print] mics\_shard\_size .............. -1 [2025-01-07 12:51:59,227] [INFO] [config.py:1000:print] monitor\_config ............... tensorboard=TensorBoardConfig(enabled=False, output\_path='', job\_name='DeepSpeedJobName') wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv\_monitor=CSVConfig(enabled=False, output\_path='', job\_name='DeepSpeedJobName') enabled=False [2025-01-07 12:51:59,227] [INFO] [config.py:1000:print] nebula\_config ................ { "enabled": false, "persistent\_storage\_path": null, "persistent\_time\_interval": 100, "num\_of\_version\_in\_retention": 2, "enable\_nebula\_load": true, "load\_path": null } [2025-01-07 12:51:59,227] [INFO] [config.py:1000:print] optimizer\_legacy\_fusion ...... False [2025-01-07 12:51:59,227] [INFO] [config.py:1000:print] optimizer\_name ............... None [2025-01-07 12:51:59,227] [INFO] [config.py:1000:print] optimizer\_params ............. None [2025-01-07 12:51:59,227] [INFO] [config.py:1000:print] pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed\_layers': False, 'activation\_checkpoint\_interval': 0, 'pipe\_partitioned': True, 'grad\_partitioned': True} [2025-01-07 12:51:59,227] [INFO] [config.py:1000:print] pld\_enabled .................. False [2025-01-07 12:51:59,227] [INFO] [config.py:1000:print] pld\_params ................... False [2025-01-07 12:51:59,227] [INFO] [config.py:1000:print] prescale\_gradients ........... False [2025-01-07 12:51:59,227] [INFO] [config.py:1000:print] scheduler\_name ............... None [2025-01-07 12:51:59,227] [INFO] [config.py:1000:print] scheduler\_params ............. None [2025-01-07 12:51:59,228] [INFO] [config.py:1000:print] seq\_parallel\_communication\_data\_type torch.float32 [2025-01-07 12:51:59,228] [INFO] [config.py:1000:print] sparse\_attention ............. None [2025-01-07 12:51:59,228] [INFO] [config.py:1000:print] sparse\_gradients\_enabled ..... False [2025-01-07 12:51:59,228] [INFO] [config.py:1000:print] steps\_per\_print .............. inf [2025-01-07 12:51:59,228] [INFO] [config.py:1000:print] train\_batch\_size ............. 24 [2025-01-07 12:51:59,228] [INFO] [config.py:1000:print] train\_micro\_batch\_size\_per\_gpu 24 [2025-01-07 12:51:59,228] [INFO] [config.py:1000:print] use\_data\_before\_expert\_parallel_ False [2025-01-07 12:51:59,228] [INFO] [config.py:1000:print] use\_node\_local\_storage ....... False [2025-01-07 12:51:59,228] [INFO] [config.py:1000:print] wall\_clock\_breakdown ......... False [2025-01-07 12:51:59,228] [INFO] [config.py:1000:print] weight\_quantization\_config ... None [2025-01-07 12:51:59,228] [INFO] [config.py:1000:print] world\_size ................... 1 [2025-01-07 12:51:59,228] [INFO] [config.py:1000:print] zero\_allow\_untested\_optimizer True [2025-01-07 12:51:59,228] [INFO] [config.py:1000:print] zero\_config .................. stage=2 contiguous\_gradients=True reduce\_scatter=True reduce\_bucket\_size=200000000 use\_multi\_rank\_bucket\_allreduce=True allgather\_partitions=True allgather\_bucket\_size=200000000 overlap\_comm=True load\_from\_fp32\_weights=True elastic\_checkpoint=False offload\_param=None offload\_optimizer=None sub\_group\_size=1000000000 cpu\_offload\_param=None cpu\_offload\_use\_pin\_memory=None cpu\_offload=None prefetch\_bucket\_size=50,000,000 param\_persistence\_threshold=100,000 model\_persistence\_threshold=sys.maxsize max\_live\_parameters=1,000,000,000 max\_reuse\_distance=1,000,000,000 gather\_16bit\_weights\_on\_model\_save=False stage3\_gather\_fp16\_weights\_on\_model\_save=False ignore\_unused\_parameters=True legacy\_stage1=False round\_robin\_gradients=False zero\_hpz\_partition\_size=1 zero\_quantized\_weights=False zero\_quantized\_nontrainable\_weights=False zero\_quantized\_gradients=False mics\_shard\_size=-1 mics\_hierarchical\_params\_gather=False memory\_efficient\_linear=True pipeline\_loading\_checkpoint=False override\_module\_apply=True [2025-01-07 12:51:59,228] [INFO] [config.py:1000:print] zero\_enabled ................. True [2025-01-07 12:51:59,228] [INFO] [config.py:1000:print] zero\_force\_ds\_cpu\_optimizer .. True [2025-01-07 12:51:59,228] [INFO] [config.py:1000:print] zero\_optimization\_stage ...... 2 [2025-01-07 12:51:59,228] [INFO] [config.py:986:print\_user\_config] json = { "bf16": { "enabled": true, "auto\_cast": true }, "zero\_optimization": { "stage": 2, "allgather\_partitions": true, "allgather\_bucket\_size": 2.000000e+08, "overlap\_comm": true, "reduce\_scatter": true, "reduce\_bucket\_size": 2.000000e+08, "contiguous\_gradients": true, "sub\_group\_size": 1.000000e+09 }, "gradient\_accumulation\_steps": 1, "train\_batch\_size": 24, "train\_micro\_batch\_size\_per\_gpu": 24, "steps\_per\_print": inf, "wall\_clock\_breakdown": false, "fp16": { "enabled": false }, "zero\_allow\_untested\_optimizer": true } 0it [00:01, ?it/s] Traceback (most recent call last): File "/content/drive/MyDrive/time-llm/Time-LLM/run\_main.py", line 211, in outputs = model(batch\_x, batch\_x\_mark, dec\_inp, batch\_y\_mark) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in \_wrapped\_call\_impl return self.\_call\_impl(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in \_call\_impl return forward\_call(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/deepspeed/utils/nvtx.py", line 15, in wrapped\_fn ret\_val = func(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/deepspeed/runtime/engine.py", line 1852, in forward loss = self.module(\*inputs, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in \_wrapped\_call\_impl return self.\_call\_impl(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in \_call\_impl return forward\_call(\*args, \*\*kwargs) File "/content/drive/MyDrive/time-llm/Time-LLM/models/TimeLLM.py", line 196, in forward dec\_out = self.forecast(x\_enc, x\_mark\_enc, x\_dec, x\_mark\_dec) File "/content/drive/MyDrive/time-llm/Time-LLM/models/TimeLLM.py", line 243, in forecast dec\_out = self.llm\_model(inputs\_embeds=llama\_enc\_out).last\_hidden\_state File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in \_wrapped\_call\_impl return self.\_call\_impl(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in \_call\_impl return forward\_call(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/transformers/models/llama/modeling\_llama.py", line 693, in forward layer\_outputs = decoder\_layer( File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in \_wrapped\_call\_impl return self.\_call\_impl(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in \_call\_impl return forward\_call(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/transformers/models/llama/modeling\_llama.py", line 408, in forward hidden\_states, self\_attn\_weights, present\_key\_value = self.self\_attn( File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in \_wrapped\_call\_impl return self.\_call\_impl(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call\_impl return forward\_call(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/transformers/models/llama/modeling\_llama.py", line 346, in forward attn\_weights = nn.functional.softmax(attn\_weights, dim=-1, dtype=torch.float32).to(query\_states.dtype) File "/usr/local/lib/python3.10/dist-packages/torch/nn/functional.py", line 1860, in softmax ret = input.softmax(dim, dtype=dtype) torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 170.00 MiB. GPU 0 has a total capacity of 39.56 GiB of which 64.81 MiB is free. Process 159212 has 39.49 GiB memory in use. Of the allocated memory 38.65 GiB is allocated by PyTorch, and 11.97 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH\_CUDA\_ALLOC\_CONF=expandable\_segments:True to avoid fragmentation. See documentation for Memory Management ([CUDA semantics — PyTorch 2.6 documentation](https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)) Traceback (most recent call last): File "/usr/local/bin/accelerate", line 8, in sys.exit(main()) File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/accelerate\_cli.py", line 46, in main args.func(args) File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py", line 1057, in launch\_command simple\_launcher(args) File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py", line 673, in simple\_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['/usr/bin/python3', 'run\_main.py', '--task\_name', 'long\_term\_forecast', '--is\_training', '1', '--root\_path', './dataset/ETT-small/', '--data\_path', 'ETTh1.csv', '--model\_id', 'ETTh1\_512\_192', '--model', 'TimeLLM', '--data', 'ETTh1', '--features', 'M', '--seq\_len', '512', '--label\_len', '48', '--pred\_len', '192', '--factor', '3', '--enc\_in', '7', '--dec\_in', '7', '--c\_out', '7', '--des', 'Exp', '--itr', '1', '--d\_model', '32', '--d\_ff', '128', '--batch\_size', '24', '--learning\_rate', '0.02', '--llm\_layers', '32', '--train\_epochs', '100', '--model\_comment', 'TimeLLM-ETTh1']' returned non-zero exit status 1. Loading checkpoint shards: 100% 2/2 [00:05\<00:00, 2.77s/it] [2025-01-07 12:53:10,060] [INFO] [real\_accelerator.py:191:get\_accelerator] Setting ds\_accelerator to cuda (auto detect) [2025-01-07 12:53:10,331] [INFO] [logging.py:96:log\_dist] [Rank -1] DeepSpeed info: version=0.14.0, git-hash=unknown, git-branch=unknown [2025-01-07 12:53:10,331] [INFO] [comm.py:637:init\_distributed] cdb=None [2025-01-07 12:53:10,331] [INFO] [comm.py:652:init\_distributed] Not using the DeepSpeed or dist launchers, attempting to detect MPI environment... [2025-01-07 12:53:11,207] [INFO] [comm.py:702:mpi\_discovery] Discovered MPI settings of world\_rank=0, local\_rank=0, world\_size=1, master\_addr=172.28.0.12, master\_port=29500 [2025-01-07 12:53:11,208] [INFO] [comm.py:668:init\_distributed] Initializing TorchBackend in DeepSpeed with backend nccl [2025-01-07 12:53:26,145] [INFO] [logging.py:96:log\_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False [2025-01-07 12:53:26,147] [INFO] [logging.py:96:log\_dist] [Rank 0] Using client Optimizer as basic optimizer [2025-01-07 12:53:26,147] [INFO] [logging.py:96:log\_dist] [Rank 0] Removing param\_group that has no 'params' in the basic Optimizer [2025-01-07 12:53:26,148] [INFO] [logging.py:96:log\_dist] [Rank 0] DeepSpeed Basic Optimizer = Adam [2025-01-07 12:53:26,148] [INFO] [utils.py:56:is\_zero\_supported\_optimizer] Checking ZeRO support for optimizer=Adam type=\ [2025-01-07 12:53:26,149] [INFO] [logging.py:96:log\_dist] [Rank 0] Creating torch.bfloat16 ZeRO stage 2 optimizer [2025-01-07 12:53:26,149] [INFO] [stage\_1\_and\_2.py:149: **init**] Reduce bucket size 200000000 [2025-01-07 12:53:26,149] [INFO] [stage\_1\_and\_2.py:150: **init**] Allgather bucket size 200000000 [2025-01-07 12:53:26,149] [INFO] [stage\_1\_and\_2.py:151: **init**] CPU Offload: False [2025-01-07 12:53:26,149] [INFO] [stage\_1\_and\_2.py:152: **init**] Round robin gradient partitioning: False [2025-01-07 12:53:26,517] [INFO] [utils.py:800:see\_memory\_usage] Before initializing optimizer states [2025-01-07 12:53:26,517] [INFO] [utils.py:801:see\_memory\_usage] MA 12.61 GB Max\_MA 12.7 GB CA 12.7 GB Max\_CA 13 GB [2025-01-07 12:53:26,518] [INFO] [utils.py:808:see\_memory\_usage] CPU Virtual Memory: used = 3.46 GB, percent = 4.1% [2025-01-07 12:53:26,687] [INFO] [utils.py:800:see\_memory\_usage] After initializing optimizer states [2025-01-07 12:53:26,688] [INFO] [utils.py:801:see\_memory\_usage] MA 12.61 GB Max\_MA 12.78 GB CA 12.88 GB Max\_CA 13 GB [2025-01-07 12:53:26,688] [INFO] [utils.py:808:see\_memory\_usage] CPU Virtual Memory: used = 3.46 GB, percent = 4.1% [2025-01-07 12:53:26,688] [INFO] [stage\_1\_and\_2.py:539: **init**] optimizer state initialized [2025-01-07 12:53:26,854] [INFO] [utils.py:800:see\_memory\_usage] After initializing ZeRO optimizer [2025-01-07 12:53:26,855] [INFO] [utils.py:801:see\_memory\_usage] MA 12.61 GB Max\_MA 12.61 GB CA 12.88 GB Max\_CA 13 GB [2025-01-07 12:53:26,855] [INFO] [utils.py:808:see\_memory\_usage] CPU Virtual Memory: used = 3.46 GB, percent = 4.1% [2025-01-07 12:53:26,856] [INFO] [logging.py:96:log\_dist] [Rank 0] DeepSpeed Final Optimizer = Adam [2025-01-07 12:53:26,856] [INFO] [logging.py:96:log\_dist] [Rank 0] DeepSpeed using client LR scheduler [2025-01-07 12:53:26,856] [INFO] [logging.py:96:log\_dist] [Rank 0] DeepSpeed LR Scheduler = None [2025-01-07 12:53:26,856] [INFO] [logging.py:96:log\_dist] [Rank 0] step=0, skipped=0, lr=[0.001], mom=[(0.9, 0.999)] [2025-01-07 12:53:26,857] [INFO] [config.py:996:print] DeepSpeedEngine configuration: [2025-01-07 12:53:26,858] [INFO] [config.py:1000:print] activation\_checkpointing\_config { "partition\_activations": false, "contiguous\_memory\_optimization": false, "cpu\_checkpointing": false, "number\_checkpoints": null, "synchronize\_checkpoint\_boundary": false, "profile": false } [2025-01-07 12:53:26,858] [INFO] [config.py:1000:print] aio\_config ................... {'block\_size': 1048576, 'queue\_depth': 8, 'thread\_count': 1, 'single\_submit': False, 'overlap\_events': True} [2025-01-07 12:53:26,858] [INFO] [config.py:1000:print] amp\_enabled .................. False [2025-01-07 12:53:26,858] [INFO] [config.py:1000:print] amp\_params ................... False [2025-01-07 12:53:26,858] [INFO] [config.py:1000:print] autotuning\_config ............ { "enabled": false, "start\_step": null, "end\_step": null, "metric\_path": null, "arg\_mappings": null, "metric": "throughput", "model\_info": null, "results\_dir": "autotuning\_results", "exps\_dir": "autotuning\_exps", "overwrite": true, "fast": true, "start\_profile\_step": 3, "end\_profile\_step": 5, "tuner\_type": "gridsearch", "tuner\_early\_stopping": 5, "tuner\_num\_trials": 50, "model\_info\_path": null, "mp\_size": 1, "max\_train\_batch\_size": null, "min\_train\_batch\_size": 1, "max\_train\_micro\_batch\_size\_per\_gpu": 1.024000e+03, "min\_train\_micro\_batch\_size\_per\_gpu": 1, "num\_tuning\_micro\_batch\_sizes": 3 } [2025-01-07 12:53:26,858] [INFO] [config.py:1000:print] bfloat16\_enabled ............. True [2025-01-07 12:53:26,858] [INFO] [config.py:1000:print] bfloat16\_immediate\_grad\_update False [2025-01-07 12:53:26,858] [INFO] [config.py:1000:print] checkpoint\_parallel\_write\_pipeline False [2025-01-07 12:53:26,858] [INFO] [config.py:1000:print] checkpoint\_tag\_validation\_enabled True [2025-01-07 12:53:26,858] [INFO] [config.py:1000:print] checkpoint\_tag\_validation\_fail False [2025-01-07 12:53:26,858] [INFO] [config.py:1000:print] comms\_config ................. \ [2025-01-07 12:53:26,858] [INFO] [config.py:1000:print] communication\_data\_type ...... None [2025-01-07 12:53:26,858] [INFO] [config.py:1000:print] compile\_config ............... enabled=False backend='inductor' kwargs={} [2025-01-07 12:53:26,858] [INFO] [config.py:1000:print] compression\_config ........... {'weight\_quantization': {'shared\_parameters': {'enabled': False, 'quantizer\_kernel': False, 'schedule\_offset': 0, 'quantize\_groups': 1, 'quantize\_verbose': False, 'quantization\_type': 'symmetric', 'quantize\_weight\_in\_forward': False, 'rounding': 'nearest', 'fp16\_mixed\_quantize': False, 'quantize\_change\_ratio': 0.001}, 'different\_groups': {}}, 'activation\_quantization': {'shared\_parameters': {'enabled': False, 'quantization\_type': 'symmetric', 'range\_calibration': 'dynamic', 'schedule\_offset': 1000}, 'different\_groups': {}}, 'sparse\_pruning': {'shared\_parameters': {'enabled': False, 'method': 'l1', 'schedule\_offset': 1000}, 'different\_groups': {}}, 'row\_pruning': {'shared\_parameters': {'enabled': False, 'method': 'l1', 'schedule\_offset': 1000}, 'different\_groups': {}}, 'head\_pruning': {'shared\_parameters': {'enabled': False, 'method': 'topk', 'schedule\_offset': 1000}, 'different\_groups': {}}, 'channel\_pruning': {'shared\_parameters': {'enabled': False, 'method': 'l1', 'schedule\_offset': 1000}, 'different\_groups': {}}, 'layer\_reduction': {'enabled': False}} [2025-01-07 12:53:26,858] [INFO] [config.py:1000:print] curriculum\_enabled\_legacy .... False [2025-01-07 12:53:26,858] [INFO] [config.py:1000:print] curriculum\_params\_legacy ..... False [2025-01-07 12:53:26,858] [INFO] [config.py:1000:print] data\_efficiency\_config ....... {'enabled': False, 'seed': 1234, 'data\_sampling': {'enabled': False, 'num\_epochs': 1000, 'num\_workers': 0, 'curriculum\_learning': {'enabled': False}}, 'data\_routing': {'enabled': False, 'random\_ltd': {'enabled': False, 'layer\_token\_lr\_schedule': {'enabled': False}}}} [2025-01-07 12:53:26,858] [INFO] [config.py:1000:print] data\_efficiency\_enabled ...... False [2025-01-07 12:53:26,858] [INFO] [config.py:1000:print] dataloader\_drop\_last ......... False [2025-01-07 12:53:26,858] [INFO] [config.py:1000:print] disable\_allgather ............ False [2025-01-07 12:53:26,858] [INFO] [config.py:1000:print] dump\_state ................... False [2025-01-07 12:53:26,858] [INFO] [config.py:1000:print] dynamic\_loss\_scale\_args ...... None [2025-01-07 12:53:26,858] [INFO] [config.py:1000:print] eigenvalue\_enabled ........... False [2025-01-07 12:53:26,858] [INFO] [config.py:1000:print] eigenvalue\_gas\_boundary\_resolution 1 [2025-01-07 12:53:26,858] [INFO] [config.py:1000:print] eigenvalue\_layer\_name ........ bert.encoder.layer [2025-01-07 12:53:26,858] [INFO] [config.py:1000:print] eigenvalue\_layer\_num ......... 0 [2025-01-07 12:53:26,859] [INFO] [config.py:1000:print] eigenvalue\_max\_iter .......... 100 [2025-01-07 12:53:26,859] [INFO] [config.py:1000:print] eigenvalue\_stability ......... 1e-06 [2025-01-07 12:53:26,859] [INFO] [config.py:1000:print] eigenvalue\_tol ............... 0.01 [2025-01-07 12:53:26,859] [INFO] [config.py:1000:print] eigenvalue\_verbose ........... False [2025-01-07 12:53:26,859] [INFO] [config.py:1000:print] elasticity\_enabled ........... False [2025-01-07 12:53:26,859] [INFO] [config.py:1000:print] flops\_profiler\_config ........ { "enabled": false, "recompute\_fwd\_factor": 0.0, "profile\_step": 1, "module\_depth": -1, "top\_modules": 1, "detailed": true, "output\_file": null } [2025-01-07 12:53:26,859] [INFO] [config.py:1000:print] fp16\_auto\_cast ............... None [2025-01-07 12:53:26,859] [INFO] [config.py:1000:print] fp16\_enabled ................. False [2025-01-07 12:53:26,859] [INFO] [config.py:1000:print] fp16\_master\_weights\_and\_gradients False [2025-01-07 12:53:26,859] [INFO] [config.py:1000:print] global\_rank .................. 0 [2025-01-07 12:53:26,859] [INFO] [config.py:1000:print] grad\_accum\_dtype ............. None [2025-01-07 12:53:26,859] [INFO] [config.py:1000:print] gradient\_accumulation\_steps .. 1 [2025-01-07 12:53:26,859] [INFO] [config.py:1000:print] gradient\_clipping ............ 0.0 [2025-01-07 12:53:26,859] [INFO] [config.py:1000:print] gradient\_predivide\_factor .... 1.0 [2025-01-07 12:53:26,859] [INFO] [config.py:1000:print] graph\_harvesting ............. False [2025-01-07 12:53:26,859] [INFO] [config.py:1000:print] hybrid\_engine ................ enabled=False max\_out\_tokens=512 inference\_tp\_size=1 release\_inference\_cache=False pin\_parameters=True tp\_gather\_partition\_size=8 [2025-01-07 12:53:26,859] [INFO] [config.py:1000:print] initial\_dynamic\_scale ........ 1 [2025-01-07 12:53:26,859] [INFO] [config.py:1000:print] load\_universal\_checkpoint .... False [2025-01-07 12:53:26,859] [INFO] [config.py:1000:print] loss\_scale ................... 1.0 [2025-01-07 12:53:26,859] [INFO] [config.py:1000:print] memory\_breakdown ............. False [2025-01-07 12:53:26,859] [INFO] [config.py:1000:print] mics\_hierarchial\_params\_gather False [2025-01-07 12:53:26,859] [INFO] [config.py:1000:print] mics\_shard\_size .............. -1 [2025-01-07 12:53:26,859] [INFO] [config.py:1000:print] monitor\_config ............... tensorboard=TensorBoardConfig(enabled=False, output\_path='', job\_name='DeepSpeedJobName') wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv\_monitor=CSVConfig(enabled=False, output\_path='', job\_name='DeepSpeedJobName') enabled=False [2025-01-07 12:53:26,859] [INFO] [config.py:1000:print] nebula\_config ................ { "enabled": false, "persistent\_storage\_path": null, "persistent\_time\_interval": 100, "num\_of\_version\_in\_retention": 2, "enable\_nebula\_load": true, "load\_path": null } [2025-01-07 12:53:26,859] [INFO] [config.py:1000:print] optimizer\_legacy\_fusion ...... False [2025-01-07 12:53:26,859] [INFO] [config.py:1000:print] optimizer\_name ............... None [2025-01-07 12:53:26,859] [INFO] [config.py:1000:print] optimizer\_params ............. None [2025-01-07 12:53:26,859] [INFO] [config.py:1000:print] pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed\_layers': False, 'activation\_checkpoint\_interval': 0, 'pipe\_partitioned': True, 'grad\_partitioned': True} [2025-01-07 12:53:26,859] [INFO] [config.py:1000:print] pld\_enabled .................. False [2025-01-07 12:53:26,859] [INFO] [config.py:1000:print] pld\_params ................... False [2025-01-07 12:53:26,859] [INFO] [config.py:1000:print] prescale\_gradients ........... False [2025-01-07 12:53:26,859] [INFO] [config.py:1000:print] scheduler\_name ............... None [2025-01-07 12:53:26,860] [INFO] [config.py:1000:print] scheduler\_params ............. None [2025-01-07 12:53:26,860] [INFO] [config.py:1000:print] seq\_parallel\_communication\_data\_type torch.float32 [2025-01-07 12:53:26,860] [INFO] [config.py:1000:print] sparse\_attention ............. None [2025-01-07 12:53:26,860] [INFO] [config.py:1000:print] sparse\_gradients\_enabled ..... False [2025-01-07 12:53:26,860] [INFO] [config.py:1000:print] steps\_per\_print .............. inf [2025-01-07 12:53:26,860] [INFO] [config.py:1000:print] train\_batch\_size ............. 24 [2025-01-07 12:53:26,860] [INFO] [config.py:1000:print] train\_micro\_batch\_size\_per\_gpu 24 [2025-01-07 12:53:26,860] [INFO] [config.py:1000:print] use\_data\_before\_expert\_parallel_ False [2025-01-07 12:53:26,860] [INFO] [config.py:1000:print] use\_node\_local\_storage ....... False [2025-01-07 12:53:26,860] [INFO] [config.py:1000:print] wall\_clock\_breakdown ......... False [2025-01-07 12:53:26,860] [INFO] [config.py:1000:print] weight\_quantization\_config ... None [2025-01-07 12:53:26,860] [INFO] [config.py:1000:print] world\_size ................... 1 [2025-01-07 12:53:26,860] [INFO] [config.py:1000:print] zero\_allow\_untested\_optimizer True [2025-01-07 12:53:26,860] [INFO] [config.py:1000:print] zero\_config .................. stage=2 contiguous\_gradients=True reduce\_scatter=True reduce\_bucket\_size=200000000 use\_multi\_rank\_bucket\_allreduce=True allgather\_partitions=True allgather\_bucket\_size=200000000 overlap\_comm=True load\_from\_fp32\_weights=True elastic\_checkpoint=False offload\_param=None offload\_optimizer=None sub\_group\_size=1000000000 cpu\_offload\_param=None cpu\_offload\_use\_pin\_memory=None cpu\_offload=None prefetch\_bucket\_size=50,000,000 param\_persistence\_threshold=100,000 model\_persistence\_threshold=sys.maxsize max\_live\_parameters=1,000,000,000 max\_reuse\_distance=1,000,000,000 gather\_16bit\_weights\_on\_model\_save=False stage3\_gather\_fp16\_weights\_on\_model\_save=False ignore\_unused\_parameters=True legacy\_stage1=False round\_robin\_gradients=False zero\_hpz\_partition\_size=1 zero\_quantized\_weights=False zero\_quantized\_nontrainable\_weights=False zero\_quantized\_gradients=False mics\_shard\_size=-1 mics\_hierarchical\_params\_gather=False memory\_efficient\_linear=True pipeline\_loading\_checkpoint=False override\_module\_apply=True [2025-01-07 12:53:26,860] [INFO] [config.py:1000:print] zero\_enabled ................. True [2025-01-07 12:53:26,860] [INFO] [config.py:1000:print] zero\_force\_ds\_cpu\_optimizer .. True [2025-01-07 12:53:26,860] [INFO] [config.py:1000:print] zero\_optimization\_stage ...... 2 [2025-01-07 12:53:26,860] [INFO] [config.py:986:print\_user\_config] json = { "bf16": { "enabled": true, "auto\_cast": true }, "zero\_optimization": { "stage": 2, "allgather\_partitions": true, "allgather\_bucket\_size": 2.000000e+08, "overlap\_comm": true, "reduce\_scatter": true, "reduce\_bucket\_size": 2.000000e+08, "contiguous\_gradients": true, "sub\_group\_size": 1.000000e+09 }, "gradient\_accumulation\_steps": 1, "train\_batch\_size": 24, "train\_micro\_batch\_size\_per\_gpu": 24, "steps\_per\_print": inf, "wall\_clock\_breakdown": false, "fp16": { "enabled": false }, "zero\_allow\_untested\_optimizer": true } 0it [00:01, ?it/s] Traceback (most recent call last): File "/content/drive/MyDrive/time-llm/Time-LLM/run\_main.py", line 211, in outputs = model(batch\_x, batch\_x\_mark, dec\_inp, batch\_y\_mark) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in \_wrapped\_call\_impl return self.\_call\_impl(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in \_call\_impl return forward\_call(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/deepspeed/utils/nvtx.py", line 15, in wrapped\_fn ret\_val = func(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/deepspeed/runtime/engine.py", line 1852, in forward loss = self.module(\*inputs, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in \_wrapped\_call\_impl return self.\_call\_impl(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in \_call\_impl return forward\_call(\*args, \*\*kwargs) File "/content/drive/MyDrive/time-llm/Time-LLM/models/TimeLLM.py", line 196, in forward dec\_out = self.forecast(x\_enc, x\_mark\_enc, x\_dec, x\_mark\_dec) File "/content/drive/MyDrive/time-llm/Time-LLM/models/TimeLLM.py", line 243, in forecast dec\_out = self.llm\_model(inputs\_embeds=llama\_enc\_out).last\_hidden\_state File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in \_wrapped\_call\_impl return self.\_call\_impl(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in \_call\_impl return forward\_call(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/transformers/models/llama/modeling\_llama.py", line 693, in forward layer\_outputs = decoder\_layer( File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in \_wrapped\_call\_impl return self.\_call\_impl(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in \_call\_impl return forward\_call(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/transformers/models/llama/modeling\_llama.py", line 408, in forward hidden\_states, self\_attn\_weights, present\_key\_value = self.self\_attn( File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in \_wrapped\_call\_impl return self.\_call\_impl(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in _call\_impl return forward\_call(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/transformers/models/llama/modeling\_llama.py", line 330, in forward attn\_weights = torch.matmul(query\_states, key\_states.transpose(2, 3)) / math.sqrt(self.head\_dim) torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 86.00 MiB. GPU 0 has a total capacity of 39.56 GiB of which 80.81 MiB is free. Process 165528 has 39.48 GiB memory in use. Of the allocated memory 38.64 GiB is allocated by PyTorch, and 13.69 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH\_CUDA\_ALLOC\_CONF=expandable\_segments:True to avoid fragmentation. See documentation for Memory Management ([CUDA semantics — PyTorch 2.6 documentation](https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)) Traceback (most recent call last): File "/usr/local/bin/accelerate", line 8, in sys.exit(main()) File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/accelerate\_cli.py", line 46, in main args.func(args) File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py", line 1057, in launch\_command simple\_launcher(args) File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py", line 673, in simple\_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['/usr/bin/python3', 'run\_main.py', '--task\_name', 'long\_term\_forecast', '--is\_training', '1', '--root\_path', './dataset/ETT-small/', '--data\_path', 'ETTh1.csv', '--model\_id', 'ETTh1\_512\_336', '--model', 'TimeLLM', '--data', 'ETTh1', '--features', 'M', '--seq\_len', '512', '--label\_len', '48', '--pred\_len', '336', '--factor', '3', '--enc\_in', '7', '--dec\_in', '7', '--c\_out', '7', '--des', 'Exp', '--itr', '1', '--d\_model', '32', '--d\_ff', '128', '--batch\_size', '24', '--lradj', 'COS', '--learning\_rate', '0.001', '--llm\_layers', '32', '--train\_epochs', '100', '--model\_comment', 'TimeLLM-ETTh1']' returned non-zero exit status 1. Loading checkpoint shards: 100% 2/2 [00:05\<00:00, 2.77s/it] [2025-01-07 12:54:37,812] [INFO] [real\_accelerator.py:191:get\_accelerator] Setting ds\_accelerator to cuda (auto detect) [2025-01-07 12:54:38,083] [INFO] [logging.py:96:log\_dist] [Rank -1] DeepSpeed info: version=0.14.0, git-hash=unknown, git-branch=unknown [2025-01-07 12:54:38,083] [INFO] [comm.py:637:init\_distributed] cdb=None [2025-01-07 12:54:38,083] [INFO] [comm.py:652:init\_distributed] Not using the DeepSpeed or dist launchers, attempting to detect MPI environment... [2025-01-07 12:54:38,959] [INFO] [comm.py:702:mpi\_discovery] Discovered MPI settings of world\_rank=0, local\_rank=0, world\_size=1, master\_addr=172.28.0.12, master\_port=29500 [2025-01-07 12:54:38,960] [INFO] [comm.py:668:init\_distributed] Initializing TorchBackend in DeepSpeed with backend nccl [2025-01-07 12:54:53,894] [INFO] [logging.py:96:log\_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False [2025-01-07 12:54:53,896] [INFO] [logging.py:96:log\_dist] [Rank 0] Using client Optimizer as basic optimizer [2025-01-07 12:54:53,896] [INFO] [logging.py:96:log\_dist] [Rank 0] Removing param\_group that has no 'params' in the basic Optimizer [2025-01-07 12:54:53,898] [INFO] [logging.py:96:log\_dist] [Rank 0] DeepSpeed Basic Optimizer = Adam [2025-01-07 12:54:53,898] [INFO] [utils.py:56:is\_zero\_supported\_optimizer] Checking ZeRO support for optimizer=Adam type=\ [2025-01-07 12:54:53,898] [INFO] [logging.py:96:log\_dist] [Rank 0] Creating torch.bfloat16 ZeRO stage 2 optimizer [2025-01-07 12:54:53,898] [INFO] [stage\_1\_and\_2.py:149: **init**] Reduce bucket size 200000000 [2025-01-07 12:54:53,898] [INFO] [stage\_1\_and\_2.py:150: **init**] Allgather bucket size 200000000 [2025-01-07 12:54:53,898] [INFO] [stage\_1\_and\_2.py:151: **init**] CPU Offload: False [2025-01-07 12:54:53,898] [INFO] [stage\_1\_and\_2.py:152: **init**] Round robin gradient partitioning: False [2025-01-07 12:54:54,277] [INFO] [utils.py:800:see\_memory\_usage] Before initializing optimizer states [2025-01-07 12:54:54,277] [INFO] [utils.py:801:see\_memory\_usage] MA 12.62 GB Max\_MA 12.72 GB CA 12.72 GB Max\_CA 13 GB [2025-01-07 12:54:54,278] [INFO] [utils.py:808:see\_memory\_usage] CPU Virtual Memory: used = 3.5 GB, percent = 4.2% [2025-01-07 12:54:54,445] [INFO] [utils.py:800:see\_memory\_usage] After initializing optimizer states [2025-01-07 12:54:54,446] [INFO] [utils.py:801:see\_memory\_usage] MA 12.62 GB Max\_MA 12.81 GB CA 12.91 GB Max\_CA 13 GB [2025-01-07 12:54:54,446] [INFO] [utils.py:808:see\_memory\_usage] CPU Virtual Memory: used = 3.5 GB, percent = 4.2% [2025-01-07 12:54:54,446] [INFO] [stage\_1\_and\_2.py:539: **init**] optimizer state initialized [2025-01-07 12:54:54,612] [INFO] [utils.py:800:see\_memory\_usage] After initializing ZeRO optimizer [2025-01-07 12:54:54,613] [INFO] [utils.py:801:see\_memory\_usage] MA 12.62 GB Max\_MA 12.62 GB CA 12.91 GB Max\_CA 13 GB [2025-01-07 12:54:54,613] [INFO] [utils.py:808:see\_memory\_usage] CPU Virtual Memory: used = 3.5 GB, percent = 4.2% [2025-01-07 12:54:54,614] [INFO] [logging.py:96:log\_dist] [Rank 0] DeepSpeed Final Optimizer = Adam [2025-01-07 12:54:54,614] [INFO] [logging.py:96:log\_dist] [Rank 0] DeepSpeed using client LR scheduler [2025-01-07 12:54:54,614] [INFO] [logging.py:96:log\_dist] [Rank 0] DeepSpeed LR Scheduler = None [2025-01-07 12:54:54,614] [INFO] [logging.py:96:log\_dist] [Rank 0] step=0, skipped=0, lr=[0.0003999999999999993], mom=[(0.95, 0.999)] [2025-01-07 12:54:54,615] [INFO] [config.py:996:print] DeepSpeedEngine configuration: [2025-01-07 12:54:54,616] [INFO] [config.py:1000:print] activation\_checkpointing\_config { "partition\_activations": false, "contiguous\_memory\_optimization": false, "cpu\_checkpointing": false, "number\_checkpoints": null, "synchronize\_checkpoint\_boundary": false, "profile": false } [2025-01-07 12:54:54,616] [INFO] [config.py:1000:print] aio\_config ................... {'block\_size': 1048576, 'queue\_depth': 8, 'thread\_count': 1, 'single\_submit': False, 'overlap\_events': True} [2025-01-07 12:54:54,616] [INFO] [config.py:1000:print] amp\_enabled .................. False [2025-01-07 12:54:54,616] [INFO] [config.py:1000:print] amp\_params ................... False [2025-01-07 12:54:54,616] [INFO] [config.py:1000:print] autotuning\_config ............ { "enabled": false, "start\_step": null, "end\_step": null, "metric\_path": null, "arg\_mappings": null, "metric": "throughput", "model\_info": null, "results\_dir": "autotuning\_results", "exps\_dir": "autotuning\_exps", "overwrite": true, "fast": true, "start\_profile\_step": 3, "end\_profile\_step": 5, "tuner\_type": "gridsearch", "tuner\_early\_stopping": 5, "tuner\_num\_trials": 50, "model\_info\_path": null, "mp\_size": 1, "max\_train\_batch\_size": null, "min\_train\_batch\_size": 1, "max\_train\_micro\_batch\_size\_per\_gpu": 1.024000e+03, "min\_train\_micro\_batch\_size\_per\_gpu": 1, "num\_tuning\_micro\_batch\_sizes": 3 } [2025-01-07 12:54:54,616] [INFO] [config.py:1000:print] bfloat16\_enabled ............. True [2025-01-07 12:54:54,616] [INFO] [config.py:1000:print] bfloat16\_immediate\_grad\_update False [2025-01-07 12:54:54,616] [INFO] [config.py:1000:print] checkpoint\_parallel\_write\_pipeline False [2025-01-07 12:54:54,616] [INFO] [config.py:1000:print] checkpoint\_tag\_validation\_enabled True [2025-01-07 12:54:54,616] [INFO] [config.py:1000:print] checkpoint\_tag\_validation\_fail False [2025-01-07 12:54:54,616] [INFO] [config.py:1000:print] comms\_config ................. \ [2025-01-07 12:54:54,616] [INFO] [config.py:1000:print] communication\_data\_type ...... None [2025-01-07 12:54:54,616] [INFO] [config.py:1000:print] compile\_config ............... enabled=False backend='inductor' kwargs={} [2025-01-07 12:54:54,616] [INFO] [config.py:1000:print] compression\_config ........... {'weight\_quantization': {'shared\_parameters': {'enabled': False, 'quantizer\_kernel': False, 'schedule\_offset': 0, 'quantize\_groups': 1, 'quantize\_verbose': False, 'quantization\_type': 'symmetric', 'quantize\_weight\_in\_forward': False, 'rounding': 'nearest', 'fp16\_mixed\_quantize': False, 'quantize\_change\_ratio': 0.001}, 'different\_groups': {}}, 'activation\_quantization': {'shared\_parameters': {'enabled': False, 'quantization\_type': 'symmetric', 'range\_calibration': 'dynamic', 'schedule\_offset': 1000}, 'different\_groups': {}}, 'sparse\_pruning': {'shared\_parameters': {'enabled': False, 'method': 'l1', 'schedule\_offset': 1000}, 'different\_groups': {}}, 'row\_pruning': {'shared\_parameters': {'enabled': False, 'method': 'l1', 'schedule\_offset': 1000}, 'different\_groups': {}}, 'head\_pruning': {'shared\_parameters': {'enabled': False, 'method': 'topk', 'schedule\_offset': 1000}, 'different\_groups': {}}, 'channel\_pruning': {'shared\_parameters': {'enabled': False, 'method': 'l1', 'schedule\_offset': 1000}, 'different\_groups': {}}, 'layer\_reduction': {'enabled': False}} [2025-01-07 12:54:54,616] [INFO] [config.py:1000:print] curriculum\_enabled\_legacy .... False [2025-01-07 12:54:54,616] [INFO] [config.py:1000:print] curriculum\_params\_legacy ..... False [2025-01-07 12:54:54,616] [INFO] [config.py:1000:print] data\_efficiency\_config ....... {'enabled': False, 'seed': 1234, 'data\_sampling': {'enabled': False, 'num\_epochs': 1000, 'num\_workers': 0, 'curriculum\_learning': {'enabled': False}}, 'data\_routing': {'enabled': False, 'random\_ltd': {'enabled': False, 'layer\_token\_lr\_schedule': {'enabled': False}}}} [2025-01-07 12:54:54,616] [INFO] [config.py:1000:print] data\_efficiency\_enabled ...... False [2025-01-07 12:54:54,616] [INFO] [config.py:1000:print] dataloader\_drop\_last ......... False [2025-01-07 12:54:54,616] [INFO] [config.py:1000:print] disable\_allgather ............ False [2025-01-07 12:54:54,616] [INFO] [config.py:1000:print] dump\_state ................... False [2025-01-07 12:54:54,617] [INFO] [config.py:1000:print] dynamic\_loss\_scale\_args ...... None [2025-01-07 12:54:54,617] [INFO] [config.py:1000:print] eigenvalue\_enabled ........... False [2025-01-07 12:54:54,617] [INFO] [config.py:1000:print] eigenvalue\_gas\_boundary\_resolution 1 [2025-01-07 12:54:54,617] [INFO] [config.py:1000:print] eigenvalue\_layer\_name ........ bert.encoder.layer [2025-01-07 12:54:54,617] [INFO] [config.py:1000:print] eigenvalue\_layer\_num ......... 0 [2025-01-07 12:54:54,617] [INFO] [config.py:1000:print] eigenvalue\_max\_iter .......... 100 [2025-01-07 12:54:54,617] [INFO] [config.py:1000:print] eigenvalue\_stability ......... 1e-06 [2025-01-07 12:54:54,617] [INFO] [config.py:1000:print] eigenvalue\_tol ............... 0.01 [2025-01-07 12:54:54,617] [INFO] [config.py:1000:print] eigenvalue\_verbose ........... False [2025-01-07 12:54:54,617] [INFO] [config.py:1000:print] elasticity\_enabled ........... False [2025-01-07 12:54:54,617] [INFO] [config.py:1000:print] flops\_profiler\_config ........ { "enabled": false, "recompute\_fwd\_factor": 0.0, "profile\_step": 1, "module\_depth": -1, "top\_modules": 1, "detailed": true, "output\_file": null } [2025-01-07 12:54:54,617] [INFO] [config.py:1000:print] fp16\_auto\_cast ............... None [2025-01-07 12:54:54,617] [INFO] [config.py:1000:print] fp16\_enabled ................. False [2025-01-07 12:54:54,617] [INFO] [config.py:1000:print] fp16\_master\_weights\_and\_gradients False [2025-01-07 12:54:54,617] [INFO] [config.py:1000:print] global\_rank .................. 0 [2025-01-07 12:54:54,617] [INFO] [config.py:1000:print] grad\_accum\_dtype ............. None [2025-01-07 12:54:54,617] [INFO] [config.py:1000:print] gradient\_accumulation\_steps .. 1 [2025-01-07 12:54:54,617] [INFO] [config.py:1000:print] gradient\_clipping ............ 0.0 [2025-01-07 12:54:54,617] [INFO] [config.py:1000:print] gradient\_predivide\_factor .... 1.0 [2025-01-07 12:54:54,617] [INFO] [config.py:1000:print] graph\_harvesting ............. False [2025-01-07 12:54:54,617] [INFO] [config.py:1000:print] hybrid\_engine ................ enabled=False max\_out\_tokens=512 inference\_tp\_size=1 release\_inference\_cache=False pin\_parameters=True tp\_gather\_partition\_size=8 [2025-01-07 12:54:54,617] [INFO] [config.py:1000:print] initial\_dynamic\_scale ........ 1 [2025-01-07 12:54:54,617] [INFO] [config.py:1000:print] load\_universal\_checkpoint .... False [2025-01-07 12:54:54,617] [INFO] [config.py:1000:print] loss\_scale ................... 1.0 [2025-01-07 12:54:54,617] [INFO] [config.py:1000:print] memory\_breakdown ............. False [2025-01-07 12:54:54,617] [INFO] [config.py:1000:print] mics\_hierarchial\_params\_gather False [2025-01-07 12:54:54,617] [INFO] [config.py:1000:print] mics\_shard\_size .............. -1 [2025-01-07 12:54:54,617] [INFO] [config.py:1000:print] monitor\_config ............... tensorboard=TensorBoardConfig(enabled=False, output\_path='', job\_name='DeepSpeedJobName') wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv\_monitor=CSVConfig(enabled=False, output\_path='', job\_name='DeepSpeedJobName') enabled=False [2025-01-07 12:54:54,617] [INFO] [config.py:1000:print] nebula\_config ................ { "enabled": false, "persistent\_storage\_path": null, "persistent\_time\_interval": 100, "num\_of\_version\_in\_retention": 2, "enable\_nebula\_load": true, "load\_path": null } [2025-01-07 12:54:54,617] [INFO] [config.py:1000:print] optimizer\_legacy\_fusion ...... False [2025-01-07 12:54:54,617] [INFO] [config.py:1000:print] optimizer\_name ............... None [2025-01-07 12:54:54,617] [INFO] [config.py:1000:print] optimizer\_params ............. None [2025-01-07 12:54:54,618] [INFO] [config.py:1000:print] pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed\_layers': False, 'activation\_checkpoint\_interval': 0, 'pipe\_partitioned': True, 'grad\_partitioned': True} [2025-01-07 12:54:54,618] [INFO] [config.py:1000:print] pld\_enabled .................. False [2025-01-07 12:54:54,618] [INFO] [config.py:1000:print] pld\_params ................... False [2025-01-07 12:54:54,618] [INFO] [config.py:1000:print] prescale\_gradients ........... False [2025-01-07 12:54:54,618] [INFO] [config.py:1000:print] scheduler\_name ............... None [2025-01-07 12:54:54,618] [INFO] [config.py:1000:print] scheduler\_params ............. None [2025-01-07 12:54:54,618] [INFO] [config.py:1000:print] seq\_parallel\_communication\_data\_type torch.float32 [2025-01-07 12:54:54,618] [INFO] [config.py:1000:print] sparse\_attention ............. None [2025-01-07 12:54:54,618] [INFO] [config.py:1000:print] sparse\_gradients\_enabled ..... False [2025-01-07 12:54:54,618] [INFO] [config.py:1000:print] steps\_per\_print .............. inf [2025-01-07 12:54:54,618] [INFO] [config.py:1000:print] train\_batch\_size ............. 24 [2025-01-07 12:54:54,618] [INFO] [config.py:1000:print] train\_micro\_batch\_size\_per\_gpu 24 [2025-01-07 12:54:54,618] [INFO] [config.py:1000:print] use\_data\_before\_expert\_parallel_ False [2025-01-07 12:54:54,618] [INFO] [config.py:1000:print] use\_node\_local\_storage ....... False [2025-01-07 12:54:54,618] [INFO] [config.py:1000:print] wall\_clock\_breakdown ......... False [2025-01-07 12:54:54,618] [INFO] [config.py:1000:print] weight\_quantization\_config ... None [2025-01-07 12:54:54,618] [INFO] [config.py:1000:print] world\_size ................... 1 [2025-01-07 12:54:54,618] [INFO] [config.py:1000:print] zero\_allow\_untested\_optimizer True [2025-01-07 12:54:54,618] [INFO] [config.py:1000:print] zero\_config .................. stage=2 contiguous\_gradients=True reduce\_scatter=True reduce\_bucket\_size=200000000 use\_multi\_rank\_bucket\_allreduce=True allgather\_partitions=True allgather\_bucket\_size=200000000 overlap\_comm=True load\_from\_fp32\_weights=True elastic\_checkpoint=False offload\_param=None offload\_optimizer=None sub\_group\_size=1000000000 cpu\_offload\_param=None cpu\_offload\_use\_pin\_memory=None cpu\_offload=None prefetch\_bucket\_size=50,000,000 param\_persistence\_threshold=100,000 model\_persistence\_threshold=sys.maxsize max\_live\_parameters=1,000,000,000 max\_reuse\_distance=1,000,000,000 gather\_16bit\_weights\_on\_model\_save=False stage3\_gather\_fp16\_weights\_on\_model\_save=False ignore\_unused\_parameters=True legacy\_stage1=False round\_robin\_gradients=False zero\_hpz\_partition\_size=1 zero\_quantized\_weights=False zero\_quantized\_nontrainable\_weights=False zero\_quantized\_gradients=False mics\_shard\_size=-1 mics\_hierarchical\_params\_gather=False memory\_efficient\_linear=True pipeline\_loading\_checkpoint=False override\_module\_apply=True [2025-01-07 12:54:54,618] [INFO] [config.py:1000:print] zero\_enabled ................. True [2025-01-07 12:54:54,618] [INFO] [config.py:1000:print] zero\_force\_ds\_cpu\_optimizer .. True [2025-01-07 12:54:54,618] [INFO] [config.py:1000:print] zero\_optimization\_stage ...... 2 [2025-01-07 12:54:54,618] [INFO] [config.py:986:print\_user\_config] json = { "bf16": { "enabled": true, "auto\_cast": true }, "zero\_optimization": { "stage": 2, "allgather\_partitions": true, "allgather\_bucket\_size": 2.000000e+08, "overlap\_comm": true, "reduce\_scatter": true, "reduce\_bucket\_size": 2.000000e+08, "contiguous\_gradients": true, "sub\_group\_size": 1.000000e+09 }, "gradient\_accumulation\_steps": 1, "train\_batch\_size": 24, "train\_micro\_batch\_size\_per\_gpu": 24, "steps\_per\_print": inf, "wall\_clock\_breakdown": false, "fp16": { "enabled": false }, "zero\_allow\_untested\_optimizer": true } 0it [00:01, ?it/s] Traceback (most recent call last): File "/content/drive/MyDrive/time-llm/Time-LLM/run\_main.py", line 211, in outputs = model(batch\_x, batch\_x\_mark, dec\_inp, batch\_y\_mark) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in \_wrapped\_call\_impl return self.\_call\_impl(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in \_call\_impl return forward\_call(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/deepspeed/utils/nvtx.py", line 15, in wrapped\_fn ret\_val = func(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/deepspeed/runtime/engine.py", line 1852, in forward loss = self.module(\*inputs, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in \_wrapped\_call\_impl return self.\_call\_impl(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in \_call\_impl return forward\_call(\*args, \*\*kwargs) File "/content/drive/MyDrive/time-llm/Time-LLM/models/TimeLLM.py", line 196, in forward dec\_out = self.forecast(x\_enc, x\_mark\_enc, x\_dec, x\_mark\_dec) File "/content/drive/MyDrive/time-llm/Time-LLM/models/TimeLLM.py", line 243, in forecast dec\_out = self.llm\_model(inputs\_embeds=llama\_enc\_out).last\_hidden\_state File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in \_wrapped\_call\_impl return self.\_call\_impl(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in \_call\_impl return forward\_call(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/transformers/models/llama/modeling\_llama.py", line 693, in forward layer\_outputs = decoder\_layer( File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in \_wrapped\_call\_impl return self.\_call\_impl(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in \_call\_impl return forward\_call(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/transformers/models/llama/modeling\_llama.py", line 408, in forward hidden\_states, self\_attn\_weights, present\_key\_value = self.self\_attn( File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1511, in \_wrapped\_call\_impl return self.\_call\_impl(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1520, in \_call\_impl return forward\_call(\*args, \*\*kwargs) File "/usr/local/lib/python3.10/dist-packages/transformers/models/llama/modeling\_llama.py", line 330, in forward attn\_weights = torch.matmul(query\_states, key\_states.transpose(2, 3)) / math.sqrt(self.head\_dim) torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 86.00 MiB. GPU 0 has a total capacity of 39.56 GiB of which 56.81 MiB is free. Process 171661 has 39.50 GiB memory in use. Of the allocated memory 38.66 GiB is allocated by PyTorch, and 13.13 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH\_CUDA\_ALLOC\_CONF=expandable\_segments:True to avoid fragmentation. See documentation for Memory Management ([CUDA semantics — PyTorch 2.6 documentation](https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)) Traceback (most recent call last): File "/usr/local/bin/accelerate", line 8, in sys.exit(main()) File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/accelerate\_cli.py", line 46, in main args.func(args) File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py", line 1057, in launch\_command simple\_launcher(args) File "/usr/local/lib/python3.10/dist-packages/accelerate/commands/launch.py", line 673, in simple\_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['/usr/bin/python3', 'run\_main.py', '--task\_name', 'long\_term\_forecast', '--is\_training', '1', '--root\_path', './dataset/ETT-small/', '--data\_path', 'ETTh1.csv', '--model\_id', 'ETTh1\_512\_720', '--model', 'TimeLLM', '--data', 'ETTh1', '--features', 'M', '--seq\_len', '512', '--label\_len', '48', '--pred\_len', '720', '--factor', '3', '--enc\_in', '7', '--dec\_in', '7', '--c\_out', '7', '--des', 'Exp', '--itr', '1', '--d\_model', '32', '--d\_ff', '128', '--batch\_size', '24', '--learning\_rate', '0.01', '--llm\_layers', '32', '--train\_epochs', '100', '--model\_comment', 'TimeLLM-ETTh1']' returned non-zero exit status 1. 이렇게 뜨고 알아본 결과로 메모리 문제라고 하는데 코랩 상에서 메모리에는 여유가 있고 심지어 GPU RAM은 계속 0/40GB 상태입니다. 런타임 유형 변경에서 A100 설정한 상황입니다. 도와주세요 ㅜㅜ ---
**Author:** ![bot](https://discuss.pytorch.kr/user_avatar/discuss.pytorch.kr/bot/32/558_2.png) [@bot](https://discuss.pytorch.kr/u/bot)\ **Post date:** [1월 7, 2025, 1:09오후 UTC](https://discuss.pytorch.kr/t/gpu-ram-0/5813/2 "2025-01-07T13:09:37Z")
- **아래 답변은 Perplexity로 자동 생성한 것으로, 정확하지 않을 수 있습니다.** - **잘못된 답변을 보셨다면 댓글로 첨삭 지도 부탁드립니다.** 😅 * * * ### 문제 이해 당신의 문제는 Google Colab에서 GPU 메모리가 부족하여 `CUDA out of memory` 오류가 발생하는 것입니다. 특히, GPU RAM이 0/40GB 상태로 표시되면서도 메모리 부족 오류가 발생하고 있습니다. ### 해결 방법 以下은 몇 가지 해결 방법을 제시합니다. #### 1. 배치 사이즈 줄이기 대부분의 경우, 배치 사이즈를 줄이는 것이 가장 간단하고 효과적인 해결 방법입니다. 배치 사이즈를 줄이면 각 배치에서 사용되는 GPU 메모리가 줄어들기 때문입니다. ```bash batch_size=24 # 현재 설정된 배치 사이즈 batch_size=16 # 또는 더 작은 값으로 줄이기 ``` #### 2. 변수 제거와 캐시 비우기 사용하지 않는 변수들을 제거하고, CUDA 캐시를 비우는 방법도 있습니다. ```python import torch import gc # 사용하지 않는 변수 제거 del a # CUDA 캐시 비우기 torch.cuda.empty_cache() # 가비지 컬렉터 실행 gc.collect() ``` #### 3. 자동 미분 끄기 자동 미분을 끄는 방법도 있습니다. 이는 특정 연산에서 메모리 사용량을 줄일 수 있습니다. ```python with torch.no_grad(): # 코드 여기서 실행 ``` #### 4. 모델 크기 줄이기 모델의 크기를 줄이는 방법도 있습니다. 예를 들어, 레이어의 수나 파라미터의 수를 줄일 수 있습니다. #### 5. 환경 재시작 Colab 환경에서 커널을 재시작하는 방법도 있습니다. 이는 모든 변수와 캐시를 초기화하여 메모리를 해제할 수 있습니다. ```python # Colab에서 커널 재시작 kill -9 -1 ``` #### 6. PYTORCH\_CUDA\_ALLOC\_CONF 설정 PyTorch의 CUDA 할당 설정을 변경하여 메모리 할당 문제를 해결할 수 있습니다. ```python import os os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "max_split_size_mb:128" ``` ### 예시 코드 아래는 위의 방법들을 적용한 예시 코드입니다. ```python import torch import gc import os # 배치 사이즈 줄이기 batch_size = 16 # 모델과 데이터 로드 후 사용하지 않는 변수 제거 del unused_variable # CUDA 캐시 비우기 torch.cuda.empty_cache() # 가비지 컬렉터 실행 gc.collect() # 자동 미분 끄기 with torch.no_grad(): # 코드 여기서 실행 outputs = model(batch_x, batch_x_mark, dec_inp, batch_y_mark) # PYTORCH_CUDA_ALLOC_CONF 설정 os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "expandable_segments:True" ``` ### 추가 참고 자료 - [PyTorch CUDA Memory Management](https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) - [Joonas' Note: PyTorch GPU 메모리가 부족할 때 확인할 내용들](https://blog.joonas.io/208) 이 방법들을 시도해 보시고, 여전히 문제가 발생한다면 모델 구조나 데이터 처리 방식을 더 효율적으로 개선해야 할 수 있습니다. * * * - _ **이 답변이 도움이 되셨나요? 괜찮았다고 생각하신다면 아래 👍 아이콘을 눌러주세요!** _ ---
**Author:** ![thdrlals135](https://discuss.pytorch.kr/user_avatar/discuss.pytorch.kr/thdrlals135/32/9450_2.png) [@thdrlals135](https://discuss.pytorch.kr/u/thdrlals135)\ **Post date:** [1월 7, 2025, 1:11오후 UTC](https://discuss.pytorch.kr/t/gpu-ram-0/5813/3 "2025-01-07T13:11:16Z")
메모리가 부족한 것이 아니라 GPU의 메모리가 0/40인 것으로 보아 코랩 상에서 GPU를 쓰지 못하고 있는 것 같습니다. ---
**Author:** ![bot](https://discuss.pytorch.kr/user_avatar/discuss.pytorch.kr/bot/32/558_2.png) [@bot](https://discuss.pytorch.kr/u/bot)\ **Post date:** [1월 7, 2025, 1:13오후 UTC](https://discuss.pytorch.kr/t/gpu-ram-0/5813/4 "2025-01-07T13:13:16Z")
- _ **아래 답변은 OpenAI의 GPT-4o 모델로 자동 생성한 것으로, 정확하지 않을 수 있습니다.** _ - _ **잘못된 답변을 보셨다면 댓글로 첨삭 지도 부탁드립니다. 😅** _ * * * 안녕하세요! 코랩에서 GPU 메모리 문제로 어려움을 겪고 계신 것 같네요. GPU 메모리가 0으로 표시되면서 `CUDA out of memory` 오류가 발생하는 상황이군요. 이 문제는 GPU 메모리가 실제로 부족한 경우뿐만 아니라, PyTorch의 메모리 관리 문제로 인해 발생할 수 있습니다. ### 문제 해결 방법 1. **메모리 초기화** : PyTorch는 GPU 메모리를 자동으로 관리하지만, 가끔 메모리가 제대로 해제되지 않을 수 있습니다. 이를 해결하기 위해, 스크립트 실행 전에 `torch.cuda.empty_cache()`를 호출하여 캐시된 메모리를 해제해보세요. 2. **배치 크기 줄이기** : 배치 크기를 줄이면 메모리 사용량을 줄일 수 있습니다. `batch_size`를 줄여서 실행해보세요. 3. **모델 크기 줄이기** : 모델의 파라미터 수를 줄이는 것도 메모리 사용량을 줄이는 방법입니다. `d_model`, `d_ff`, `llama_layers` 등의 값을 줄여보세요. 4. **Mixed Precision 사용** : 이미 `--mixed_precision bf16` 옵션을 사용하고 계신데, 이 옵션이 제대로 작동하는지 확인해보세요. `bf16` 대신 `fp16`을 시도해볼 수도 있습니다. 5. **환경 변수 설정** : PyTorch의 메모리 할당 방식을 조정하기 위해 환경 변수를 설정할 수 있습니다. `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`를 설정하여 메모리 조각화를 줄일 수 있습니다. 6. **런타임 재시작** : 코랩 런타임을 재시작하여 모든 메모리를 초기화하는 것도 도움이 될 수 있습니다. 7. **GPU 사용 확인** : `nvidia-smi` 명령어를 사용하여 실제 GPU 메모리 사용량을 확인해보세요. 코랩 셀에서 다음 명령어를 실행할 수 있습니다. 이 방법들을 시도해보시고, 문제가 지속된다면 다른 설정이나 코드 부분에서 문제가 있을 수 있으니, 추가적인 정보를 제공해주시면 더 도와드릴 수 있을 것 같습니다. 더 많은 정보는 PyTorch의 [CUDA 메모리 관리 문서](https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)를 참고하실 수 있습니다. 문제가 해결되길 바라며, 제 답변이 틀릴 수 있으니 다른 방법도 고려해보세요. 도움이 필요하시면 언제든지 질문해 주세요! * * * - _ **이 답변이 도움이 되셨나요? 괜찮았다고 생각하신다면 아래 ❤를 한 번 눌러주시길 부탁드려요!** _ ---
**Author:** ![bot](https://discuss.pytorch.kr/user_avatar/discuss.pytorch.kr/bot/32/558_2.png) [@bot](https://discuss.pytorch.kr/u/bot)\ **Post date:** [1월 7, 2025, 1:31오후 UTC](https://discuss.pytorch.kr/t/gpu-ram-0/5813/5 "2025-01-07T13:31:32Z")
- _ **아래 답변은 Anthropic의 Claude-3 Opus 모델로 자동 생성한 것으로, 정확하지 않을 수 있습니다.** _ - _ **잘못된 답변을 보셨다면 댓글로 첨삭 지도 부탁드립니다. 😅** _ * * * 안녕하세요 thdrlals135님, GPU RAM 사용률이 계속 0으로 나오는 문제에 대해 글을 남겨주셨네요. 코드 실행시 발생하는 CUDA out of memory 에러를 해결하기 위해 노력해보겠습니다. 질문 내용을 정리해보면: - Colab에서 GPU 환경(A100)으로 설정했음에도 불구하고 GPU RAM 사용량이 계속 0으로 표시됨 - TimeLLM 모델 학습 코드 실행 중 CUDA out of memory 에러 발생 - Colab에서 메모리 여유는 있는데 GPU 메모리를 사용하지 못하는 상황 에러 메시지를 보면 GPU 메모리가 부족해서 할당하는데 실패한 것으로 보입니다. "CUDA out of memory. Tried to allocate 86.00 MiB. GPU 0 has a total capacity of 39.56 GiB of which 56.81 MiB is free." 39.56GB 중 56.81MB만 남은 상태에서 추가로 86MB를 더 할당하려다 실패한 것 같네요. GPU 모니터링 값이 0으로 나오는 것은 Colab UI와 실제 GPU 사용량 간에 갭이 있어서 그런 것 같습니다. 해결 방안은 다음과 같이 제안드려 볼게요: 1. 배치 크기를 줄여서 한번에 처리하는 데이터 양을 줄이는 방법이 있습니다. batch\_size를 현재 24에서 16이나 8 정도로 줄여서 실행해보시길 추천드립니다. "batch\_size=24" 부분을 수정하시면 됩니다. 2. 모델 크기를 작게 만드는 것도 도움이 될 수 있습니다. d\_model, d\_ff, llm\_layers 등 모델 사이즈와 관련된 인자값들을 조정해보세요. 메모리 사용량을 줄일 수 있을 것 같습니다. 3. Colab 외에 다른 GPU 환경에서 시도해보는 것도 좋겠네요. 로컬 머신의 GPU를 이용하거나, 클라우드 플랫폼 중에서는 AWS EC2의 p3.2xlarge 등 고성능 GPU가 장착된 인스턴스를 활용해볼 수 있습니다. 4. PyTorch에서 제공하는 amp를 활용해 FP16 mixed precision으로 학습 시 메모리 사용량을 줄일 수 있습니다. [Automatic Mixed Precision — 파이토치 한국어 튜토리얼 (PyTorch tutorials in Korean)](https://tutorials.pytorch.kr/recipes/recipes/amp_recipe.html) 마지막으로 제 답변이 잘못되었거나 부족할 수 있다는 점 양해 부탁드립니다. 더 자세한 내용은 PyTorch 공식 문서의 CUDA 관련 설명을 참고해보시길 추천드려요. - 영어: [CUDA semantics — PyTorch 2.6 documentation](https://pytorch.org/docs/stable/notes/cuda.html#memory-management) - 한국어: [선택 사항: 데이터 병렬 처리 (Data Parallelism) — 파이토치 한국어 튜토리얼 (PyTorch tutorials in Korean)](https://tutorials.pytorch.kr/beginner/blitz/data_parallel_tutorial.html) 긴 답변 읽어주셔서 감사하고, 도움이 되었기를 바랍니다! thdrlals135님의 문제가 빠르게 해결되길 기원하겠습니다. * * * - _ **답변이 어떠셨나요? 괜찮았다고 생각하신다면 아래 ❤를 한 번 눌러주시길 부탁드려요!** _