안녕하세요.
ResNet50을 사용하여 ImageNet을 학습시키다가 오류가 떠서 질문드립니다.
우선 A100 8장을 이용하여 ResNet50을 학습시켰구요. 방식은 DDP로 사용하였습니다.
그 다음 A100 8장을 이용하여 Linear evaluation을 하려했는데 오류가 뜨더군요. 방식은 DP방식이었습니다.
제가 짠 코드가 아니라 다른 분이 짠 깃헙의 코드였는데 linear evaluation에서 DDP는 구현이 안되어있더군요.
오류 내용은 다음과 같습니다.
Traceback (most recent call last):
File “train.py”, line 253, in
main()
File “train.py”, line 103, in main
main_worker(args.gpu, -1, args)
File “train.py”, line 137, in main_worker
model = models.REGISTERED_MODELS[args.problem](args, device=device)
File “/scratch/a1613a01/simclr-pytorch-master/models/ssl.py”, line 364, in init
hdim = self.encode(torch.ones(10, 3, 224, 224).to(device)).shape[1]
File “/scratch/a1613a01/simclr-pytorch-master/models/ssl.py”, line 379, in encode
return self.encoder.model(x, out=‘h’)
File “/home01/a1613a01/.conda/envs/torch/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 1130, in _call_impl
return forward_call(*input, **kwargs)
File “/home01/a1613a01/.conda/envs/torch/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py”, line 168, in forward
outputs = self.parallel_apply(replicas, inputs, kwargs)
File “/home01/a1613a01/.conda/envs/torch/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py”, line 178, in parallel_apply
return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
File “/home01/a1613a01/.conda/envs/torch/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py”, line 86, in parallel_apply
output.reraise()
File “/home01/a1613a01/.conda/envs/torch/lib/python3.8/site-packages/torch/_utils.py”, line 461, in reraise
raise exception
TypeError: Caught TypeError in replica 5 on device 5.
Original Traceback (most recent call last):
File “/home01/a1613a01/.conda/envs/torch/lib/python3.8/site-packages/torch/nn/parallel/parallel_apply.py”, line 61, in _worker
output = module(*input, **kwargs)
File “/home01/a1613a01/.conda/envs/torch/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 1130, in _call_impl
return forward_call(*input, **kwargs)
TypeError: forward() missing 1 required positional argument: ‘x’
srun: error: gpu41: task 0: Exited with exit code 1
인터넷 검색하다보니 GPU의 개수를 줄이면 된다던데 GPU를 4장으로 줄이면 돌아가긴하나
근본적인 이유를 알고싶습니다.
혹시 이러한 오류 해결방법을 아시면 조언 좀 해주시면 감사하겠습니다.
