CUDA Error

[Predrag Punosevac]

Vincent Jeanselme <vjeansel at andrew.cmu.edu> wrote:

> Good Morning,

Lets try users at autonlab.org


Predrag

> 
> Since the change of the hard drive, I have the following error when I 
> run it on the GPUs (I have reinstalled pytorch but does not solve my 
> problem). I think that the problem comes from the Cuda library.
> 
>     THCudaCheck FAIL
>     file=/opt/conda/conda-bld/pytorch_1524577177097/work/aten/src/THC/THCTensorRandom.cu
>     line=25 error=30 : unknown error
>     Traceback (most recent call last):
>      ?? File "./train.py", line 519, in <module>
>      ?????? main(args)
>      ?? File "./train.py", line 61, in main
>      ?????? model = nn.DataParallel(model).cuda()
>      ?? File
>     "/zfsauton/home/vjeanselme/anaconda3/envs/lstmpy27/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py",
>     line 102, in __init__
>      ?????? _check_balance(self.device_ids)
>      ?? File
>     "/zfsauton/home/vjeanselme/anaconda3/envs/lstmpy27/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py",
>     line 17, in _check_balance
>      ?????? dev_props = [torch.cuda.get_device_properties(i) for i in
>     device_ids]
>      ?? File
>     "/zfsauton/home/vjeanselme/anaconda3/envs/lstmpy27/lib/python2.7/site-packages/torch/cuda/__init__.py",
>     line 290, in get_device_properties
>      ?????? init()?? # will define _get_device_properties and
>     _CudaDeviceProperties
>      ?? File
>     "/zfsauton/home/vjeanselme/anaconda3/envs/lstmpy27/lib/python2.7/site-packages/torch/cuda/__init__.py",
>     line 143, in init
>      ?????? _lazy_init()
>      ?? File
>     "/zfsauton/home/vjeanselme/anaconda3/envs/lstmpy27/lib/python2.7/site-packages/torch/cuda/__init__.py",
>     line 161, in _lazy_init
>      ?????? torch._C._cuda_init()
>     RuntimeError: cuda runtime error (30) : unknown error at
>     /opt/conda/conda-bld/pytorch_1524577177097/work/aten/src/THC/THCTensorRandom.cu:25
> 
> I don't know how to fix it, would you have any suggestions ?
> 
> Thank you,
> 
> -- 
> Vincent Jeanselme
> -----------------
> Analyst Researcher
> Auton Lab - Robotics Institute
> Carnegie Mellon University
>