<div dir="ltr">I think this is an issue with the CUDA install. I'm unable to run Tensorflow jobs on GPU9 as of last night (have not checked the others, but I suspect similar).<div><br></div><div><div>2018-03-26 14:54:49.214493: E tensorflow/stream_executor/cuda/cuda_driver.cc:406] failed call to cuInit: CUDA_ERROR_UNKNOWN</div><div>2018-03-26 14:54:49.214599: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:158] retrieving CUDA diagnostic information for host: <a href="http://gpu9.int.autonlab.org">gpu9.int.autonlab.org</a></div><div>2018-03-26 14:54:49.214617: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:165] hostname: <a href="http://gpu9.int.autonlab.org">gpu9.int.autonlab.org</a></div><div>2018-03-26 14:54:49.214685: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:189] libcuda reported version is: 390.30.0</div><div>2018-03-26 14:54:49.214747: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:193] kernel reported version is: 390.30.0</div><div>2018-03-26 14:54:49.214762: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:300] kernel version seems to match DSO: 390.30.0</div></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Mar 27, 2018 at 1:47 AM Manzil Zaheer <<a href="mailto:manzil@cmu.edu">manzil@cmu.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Predrag,<br>
<br>
Thanks for pointing out the links. From the link you provided, we can see that FB engineers mention that "error 30 is usually unrelated to pytorch issues (or your code change)".<br>
<br>
Thanks,<br>
Manzil<br>
________________________________________<br>
From: Predrag Punosevac <<a href="mailto:predragp@andrew.cmu.edu" target="_blank">predragp@andrew.cmu.edu</a>><br>
Sent: 27 March 2018 01:31<br>
To: Manzil Zaheer<br>
Cc: Barnabas Poczos; <a href="mailto:users@autonlab.org" target="_blank">users@autonlab.org</a><br>
Subject: Re: PyTorch<br>
<br>
Manzil Zaheer <<a href="mailto:manzil@cmu.edu" target="_blank">manzil@cmu.edu</a>> wrote:<br>
<br>
> Hi Pregrad,<br>
><br>
> Thanks again for your help. But I still can not get anything running on GPU5,6,7,9. Also notice that GPU1,2,3,4,8 almost all GPUs are full, while no one is using GPU5,6,7,9. This might mean no one else is also able to run anything as well.<br>
><br>
<br>
7 if off limit used for the special project. How did you figure out that nobody is using it when<br>
you can't even log there?<br>
<br>
> So I tried many things. Everything installs without issue. But when i try to run the simple code like:<br>
><br>
<br>
PyTorch is a research grade software. They have a mailing list. 3 sec Googling reveals<br>
<br>
<br>
<a href="https://github.com/pytorch/pytorch/issues/2527" rel="noreferrer" target="_blank">https://github.com/pytorch/pytorch/issues/2527</a><br>
<br>
also<br>
<br>
<a href="https://stackoverflow.com/questions/45861767/pytorch-giving-cuda-runtime-error" rel="noreferrer" target="_blank">https://stackoverflow.com/questions/45861767/pytorch-giving-cuda-runtime-error</a><br>
<br>
I will look at this more but it would be helpful if you get on PyTorch mailing list and ask<br>
developers what they think. I see this once every 9 months they are looking at this bugs every<br>
day.<br>
<br>
Predrag<br>
<br>
> import torch<br>
> x = torch.cuda.FloatTensor(2,3,4)<br>
> print(x)<br>
><br>
><br>
> I get the following error:<br>
> THCudaCheck FAIL file=/pytorch/torch/lib/THC/THCGeneral.c line=70 error=30 : unknown error<br>
> Traceback (most recent call last):<br>
> File "<stdin>", line 1, in <module><br>
> File "/zfsauton/home/manzilz/.local/lib/python3.6/site-packages/torch/_utils.py", line 69, in _cuda<br>
> return new_type(self.size()).copy_(self, async)<br>
> File "/zfsauton/home/manzilz/.local/lib/python3.6/site-packages/torch/cuda/__init__.py", line 384, in _lazy_new<br>
> _lazy_init()<br>
> File "/zfsauton/home/manzilz/.local/lib/python3.6/site-packages/torch/cuda/__init__.py", line 142, in _lazy_init<br>
> torch._C._cuda_init()<br>
> RuntimeError: cuda runtime error (30) : unknown error at /pytorch/torch/lib/THC/THCGeneral.c:70<br>
><br>
> Thanks,<br>
> Manzil<br>
><br>
> ________________________________________<br>
> From: Predrag Punosevac <<a href="mailto:predragp@andrew.cmu.edu" target="_blank">predragp@andrew.cmu.edu</a>><br>
> Sent: 26 March 2018 22:50<br>
> To: Manzil Zaheer<br>
> Cc: Barnabas Poczos; <a href="mailto:users@autonlab.org" target="_blank">users@autonlab.org</a><br>
> Subject: Re: PyTorch<br>
><br>
> Manzil Zaheer <<a href="mailto:manzil@cmu.edu" target="_blank">manzil@cmu.edu</a>> wrote:<br>
><br>
> > Thanks for the detailed analysis. But I am using pytorch. I have not tried Lua torch. Can you please check? Thanks again!<br>
> ><br>
><br>
> I did. You have Python 3.6.4 in /opt/miniconda3/bin/python3.6<br>
><br>
> predrag@gpu3$ /opt/miniconda3/bin/python3.6<br>
> Python 3.6.4 |Anaconda, Inc.| (default, Jan 16 2018, 18:10:19)<br>
> [GCC 7.2.0] on linux<br>
> Type "help", "copyright", "credits" or "license" for more information.<br>
><br>
><br>
> Try reinstalling thing in your scratch directory as<br>
><br>
> /opt/miniconda3/bin/conda install pytorch torchvision cuda91 -c pytorch<br>
><br>
> You should see something like<br>
><br>
> The following packages will be downloaded:<br>
><br>
> package | build<br>
> ---------------------------|-----------------<br>
> pillow-5.0.0 | py36h3deb7b8_0 561 KB<br>
> mkl-2018.0.2 | 1 205.2 MB<br>
> cuda91-1.0 | h4c16780_0 3 KB<br>
> pytorch<br>
> libpng-1.6.34 | hb9fc6fc_0 334 KB<br>
> freetype-2.8 | hab7d2ae_1 804 KB<br>
> libgfortran-ng-7.2.0 | hdf63c60_3 1.2 MB<br>
> intel-openmp-2018.0.0 | 8 620 KB<br>
> libtiff-4.0.9 | h28f6b97_0 586 KB<br>
> pytorch-0.3.1 |py36_cuda9.1.85_cudnn7.0.5_2 475.0<br>
> MB pytorch<br>
> torchvision-0.2.0 | py36h17b6947_1 102 KB<br>
> pytorch<br>
> jpeg-9b | h024ee3a_2 248 KB<br>
> numpy-1.14.2 | py36hdbf6ddf_0 4.0 MB<br>
> olefile-0.45.1 | py36_0 47 KB<br>
> ------------------------------------------------------------<br>
> Total: 688.7 MB<br>
><br>
><br>
> Make sure you put your scratch as a path since file server is full. I<br>
> got clean installation but I didn't play further. One thing that worries<br>
> me is this line<br>
><br>
> pytorch-0.3.1 |py36_cuda9.1.85_cudnn7.0.5_2 475.0 MB<br>
> pytorch<br>
><br>
> We had problems with cudnn on 9.1 apparently because the upstream was<br>
> assuming 7.0.5 when in reality I have 7.1.1 CUDA 9 or even 7.1.5. CUDA<br>
> 9.1<br>
><br>
> GPU3 has CUDNN library 7.0.5 in cuda-9.0 so try adjusting conda command<br>
> accordingly.<br>
><br>
><br>
> Best,<br>
> Predrag<br>
><br>
><br>
><br>
><br>
><br>
><br>
> ><br>
> ><br>
> > Sent from my Samsung Galaxy smartphone.<br>
> ><br>
> ><br>
> > -------- Original message --------<br>
> > From: Predrag Punosevac <<a href="mailto:predragp@andrew.cmu.edu" target="_blank">predragp@andrew.cmu.edu</a>><br>
> > Date: 3/26/18 9:00 PM (GMT-05:00)<br>
> > To: Manzil Zaheer <<a href="mailto:manzil@cmu.edu" target="_blank">manzil@cmu.edu</a>><br>
> > Cc: Barnabas Poczos <<a href="mailto:bapoczos@andrew.cmu.edu" target="_blank">bapoczos@andrew.cmu.edu</a>>, <a href="mailto:users@autonlab.org" target="_blank">users@autonlab.org</a><br>
> > Subject: Re: Lua Torch<br>
> ><br>
> > Manzil Zaheer <<a href="mailto:manzil@cmu.edu" target="_blank">manzil@cmu.edu</a>> wrote:<br>
> ><br>
> > > Hi Predrag,<br>
> > ><br>
> > > I am not able to use any GPUSs on gpu5,6,7,9. I tried all 3 versions of cuda, but I get the following error:<br>
> > ><br>
> ><br>
> ><br>
> > I was able to build it after adding this<br>
> ><br>
> > export TORCH_NVCC_FLAGS="-D__CUDA_NO_HALF_OPERATORS__"<br>
> ><br>
> > per<br>
> ><br>
> > <a href="https://github.com/torch/torch7/issues/1086" rel="noreferrer" target="_blank">https://github.com/torch/torch7/issues/1086</a><br>
> ><br>
> > When I try to run it I get errors that Lua packages are missing (probably<br>
> > due to my path variables). I have a vague recollection that Simon and I<br>
> > halped you once with this thing in the past. IIRC it was very picky about<br>
> > the version of some Lua package and required their version not the one<br>
> > which comes with yum .<br>
> ><br>
> > Anyhow I am forwarding this to users@autonlab in hope somebody is using<br>
> > it and might be of more help. Please stop by NSH 3119 and let us try to<br>
> > debug this.<br>
> ><br>
> > Predrag<br>
> ><br>
> ><br>
> ><br>
> ><br>
> > > THCudaCheck FAIL file=/pytorch/torch/lib/THC/THCGeneral.c line=70 error=30 : unknown error<br>
> > > Traceback (most recent call last):<br>
> > > File "<stdin>", line 1, in <module><br>
> > > File "/zfsauton/home/manzilz/local/lib/python3.6/site-packages/torch/cuda/__init__.py", line 384, in _lazy_new<br>
> > > _lazy_init()<br>
> > > File "/zfsauton/home/manzilz/local/lib/python3.6/site-packages/torch/cuda/__init__.py", line 142, in _lazy_init<br>
> > > torch._C._cuda_init()<br>
> > > RuntimeError: cuda runtime error (30) : unknown error at /pytorch/torch/lib/THC/THCGeneral.c:70<br>
> > ><br>
> > > Can you kindly look into it?<br>
> > ><br>
> > > Thanks,<br>
> > > Manzil<br>
<br>
</blockquote></div>