CUDA hangs

[Matthew Barnes]

The CUDA_CACHE_PATH works! Thanks for the quick fix.

On Tue, Nov 6, 2018 at 9:44 PM Yichong Xu <yichongx at cs.cmu.edu> wrote:

> Previously we have encountered this issue: Basically somehow you cannot
> put your cuda cache on nfs server now. Doing this will resolve the problem
> (works for me):
> export CUDA_CACHE_PATH=/home/scratch/[your_id]/[some_folder]
>
> *Thanks,*
> *Yichong*
>
>
>
> On Nov 6, 2018, at 7:41 PM, Emre Yolcu <eyolcu at cs.cmu.edu> wrote:
>
> Could you try setting up everything in the scratch directory and test that
> way (if that's not what you're already doing)? The last time we had a CUDA
> problem I moved everything from /zfsauton/home to /home/scratch directories
> and I cannot reproduce the error on gpu{6,8,9}.
>
> On Tue, Nov 6, 2018 at 6:41 PM, <qiong.zhang at stat.ubc.ca> wrote:
>
>> I have a similar issue. When I submit the job, it says Runtime error:
>> CUDA error: unknown error. I tried the simple commands that you provided,
>> doesn't work as well.
>>
>> Qiong
>>
>> November 6, 2018 3:02 PM, "Matthew Barnes" <mbarnes1 at andrew.cmu.edu
>> <%22Matthew%20Barnes%22%20%3Cmbarnes1 at andrew.cmu.edu%3E>> wrote:
>>
>> Is anyone else having issues with CUDA since this week? Even simple
>> pytorch commands hang:
>> (torch) bash-4.2$ python
>> Python 2.7.5 (default, Jul 3 2018, 19:30:05)
>> [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux2
>> Type "help", "copyright", "credits" or "license" for more information.
>> >>> import torch
>> x>>> x = torch.zeros(4)
>> >>> x.cuda()
>> nvidia-smi works, and torch.cuda.is_available() returns True.
>>
>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/autonlab-users/attachments/20181106/596c5dd5/attachment.html>