<div dir="ltr">The CUDA_CACHE_PATH works! Thanks for the quick fix.</div><br><div class="gmail_quote"><div dir="ltr">On Tue, Nov 6, 2018 at 9:44 PM Yichong Xu <<a href="mailto:yichongx@cs.cmu.edu">yichongx@cs.cmu.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">



<div style="word-wrap:break-word;line-break:after-white-space">
<div>Previously we have encountered this issue: Basically somehow you cannot put your cuda cache on nfs server now. Doing this will resolve the problem (works for me):</div>
export CUDA_CACHE_PATH=/home/scratch/[your_id]/[some_folder]
<div><br>
<div>
<div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<i>Thanks,</i></div>
<div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<i>Yichong</i></div>
<div style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<br>
</div>
<br class="m_2796597251052082966Apple-interchange-newline">
</div></div></div><div style="word-wrap:break-word;line-break:after-white-space"><div>
<div><br>
<blockquote type="cite">
<div>On Nov 6, 2018, at 7:41 PM, Emre Yolcu <<a href="mailto:eyolcu@cs.cmu.edu" target="_blank">eyolcu@cs.cmu.edu</a>> wrote:</div>
<br class="m_2796597251052082966Apple-interchange-newline">
<div>
<div dir="ltr">Could you try setting up everything in the scratch directory and test that way (if that's not what you're already doing)? The last time we had a CUDA problem I moved everything from /zfsauton/home to /home/scratch directories and I cannot
 reproduce the error on gpu{6,8,9}.<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Nov 6, 2018 at 6:41 PM, <span dir="ltr">
<<a href="mailto:qiong.zhang@stat.ubc.ca" target="_blank">qiong.zhang@stat.ubc.ca</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<u></u>
<div>
<div style="font-family:arial,sans-serif;font-size:13px">
<p>I have a similar issue. When I submit the job, it says Runtime error: CUDA error: unknown error. I tried the simple commands that you provided, doesn't work as well.<br>
<br>
Qiong</p>
<div>
<div class="m_2796597251052082966h5"><br>
November 6, 2018 3:02 PM, "Matthew Barnes" <<a href="mailto:%22Matthew%20Barnes%22%20%3Cmbarnes1@andrew.cmu.edu%3E" target="_blank">mbarnes1@andrew.cmu.edu</a>> wrote:<br>
<blockquote>
<div>
<div>
<div dir="ltr">Is anyone else having issues with CUDA since this week? Even simple pytorch commands hang:
<div></div>
<div>
<div>(torch) bash-4.2$ python</div>
<div>Python 2.7.5 (default, Jul 3 2018, 19:30:05)</div>
<div>[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux2</div>
<div>Type "help", "copyright", "credits" or "license" for more information.</div>
<div>>>> import torch</div>
<div>x>>> x = torch.zeros(4)</div>
<div>>>> x.cuda()</div>
</div>
<div></div>
<div></div>
<div>nvidia-smi works, and torch.cuda.is_available() returns True.</div>
</div>
</div>
</div>
</blockquote>
<br>
<br>
<u></u><u></u></div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div></div></blockquote></div>