<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class="">Previously we have encountered this issue: Basically somehow you cannot put your cuda cache on nfs server now. Doing this will resolve the problem (works for me):</div>
export CUDA_CACHE_PATH=/home/scratch/[your_id]/[some_folder]
<div class=""><br class="">
<div class="">
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;">
<i class="">Thanks,</i></div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;">
<i class="">Yichong</i></div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">
<br class="">
</div>
<br class="Apple-interchange-newline">
</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Nov 6, 2018, at 7:41 PM, Emre Yolcu <<a href="mailto:eyolcu@cs.cmu.edu" class="">eyolcu@cs.cmu.edu</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">Could you try setting up everything in the scratch directory and test that way (if that's not what you're already doing)? The last time we had a CUDA problem I moved everything from /zfsauton/home to /home/scratch directories and I cannot
reproduce the error on gpu{6,8,9}.<br class="">
</div>
<div class="gmail_extra"><br class="">
<div class="gmail_quote">On Tue, Nov 6, 2018 at 6:41 PM, <span dir="ltr" class="">
<<a href="mailto:qiong.zhang@stat.ubc.ca" target="_blank" class="">qiong.zhang@stat.ubc.ca</a>></span> wrote:<br class="">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<u class=""></u>
<div class="">
<div style="font-family:arial,sans-serif;font-size:13px" class="">
<p class="">I have a similar issue. When I submit the job, it says Runtime error: CUDA error: unknown error. I tried the simple commands that you provided, doesn't work as well.<br class="">
<br class="">
Qiong</p>
<div class="">
<div class="h5"><br class="">
November 6, 2018 3:02 PM, "Matthew Barnes" <<a href="mailto:%22Matthew%20Barnes%22%20%3Cmbarnes1@andrew.cmu.edu%3E" target="_blank" class="">mbarnes1@andrew.cmu.edu</a>> wrote:<br class="">
<blockquote class="">
<div class="">
<div class="">
<div dir="ltr" class="">Is anyone else having issues with CUDA since this week? Even simple pytorch commands hang:
<div class=""></div>
<div class="">
<div class="">(torch) bash-4.2$ python</div>
<div class="">Python 2.7.5 (default, Jul 3 2018, 19:30:05)</div>
<div class="">[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux2</div>
<div class="">Type "help", "copyright", "credits" or "license" for more information.</div>
<div class="">>>> import torch</div>
<div class="">x>>> x = torch.zeros(4)</div>
<div class="">>>> x.cuda()</div>
</div>
<div class=""></div>
<div class=""></div>
<div class="">nvidia-smi works, and torch.cuda.is_available() returns True.</div>
</div>
</div>
</div>
</blockquote>
<br class="">
<br class="">
<u class=""></u><u class=""></u></div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>