<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Unfortunately not for me, I already had this path ...</p>
<div class="moz-cite-prefix">Le 06/11/2018 à 21:51, Matthew Barnes a
écrit :<br>
</div>
<blockquote type="cite"
cite="mid:CAB7OVwB=kYQxY7PkCFxZwZDpLuZ2ES6i7ELj39d_02YFx-DkMw@mail.gmail.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<div dir="ltr">The CUDA_CACHE_PATH works! Thanks for the quick
fix.</div>
<br>
<div class="gmail_quote">
<div dir="ltr">On Tue, Nov 6, 2018 at 9:44 PM Yichong Xu <<a
href="mailto:yichongx@cs.cmu.edu" moz-do-not-send="true">yichongx@cs.cmu.edu</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word;line-break:after-white-space">
<div>Previously we have encountered this issue: Basically
somehow you cannot put your cuda cache on nfs server now.
Doing this will resolve the problem (works for me):</div>
export CUDA_CACHE_PATH=/home/scratch/[your_id]/[some_folder]
<div><br>
<div>
<div
style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><i>Thanks,</i></div>
<div
style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><i>Yichong</i></div>
<div
style="color:rgb(0,0,0);font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><br>
</div>
<br
class="m_2796597251052082966Apple-interchange-newline">
</div>
</div>
</div>
<div style="word-wrap:break-word;line-break:after-white-space">
<div>
<div><br>
<blockquote type="cite">
<div>On Nov 6, 2018, at 7:41 PM, Emre Yolcu <<a
href="mailto:eyolcu@cs.cmu.edu" target="_blank"
moz-do-not-send="true">eyolcu@cs.cmu.edu</a>>
wrote:</div>
<br
class="m_2796597251052082966Apple-interchange-newline">
<div>
<div dir="ltr">Could you try setting up everything
in the scratch directory and test that way (if
that's not what you're already doing)? The last
time we had a CUDA problem I moved everything from
/zfsauton/home to /home/scratch directories and I
cannot reproduce the error on gpu{6,8,9}.<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Nov 6, 2018 at
6:41 PM, <span dir="ltr">
<<a href="mailto:qiong.zhang@stat.ubc.ca"
target="_blank" moz-do-not-send="true">qiong.zhang@stat.ubc.ca</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div>
<div
style="font-family:arial,sans-serif;font-size:13px">
<p>I have a similar issue. When I submit
the job, it says Runtime error: CUDA
error: unknown error. I tried the simple
commands that you provided, doesn't work
as well.<br>
<br>
Qiong</p>
<div>
<div class="m_2796597251052082966h5"><br>
November 6, 2018 3:02 PM, "Matthew
Barnes" <<a
href="mailto:%22Matthew%20Barnes%22%20%3Cmbarnes1@andrew.cmu.edu%3E"
target="_blank"
moz-do-not-send="true">mbarnes1@andrew.cmu.edu</a>>
wrote:<br>
<blockquote>
<div>
<div>
<div dir="ltr">Is anyone else
having issues with CUDA since
this week? Even simple pytorch
commands hang:
<div>
<div>(torch) bash-4.2$
python</div>
<div>Python 2.7.5 (default,
Jul 3 2018, 19:30:05)</div>
<div>[GCC 4.8.5 20150623
(Red Hat 4.8.5-28)] on
linux2</div>
<div>Type "help",
"copyright", "credits" or
"license" for more
information.</div>
<div>>>> import
torch</div>
<div>x>>> x =
torch.zeros(4)</div>
<div>>>> x.cuda()</div>
</div>
<div>nvidia-smi works, and
torch.cuda.is_available()
returns True.</div>
</div>
</div>
</div>
</blockquote>
<br>
<br>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
Vincent Jeanselme
-----------------
Analyst Researcher
Auton Lab - Robotics Institute
Carnegie Mellon University</pre>
</body>
</html>