<div dir="ltr">a) You might also want to do<div><br></div><div><div>export JUPYTER_CONFIG_DIR=/home/scratch/$USER/.jupyter</div><div><br></div><div>if you use jupyter notebooks and whatnot.</div><div><br></div><div><br></div><div>b) I'm also not able to import theano when using GPUs on gpu2/gpu3 as of this afternoon. It hangs in a similar way, though I've set the compiledir to be in scratch in my theanorc. I've tracked it down in the debugger to <a href="https://github.com/Theano/Theano/blob/25f0dee338b901070e021d431c938102890bc69f/theano/sandbox/cuda/__init__.py#L556">this line</a> or <a href="https://github.com/Theano/Theano/blob/25f0dee338b901070e021d431c938102890bc69f/theano/sandbox/cuda/__init__.py#L571">this one</a> (both hang, which one is called depends on theano settings), which call <a href="https://github.com/Theano/Theano/blob/140d0a064523349b630a284247c7cddd767fc46e/theano/sandbox/cuda/cuda_ndarray.cu#L3170">this function</a> or <a href="https://github.com/Theano/Theano/blob/140d0a064523349b630a284247c7cddd767fc46e/theano/sandbox/cuda/cuda_ndarray.cu#L2947">this one</a>. No idea why this started happening or if it's related -- it doesn't seem like it should be hitting nfs at all -- but it seemed to start at the same time.</div><div><br></div><div><br></div><br><div class="gmail_quote"><div dir="ltr">On Wed, Oct 26, 2016 at 7:20 PM Junier Oliva <<a href="mailto:junieroliva@gmail.com">junieroliva@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg">Awesome, seems to work for me too. Thanks!!</div><div dir="ltr" class="gmail_msg"><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">-Junier</div></div><div class="gmail_extra gmail_msg"><br class="gmail_msg"><div class="gmail_quote gmail_msg">On Wed, Oct 26, 2016 at 2:08 PM, Dougal Sutherland <span dir="ltr" class="gmail_msg"><<a href="mailto:dougal@gmail.com" class="gmail_msg" target="_blank">dougal@gmail.com</a>></span> wrote:<br class="gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg">Here's a workaround that avoids ipython having its config files / etc on nfs, it seems to work for me:<div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">export IPYTHONDIR=/home/scratch/$USER/.ipython</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">You can do this in a terminal or put it your .bash_profile / similar to make it permanent.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">(I guess this means something changed about the nfs server yesterday/today that broke this.)</div><div class="gmail_msg"><div class="m_-6047073204725867772h5 gmail_msg"><div class="gmail_msg"><br class="gmail_msg"><br class="gmail_msg"><div class="gmail_quote gmail_msg"><div dir="ltr" class="gmail_msg">On Wed, Oct 26, 2016 at 7:03 PM yifei ma <<a href="mailto:mayifei1012@gmail.com" class="gmail_msg" target="_blank">mayifei1012@gmail.com</a>> wrote:<br class="gmail_msg"></div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000" class="m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">
Second that on foxconn. It launches but the ipython client won't
start.<br class="m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">
<br class="m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">
Thanks,<br class="m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">
Yifei</div><div bgcolor="#FFFFFF" text="#000000" class="m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg"><br class="m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">
<br class="m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">
<div class="m_-6047073204725867772m_-424763064694038349m_-3009744765975672134moz-cite-prefix m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">On 10/26/2016 01:03 PM, Junier Oliva
wrote:<br class="m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">
</div>
<blockquote type="cite" class="m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">
<div dir="ltr" class="m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">Not sure if this at all related, but is ipython
broken for anyone else? It seems to just hang upon launching it
on several auton machines (GPU1, GPU2, LOV4, LOW1).
<div class="m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg"><br class="m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">
</div>
<div class="m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">Thanks,</div>
<div class="m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">Junier</div>
</div>
<div class="gmail_extra m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg"><br class="m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">
<div class="gmail_quote m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">On Tue, Oct 25, 2016 at 2:14 PM, Dougal
Sutherland <span dir="ltr" class="m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg"><<a href="mailto:dougal@gmail.com" class="m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg" target="_blank">dougal@gmail.com</a>></span>
wrote:<br class="m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">
<blockquote class="gmail_quote m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr" class="m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">The same version of TensorFlow as on gpu3 is
now installed on gpu2, along with cudnn; let me know if
there are issues.
<div class="m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg"><br class="m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">
</div>
<div class="m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">I didn't do a global install of Caffe on either
machine, because Caffe is kind of dumb and doesn't
really do global installs. If anyone wants this, talk to
me and we can figure out what makes sense.</div>
<span class="m_-6047073204725867772m_-424763064694038349m_-3009744765975672134HOEnZb m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg"><font color="#888888" class="m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">
<div class="m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg"><br class="m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">
</div>
<div class="m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">- Dougal</div>
</font></span></div>
<div class="m_-6047073204725867772m_-424763064694038349m_-3009744765975672134HOEnZb m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">
<div class="m_-6047073204725867772m_-424763064694038349m_-3009744765975672134h5 m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg"><br class="m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">
<div class="gmail_quote m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">
<div dir="ltr" class="m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">On Tue, Oct 25, 2016 at 6:09 PM Predrag
Punosevac <<a href="mailto:predragp@imap.srv.cs.cmu.edu" class="m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg" target="_blank">predragp@imap.srv.cs.cmu.edu</a>>
wrote:<br class="m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">
</div>
<blockquote class="gmail_quote m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Dear
Autonians,<br class="m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">
<br class="m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">
As scheduled I upgraded GPU2 to 256 GB of RAM and 4
Titan (Pascal) cards<br class="m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">
per Barnabas. Dugal is currently upgrading
TensorFlow and Caffe to the<br class="m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">
latest and the greatest. Please wait until his
e-mail until you hit the<br class="m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">
machine. MATLAB is broken just like on GPU3 due to
cuda-8.0. I am<br class="m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">
working now on GPU1.<br class="m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">
<br class="m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">
<br class="m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">
Predrag<br class="m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">
<br class="m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">
P.S. I will escalate MATLAB issue with MathWorks but
I don't expect to<br class="m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">
fixed before R2017a.<br class="m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">
<br class="m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<br class="m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">
</div>
</blockquote>
<br class="m_-6047073204725867772m_-424763064694038349gmail_msg gmail_msg">
</div></blockquote></div></div></div></div></div>
</blockquote></div><br class="gmail_msg"></div>
</blockquote></div></div></div>