<div dir="ltr">Hi,<div><br></div><div>I am having similar issues with tensorflow as well.</div><div><br></div><div>When I run a tensorflow program (like python XX.py), it hangs after loading the graph (after executing session.run()). I cannot quit it by hitting Ctrl-Z/Ctrl-C and nothing is printed to the stdout/stderr.</div><div><br></div><div>Strangely, my program will terminate and do the job (like writing results to files). And after it terminates I am able to see all the strings I print to stdout/stderr. But before its termination I can do nothing about it.</div><div><br></div><div>I have tried both my local tensorflow and the global one on all three GPU machines. They all had the same problem.</div><div><br></div><div>Best,</div><div>Hanqi</div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Oct 26, 2016 at 3:11 PM, Dougal Sutherland <span dir="ltr"><<a href="mailto:dougal@gmail.com" target="_blank">dougal@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">a) You might also want to do<div><br></div><div><div>export JUPYTER_CONFIG_DIR=/home/scrat<wbr>ch/$USER/.jupyter</div><div><br></div><div>if you use jupyter notebooks and whatnot.</div><div><br></div><div><br></div><div>b) I'm also not able to import theano when using GPUs on gpu2/gpu3 as of this afternoon. It hangs in a similar way, though I've set the compiledir to be in scratch in my theanorc. I've tracked it down in the debugger to <a href="https://github.com/Theano/Theano/blob/25f0dee338b901070e021d431c938102890bc69f/theano/sandbox/cuda/__init__.py#L556" target="_blank">this line</a> or <a href="https://github.com/Theano/Theano/blob/25f0dee338b901070e021d431c938102890bc69f/theano/sandbox/cuda/__init__.py#L571" target="_blank">this one</a> (both hang, which one is called depends on theano settings), which call <a href="https://github.com/Theano/Theano/blob/140d0a064523349b630a284247c7cddd767fc46e/theano/sandbox/cuda/cuda_ndarray.cu#L3170" target="_blank">this function</a> or <a href="https://github.com/Theano/Theano/blob/140d0a064523349b630a284247c7cddd767fc46e/theano/sandbox/cuda/cuda_ndarray.cu#L2947" target="_blank">this one</a>. No idea why this started happening or if it's related -- it doesn't seem like it should be hitting nfs at all -- but it seemed to start at the same time.</div><div><div class="m_2332191535745368018m_7628515876608224467h5"><div><br></div><div><br></div><br><div class="gmail_quote"><div dir="ltr">On Wed, Oct 26, 2016 at 7:20 PM Junier Oliva <<a href="mailto:junieroliva@gmail.com" target="_blank">junieroliva@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">Awesome, seems to work for me too. Thanks!!</div><div dir="ltr" class="m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg"><div class="m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg"><br class="m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg"></div><div class="m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">-Junier</div></div><div class="gmail_extra m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg"><br class="m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg"><div class="gmail_quote m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">On Wed, Oct 26, 2016 at 2:08 PM, Dougal Sutherland <span dir="ltr" class="m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg"><<a href="mailto:dougal@gmail.com" class="m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg" target="_blank">dougal@gmail.com</a>></span> wrote:<br class="m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg"><blockquote class="gmail_quote m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">Here's a workaround that avoids ipython having its config files / etc on nfs, it seems to work for me:<div class="m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg"><br class="m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg"></div><div class="m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">export IPYTHONDIR=/home/scratch/$USER<wbr>/.ipython</div><div class="m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg"><br class="m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg"></div><div class="m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">You can do this in a terminal or put it your .bash_profile / similar to make it permanent.</div><div class="m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg"><br class="m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg"></div><div class="m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">(I guess this means something changed about the nfs server yesterday/today that broke this.)</div><div class="m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg"><div class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772h5 m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg"><div class="m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg"><br class="m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg"><br class="m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg"><div class="gmail_quote m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg"><div dir="ltr" class="m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">On Wed, Oct 26, 2016 at 7:03 PM yifei ma <<a href="mailto:mayifei1012@gmail.com" class="m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg" target="_blank">mayifei1012@gmail.com</a>> wrote:<br class="m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg"></div><blockquote class="gmail_quote m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000" class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">
Second that on foxconn. It launches but the ipython client won't
start.<br class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">
<br class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">
Thanks,<br class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">
Yifei</div><div bgcolor="#FFFFFF" text="#000000" class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg"><br class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">
<br class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">
<div class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349m_-3009744765975672134moz-cite-prefix m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">On 10/26/2016 01:03 PM, Junier Oliva
wrote:<br class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">
</div>
<blockquote type="cite" class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">
<div dir="ltr" class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">Not sure if this at all related, but is ipython
broken for anyone else? It seems to just hang upon launching it
on several auton machines (GPU1, GPU2, LOV4, LOW1).
<div class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg"><br class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">
</div>
<div class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">Thanks,</div>
<div class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">Junier</div>
</div>
<div class="gmail_extra m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg"><br class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">
<div class="gmail_quote m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">On Tue, Oct 25, 2016 at 2:14 PM, Dougal
Sutherland <span dir="ltr" class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg"><<a href="mailto:dougal@gmail.com" class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg" target="_blank">dougal@gmail.com</a>></span>
wrote:<br class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">
<blockquote class="gmail_quote m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr" class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">The same version of TensorFlow as on gpu3 is
now installed on gpu2, along with cudnn; let me know if
there are issues.
<div class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg"><br class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">
</div>
<div class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">I didn't do a global install of Caffe on either
machine, because Caffe is kind of dumb and doesn't
really do global installs. If anyone wants this, talk to
me and we can figure out what makes sense.</div>
<span class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349m_-3009744765975672134HOEnZb m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg"><font color="#888888" class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">
<div class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg"><br class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">
</div>
<div class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">- Dougal</div>
</font></span></div>
<div class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349m_-3009744765975672134HOEnZb m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">
<div class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349m_-3009744765975672134h5 m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg"><br class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">
<div class="gmail_quote m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">
<div dir="ltr" class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">On Tue, Oct 25, 2016 at 6:09 PM Predrag
Punosevac <<a href="mailto:predragp@imap.srv.cs.cmu.edu" class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg" target="_blank">predragp@imap.srv.cs.cmu.edu</a>>
wrote:<br class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">
</div>
<blockquote class="gmail_quote m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Dear
Autonians,<br class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">
<br class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">
As scheduled I upgraded GPU2 to 256 GB of RAM and 4
Titan (Pascal) cards<br class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">
per Barnabas. Dugal is currently upgrading
TensorFlow and Caffe to the<br class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">
latest and the greatest. Please wait until his
e-mail until you hit the<br class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">
machine. MATLAB is broken just like on GPU3 due to
cuda-8.0. I am<br class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">
working now on GPU1.<br class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">
<br class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">
<br class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">
Predrag<br class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">
<br class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">
P.S. I will escalate MATLAB issue with MathWorks but
I don't expect to<br class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">
fixed before R2017a.<br class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">
<br class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<br class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">
</div>
</blockquote>
<br class="m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg">
</div></blockquote></div></div></div></div></div>
</blockquote></div><br class="m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg"></div>
</blockquote></div></div></div></div></div>
</blockquote></div><br></div></div>