<div dir="ltr">I'm not sure if anyone changed anything, but ipython (without the IPYTHONDIR workaround above), theano with GPUs, and tensorflow -m tensorflow.models.image.mnist.convolutional are all working for me now.</div><br><div class="gmail_quote"><div dir="ltr">On Wed, Oct 26, 2016 at 8:38 PM Dougal Sutherland <<a href="mailto:dougal@gmail.com">dougal@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg">Hmm - I tested that tensorflow worked on the mnist example when I built it, but when I run<div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">python -m tensorflow.models.image.mnist.convolutional</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">on gpu3 now, it gets to "Initialized!" and then just hangs, not responding to ^C or ^Z and also not leading to any GPU utiliziation according to nvidia-smi; it takes a kill -9 to stop it.</div><div class="gmail_msg"><div class="gmail_msg"><br class="gmail_msg"></div></div></div><br class="gmail_msg"><div class="gmail_quote gmail_msg"><div dir="ltr" class="gmail_msg">On Wed, Oct 26, 2016 at 8:25 PM Hanqi Sun <<a href="mailto:hanqis@andrew.cmu.edu" class="gmail_msg" target="_blank">hanqis@andrew.cmu.edu</a>> wrote:<br class="gmail_msg"></div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg">Hi,<div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">I am having similar issues with tensorflow as well.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">When I run a tensorflow program (like python XX.py), it hangs after loading the graph (after executing session.run()). I cannot quit it by hitting Ctrl-Z/Ctrl-C and nothing is printed to the stdout/stderr.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">Strangely, my program will terminate and do the job (like writing results to files). And after it terminates I am able to see all the strings I print to stdout/stderr. But before its termination I can do nothing about it.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">I have tried both my local tensorflow and the global one on all three GPU machines. They all had the same problem.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">Best,</div><div class="gmail_msg">Hanqi</div></div><div dir="ltr" class="gmail_msg"><div class="gmail_extra gmail_msg"><br class="gmail_msg"><div class="gmail_quote gmail_msg">On Wed, Oct 26, 2016 at 3:11 PM, Dougal Sutherland <span dir="ltr" class="gmail_msg"><<a href="mailto:dougal@gmail.com" class="gmail_msg" target="_blank">dougal@gmail.com</a>></span> wrote:<br class="gmail_msg"><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="gmail_msg">a) You might also want to do<div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg"><div class="gmail_msg">export JUPYTER_CONFIG_DIR=/home/scratch/$USER/.jupyter</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">if you use jupyter notebooks and whatnot.</div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg">b) I'm also not able to import theano when using GPUs on gpu2/gpu3 as of this afternoon. It hangs in a similar way, though I've set the compiledir to be in scratch in my theanorc. I've tracked it down in the debugger to <a href="https://github.com/Theano/Theano/blob/25f0dee338b901070e021d431c938102890bc69f/theano/sandbox/cuda/__init__.py#L556" class="gmail_msg" target="_blank">this line</a> or <a href="https://github.com/Theano/Theano/blob/25f0dee338b901070e021d431c938102890bc69f/theano/sandbox/cuda/__init__.py#L571" class="gmail_msg" target="_blank">this one</a> (both hang, which one is called depends on theano settings), which call <a href="https://github.com/Theano/Theano/blob/140d0a064523349b630a284247c7cddd767fc46e/theano/sandbox/cuda/cuda_ndarray.cu#L3170" class="gmail_msg" target="_blank">this function</a> or <a href="https://github.com/Theano/Theano/blob/140d0a064523349b630a284247c7cddd767fc46e/theano/sandbox/cuda/cuda_ndarray.cu#L2947" class="gmail_msg" target="_blank">this one</a>. No idea why this started happening or if it's related -- it doesn't seem like it should be hitting nfs at all -- but it seemed to start at the same time.</div><div class="gmail_msg"><div class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467h5 gmail_msg"><div class="gmail_msg"><br class="gmail_msg"></div><div class="gmail_msg"><br class="gmail_msg"></div><br class="gmail_msg"><div class="gmail_quote gmail_msg"><div dir="ltr" class="gmail_msg">On Wed, Oct 26, 2016 at 7:20 PM Junier Oliva <<a href="mailto:junieroliva@gmail.com" class="gmail_msg" target="_blank">junieroliva@gmail.com</a>> wrote:<br class="gmail_msg"></div><blockquote class="gmail_quote gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">Awesome, seems to work for me too. Thanks!!</div><div dir="ltr" class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg"><div class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg"><br class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg"></div><div class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">-Junier</div></div><div class="gmail_extra m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg"><br class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg"><div class="gmail_quote m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">On Wed, Oct 26, 2016 at 2:08 PM, Dougal Sutherland <span dir="ltr" class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg"><<a href="mailto:dougal@gmail.com" class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg" target="_blank">dougal@gmail.com</a>></span> wrote:<br class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg"><blockquote class="gmail_quote m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">Here's a workaround that avoids ipython having its config files / etc on nfs, it seems to work for me:<div class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg"><br class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg"></div><div class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">export IPYTHONDIR=/home/scratch/$USER/.ipython</div><div class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg"><br class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg"></div><div class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">You can do this in a terminal or put it your .bash_profile / similar to make it permanent.</div><div class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg"><br class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg"></div><div class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">(I guess this means something changed about the nfs server yesterday/today that broke this.)</div><div class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg"><div class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772h5 m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg"><div class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg"><br class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg"><br class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg"><div class="gmail_quote m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg"><div dir="ltr" class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">On Wed, Oct 26, 2016 at 7:03 PM yifei ma <<a href="mailto:mayifei1012@gmail.com" class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg" target="_blank">mayifei1012@gmail.com</a>> wrote:<br class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg"></div><blockquote class="gmail_quote m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000" class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">
Second that on foxconn. It launches but the ipython client won't
start.<br class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">
<br class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">
Thanks,<br class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">
Yifei</div><div bgcolor="#FFFFFF" text="#000000" class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg"><br class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">
<br class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">
<div class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349m_-3009744765975672134moz-cite-prefix m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">On 10/26/2016 01:03 PM, Junier Oliva
wrote:<br class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">
</div>
<blockquote type="cite" class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">
<div dir="ltr" class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">Not sure if this at all related, but is ipython
broken for anyone else? It seems to just hang upon launching it
on several auton machines (GPU1, GPU2, LOV4, LOW1).
<div class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg"><br class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">
</div>
<div class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">Thanks,</div>
<div class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">Junier</div>
</div>
<div class="gmail_extra m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg"><br class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">
<div class="gmail_quote m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">On Tue, Oct 25, 2016 at 2:14 PM, Dougal
Sutherland <span dir="ltr" class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg"><<a href="mailto:dougal@gmail.com" class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg" target="_blank">dougal@gmail.com</a>></span>
wrote:<br class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">
<blockquote class="gmail_quote m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr" class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">The same version of TensorFlow as on gpu3 is
now installed on gpu2, along with cudnn; let me know if
there are issues.
<div class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg"><br class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">
</div>
<div class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">I didn't do a global install of Caffe on either
machine, because Caffe is kind of dumb and doesn't
really do global installs. If anyone wants this, talk to
me and we can figure out what makes sense.</div>
<span class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349m_-3009744765975672134HOEnZb m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg"><font color="#888888" class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">
<div class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg"><br class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">
</div>
<div class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">- Dougal</div>
</font></span></div>
<div class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349m_-3009744765975672134HOEnZb m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">
<div class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349m_-3009744765975672134h5 m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg"><br class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">
<div class="gmail_quote m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">
<div dir="ltr" class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">On Tue, Oct 25, 2016 at 6:09 PM Predrag
Punosevac <<a href="mailto:predragp@imap.srv.cs.cmu.edu" class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg" target="_blank">predragp@imap.srv.cs.cmu.edu</a>>
wrote:<br class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">
</div>
<blockquote class="gmail_quote m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Dear
Autonians,<br class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">
<br class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">
As scheduled I upgraded GPU2 to 256 GB of RAM and 4
Titan (Pascal) cards<br class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">
per Barnabas. Dugal is currently upgrading
TensorFlow and Caffe to the<br class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">
latest and the greatest. Please wait until his
e-mail until you hit the<br class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">
machine. MATLAB is broken just like on GPU3 due to
cuda-8.0. I am<br class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">
working now on GPU1.<br class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">
<br class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">
<br class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">
Predrag<br class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">
<br class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">
P.S. I will escalate MATLAB issue with MathWorks but
I don't expect to<br class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">
fixed before R2017a.<br class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">
<br class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349m_-3009744765975672134m_-6204286028548759537gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
<br class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">
</div>
</blockquote>
<br class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878m_-6047073204725867772m_-424763064694038349gmail_msg m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg">
</div></blockquote></div></div></div></div></div>
</blockquote></div><br class="m_1219516275707823990m_-5461075307090203425m_2332191535745368018m_7628515876608224467m_3863702182978257878gmail_msg gmail_msg"></div>
</blockquote></div></div></div></div></div>
</blockquote></div><br class="gmail_msg"></div></div></blockquote></div></blockquote></div>