GPU2 upgraded

[Dougal Sutherland]

a) You might also want to do

export JUPYTER_CONFIG_DIR=/home/scratch/$USER/.jupyter

if you use jupyter notebooks and whatnot.


b) I'm also not able to import theano when using GPUs on gpu2/gpu3 as of
this afternoon. It hangs in a similar way, though I've set the compiledir
to be in scratch in my theanorc. I've tracked it down in the debugger to this
line
<https://github.com/Theano/Theano/blob/25f0dee338b901070e021d431c938102890bc69f/theano/sandbox/cuda/__init__.py#L556>
or this one
<https://github.com/Theano/Theano/blob/25f0dee338b901070e021d431c938102890bc69f/theano/sandbox/cuda/__init__.py#L571>
(both
hang, which one is called depends on theano settings), which call this
function
<https://github.com/Theano/Theano/blob/140d0a064523349b630a284247c7cddd767fc46e/theano/sandbox/cuda/cuda_ndarray.cu#L3170>
or this one
<https://github.com/Theano/Theano/blob/140d0a064523349b630a284247c7cddd767fc46e/theano/sandbox/cuda/cuda_ndarray.cu#L2947>.
No idea why this started happening or if it's related -- it doesn't seem
like it should be hitting nfs at all -- but it seemed to start at the same
time.



On Wed, Oct 26, 2016 at 7:20 PM Junier Oliva <junieroliva at gmail.com> wrote:

> Awesome, seems to work for me too. Thanks!!
>
> -Junier
>
> On Wed, Oct 26, 2016 at 2:08 PM, Dougal Sutherland <dougal at gmail.com>
> wrote:
>
> Here's a workaround that avoids ipython having its config files / etc on
> nfs, it seems to work for me:
>
> export IPYTHONDIR=/home/scratch/$USER/.ipython
>
> You can do this in a terminal or put it your .bash_profile / similar to
> make it permanent.
>
> (I guess this means something changed about the nfs server yesterday/today
> that broke this.)
>
>
> On Wed, Oct 26, 2016 at 7:03 PM yifei ma <mayifei1012 at gmail.com> wrote:
>
> Second that on foxconn. It launches but the ipython client won't start.
>
> Thanks,
> Yifei
>
>
> On 10/26/2016 01:03 PM, Junier Oliva wrote:
>
> Not sure if this at all related, but is ipython broken for anyone else? It
> seems to just hang upon launching it on several auton machines (GPU1, GPU2,
> LOV4, LOW1).
>
> Thanks,
> Junier
>
> On Tue, Oct 25, 2016 at 2:14 PM, Dougal Sutherland <dougal at gmail.com>
> wrote:
>
> The same version of TensorFlow as on gpu3 is now installed on gpu2, along
> with cudnn; let me know if there are issues.
>
> I didn't do a global install of Caffe on either machine, because Caffe is
> kind of dumb and doesn't really do global installs. If anyone wants this,
> talk to me and we can figure out what makes sense.
>
> - Dougal
>
> On Tue, Oct 25, 2016 at 6:09 PM Predrag Punosevac <
> predragp at imap.srv.cs.cmu.edu> wrote:
>
> Dear Autonians,
>
> As scheduled I upgraded GPU2 to 256 GB of RAM and 4 Titan (Pascal) cards
> per Barnabas. Dugal is currently upgrading TensorFlow and Caffe to the
> latest and the greatest. Please wait until his e-mail until you hit the
> machine. MATLAB is broken just like on GPU3 due to cuda-8.0. I am
> working now on GPU1.
>
>
> Predrag
>
> P.S. I will escalate MATLAB issue with MathWorks but I don't expect to
> fixed before R2017a.
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/autonlab-users/attachments/20161026/90074556/attachment-0001.html>