Fwd: Driver/library version mismatch on gpu nodes

[Predrag Punosevac]

Michael Andrews <mbandrews at cmu.edu> wrote:

> Forwarding for Predrag:

Why did you forward me my own message?

Predrag

> 
> ---------- Forwarded message ----------
> From: Predrag Punosevac <predragp at andrew.cmu.edu>
> Date: Thu, Mar 22, 2018 at 7:21 PM
> Subject: Re: Driver/library version mismatch on gpu nodes
> To: mbandrews at cmu.edu
> 
> Dear Autonians,
> 
> This turned to be little bit bigger job than originally anticipated.
> This is the summary of what has being done:
> 
> gpu[1-9] with the exception of GPU7 which is used to serve a client have
> being upgraded to the latest 3.10.0-693.21.1.el7 including all packages
> 
> nvidia-smi works as expected on all of those machines now
> 
> devtool-4 tools were replaced with devtool-6 which means that on all
> those machines you have gcc 6
> 
> miniconda3 (python 3.6.3.) in /opt/miniconda3
> 
> All these machines now have /opt/rh-git29
> 
> 
> MATLAB has to be removed from GPU1 and GPU2 due to the space issue. I
> will be reinstalling it tomorrow on the different location. As a bonus
> you will get MATLAB R2018a which I am not planning to install on other
> servers unless requested (I will be waiting for R2018b).
> 
> For some reason MATLAB no longer works with GPUs on servers GPU5 and
> GPU9. Those two servers will have the same treatment as GPU1 and GPU2
> and I will install the latest version of MATLAB in order to fix the
> problem (IIRC GPU8 and GPU9 are still only available to designated
> users).
> 
> Finally Tensorflow is possibly broken due to a symlink. Please see my
> next e-mail. I plan to fix this later tonight.
> 
> Predrag