nvidia-smi not available on gpu1, gpu5-7

[Predrag Punosevac]

Yichong Xu <yichongx at cs.cmu.edu> wrote:

> Thank you Predrag!
> Thanks,
> Yichong
> 

Dear Autonians,

I know how to fix this. Upgrade process have upgraded CUDA to 9.0 which
requires new 

NVIDIA-Linux-x86_64-384.98

driver instead of 

NVIDIA-Linux-x86_64-384.90

GPU7 now works as expected (I tested MATLAB). Be mindful that some other
peaces of custom compiled software (TensorFlow, Caffe) might have gotten
broken and need to be recompiled. 

I am upgrading driver right now on GPU5 and GPU6


GPU1 works also as exected (Tesla K80 cards) but CUDA is upgraded to 9.0
so crazy things might happen until dust settles.

Best,
Predrag

P.S. CUDA-9 is supposedly much faster than CUDA-8




> 
> 
> > On Nov 30, 2017, at 5:45 PM, Predrag Punosevac <predragp at andrew.cmu.edu> wrote:
> > 
> > Yichong Xu <yichongx at cs.cmu.edu> wrote:
> > 
> >> Hi Predrag,
> >> It seems nvidia-smi is not running properly on gpu1 and gpu5-7. (My programs are still running but I???m not sure whether they???re still using the gpu or not.) Could you check about this? Thank you very much!
> >> 
> > 
> > Thanks for the report. Apparently CUDA upgrade broke the something
> > 
> > root at gpu7$ nvidia-smi
> > Failed to initialize NVML: Driver/library version mismatch
> > 
> > I am working to fix the issue.
> > 
> > Predrag
> > 
> >> Yichong Xu
> >> Machine Learning Department, CMU
> >> yichongx at cs.cmu.edu
> >> 412-652-8309
> >> 
> >> 
> >> 
> >> 
>