GPUs 11/13/14 nvidia-smi not working

[Predrag Punosevac]

Fixed. The only problematic machines were 11, 13, and 14.

On Mon, Aug 26, 2019 at 3:53 PM Predrag Punosevac
<predragp at andrew.cmu.edu> wrote:
>
> Thanks for the report! I can reproduce the problem. This is typically
> due to the upgraded kernel for which nvidia module has to be rebuilt
> (If i reboot the machine that will happen automatically). Welcome to
> the wonderful world of proprietary software. So this is the notice to
> the rest of the group that I will be rebooting non functional GPU
> nodes in next hour or two until things are fixed.
>
> Cheers,
> Predrag
>
> On Mon, Aug 26, 2019 at 3:47 PM Abhay Gupta <abhayg at andrew.cmu.edu> wrote:
> >
> > Hi Predrag,
> >
> > I was trying to access GPUs on servers 11/13/14 got this error while running 'nvidia-smi':
> > 'Failed to initialize NVML: Driver/library version mismatch'
> >
> > I think either the driver or the library needs to be updated for Nvidia drivers on these servers. Can you please have a look into this? Thanks.
> >
> > --
> > Regards,
> > Abhay Gupta