Zombie processes running on GPU9 and GPU4

[Predrag Punosevac]

Hi Tanmay,

The only sure thing would be to reboot the machines. It is against my
religion to do that on Friday evening. Rebooting will likely brake
NVidia driver as the machines will boot into Springdale 7.7 instead of
current 7.6. I will need to get confirmation that these machines are
not usable  at least from a few other people who were recently using
them before I reschedule reboot.

Best,
Predrag

On Fri, Nov 1, 2019 at 9:41 PM Tanmay Agarwal <tanmaya at andrew.cmu.edu> wrote:
>
> Hi Predrag,
>
> Hope you are doing well. I wanted to bring to your attention that I have noticed some kind of a zombie process that are running on GPU9 (0) and GPU4 (0) which are not even visible in nvidia-smi and are consuming the entire gpu memory. Can you help us reset those gpus?
>
> Also, i have some runs running on these GPU machines, so "in case" you need to reboot the machine, kindly let me know before you have to reset the machines.
>
> P.S Attaching a screenshot of one of those GPU machines.
>
> Thanking you,
>
> Best Regards,
>
> Tanmay Agarwal | MSR Graduate Student
> Robotics Institute @ CMU
> mailto: tanmaya at andrew.cmu.edu