RuntimeError: CUDNN_STATUS_INTERNAL_ERROR

[Siyu He]

Hi Chirag,

Thanks for your reply! It seems it’s now working. 
But I did use "export CUDA_VISIBLE_DEVICES=#GPUNUMBER” before to specify what GPU I want to use and check the available memory. I am still a little confused why I had the issue on Sunday. But anyway it’s working. Yay!

Thanks,
Siyu

> On Apr 23, 2018, at 1:39 PM, Chirag Nagpal <chiragn at cs.cmu.edu> wrote:
> 
> This seems to be an issue with the way your code is utilizing the GPU RAM. It seems as the Card is running out of RAM.
> 
> Please make sure you are setting the GPU card environment variable where you want to execute your CUDA code using 
> 
> $export CUDA_VISIBLE_DEVICES=#GPUNUMBER
> 
> Also try to calculate how much RAM is your card using. If its more than the maximum RAM of the GPU, try to use smaller batch sizes.
> 
> Chirag
> 
> 
> 
> 
> On Sun, Apr 22, 2018 at 8:06 PM, Predrag Punosevac <predragp at andrew.cmu.edu <mailto:predragp at andrew.cmu.edu>> wrote:
> Siyu He <siyu.cosmo at gmail.com <mailto:siyu.cosmo at gmail.com>> wrote:
> 
> > Dear all,
> > 
> > Sorry to bother you on this. I am running some jobs on autonlab. It works all fine but all of a sudden I receive the following error:
> > RuntimeError: CUDNN_STATUS_INTERNAL_ERROR
> > Then all my jobs cannot be run even for those which are working well before. 
> > Do you know why does this happen? 
> 
> This is a poor problem report. When was the last time program was
> working OK? When did you notice the problem. Any changes in your local
> path etc etc... What are you trying to do.
> 
> Predrag
> 
> > 
> > Thanks!!
> > Siyu
> > 
> > 
> 
> 
> 
> -- 
> Chirag Nagpal
> Graduate Student, Language Technologies Institute
> School of Computer Science
> Carnegie Mellon University
> cs.cmu.edu/~chiragn <http://cs.cmu.edu/~chiragn>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/autonlab-users/attachments/20180423/9c3caec5/attachment-0001.html>