RuntimeError: CUDNN_STATUS_INTERNAL_ERROR

[Chirag Nagpal]

This seems to be an issue with the way your code is utilizing the GPU RAM.
It seems as the Card is running out of RAM.

Please make sure you are setting the GPU card environment variable where
you want to execute your CUDA code using

$export CUDA_VISIBLE_DEVICES=#GPUNUMBER

Also try to calculate how much RAM is your card using. If its more than the
maximum RAM of the GPU, try to use smaller batch sizes.

Chirag




On Sun, Apr 22, 2018 at 8:06 PM, Predrag Punosevac <predragp at andrew.cmu.edu>
wrote:

> Siyu He <siyu.cosmo at gmail.com> wrote:
>
> > Dear all,
> >
> > Sorry to bother you on this. I am running some jobs on autonlab. It
> works all fine but all of a sudden I receive the following error:
> > RuntimeError: CUDNN_STATUS_INTERNAL_ERROR
> > Then all my jobs cannot be run even for those which are working well
> before.
> > Do you know why does this happen?
>
> This is a poor problem report. When was the last time program was
> working OK? When did you notice the problem. Any changes in your local
> path etc etc... What are you trying to do.
>
> Predrag
>
> >
> > Thanks!!
> > Siyu
> >
> >
>



-- 

*Chirag Nagpal* Graduate Student, Language Technologies Institute
School of Computer Science
Carnegie Mellon University
cs.cmu.edu/~chiragn
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/autonlab-users/attachments/20180423/cb551f49/attachment.html>