<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Hi Chirag,<div class=""><br class=""></div><div class="">Thanks for your reply! It seems it’s now working. </div><div class="">But I did use "export CUDA_VISIBLE_DEVICES=#<wbr class="">GPUNUMBER” before to specify what GPU I want to use and check the available memory. I am still a little confused why I had the issue on Sunday. But anyway it’s working. Yay!</div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Siyu</div><div class=""><br class=""></div><div class=""><div><blockquote type="cite" class=""><div class="">On Apr 23, 2018, at 1:39 PM, Chirag Nagpal <<a href="mailto:chiragn@cs.cmu.edu" class="">chiragn@cs.cmu.edu</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class="">This seems to be an issue with the way your code is utilizing the GPU RAM. It seems as the Card is running out of RAM.</div><div class=""><br class=""></div>Please make sure you are setting the GPU card environment variable where you want to execute your CUDA code using <div class=""><br class=""><div class="">$export CUDA_VISIBLE_DEVICES=#<wbr class="">GPUNUMBER</div><div class=""><br class=""></div><div class="">Also try to calculate how much RAM is your card using. If its more than the maximum RAM of the GPU, try to use smaller batch sizes.</div><div class=""><br class=""></div><div class="">Chirag</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Sun, Apr 22, 2018 at 8:06 PM, Predrag Punosevac <span dir="ltr" class=""><<a href="mailto:predragp@andrew.cmu.edu" target="_blank" class="">predragp@andrew.cmu.edu</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">Siyu He <<a href="mailto:siyu.cosmo@gmail.com" target="_blank" class="">siyu.cosmo@gmail.com</a>> wrote:<br class="">
<br class="">
> Dear all,<br class="">
> <br class="">
> Sorry to bother you on this. I am running some jobs on autonlab. It works all fine but all of a sudden I receive the following error:<br class="">
> RuntimeError: CUDNN_STATUS_INTERNAL_ERROR<br class="">
> Then all my jobs cannot be run even for those which are working well before. <br class="">
> Do you know why does this happen? <br class="">
<br class="">
</span>This is a poor problem report. When was the last time program was<br class="">
working OK? When did you notice the problem. Any changes in your local<br class="">
path etc etc... What are you trying to do.<br class="">
<br class="">
Predrag<br class="">
<br class="">
> <br class="">
> Thanks!!<br class="">
> Siyu<br class="">
> <br class="">
> <br class="">
</blockquote></div><br class=""><br clear="all" class=""><div class=""><br class=""></div>-- <br class=""><div class="m_-5039788142851939176gmail_signature" data-smartmail="gmail_signature"><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><span style="" class=""><span style="background-color:rgb(255,255,255)" class=""><span class=""><b class="">Chirag Nagpal<br class=""></b>
Graduate Student, Language Technologies Institute<br class="">
School of Computer Science<br class="">
Carnegie Mellon University<br class=""></span></span></span></div><div class=""><span style="" class=""><span style="background-color:rgb(255,255,255)" class=""><span class=""><a href="http://cs.cmu.edu/~chiragn" target="_blank" class="">cs.cmu.edu/~chiragn</a><br class=""></span></span></span></div></div></div></div></div></div>
</div></div>
</div></blockquote></div><br class=""></div></body></html>