<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta content="text/html; charset=utf-8">
</head>
<body>
<div>It does work for me and my friends</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>-------- Original message --------</div>
<div>From: Predrag Punosevac <predragp@andrew.cmu.edu> </div>
<div>Date: 9/5/18 4:40 PM (GMT-05:00) </div>
<div>To: Biswajit Paria <bparia@cs.cmu.edu> </div>
<div>Cc: Manzil Zaheer <manzil@cmu.edu>, Yichong Xu <yichongx@cs.cmu.edu>, Emre Yolcu <eyolcu@cs.cmu.edu>, users@autonlab.org
</div>
<div>Subject: Re: PyTorch problem </div>
<div><br>
</div>
<div>
<div dir="ltr">I just rebooted GPU8. All packages are up to date. NVidia driver appears to be working properly and I can do GPU computations from MATLAB. Let's try now to get pytorch working on GPU8.
<div><br>
</div>
<div>Predrag</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, Sep 5, 2018 at 12:19 AM, Biswajit Paria <span dir="ltr">
<<a href="mailto:bparia@cs.cmu.edu" target="_blank">bparia@cs.cmu.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
<div dir="ltr">I am facing a similar error on all GPU machines. Did someone find a solution yet?
<div><br>
</div>
<div>
<p class="m_7799108538010190598gmail-p1" style="margin:0px; font-size:13px; line-height:normal; font-family:Menlo; color:rgb(244,244,244)">
<span class="m_7799108538010190598gmail-s1" style="">2018-09-05 00:27:41.546064: E tensorflow/stream_executor/<wbr>cuda/cuda_blas.cc:459] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED</span></p>
</div>
</div>
<div class="HOEnZb">
<div class="h5"><br>
<div class="gmail_quote">
<div dir="ltr">On Tue, Sep 4, 2018 at 10:03 PM Manzil Zaheer <<a href="mailto:manzil@cmu.edu" target="_blank">manzil@cmu.edu</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
<div style="word-wrap:break-word; line-break:after-white-space">Hi Yichong
<div><br>
</div>
<div>Yes I am able to run TF and PyTorch on these machines. Recently someone else also had similar issue, but it got fixed by reinstalling some local packages.</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Manzil </div>
<div><br>
</div>
<div><br>
</div>
<div>-------- Original message --------</div>
<div>From: Yichong Xu <<a href="mailto:yichongx@cs.cmu.edu" target="_blank">yichongx@cs.cmu.edu</a>>
</div>
<div>Date: 9/4/18 9:58 PM (GMT-05:00) </div>
<div>To: Emre Yolcu <<a href="mailto:eyolcu@cs.cmu.edu" target="_blank">eyolcu@cs.cmu.edu</a>>, Predrag Punosevac <<a href="mailto:predragp@andrew.cmu.edu" target="_blank">predragp@andrew.cmu.edu</a>>
</div>
<div>Cc: <a href="mailto:users@autonlab.org" target="_blank">users@autonlab.org</a>
</div>
<div>Subject: Re: PyTorch problem </div>
<div><br>
</div>
<div>
<div>Just wondering - can Tensorflow run well on these machines? I hope someone to confirm about this so that we can isolate the problem.</div>
OK so here’s a further test: I tried running the cuda examples from the cuda installation (in /usr/local/cuda/sample), on gpu2 in my scratch directory. Simple jobs like deviceQuery succeeds, but simpleCUBLAS failed:
<div>
<div style="margin:0px; font-size:11px; line-height:normal; font-family:Menlo"><span>yichongx@gpu2$ cd /home/scratch/yichongx/</span></div>
<div style="margin:0px; font-size:11px; line-height:normal; font-family:Menlo"><span>yichongx@gpu2$ cd </span></div>
<div style="margin:0px; font-size:11px; line-height:normal; font-family:Menlo"><span>0_Simple/ 2_Graphics/ 4_Finance/ 6_Advanced/ bin/ conda/ </span></div>
<div style="margin:0px; font-size:11px; line-height:normal; font-family:Menlo"><span>1_Utilities/ 3_Imaging/ 5_Simulations/ 7_CUDALibraries/ common/ miniconda3/ </span></div>
<div style="margin:0px; font-size:11px; line-height:normal; font-family:Menlo"><span>yichongx@gpu2$ cd 7_CUDALibraries/</span></div>
<div style="margin:0px; font-size:11px; line-height:normal; font-family:Menlo"><span>yichongx@gpu2$ cd simpleCUBLAS</span></div>
<div style="margin:0px; font-size:11px; line-height:normal; font-family:Menlo"><span>yichongx@gpu2$ CUDA_VISIBLE_DEVICES=3 ./simpleCUBLAS</span></div>
<div style="margin:0px; font-size:11px; line-height:normal; font-family:Menlo"><span>GPU Device 0: "TITAN X (Pascal)" with compute capability 6.1</span></div>
<div style="margin:0px; font-size:11px; line-height:normal; font-family:Menlo; min-height:13px">
<span></span><br>
</div>
<div style="margin:0px; font-size:11px; line-height:normal; font-family:Menlo"><span>simpleCUBLAS test running..</span></div>
<div style="margin:0px; font-size:11px; line-height:normal; font-family:Menlo"><span>!!!! CUBLAS initialization error</span></div>
<div style="margin:0px; font-size:11px; line-height:normal; font-family:Menlo"><span>yichongx@gpu2$ </span></div>
<div>
<div><br>
<div style="color:rgb(0,0,0); font-family:Helvetica; font-size:12px; font-style:normal; font-weight:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px">
<br>
</div>
</div>
<div>This is also consistent with our previous errors from pytorch, which say cublas library not initialized.</div>
<div><br>
</div>
<div>So this means at least there is some problem with CUBLAS on gpu2. This post suggests that using sudo can resolve this problem, and this is probably because of some permission problems on CUBLAS libraries:</div>
<div><a href="https://devtalk.nvidia.com/default/topic/1027602/cuda-setup-and-installation/cublas-libraries-with-incorrect-permissions/" target="_blank">https://devtalk.nvidia.com/<wbr>default/topic/1027602/cuda-<wbr>setup-and-installation/cublas-<wbr>libraries-with-incorrect-<wbr>permissions/</a></div>
<div>@Predrag: Can you try running the simpleCUBLAS example from the CUDA library, with and without root privilege? I think that might be something that you are more familiar with. Thank you very much!</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div><i>Thanks,</i></div>
<div><i>Yichong</i></div>
<div><i><br>
</i></div>
<blockquote type="cite">
<div>On Sep 4, 2018, at 3:18 PM, Emre Yolcu <<a href="mailto:eyolcu@cs.cmu.edu" target="_blank">eyolcu@cs.cmu.edu</a>> wrote:</div>
<br class="m_7799108538010190598m_-2294202965647517299Apple-interchange-newline">
<div>
<div dir="ltr">
<div>Hi,</div>
<div><br>
</div>
<div>We are trying to troubleshoot the PyTorch issue with Predrag and were wondering:</div>
<div><br>
</div>
<div>Is anybody able to run PyTorch GPU models on gpu1-9? If you can, we would appreciate if you can respond.</div>
<div><br>
</div>
<div>Also, is it a problem for anyone if gpu8 is rebooted today?</div>
<div><br>
</div>
<div>Thanks,</div>
<div><br>
</div>
<div>Emre<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
</div>
</div>
<span class="HOEnZb"><font color="#888888">-- <br>
<div dir="ltr" class="m_7799108538010190598gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr">Biswajit Paria
<div>PhD in ML @ CMU</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</font></span></blockquote>
</div>
<br>
</div>
</div>
</body>
</html>