<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.m-3845574559137153614m6968877778934746229im
{mso-style-name:m_-3845574559137153614m_6968877778934746229im;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Here is my related env variables:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">CUDA_HOME=/zfsauton/home/manzilz/local/cuda-9.0/<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">LD_LIBRARY_PATH=/zfsauton/home/manzilz/local/lib64:/zfsauton/home/manzilz/local/lib:/zfsauton/home/manzilz/local/cuda-9.0/lib64:/usr/local/cuda/lib64:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">PATH=/zfsauton/home/manzilz/local/bin:/zfsauton/home/manzilz/.local/bin:/zfsauton/home/manzilz/local/cuda-9.0/bin:/usr/local/cuda/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">C_INCLUDE_PATH=/zfsauton/home/manzilz/local/include:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Biswajit Paria <bparia@cs.cmu.edu>
<br>
<b>Sent:</b> Wednesday, September 05, 2018 5:29 PM<br>
<b>To:</b> Yichong Xu <yichongx@cs.cmu.edu><br>
<b>Cc:</b> Biswajit Paria <bparia@cs.cmu.edu>; eyolcu@cs.cmu.edu; Predrag Punosevac <predragp@andrew.cmu.edu>; Manzil Zaheer <manzil@cmu.edu>; users@autonlab.org<br>
<b>Subject:</b> Re: PyTorch problem<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">If the CUDA examples work for anyone, can they share their PATH and LD_LIBRARY_PATH variables?<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Wed, Sep 5, 2018 at 5:27 PM Yichong Xu <<a href="mailto:yichongx@cs.cmu.edu">yichongx@cs.cmu.edu</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal">I think with Biswajit’s and my problem with cuda, we should isolate the problem with just CUDA (and drivers) instead of wandering around python or pytorch.
<o:p></o:p></p>
<div>
<p class="MsoNormal">Predrag can you test the CUDA examples? I sort of agree with Manzil that this might be a user account problem.<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><i><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif;color:black">Thanks,</span></i><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><i><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif;color:black">Yichong</span></i><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Sep 5, 2018, at 5:14 PM, Biswajit Paria <<a href="mailto:bparia@cs.cmu.edu" target="_blank">bparia@cs.cmu.edu</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<p class="MsoNormal">I just tried Yichong's way of testing cuBLAS, and get the same error as earlier:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<blockquote style="margin-left:30.0pt;margin-right:0in">
<div>
<div>
<div>
<p class="MsoNormal">[Matrix Multiply CUBLAS] - Starting...<o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal">GPU Device 0: "TITAN Xp" with compute capability 6.1<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">MatrixA(640,480), MatrixB(480,320), MatrixC(640,320)<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal">CUDA error at matrixMulCUBLAS.cpp:275 code=1(CUBLAS_STATUS_NOT_INITIALIZED) "cublasCreate(&handle)" <o:p></o:p></p>
</div>
</div>
</blockquote>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">So I believe it is not a conda error. I also tried removing .nv, doesn't help either. Maybe someone can share the PATH env variable?<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Wed, Sep 5, 2018 at 5:08 PM Emre Yolcu <<a href="mailto:eyolcu@cs.cmu.edu" target="_blank">eyolcu@cs.cmu.edu</a>> wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal">Manzil, could you share your `conda env export` (or equivalent) output for the environment you use for pytorch? It's still not working for me after reboot, maybe I can try replicating your exact setup and try with that.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Emre<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Wed, Sep 5, 2018 at 4:56 PM, Predrag Punosevac <<a href="mailto:predragp@andrew.cmu.edu" target="_blank">predragp@andrew.cmu.edu</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal">Manzil Zaheer <<a href="mailto:manzil@cmu.edu" target="_blank">manzil@cmu.edu</a>> wrote:<br>
<br>
> It was working me before reboot as well. PyTorch does work on all<br>
> nodes for me.<br>
<br>
Aha! Gotcha. <br>
<br>
> <br>
> I am trying to say is that i think it is not issue at system level but<br>
> at user account level. I might be wrong though.<br>
<br>
That was my hunch as well. They were trying to convince me in a 150<br>
e-mails chain over the weekend that pytorch was broken when I replaced a<br>
failed HDD on the main file server. That didn't make any sense.<br>
<br>
Could you please share your binaries and setup with outher pytorch<br>
users?<br>
<br>
Cheers,<br>
Predrag<br>
<br>
> <br>
> <br>
> -------- Original message --------<br>
> From: Predrag Punosevac <<a href="mailto:predragp@andrew.cmu.edu" target="_blank">predragp@andrew.cmu.edu</a>><br>
> Date: 9/5/18 4:44 PM (GMT-05:00)<br>
> To: Manzil Zaheer <<a href="mailto:manzil@cmu.edu" target="_blank">manzil@cmu.edu</a>><br>
> Cc: Biswajit Paria <<a href="mailto:bparia@cs.cmu.edu" target="_blank">bparia@cs.cmu.edu</a>>, Yichong Xu <<a href="mailto:yichongx@cs.cmu.edu" target="_blank">yichongx@cs.cmu.edu</a>>, Emre Yolcu <<a href="mailto:eyolcu@cs.cmu.edu" target="_blank">eyolcu@cs.cmu.edu</a>>,
<a href="mailto:users@autonlab.org" target="_blank">users@autonlab.org</a><br>
> Subject: Re: PyTorch problem<br>
> <br>
> Should I go ahead and reboot all GPU computing nodes? Can somebody else confirm that a reboot fixes the issue?<br>
> <br>
> Predrag<br>
> <br>
> On Wed, Sep 5, 2018 at 4:42 PM, Manzil Zaheer <<a href="mailto:manzil@cmu.edu" target="_blank">manzil@cmu.edu</a><mailto:<a href="mailto:manzil@cmu.edu" target="_blank">manzil@cmu.edu</a>>> wrote:<br>
> It does work for me and my friends<br>
> <br>
> <br>
> <br>
> <br>
> -------- Original message --------<br>
> From: Predrag Punosevac <<a href="mailto:predragp@andrew.cmu.edu" target="_blank">predragp@andrew.cmu.edu</a><mailto:<a href="mailto:predragp@andrew.cmu.edu" target="_blank">predragp@andrew.cmu.edu</a>>><br>
> Date: 9/5/18 4:40 PM (GMT-05:00)<br>
> To: Biswajit Paria <<a href="mailto:bparia@cs.cmu.edu" target="_blank">bparia@cs.cmu.edu</a><mailto:<a href="mailto:bparia@cs.cmu.edu" target="_blank">bparia@cs.cmu.edu</a>>><br>
> Cc: Manzil Zaheer <<a href="mailto:manzil@cmu.edu" target="_blank">manzil@cmu.edu</a><mailto:<a href="mailto:manzil@cmu.edu" target="_blank">manzil@cmu.edu</a>>>, Yichong Xu <<a href="mailto:yichongx@cs.cmu.edu" target="_blank">yichongx@cs.cmu.edu</a><mailto:<a href="mailto:yichongx@cs.cmu.edu" target="_blank">yichongx@cs.cmu.edu</a>>>,
Emre Yolcu <<a href="mailto:eyolcu@cs.cmu.edu" target="_blank">eyolcu@cs.cmu.edu</a><mailto:<a href="mailto:eyolcu@cs.cmu.edu" target="_blank">eyolcu@cs.cmu.edu</a>>>,
<a href="mailto:users@autonlab.org" target="_blank">users@autonlab.org</a><mailto:<a href="mailto:users@autonlab.org" target="_blank">users@autonlab.org</a>><br>
> Subject: Re: PyTorch problem<br>
> <br>
> I just rebooted GPU8. All packages are up to date. NVidia driver appears to be working properly and I can do GPU computations from MATLAB. Let's try now to get pytorch working on GPU8.<br>
> <br>
> Predrag<br>
> <br>
> On Wed, Sep 5, 2018 at 12:19 AM, Biswajit Paria <<a href="mailto:bparia@cs.cmu.edu" target="_blank">bparia@cs.cmu.edu</a><mailto:<a href="mailto:bparia@cs.cmu.edu" target="_blank">bparia@cs.cmu.edu</a>>> wrote:<br>
> I am facing a similar error on all GPU machines. Did someone find a solution yet?<br>
> <br>
> <br>
> 2018-09-05 00:27:41.546064: E tensorflow/stream_executor/cuda/<a href="http://cuda_blas.cc:459" target="_blank">cuda_blas.cc:459</a>] failed to create cublas handle: CUBLAS_STATUS_NOT_INITIALIZED<br>
> <br>
> On Tue, Sep 4, 2018 at 10:03 PM Manzil Zaheer <<a href="mailto:manzil@cmu.edu" target="_blank">manzil@cmu.edu</a><mailto:<a href="mailto:manzil@cmu.edu" target="_blank">manzil@cmu.edu</a>>> wrote:<br>
> Hi Yichong<br>
> <br>
> Yes I am able to run TF and PyTorch on these machines. Recently someone else also had similar issue, but it got fixed by reinstalling some local packages.<br>
> <br>
> Thanks,<br>
> Manzil<br>
> <br>
> <br>
> -------- Original message --------<br>
> From: Yichong Xu <<a href="mailto:yichongx@cs.cmu.edu" target="_blank">yichongx@cs.cmu.edu</a><mailto:<a href="mailto:yichongx@cs.cmu.edu" target="_blank">yichongx@cs.cmu.edu</a>>><br>
> Date: 9/4/18 9:58 PM (GMT-05:00)<br>
> To: Emre Yolcu <<a href="mailto:eyolcu@cs.cmu.edu" target="_blank">eyolcu@cs.cmu.edu</a><mailto:<a href="mailto:eyolcu@cs.cmu.edu" target="_blank">eyolcu@cs.cmu.edu</a>>>, Predrag Punosevac <<a href="mailto:predragp@andrew.cmu.edu" target="_blank">predragp@andrew.cmu.edu</a><mailto:<a href="mailto:predragp@andrew.cmu.edu" target="_blank">predragp@andrew.cmu.edu</a>>><br>
> Cc: <a href="mailto:users@autonlab.org" target="_blank">users@autonlab.org</a><mailto:<a href="mailto:users@autonlab.org" target="_blank">users@autonlab.org</a>><br>
> Subject: Re: PyTorch problem<br>
> <br>
> Just wondering - can Tensorflow run well on these machines? I hope someone to confirm about this so that we can isolate the problem.<br>
> OK so here?s a further test: I tried running the cuda examples from the cuda installation (in /usr/local/cuda/sample), on gpu2 in my scratch directory. Simple jobs like deviceQuery succeeds, but simpleCUBLAS failed:<br>
<span class="m-3845574559137153614m6968877778934746229im">> yichongx@gpu2$ cd /home/scratch/yichongx/</span><br>
<span class="m-3845574559137153614m6968877778934746229im">> yichongx@gpu2$ cd</span><br>
<span class="m-3845574559137153614m6968877778934746229im">> 0_Simple/ 2_Graphics/ 4_Finance/ 6_Advanced/ bin/ conda/</span><br>
<span class="m-3845574559137153614m6968877778934746229im">> 1_Utilities/ 3_Imaging/ 5_Simulations/ 7_CUDALibraries/ common/ miniconda3/</span><br>
<span class="m-3845574559137153614m6968877778934746229im">> yichongx@gpu2$ cd 7_CUDALibraries/</span><br>
<span class="m-3845574559137153614m6968877778934746229im">> yichongx@gpu2$ cd simpleCUBLAS</span><br>
<span class="m-3845574559137153614m6968877778934746229im">> yichongx@gpu2$ CUDA_VISIBLE_DEVICES=3 ./simpleCUBLAS</span><br>
<span class="m-3845574559137153614m6968877778934746229im">> GPU Device 0: "TITAN X (Pascal)" with compute capability 6.1</span><br>
<span class="m-3845574559137153614m6968877778934746229im">> </span><br>
<span class="m-3845574559137153614m6968877778934746229im">> simpleCUBLAS test running..</span><br>
<span class="m-3845574559137153614m6968877778934746229im">> !!!! CUBLAS initialization error</span><br>
<span class="m-3845574559137153614m6968877778934746229im">> yichongx@gpu2$</span><br>
<span class="m-3845574559137153614m6968877778934746229im">> </span><br>
<span class="m-3845574559137153614m6968877778934746229im">> </span><br>
<span class="m-3845574559137153614m6968877778934746229im">> This is also consistent with our previous errors from pytorch, which say cublas library not initialized.</span><br>
<span class="m-3845574559137153614m6968877778934746229im">> </span><br>
<span class="m-3845574559137153614m6968877778934746229im">> So this means at least there is some problem with CUBLAS on gpu2. This post suggests that using sudo can resolve this problem, and this is probably because of some permission problems on CUBLAS libraries:</span><br>
<span class="m-3845574559137153614m6968877778934746229im">> <a href="https://devtalk.nvidia.com/default/topic/1027602/cuda-setup-and-installation/cublas-libraries-with-incorrect-permissions/" target="_blank">
https://devtalk.nvidia.com/default/topic/1027602/cuda-setup-and-installation/cublas-libraries-with-incorrect-permissions/</a></span><br>
<span class="m-3845574559137153614m6968877778934746229im">> @Predrag: Can you try running the simpleCUBLAS example from the CUDA library, with and without root privilege? I think that might be something that you are more familiar with. Thank you very much!</span><br>
<span class="m-3845574559137153614m6968877778934746229im">> </span><br>
<span class="m-3845574559137153614m6968877778934746229im">> </span><br>
<span class="m-3845574559137153614m6968877778934746229im">> Thanks,</span><br>
<span class="m-3845574559137153614m6968877778934746229im">> Yichong</span><br>
<span class="m-3845574559137153614m6968877778934746229im">> </span><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal">> On Sep 4, 2018, at 3:18 PM, Emre Yolcu <<a href="mailto:eyolcu@cs.cmu.edu" target="_blank">eyolcu@cs.cmu.edu</a><mailto:<a href="mailto:eyolcu@cs.cmu.edu" target="_blank">eyolcu@cs.cmu.edu</a>>> wrote:<br>
> <br>
> Hi,<br>
> <br>
> We are trying to troubleshoot the PyTorch issue with Predrag and were wondering:<br>
> <br>
> Is anybody able to run PyTorch GPU models on gpu1-9? If you can, we would appreciate if you can respond.<br>
> <br>
> Also, is it a problem for anyone if gpu8 is rebooted today?<br>
> <br>
> Thanks,<br>
> <br>
> Emre<br>
> <br>
> <br>
> <br>
> --<br>
> Biswajit Paria<br>
> PhD in ML @ CMU<br>
> <br>
> <o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">Biswajit Paria <o:p></o:p></p>
<div>
<p class="MsoNormal">PhD in ML @ CMU<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">Biswajit Paria<o:p></o:p></p>
<div>
<p class="MsoNormal">PhD in ML @ CMU<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>