<div dir="ltr"><div dir="ltr"><div><div><div>Hi Predrag,<br><br></div>Is cuDNN properly installed?<br></div>I can't see it inside the /usr/local/cuda.<br><br></div>Also <b>import tensorflow</b> provides:</div><div dir="ltr"><b><i><br></i></b></div><div dir="ltr"><b><i>ImportError: libcudnn.so.7: cannot open shared object file: No such file or directory</i></b><br></div><div dir="ltr"><br></div><div>Thanks,<br></div><div>Yotam.<br></div></div><br><div class="gmail_quote"><div dir="ltr">On Mon, Jan 7, 2019 at 10:43 PM Predrag Punosevac <<a href="mailto:predragp@andrew.cmu.edu">predragp@andrew.cmu.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Ok. I found one problem. CUDA 10 was not properly installed on GPU10<br>
due to the dependency problems. I had to disable rpmfusion repos<br>
(both free and non-free) which I considered safe in the past. Now CUDA<br>
10 is installed from NVidia repo and is in /usr/local and<br>
/usr/local/cuda is the symbolic link to actual /usr/local/cuda-10.0<br>
folder. Please try now.<br>
<br>
Predrag<br>
<br>
On Mon, Jan 7, 2019 at 2:28 PM Yotam Hechtlinger<br>
<<a href="mailto:yhechtli@andrew.cmu.edu" target="_blank">yhechtli@andrew.cmu.edu</a>> wrote:<br>
><br>
> Hi Predrag,<br>
><br>
> With GPU10 the problem is probably because LD_LIBRARY_PATH directs to /usr/local/cuda/lib64 but that's not where CUDA is installed (where is it?).<br>
><br>
> Yotam.<br>
><br>
><br>
> On Mon, Jan 7, 2019 at 9:00 PM Predrag Punosevac <<a href="mailto:predragp@andrew.cmu.edu" target="_blank">predragp@andrew.cmu.edu</a>> wrote:<br>
>><br>
>> Yotam,<br>
>><br>
>> Thank you so much for this report! I am CC-ing <a href="mailto:users@autonlab.org" target="_blank">users@autonlab.org</a> so<br>
>> that everyone is on the same page. Could you please work with me on<br>
>> this one? Let's try to fix GPU10 first. GPU10 was recently<br>
>> provisioned. It has three (one was DoA) GeForce 1080Ti. I am running<br>
>> the latest NVIDIA-Linux-x86_64-410.78 driver and the latest<br>
>> cuda-10.0.130-1. You have two versions of Python. /opt/rh/rh-python36<br>
>> will give you the latest 3.6.7. While /opt/miniconda3 will install<br>
>> python-3.7.2. Once we fix GPU10 we will move to other machines. Note<br>
>> that other machines are still running older version of NVidia driver<br>
>> and CUDA-9.2. I have changed nothing on them so whatever is broken it<br>
>> is broken upstream (Python,TensorFlow, NVidia, or CUDA).<br>
>><br>
>> Please keep CC-ing users to this discussion so that people know what<br>
>> is going on.<br>
>><br>
>> Predrag<br>
>><br>
>><br>
>> On Mon, Jan 7, 2019 at 8:02 AM Yotam Hechtlinger<br>
>> <<a href="mailto:yhechtli@andrew.cmu.edu" target="_blank">yhechtli@andrew.cmu.edu</a>> wrote:<br>
>> ><br>
>> > Hi Predrag,<br>
>> ><br>
>> > There might be some CUDA problem on GPU 5,6 & 10.<br>
>> > I get the following message when I try to import tensorflow:<br>
>> ><br>
>> ><br>
>> ><br>
>> > >>> import tensorflow<br>
>> > Traceback (most recent call last):<br>
>> > File "/zfsauton/home/yhechtli/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in <module><br>
>> > from tensorflow.python.pywrap_tensorflow_internal import *<br>
>> > File "/zfsauton/home/yhechtli/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module><br>
>> > _pywrap_tensorflow_internal = swig_import_helper()<br>
>> > File "/zfsauton/home/yhechtli/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper<br>
>> > _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)<br>
>> > File "/zfsauton/home/yhechtli/anaconda3/lib/python3.6/imp.py", line 243, in load_module<br>
>> > return load_dynamic(name, filename, file)<br>
>> > File "/zfsauton/home/yhechtli/anaconda3/lib/python3.6/imp.py", line 343, in load_dynamic<br>
>> > return _load(spec)<br>
>> > ImportError: libcublas.so.9.0: cannot open shared object file: No such file or directory<br>
>> ><br>
>> > During handling of the above exception, another exception occurred:<br>
>> ><br>
>> > Traceback (most recent call last):<br>
>> > File "<stdin>", line 1, in <module><br>
>> > File "/zfsauton/home/yhechtli/anaconda3/lib/python3.6/site-packages/tensorflow/__init__.py", line 24, in <module><br>
>> > from tensorflow.python import pywrap_tensorflow # pylint: disable=unused-import<br>
>> > File "/zfsauton/home/yhechtli/anaconda3/lib/python3.6/site-packages/tensorflow/python/__init__.py", line 49, in <module><br>
>> > from tensorflow.python import pywrap_tensorflow<br>
>> > File "/zfsauton/home/yhechtli/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 74, in <module><br>
>> > raise ImportError(msg)<br>
>> > ImportError: Traceback (most recent call last):<br>
>> > File "/zfsauton/home/yhechtli/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow.py", line 58, in <module><br>
>> > from tensorflow.python.pywrap_tensorflow_internal import *<br>
>> > File "/zfsauton/home/yhechtli/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module><br>
>> > _pywrap_tensorflow_internal = swig_import_helper()<br>
>> > File "/zfsauton/home/yhechtli/anaconda3/lib/python3.6/site-packages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper<br>
>> > _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)<br>
>> > File "/zfsauton/home/yhechtli/anaconda3/lib/python3.6/imp.py", line 243, in load_module<br>
>> > return load_dynamic(name, filename, file)<br>
>> > File "/zfsauton/home/yhechtli/anaconda3/lib/python3.6/imp.py", line 343, in load_dynamic<br>
>> > return _load(spec)<br>
>> > ImportError: libcublas.so.9.0: cannot open shared object file: No such file or directory<br>
>> ><br>
>> ><br>
>> > Failed to load the native TensorFlow runtime.<br>
>> ><br>
>> > See <a href="https://www.tensorflow.org/install/errors" rel="noreferrer" target="_blank">https://www.tensorflow.org/install/errors</a><br>
>> ><br>
>> > for some common reasons and solutions. Include the entire stack trace<br>
>> > above this error message when asking for help.<br>
>> ><br>
</blockquote></div>