<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Dear Predrag,<div class=""><br class=""></div><div class="">Thanks for replying!</div><div class="">I did this and it shows an error message: <span style="color: rgb(244, 244, 244); font-family: Monaco; font-size: 10px; background-color: rgba(0, 0, 0, 0.85098);" class="">Operation was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/cpu:0 ].</span></div><div class="">Below is the full message:</div><div class=""><div style="margin: 0px; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Traceback (most recent call last):</span></div><div style="margin: 0px; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> File "test.py", line 8, in <module></span></div><div style="margin: 0px; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> print (sess.run(c))</span></div><div style="margin: 0px; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> File "/zfsauton/home/siyuh/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 895, in run</span></div><div style="margin: 0px; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> run_metadata_ptr)</span></div><div style="margin: 0px; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> File "/zfsauton/home/siyuh/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1124, in _run</span></div><div style="margin: 0px; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> feed_dict_tensor, options, run_metadata)</span></div><div style="margin: 0px; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> File "/zfsauton/home/siyuh/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1321, in _do_run</span></div><div style="margin: 0px; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> options, run_metadata)</span></div><div style="margin: 0px; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> File "/zfsauton/home/siyuh/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1340, in _do_call</span></div><div style="margin: 0px; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> raise type(e)(node_def, op, message)</span></div><div style="margin: 0px; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation 'MatMul': Operation was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/cpu:0 ]. Make sure the device specification refers to a valid device.</span></div><div style="margin: 0px; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><span class="Apple-tab-span" style="white-space:pre"> </span> [[Node: MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/device:GPU:0"](a, b)]]</span></div><div style="margin: 0px; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098); min-height: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class=""></div><div style="margin: 0px; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">Caused by op u'MatMul', defined at:</span></div><div style="margin: 0px; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> File "test.py", line 5, in <module></span></div><div style="margin: 0px; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> c = tf.matmul(a, b)</span></div><div style="margin: 0px; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> File "/zfsauton/home/siyuh/.local/lib/python2.7/site-packages/tensorflow/python/ops/math_ops.py", line 1844, in matmul</span></div><div style="margin: 0px; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)</span></div><div style="margin: 0px; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> File "/zfsauton/home/siyuh/.local/lib/python2.7/site-packages/tensorflow/python/ops/gen_math_ops.py", line 1289, in _mat_mul</span></div><div style="margin: 0px; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> transpose_b=transpose_b, name=name)</span></div><div style="margin: 0px; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> File "/zfsauton/home/siyuh/.local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op</span></div><div style="margin: 0px; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> op_def=op_def)</span></div><div style="margin: 0px; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> File "/zfsauton/home/siyuh/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2630, in create_op</span></div><div style="margin: 0px; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> original_op=self._default_original_op, op_def=op_def)</span></div><div style="margin: 0px; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> File "/zfsauton/home/siyuh/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1204, in __init__</span></div><div style="margin: 0px; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""> self._traceback = self._graph._extract_stack() # pylint: disable=protected-access</span></div><div style="margin: 0px; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098); min-height: 14px;" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""></span><br class=""></div><div style="margin: 0px; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class="">InvalidArgumentError (see above for traceback): Cannot assign a device for operation 'MatMul': Operation was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/cpu:0 ]. Make sure the device specification refers to a valid device.</span></div><div style="margin: 0px; font-size: 10px; line-height: normal; font-family: Monaco; color: rgb(244, 244, 244); background-color: rgba(0, 0, 0, 0.85098);" class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><span class="Apple-tab-span" style="white-space:pre"> </span> [[Node: MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/device:GPU:0"](a, b)]]</span></div></div><div class=""><br class=""></div><div class="">Then I tried nvidia-smi but I find most of the GPU memory is free. </div><div class=""><br class=""></div><div class="">Thanks,</div><div class="">Siyu</div><div class=""><br class=""></div><div class=""><div><blockquote type="cite" class=""><div class="">On Oct 4, 2017, at 6:22 PM, Chirag Nagpal <<a href="mailto:chiragn@cs.cmu.edu" class="">chiragn@cs.cmu.edu</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Try running the following on one of the GPUs. It should multiply the matrices and give you an output, if it does means its working. If it doesn't mail whatever output you get (The entire output) back on this thread.<br class=""><br class=""><pre class="gmail-prettyprint gmail-lang-py gmail-prettyprinted"><code class=""><span class="gmail-kwd">import</span><span class="gmail-pln"> tensorflow </span><span class="gmail-kwd">as</span><span class="gmail-pln"> tf
</span><span class="gmail-kwd">with</span><span class="gmail-pln"> tf</span><span class="gmail-pun">.</span><span class="gmail-pln">device</span><span class="gmail-pun">(</span><span class="gmail-str">'/gpu:0'</span><span class="gmail-pun">):</span><span class="gmail-pln">
a </span><span class="gmail-pun">=</span><span class="gmail-pln"> tf</span><span class="gmail-pun">.</span><span class="gmail-pln">constant</span><span class="gmail-pun">([</span><span class="gmail-lit">1.0</span><span class="gmail-pun">,</span><span class="gmail-pln"> </span><span class="gmail-lit">2.0</span><span class="gmail-pun">,</span><span class="gmail-pln"> </span><span class="gmail-lit">3.0</span><span class="gmail-pun">,</span><span class="gmail-pln"> </span><span class="gmail-lit">4.0</span><span class="gmail-pun">,</span><span class="gmail-pln"> </span><span class="gmail-lit">5.0</span><span class="gmail-pun">,</span><span class="gmail-pln"> </span><span class="gmail-lit">6.0</span><span class="gmail-pun">],</span><span class="gmail-pln"> shape</span><span class="gmail-pun">=[</span><span class="gmail-lit">2</span><span class="gmail-pun">,</span><span class="gmail-pln"> </span><span class="gmail-lit">3</span><span class="gmail-pun">],</span><span class="gmail-pln"> name</span><span class="gmail-pun">=</span><span class="gmail-str">'a'</span><span class="gmail-pun">)</span><span class="gmail-pln">
b </span><span class="gmail-pun">=</span><span class="gmail-pln"> tf</span><span class="gmail-pun">.</span><span class="gmail-pln">constant</span><span class="gmail-pun">([</span><span class="gmail-lit">1.0</span><span class="gmail-pun">,</span><span class="gmail-pln"> </span><span class="gmail-lit">2.0</span><span class="gmail-pun">,</span><span class="gmail-pln"> </span><span class="gmail-lit">3.0</span><span class="gmail-pun">,</span><span class="gmail-pln"> </span><span class="gmail-lit">4.0</span><span class="gmail-pun">,</span><span class="gmail-pln"> </span><span class="gmail-lit">5.0</span><span class="gmail-pun">,</span><span class="gmail-pln"> </span><span class="gmail-lit">6.0</span><span class="gmail-pun">],</span><span class="gmail-pln"> shape</span><span class="gmail-pun">=[</span><span class="gmail-lit">3</span><span class="gmail-pun">,</span><span class="gmail-pln"> </span><span class="gmail-lit">2</span><span class="gmail-pun">],</span><span class="gmail-pln"> name</span><span class="gmail-pun">=</span><span class="gmail-str">'b'</span><span class="gmail-pun">)</span><span class="gmail-pln">
c </span><span class="gmail-pun">=</span><span class="gmail-pln"> tf</span><span class="gmail-pun">.</span><span class="gmail-pln">matmul</span><span class="gmail-pun">(</span><span class="gmail-pln">a</span><span class="gmail-pun">,</span><span class="gmail-pln"> b</span><span class="gmail-pun">)</span><span class="gmail-pln">
</span><span class="gmail-kwd">with</span><span class="gmail-pln"> tf</span><span class="gmail-pun">.</span><span class="gmail-typ">Session</span><span class="gmail-pun">()</span><span class="gmail-pln"> </span><span class="gmail-kwd">as</span><span class="gmail-pln"> sess</span><span class="gmail-pun">:</span><span class="gmail-pln">
</span><span class="gmail-kwd">print</span><span class="gmail-pln"> </span><span class="gmail-pun">(</span><span class="gmail-pln">sess</span><span class="gmail-pun">.</span><span class="gmail-pln">run</span><span class="gmail-pun">(</span><span class="gmail-pln">c</span><span class="gmail-pun">))<br class=""><br class=""></span></code></pre><pre class="gmail-prettyprint gmail-lang-py gmail-prettyprinted"><code class=""><span class="gmail-pun"><br class=""></span></code></pre><pre class="gmail-prettyprint gmail-lang-py gmail-prettyprinted"><code class=""><span class="gmail-pun"><br class=""></span></code></pre><pre class="gmail-prettyprint gmail-lang-py gmail-prettyprinted"><code class=""><span class="gmail-pun"><br class=""><br class=""><br class=""></span></code></pre></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Wed, Oct 4, 2017 at 9:15 PM, Predrag Punosevac <span dir="ltr" class=""><<a href="mailto:predragp@andrew.cmu.edu" target="_blank" class="">predragp@andrew.cmu.edu</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><a href="mailto:siyuh@andrew.cmu.edu" class="">siyuh@andrew.cmu.edu</a> wrote:<br class="">
<br class="">
> Dear Predrag,<br class="">
><br class="">
> Good evening!<br class="">
> I am now trying to run tensorflow on Autonlab, it seems I am always using CPU instead of GPU. Then I try to add tf.device(???gnu:0???) in my code, then it shows that the GPU is not available. May I ask if you have any idea on this?<br class="">
><br class="">
<br class="">
I don't user TensorFlow but I know for the fact that it works as<br class="">
expected on all 7 GPU servers. You should have sent the e-mail to the<br class="">
mailing list so that you can get help from other TensorFlow users.<br class="">
<br class="">
Predrag<br class="">
<br class="">
<br class="">
> Thanks,<br class="">
> Siyu<br class="">
</blockquote></div><br class=""><br clear="all" class=""><br class="">-- <br class=""><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><div class=""><div dir="ltr" class=""><span style="" class=""><span style="background-color:rgb(255,255,255)" class=""><span class=""><b class="">Chirag Nagpal<br class=""></b>
Graduate Student, Language Technologies Institute<br class="">
School of Computer Science<br class="">
Carnegie Mellon University<br class=""></span></span></span></div><div class=""><span style="" class=""><span style="background-color:rgb(255,255,255)" class=""><span class=""><a href="http://cs.cmu.edu/~chiragn" target="_blank" class="">cs.cmu.edu/~chiragn</a><br class=""></span></span></span></div></div></div></div></div></div>
</div>
</div></blockquote></div><br class=""></div></body></html>