Naive Tensorflow/GPU1 question

[Predrag Punosevac]

Kirthevasan Kandasamy <kandasamy at cmu.edu> wrote:

> Hi Predrag,
> 
> I am re-running a tensorflow project on GPU1 - I haven't touched it in 4/5
> months, and the last time I ran it it worked fine, but when I try now I
> seem to be getting the following error.
> 

This is the first time I hear about it. I was under impression that GPU
nodes were usable.  I am redirecting your e-mail to users at autonlab.org
in the hope that somebody who is using TensorFlow on the regular basis
can be of more help.

Predrag




> Can you please tell me what the issue might be or direct me to someone who
> might know?
> 
> This is for the NIPS deadline, so I would appreciate a quick response.
> 
> thanks,
> Samy
> 
> 
> I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with
> properties:
> name: Tesla K80
> major: 3 minor: 7 memoryClockRate (GHz) 0.8235
> pciBusID 0000:05:00.0
> Total memory: 11.17GiB
> Free memory: 11.11GiB
> I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0
> I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0:   Y
> I tensorflow/core/common_runtime/gpu/gpu_device.cc:838] Creating TensorFlow
> device (/gpu:0) -> (device: 0, name: Tesla K80, pci bus id: 0000:05:00.0)
> E tensorflow/stream_executor/cuda/cuda_dnn.cc:347] Loaded runtime CuDNN
> library: 4007 (compatibility version 4000) but source was compiled with
> 5103 (compatibility version 5100).  If using a binary install, upgrade your
> CuDNN library to match.  If building from sources, make sure the library
> loaded at runtime matches a compatible version specified during compile
> configuration.
> F tensorflow/core/kernels/conv_ops.cc:457] Check failed:
> stream->parent()->GetConvolveAlgorithms(&algorithms)
> run_resnet.sh: line 49: 22665 Aborted                 (core dumped)
> CUDA_VISIBLE_DEVICES=$GPU python ../resnettf/resnet_main.py --data_dir
> $DATA_DIR --max_batch_iters $NUM_ITERS --report_results_every
> $REPORT_RESULTS_EVERY --log_root $LOG_ROOT --dataset $DATASET --num_gpus 1
> --save_model_dir $SAVE_MODEL_DIR --save_model_every $SAVE_MODEL_EVERY
> --skip_add_method $SKIP_ADD_METHOD --architecture $ARCHITECTURE --skip_size
> $SKIP_SIZE