<div dir="ltr">Hi all,<div><br></div><div>FYI, almost all of the dependencies built in this script are actually already installed on the server. This should work:</div><div><div><br></div><div><b>Base setup</b></div><div><font face="monospace, monospace">cd /home/scratch/$USER<br></font></div><div><font face="monospace, monospace">git clone <a href="https://github.com/BLVC/caffe" target="_blank">https://github.com/BLVC/caffe</a></font></div><div><font face="monospace, monospace">cd caffe</font></div><div><font face="monospace, monospace">cp Makefile.config.example Makefile.config</font></div><div><font face="monospace, monospace"># do configuration changes here</font></div><div><font face="monospace, monospace"># you'll need to do either openblas or atlas below</font></div><div><font face="monospace, monospace"># you probably want cudnn as well</font></div><div><font face="monospace, monospace">make -j23<br></font></div></div><div><font face="monospace, monospace">make -j23 test</font></div><div><font face="monospace, monospace">make runtest</font></div><div><br></div><div><br></div><div><b>Using openblas</b></div><div>This requires the openblas-devel package, which I just asked Predrag to install. After he does, it should work with just:</div><div><br></div><div><div><font face="monospace, monospace">sed -i 's/BLAS := atlas/BLAS := open/' Makefile.config</font></div></div><div><br></div><div><br></div><div><b>Using atlas</b></div><div>Openblas is faster than atlas, but you'll probably be doing everything on the GPU anyway, so it shouldn't really matter. Here's how to get Caffe to work with red hat distributions' nonstandard atlas layout. (For some reason, the Caffe developers rejected a pull request making this an option in Makefile.config....)</div><div><br></div><div><div><font face="monospace, monospace">sed -i 's/LIBRARIES += cblas atlas/LIBRARIES += tatlas/' Makefile</font></div><div><font face="monospace, monospace">sed -i 's|# BLAS_INCLUDE := /path/to/your/blas|BLAS_INCLUDE := /usr/include/atlas|' Makefile.config<br></font></div><div><font face="monospace, monospace">sed -i 's|# BLAS_LIB := /path/to/your/blas|BLAS_LIB := /usr/lib64/atlas|' Makefile.config</font></div></div><div><br></div><div><br></div><div><b>Using cudnn</b></div><div>If you want to use the cudnn library, which makes things faster (sometimes substantially):</div><div><br></div><div>Register as a developer at <a href="https://developer.nvidia.com/cudnn" target="_blank">https://developer.nvidia.com/cudnn</a>.</div><div>Wait a day for them to approve you. Or, just get the file from ~dsutherl/cudnn-7.0-linux-x64-v3.0-prod.tgz.</div><div><br></div><div><font face="monospace, monospace">cd /home/scratch/$USER</font></div><div><font face="monospace, monospace">tar xf ~dsutherl/cudnn-7.0-linux-x64-v3.0-prod.tgz</font></div><div><font face="monospace, monospace">mv cuda cudnn</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">cd caffe</font></div><div><font face="monospace, monospace">sed -i 's/# USE_CUDNN := 1/USE_CUDNN := 1/' Makefile.config</font></div><div><font face="monospace, monospace">perl -ipe 's|$| /home/scratch/'$USER'/cudnn/include| if /INCLUDE_DIRS :=/' Makefile.config</font></div><div><font face="monospace, monospace">perl -ipe 's|$| /home/scratch/'$USER'/cudnn/lib64| if /LIBRARY_DIRS :=/' Makefile.config<br></font></div><div><br></div><div>Then, whenever you run a caffe binary you'll have to have run</div><div><font face="monospace, monospace">export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/home/scratch/$USER/cudnn/lib64"</font></div><div>in that shell (or set it wherever).</div><div><br></div><div><b>Python interface</b></div><div><font face="monospace, monospace">make py</font> works. You'll need some requirements to import the package, though, because several of the system-installed packages are too old. One option is</div><div><br></div><div><font face="monospace, monospace">export PYTHONUSERBASE=/home/scratch/$USER/.local</font></div><div><font face="monospace, monospace">pip install --user -r python/requirements.txt</font></div><div><br></div><div>This will take a long time, since it has to compile scipy and a bunch of other stuff. You'll also need to set the <font face="monospace, monospace">PYTHONUSERBASE</font> variable before launching any python process that needs to use this.</div><div><br></div><div><b>Matlab interface</b></div><div><font face="monospace, monospace">sed -i 's|# MATLAB_DIR := /usr/local|MATLAB_DIR := /usr/local/MATLAB/R2015b|' Makefile.config</font></div><div><font face="monospace, monospace">make matcaffe</font></div><div><br></div><div>Haven't tested this at all, matlab is gross. :)</div><div><br></div><div><br></div><div>Hope that's helpful!</div><div>- Dougal</div><div><br></div><br><div class="gmail_quote"><div dir="ltr">On Wed, Dec 2, 2015 at 2:13 PM Predrag Punosevac <<a href="mailto:predragp@cs.cmu.edu" target="_blank">predragp@cs.cmu.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Dear Autonians,<br>
<br>
I have being approached by several of you recently regarding<br>
availability of Caffe on GPU nodes. Instead of e-mailing individually<br>
people I opted out to reach everyone in this fashion.<br>
<br>
I tried to build Caffe on Monday myself but failed due to some linker<br>
errors with ATLAS library. This morning I run into our own Arne who has<br>
already built Caffe for himself on GPU2 and who was kind enough to share<br>
his knowledge with me. I am also sharing his scripts with you. Long<br>
story short it looks like OpenBLAS which is installed on all computing<br>
nodes is the way for us to proceed. I will try to adjust<br>
Makefile.config tonight and built Caffe on both machines. In the mean<br>
time feel free to play at your own scratch space.<br>
<br>
Cheers,<br>
Predrag<br>
<br>
This is setupAuton.sh<br>
#!/bin/bash<br>
hn=`hostname`<br>
USE_GPU=0<br>
if [ $hn = "<a href="http://ari.int.autonlab.org" rel="noreferrer" target="_blank">ari.int.autonlab.org</a>" ]; then<br>
ROOT_DIR="/home/scratch/suppe"<br>
elif [ $hn = "<a href="http://gpu2.int.autonlab.org" rel="noreferrer" target="_blank">gpu2.int.autonlab.org</a>" ]; then<br>
ROOT_DIR="/home/scratch/suppe"<br>
USE_GPU=1<br>
else<br>
echo "This build script is for the ARI and GPU2 servers."<br>
exit 1<br>
fi<br>
<br>
INSTALL_DIR=$ROOT_DIR/private<br>
BUILD_DIR=$ROOT_DIR/private/build<br>
<br>
This is buildCaffe.sh<br>
#!/bin/bash<br>
owd=`pwd`<br>
<br>
source ./setupAuton.sh<br>
<br>
if [ ! -d $BUILD_DIR ]; then<br>
mkdir -p $BUILD_DIR<br>
fi<br>
cd $BUILD_DIR<br>
echo "Installing to: " $INSTALL_DIR<br>
<br>
temp=`pkg-config --cflags protobuf --silence-errors`<br>
if [ $? -ne 0 ]; then<br>
echo "Building Google Protocol Buffers"<br>
git clone --branch v2.6.0 <a href="https://github.com/google/protobuf" rel="noreferrer" target="_blank">https://github.com/google/protobuf</a><br>
cd protobuf<br>
./autogen.sh && ./configure --prefix=$INSTALL_DIR && make -j 10 && make install<br>
if [ $? -ne 0 ]; then<br>
exit 1<br>
fi<br>
ADD_PKG_PATH=$INSTALL_DIR/bin/pkgconfig<br>
PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$ADD_PKG_PATH<br>
fi<br>
<br>
cd $BUILD_DIR<br>
aclocal-1.14 --help >& /dev/null<br>
if [ $? -ne 0 ]; then<br>
echo "Building Autotools"<br>
wget <a href="http://ftp.gnu.org/gnu/automake/automake-1.14.1.tar.gz" rel="noreferrer" target="_blank">http://ftp.gnu.org/gnu/automake/automake-1.14.1.tar.gz</a><br>
tar -xzf automake-1.14.1.tar.gz<br>
rm automake-1.14.1.tar.gz<br>
cd automake-1.14.1<br>
./configure --prefix=$INSTALL_DIR<br>
make -j 10 && make install<br>
if [ $? -ne 0 ]; then<br>
echo "Unable to build automake"<br>
exit 1<br>
fi<br>
ADDPATH=$ADDPATH:/$INSTALL_DIR/bin<br>
PATH=$PATH:$ADD_PATH<br>
fi<br>
<br>
cd $BUILD_DIR<br>
temp=`pkg-config --cflags libglog --silence-errors`<br>
if [ $? -ne 0 ]; then<br>
echo "Building GLOG"<br>
wget <a href="http://google-glog.googlecode.com/files/glog-0.3.3.tar.gz" rel="noreferrer" target="_blank">http://google-glog.googlecode.com/files/glog-0.3.3.tar.gz</a><br>
tar -xzf glog-0.3.3.tar.gz<br>
rm glog-0.3.3.tar.gz<br>
cd glog-0.3.3<br>
./configure --prefix=$INSTALL_DIR && make -j 10 && make install<br>
if [ $? -ne 0 ]; then<br>
exit 1<br>
fi<br>
ADD_PKG_PATH=$INSTALL_DIR/bin/pkgconfig<br>
PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$ADD_PKG_PATH<br>
fi<br>
<br>
<br>
cd $BUILD_DIR<br>
temp=`pkg-config --cflags libgflags --silence-errors`<br>
if [ $? -ne 0 ]; then<br>
echo "Building gflags"<br>
git clone --branch v2.0 <a href="https://github.com/gflags/gflags" rel="noreferrer" target="_blank">https://github.com/gflags/gflags</a><br>
cd gflags<br>
./configure --prefix=$INSTALL_DIR && make -j 10 && make install<br>
if [ $? -ne 0 ]; then<br>
exit 1<br>
fi<br>
ADD_PKG_PATH=$INSTALL_DIR/bin/pkgconfig<br>
PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$ADD_PKG_PATH<br>
fi<br>
<br>
if [ ! -d $INSTALL_DIR/include/leveldb ]; then<br>
cd $BUILD_DIR<br>
git clone <a href="https://github.com/google/leveldb" rel="noreferrer" target="_blank">https://github.com/google/leveldb</a><br>
cd leveldb<br>
make -j 10<br>
if [ $? -ne 0 ]; then<br>
echo "LevelDB build failed"<br>
exit 1<br>
fi<br>
cp lib* $INSTALL_DIR/lib<br>
cp -rp include/leveldb $INSTALL_DIR/include<br>
fi<br>
cd $BUILD_DIR<br>
if [ ! -e $INSTALL_DIR/lib/liblmdb.a ]; then<br>
git clone <a href="https://github.com/LMDB/lmdb" rel="noreferrer" target="_blank">https://github.com/LMDB/lmdb</a><br>
cd lmdb/libraries/liblmdb<br>
make -j 20<br>
if [ $? -ne 0 ]; then<br>
echo "Error building LMDB"<br>
exit 1<br>
fi<br>
cp lmdb.h $INSTALL_DIR/include<br>
cp liblmdb* $INSTALL_DIR/lib<br>
fi<br>
<br>
<br>
###########INSTALL HDF5##############<br>
cd $BUILD_DIR<br>
./buildHDF5.sh $INSTALL_DIR<br>
if [ $? -ne 0 ]; then<br>
exit 1<br>
fi<br>
<br>
if [ ! -e $INSTALL_DIR/lib/libsnappy.a ]; then<br>
echo "Building libsnappy"<br>
wget --no-check-certificate <a href="https://github.com/google/snappy/releases/download/1.1.3/snappy-1.1.3.tar.gz" rel="noreferrer" target="_blank">https://github.com/google/snappy/releases/download/1.1.3/snappy-1.1.3.tar.gz</a><br>
tar -xzpf snappy-1.1.3.tar.gz<br>
rm snappy*.tar.gz<br>
cd snappy-1.1.3<br>
./configure --prefix=$INSTALL_DIR && make -j 10 && make install<br>
if [ $? -ne 0 ]; then<br>
echo "libsnappy build failed"<br>
exit 1<br>
fi<br>
fi<br>
<br>
<br>
if [ ! -d $BUILD_DIR/OpenBLAS ]; then<br>
cd $BUILD_DIR<br>
echo "Building OpenBlas"<br>
git clone <a href="https://github.com/xianyi/OpenBLAS" rel="noreferrer" target="_blank">https://github.com/xianyi/OpenBLAS</a><br>
cd OpenBLAS<br>
make TARGET=SANDYBRIDGE -j 20 && make install PREFIX=$INSTALL_DIR<br>
if [ $? -ne 0 ]; then<br>
exit 1<br>
fi<br>
fi<br>
<br>
cd $BUILD_DIR<br>
echo "Building Caffe"<br>
if [ ! -e caffe ]; then<br>
# git clone ssh://<a href="http://suppe@alfa.ahs.ri.cmu.edu/home/suppe/repo/caffe" rel="noreferrer" target="_blank">suppe@alfa.ahs.ri.cmu.edu/home/suppe/repo/caffe</a><br>
git clone <a href="https://github.com/bvlc/caffe" rel="noreferrer" target="_blank">https://github.com/bvlc/caffe</a><br>
fi<br>
cd caffe<br>
if [ $USE_GPU -eq 0 ]; then<br>
sed -e 's/# CPU_ONLY/CPU_ONLY/g' Makefile.config.example > Makefile.config<br>
else<br>
cp Makefile.config.example Makefile.config<br>
fi<br>
sed --in-place '\|^INCLUDE_DIRS|s|$| '$INSTALL_DIR'/include|' Makefile.config<br>
sed --in-place '\|^LIBRARY_DIRS|s|$| '$INSTALL_DIR'/lib|' Makefile.config<br>
sed --in-place 's/BLAS := atlas/BLAS := open/g' Makefile.config<br>
printf 'BLAS_INCLUDE := %s/include\n' $INSTALL_DIR >> Makefile.config<br>
printf 'BLAS_LIB := %s/lib\n' $INSTALL_DIR >> Makefile.config<br>
<br>
make -j 100 && make distribute<br>
if [ $? -ne 0 ]; then<br>
echo "Caffe build failed!"<br>
exit 1<br>
fi<br>
<br>
<br>
<br>
<br>
-------- Original Message --------<br>
From: Arne Suppe <<a href="mailto:suppe@andrew.cmu.edu" target="_blank">suppe@andrew.cmu.edu</a>><br>
Subject: CAFFE Script<br>
Date: Wed, 2 Dec 2015 10:38:43 -0500<br>
To: Predrag Punosevac <<a href="mailto:predragp@cs.cmu.edu" target="_blank">predragp@cs.cmu.edu</a>><br>
<br>
<br>
If you use the buildCaffe.sh script, you will need to update the setupAuton.sh for your username. In general, I use the Springdale version of a package if its available, and only download and compile if its not. You probably already installed versions of these packages. Towards the end, I build OpenBLAS and then used sed to update the Makefile.config.<br>
<br>
Hope this helps!<br>
Arne<br>
<br>
<br>
<br>
<br>
</blockquote></div></div>