From predragp at cs.cmu.edu  Wed Dec  2 14:12:51 2015
From: predragp at cs.cmu.edu (Predrag Punosevac)
Date: Wed, 02 Dec 2015 14:12:51 -0500
Subject: Building Caffe on GPU machines
Message-ID: <20151202191251.tjZUf2dxS1y3%predragp@cs.cmu.edu>

Dear Autonians,

I have being approached by several of you recently regarding
availability of Caffe on GPU nodes. Instead of e-mailing individually
people I opted out to reach everyone in this fashion. 

I tried to build Caffe on Monday myself but failed due to some linker
errors with ATLAS library. This morning I run into our own Arne who has
already built Caffe for himself on GPU2 and who was kind enough to share
his knowledge with me. I am also sharing his scripts with you. Long
story short it looks like OpenBLAS which is installed on all computing
nodes is the way for us to proceed. I will try to adjust
Makefile.config tonight and built Caffe on both machines. In the mean
time feel free to play at your own scratch space.

Cheers,
Predrag

This is setupAuton.sh
#!/bin/bash
hn=`hostname`
USE_GPU=0
if [ $hn = "ari.int.autonlab.org" ]; then
    ROOT_DIR="/home/scratch/suppe"
elif [ $hn = "gpu2.int.autonlab.org" ]; then
    ROOT_DIR="/home/scratch/suppe"
    USE_GPU=1
else
    echo "This build script is for the ARI and GPU2 servers."
    exit 1
fi

INSTALL_DIR=$ROOT_DIR/private
BUILD_DIR=$ROOT_DIR/private/build

This is buildCaffe.sh
#!/bin/bash
owd=`pwd`

source ./setupAuton.sh

if [ ! -d $BUILD_DIR ]; then
    mkdir -p $BUILD_DIR
fi
cd $BUILD_DIR
echo "Installing to: " $INSTALL_DIR

temp=`pkg-config --cflags protobuf --silence-errors`
if [ $? -ne 0 ]; then
    echo "Building Google Protocol Buffers"
    git clone --branch v2.6.0 https://github.com/google/protobuf
    cd protobuf
    ./autogen.sh && ./configure --prefix=$INSTALL_DIR && make -j 10 && make install
    if [ $? -ne 0 ]; then
	exit 1
    fi
    ADD_PKG_PATH=$INSTALL_DIR/bin/pkgconfig
    PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$ADD_PKG_PATH
fi

cd $BUILD_DIR
aclocal-1.14 --help >& /dev/null
if [ $? -ne 0 ]; then
    echo "Building Autotools"
    wget http://ftp.gnu.org/gnu/automake/automake-1.14.1.tar.gz
    tar -xzf automake-1.14.1.tar.gz
    rm automake-1.14.1.tar.gz
    cd automake-1.14.1
    ./configure --prefix=$INSTALL_DIR
    make -j 10 && make install
    if [ $? -ne 0 ]; then
	echo "Unable to build automake"
	exit 1
    fi
    ADDPATH=$ADDPATH:/$INSTALL_DIR/bin
    PATH=$PATH:$ADD_PATH
fi

cd $BUILD_DIR
temp=`pkg-config --cflags libglog --silence-errors`
if [ $? -ne 0 ]; then
    echo "Building GLOG"
    wget http://google-glog.googlecode.com/files/glog-0.3.3.tar.gz
    tar -xzf glog-0.3.3.tar.gz
    rm glog-0.3.3.tar.gz
    cd glog-0.3.3
    ./configure --prefix=$INSTALL_DIR && make -j 10 && make install
    if [ $? -ne 0 ]; then
	exit 1
    fi
    ADD_PKG_PATH=$INSTALL_DIR/bin/pkgconfig
    PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$ADD_PKG_PATH
fi


cd $BUILD_DIR
temp=`pkg-config --cflags libgflags --silence-errors`
if [ $? -ne 0 ]; then
    echo "Building gflags"
    git clone --branch v2.0 https://github.com/gflags/gflags
    cd gflags
    ./configure --prefix=$INSTALL_DIR && make -j 10 && make install
    if [ $? -ne 0 ]; then
	exit 1
    fi
    ADD_PKG_PATH=$INSTALL_DIR/bin/pkgconfig
    PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$ADD_PKG_PATH
fi

if [ ! -d $INSTALL_DIR/include/leveldb ]; then
    cd $BUILD_DIR
    git clone https://github.com/google/leveldb
    cd leveldb
    make -j 10
    if [ $? -ne 0 ]; then
	echo "LevelDB build failed"
	exit 1
    fi
    cp lib* $INSTALL_DIR/lib
    cp -rp include/leveldb $INSTALL_DIR/include
fi
cd $BUILD_DIR
if [ ! -e $INSTALL_DIR/lib/liblmdb.a ]; then
    git clone https://github.com/LMDB/lmdb
    cd lmdb/libraries/liblmdb
    make -j 20
    if [ $? -ne 0 ]; then
	echo "Error building LMDB"
	exit 1
    fi
    cp lmdb.h $INSTALL_DIR/include
    cp liblmdb* $INSTALL_DIR/lib
fi


###########INSTALL HDF5##############
cd $BUILD_DIR
./buildHDF5.sh $INSTALL_DIR
if [ $? -ne 0 ]; then
    exit 1
fi

if [ ! -e $INSTALL_DIR/lib/libsnappy.a ]; then
    echo "Building libsnappy"
    wget --no-check-certificate https://github.com/google/snappy/releases/download/1.1.3/snappy-1.1.3.tar.gz
    tar -xzpf snappy-1.1.3.tar.gz
    rm snappy*.tar.gz
    cd snappy-1.1.3
    ./configure --prefix=$INSTALL_DIR && make -j 10 && make install
    if [ $? -ne 0 ]; then
	echo "libsnappy build failed"
    exit 1
    fi
fi


if [ ! -d $BUILD_DIR/OpenBLAS ]; then
    cd $BUILD_DIR
    echo "Building OpenBlas"
    git clone https://github.com/xianyi/OpenBLAS
    cd OpenBLAS
    make TARGET=SANDYBRIDGE -j 20 && make install PREFIX=$INSTALL_DIR
    if [ $? -ne 0 ]; then
    exit 1
    fi
fi

cd $BUILD_DIR
echo "Building Caffe"
if [ ! -e caffe ]; then
#    git clone ssh://suppe at alfa.ahs.ri.cmu.edu/home/suppe/repo/caffe
    git clone https://github.com/bvlc/caffe
fi
cd caffe
if [ $USE_GPU -eq 0 ]; then
    sed -e 's/# CPU_ONLY/CPU_ONLY/g' Makefile.config.example > Makefile.config
else
    cp Makefile.config.example Makefile.config
fi
sed --in-place '\|^INCLUDE_DIRS|s|$| '$INSTALL_DIR'/include|' Makefile.config
sed --in-place '\|^LIBRARY_DIRS|s|$| '$INSTALL_DIR'/lib|' Makefile.config
sed --in-place 's/BLAS := atlas/BLAS := open/g' Makefile.config
printf 'BLAS_INCLUDE := %s/include\n' $INSTALL_DIR >> Makefile.config
printf 'BLAS_LIB := %s/lib\n' $INSTALL_DIR >> Makefile.config

make -j 100 && make distribute
if [ $? -ne 0 ]; then
    echo "Caffe build failed!"
    exit 1
fi




-------- Original Message --------
From: Arne Suppe <suppe at andrew.cmu.edu>
Subject: CAFFE Script
Date: Wed, 2 Dec 2015 10:38:43 -0500
To: Predrag Punosevac <predragp at cs.cmu.edu>


If you use the buildCaffe.sh script, you will need to update the setupAuton.sh for your username.  In general, I use the Springdale version of a package if its available, and only download and compile if its not.  You probably already installed versions of these packages.  Towards the end, I build OpenBLAS and then used sed to update the Makefile.config.

Hope this helps!
Arne





From dsutherl at cs.cmu.edu  Wed Dec  2 15:58:57 2015
From: dsutherl at cs.cmu.edu (Dougal Sutherland)
Date: Wed, 2 Dec 2015 15:58:57 -0500
Subject: Building Caffe on GPU machines
In-Reply-To: <20151202191251.tjZUf2dxS1y3%predragp@cs.cmu.edu>
References: <20151202191251.tjZUf2dxS1y3%predragp@cs.cmu.edu>
Message-ID: <CA+HXLPDqhW0B-yHS2gnUrrFmzADOTLy0gAUUOXzOYESYp65Khw@mail.gmail.com>

Hi all,

FYI, almost all of the dependencies built in this script are actually
already installed on the server. This should work:

*Base setup*
cd /home/scratch/$USER
git clone https://github.com/BLVC/caffe
cd caffe
cp Makefile.config.example Makefile.config
# do configuration changes here
# you'll need to do either openblas or atlas below
# you probably want cudnn as well
make -j23
make -j23 test
make runtest


*Using openblas*
This requires the openblas-devel package, which I just asked Predrag to
install. After he does, it should work with just:

sed -i 's/BLAS := atlas/BLAS := open/' Makefile.config


*Using atlas*
Openblas is faster than atlas, but you'll probably be doing everything on
the GPU anyway, so it shouldn't really matter. Here's how to get Caffe to
work with red hat distributions' nonstandard atlas layout. (For some
reason, the Caffe developers rejected a pull request making this an option
in Makefile.config....)

sed -i 's/LIBRARIES += cblas atlas/LIBRARIES += tatlas/' Makefile
sed -i 's|# BLAS_INCLUDE := /path/to/your/blas|BLAS_INCLUDE :=
/usr/include/atlas|' Makefile.config
sed -i 's|# BLAS_LIB := /path/to/your/blas|BLAS_LIB := /usr/lib64/atlas|'
Makefile.config


*Using cudnn*
If you want to use the cudnn library, which makes things faster (sometimes
substantially):

Register as a developer at https://developer.nvidia.com/cudnn.
Wait a day for them to approve you. Or, just get the file from
~dsutherl/cudnn-7.0-linux-x64-v3.0-prod.tgz.

cd /home/scratch/$USER
tar xf ~dsutherl/cudnn-7.0-linux-x64-v3.0-prod.tgz
mv cuda cudnn

cd caffe
sed -i 's/# USE_CUDNN := 1/USE_CUDNN := 1/' Makefile.config
perl -ipe 's|$| /home/scratch/'$USER'/cudnn/include| if /INCLUDE_DIRS :=/'
Makefile.config
perl -ipe 's|$| /home/scratch/'$USER'/cudnn/lib64| if /LIBRARY_DIRS :=/'
Makefile.config

Then, whenever you run a caffe binary you'll have to have run
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/home/scratch/$USER/cudnn/lib64"
in that shell (or set it wherever).

*Python interface*
make py works. You'll need some requirements to import the package, though,
because several of the system-installed packages are too old. One option is

export PYTHONUSERBASE=/home/scratch/$USER/.local
pip install --user -r python/requirements.txt

This will take a long time, since it has to compile scipy and a bunch of
other stuff. You'll also need to set the PYTHONUSERBASE variable before
launching any python process that needs to use this.

*Matlab interface*
sed -i 's|# MATLAB_DIR := /usr/local|MATLAB_DIR
:= /usr/local/MATLAB/R2015b|' Makefile.config
make matcaffe

Haven't tested this at all, matlab is gross. :)


Hope that's helpful!
- Dougal


On Wed, Dec 2, 2015 at 2:13 PM Predrag Punosevac <predragp at cs.cmu.edu>
wrote:

> Dear Autonians,
>
> I have being approached by several of you recently regarding
> availability of Caffe on GPU nodes. Instead of e-mailing individually
> people I opted out to reach everyone in this fashion.
>
> I tried to build Caffe on Monday myself but failed due to some linker
> errors with ATLAS library. This morning I run into our own Arne who has
> already built Caffe for himself on GPU2 and who was kind enough to share
> his knowledge with me. I am also sharing his scripts with you. Long
> story short it looks like OpenBLAS which is installed on all computing
> nodes is the way for us to proceed. I will try to adjust
> Makefile.config tonight and built Caffe on both machines. In the mean
> time feel free to play at your own scratch space.
>
> Cheers,
> Predrag
>
> This is setupAuton.sh
> #!/bin/bash
> hn=`hostname`
> USE_GPU=0
> if [ $hn = "ari.int.autonlab.org" ]; then
>     ROOT_DIR="/home/scratch/suppe"
> elif [ $hn = "gpu2.int.autonlab.org" ]; then
>     ROOT_DIR="/home/scratch/suppe"
>     USE_GPU=1
> else
>     echo "This build script is for the ARI and GPU2 servers."
>     exit 1
> fi
>
> INSTALL_DIR=$ROOT_DIR/private
> BUILD_DIR=$ROOT_DIR/private/build
>
> This is buildCaffe.sh
> #!/bin/bash
> owd=`pwd`
>
> source ./setupAuton.sh
>
> if [ ! -d $BUILD_DIR ]; then
>     mkdir -p $BUILD_DIR
> fi
> cd $BUILD_DIR
> echo "Installing to: " $INSTALL_DIR
>
> temp=`pkg-config --cflags protobuf --silence-errors`
> if [ $? -ne 0 ]; then
>     echo "Building Google Protocol Buffers"
>     git clone --branch v2.6.0 https://github.com/google/protobuf
>     cd protobuf
>     ./autogen.sh && ./configure --prefix=$INSTALL_DIR && make -j 10 &&
> make install
>     if [ $? -ne 0 ]; then
>         exit 1
>     fi
>     ADD_PKG_PATH=$INSTALL_DIR/bin/pkgconfig
>     PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$ADD_PKG_PATH
> fi
>
> cd $BUILD_DIR
> aclocal-1.14 --help >& /dev/null
> if [ $? -ne 0 ]; then
>     echo "Building Autotools"
>     wget http://ftp.gnu.org/gnu/automake/automake-1.14.1.tar.gz
>     tar -xzf automake-1.14.1.tar.gz
>     rm automake-1.14.1.tar.gz
>     cd automake-1.14.1
>     ./configure --prefix=$INSTALL_DIR
>     make -j 10 && make install
>     if [ $? -ne 0 ]; then
>         echo "Unable to build automake"
>         exit 1
>     fi
>     ADDPATH=$ADDPATH:/$INSTALL_DIR/bin
>     PATH=$PATH:$ADD_PATH
> fi
>
> cd $BUILD_DIR
> temp=`pkg-config --cflags libglog --silence-errors`
> if [ $? -ne 0 ]; then
>     echo "Building GLOG"
>     wget http://google-glog.googlecode.com/files/glog-0.3.3.tar.gz
>     tar -xzf glog-0.3.3.tar.gz
>     rm glog-0.3.3.tar.gz
>     cd glog-0.3.3
>     ./configure --prefix=$INSTALL_DIR && make -j 10 && make install
>     if [ $? -ne 0 ]; then
>         exit 1
>     fi
>     ADD_PKG_PATH=$INSTALL_DIR/bin/pkgconfig
>     PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$ADD_PKG_PATH
> fi
>
>
> cd $BUILD_DIR
> temp=`pkg-config --cflags libgflags --silence-errors`
> if [ $? -ne 0 ]; then
>     echo "Building gflags"
>     git clone --branch v2.0 https://github.com/gflags/gflags
>     cd gflags
>     ./configure --prefix=$INSTALL_DIR && make -j 10 && make install
>     if [ $? -ne 0 ]; then
>         exit 1
>     fi
>     ADD_PKG_PATH=$INSTALL_DIR/bin/pkgconfig
>     PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$ADD_PKG_PATH
> fi
>
> if [ ! -d $INSTALL_DIR/include/leveldb ]; then
>     cd $BUILD_DIR
>     git clone https://github.com/google/leveldb
>     cd leveldb
>     make -j 10
>     if [ $? -ne 0 ]; then
>         echo "LevelDB build failed"
>         exit 1
>     fi
>     cp lib* $INSTALL_DIR/lib
>     cp -rp include/leveldb $INSTALL_DIR/include
> fi
> cd $BUILD_DIR
> if [ ! -e $INSTALL_DIR/lib/liblmdb.a ]; then
>     git clone https://github.com/LMDB/lmdb
>     cd lmdb/libraries/liblmdb
>     make -j 20
>     if [ $? -ne 0 ]; then
>         echo "Error building LMDB"
>         exit 1
>     fi
>     cp lmdb.h $INSTALL_DIR/include
>     cp liblmdb* $INSTALL_DIR/lib
> fi
>
>
> ###########INSTALL HDF5##############
> cd $BUILD_DIR
> ./buildHDF5.sh $INSTALL_DIR
> if [ $? -ne 0 ]; then
>     exit 1
> fi
>
> if [ ! -e $INSTALL_DIR/lib/libsnappy.a ]; then
>     echo "Building libsnappy"
>     wget --no-check-certificate
> https://github.com/google/snappy/releases/download/1.1.3/snappy-1.1.3.tar.gz
>     tar -xzpf snappy-1.1.3.tar.gz
>     rm snappy*.tar.gz
>     cd snappy-1.1.3
>     ./configure --prefix=$INSTALL_DIR && make -j 10 && make install
>     if [ $? -ne 0 ]; then
>         echo "libsnappy build failed"
>     exit 1
>     fi
> fi
>
>
> if [ ! -d $BUILD_DIR/OpenBLAS ]; then
>     cd $BUILD_DIR
>     echo "Building OpenBlas"
>     git clone https://github.com/xianyi/OpenBLAS
>     cd OpenBLAS
>     make TARGET=SANDYBRIDGE -j 20 && make install PREFIX=$INSTALL_DIR
>     if [ $? -ne 0 ]; then
>     exit 1
>     fi
> fi
>
> cd $BUILD_DIR
> echo "Building Caffe"
> if [ ! -e caffe ]; then
> #    git clone ssh://suppe at alfa.ahs.ri.cmu.edu/home/suppe/repo/caffe
>     git clone https://github.com/bvlc/caffe
> fi
> cd caffe
> if [ $USE_GPU -eq 0 ]; then
>     sed -e 's/# CPU_ONLY/CPU_ONLY/g' Makefile.config.example >
> Makefile.config
> else
>     cp Makefile.config.example Makefile.config
> fi
> sed --in-place '\|^INCLUDE_DIRS|s|$| '$INSTALL_DIR'/include|'
> Makefile.config
> sed --in-place '\|^LIBRARY_DIRS|s|$| '$INSTALL_DIR'/lib|' Makefile.config
> sed --in-place 's/BLAS := atlas/BLAS := open/g' Makefile.config
> printf 'BLAS_INCLUDE := %s/include\n' $INSTALL_DIR >> Makefile.config
> printf 'BLAS_LIB := %s/lib\n' $INSTALL_DIR >> Makefile.config
>
> make -j 100 && make distribute
> if [ $? -ne 0 ]; then
>     echo "Caffe build failed!"
>     exit 1
> fi
>
>
>
>
> -------- Original Message --------
> From: Arne Suppe <suppe at andrew.cmu.edu>
> Subject: CAFFE Script
> Date: Wed, 2 Dec 2015 10:38:43 -0500
> To: Predrag Punosevac <predragp at cs.cmu.edu>
>
>
> If you use the buildCaffe.sh script, you will need to update the
> setupAuton.sh for your username.  In general, I use the Springdale version
> of a package if its available, and only download and compile if its not.
> You probably already installed versions of these packages.  Towards the
> end, I build OpenBLAS and then used sed to update the Makefile.config.
>
> Hope this helps!
> Arne
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/autonlab-users/attachments/20151202/a01e2d8b/attachment-0001.html>

From dsutherl at cs.cmu.edu  Wed Dec  2 16:02:06 2015
From: dsutherl at cs.cmu.edu (Dougal Sutherland)
Date: Wed, 2 Dec 2015 16:02:06 -0500
Subject: Building Caffe on GPU machines
In-Reply-To: <CA+HXLPDqhW0B-yHS2gnUrrFmzADOTLy0gAUUOXzOYESYp65Khw@mail.gmail.com>
References: <20151202191251.tjZUf2dxS1y3%predragp@cs.cmu.edu>
 <CA+HXLPDqhW0B-yHS2gnUrrFmzADOTLy0gAUUOXzOYESYp65Khw@mail.gmail.com>
Message-ID: <CA+HXLPD3KQ9+5reUgViE65_nAXn=HybktV7Uz2-nCgq=qiuQJg@mail.gmail.com>

Also: if you're running GPU stuff like caffe, nvidia-smi is the command to
run to check which GPUs are available (htop or whatever won't show anything
useful there). Unlike CPU jobs, where multiple people running stuff will
just slow everything down, GPU jobs are often memory-bound and very likely
to crash if multiple jobs ask for more memory than is available. Probably
better to stick to one job per GPU unless you've coordinated with the
person running the other job.

On Wed, Dec 2, 2015 at 3:58 PM, Dougal Sutherland <dsutherl at cs.cmu.edu>
wrote:

> Hi all,
>
> FYI, almost all of the dependencies built in this script are actually
> already installed on the server. This should work:
>
> *Base setup*
> cd /home/scratch/$USER
> git clone https://github.com/BLVC/caffe
> cd caffe
> cp Makefile.config.example Makefile.config
> # do configuration changes here
> # you'll need to do either openblas or atlas below
> # you probably want cudnn as well
> make -j23
> make -j23 test
> make runtest
>
>
> *Using openblas*
> This requires the openblas-devel package, which I just asked Predrag to
> install. After he does, it should work with just:
>
> sed -i 's/BLAS := atlas/BLAS := open/' Makefile.config
>
>
> *Using atlas*
> Openblas is faster than atlas, but you'll probably be doing everything on
> the GPU anyway, so it shouldn't really matter. Here's how to get Caffe to
> work with red hat distributions' nonstandard atlas layout. (For some
> reason, the Caffe developers rejected a pull request making this an option
> in Makefile.config....)
>
> sed -i 's/LIBRARIES += cblas atlas/LIBRARIES += tatlas/' Makefile
> sed -i 's|# BLAS_INCLUDE := /path/to/your/blas|BLAS_INCLUDE :=
> /usr/include/atlas|' Makefile.config
> sed -i 's|# BLAS_LIB := /path/to/your/blas|BLAS_LIB := /usr/lib64/atlas|'
> Makefile.config
>
>
> *Using cudnn*
> If you want to use the cudnn library, which makes things faster (sometimes
> substantially):
>
> Register as a developer at https://developer.nvidia.com/cudnn.
> Wait a day for them to approve you. Or, just get the file from
> ~dsutherl/cudnn-7.0-linux-x64-v3.0-prod.tgz.
>
> cd /home/scratch/$USER
> tar xf ~dsutherl/cudnn-7.0-linux-x64-v3.0-prod.tgz
> mv cuda cudnn
>
> cd caffe
> sed -i 's/# USE_CUDNN := 1/USE_CUDNN := 1/' Makefile.config
> perl -ipe 's|$| /home/scratch/'$USER'/cudnn/include| if /INCLUDE_DIRS :=/'
> Makefile.config
> perl -ipe 's|$| /home/scratch/'$USER'/cudnn/lib64| if /LIBRARY_DIRS :=/'
> Makefile.config
>
> Then, whenever you run a caffe binary you'll have to have run
> export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/home/scratch/$USER/cudnn/lib64"
> in that shell (or set it wherever).
>
> *Python interface*
> make py works. You'll need some requirements to import the package,
> though, because several of the system-installed packages are too old. One
> option is
>
> export PYTHONUSERBASE=/home/scratch/$USER/.local
> pip install --user -r python/requirements.txt
>
> This will take a long time, since it has to compile scipy and a bunch of
> other stuff. You'll also need to set the PYTHONUSERBASE variable before
> launching any python process that needs to use this.
>
> *Matlab interface*
> sed -i 's|# MATLAB_DIR := /usr/local|MATLAB_DIR
> := /usr/local/MATLAB/R2015b|' Makefile.config
> make matcaffe
>
> Haven't tested this at all, matlab is gross. :)
>
>
> Hope that's helpful!
> - Dougal
>
>
> On Wed, Dec 2, 2015 at 2:13 PM Predrag Punosevac <predragp at cs.cmu.edu>
> wrote:
>
>> Dear Autonians,
>>
>> I have being approached by several of you recently regarding
>> availability of Caffe on GPU nodes. Instead of e-mailing individually
>> people I opted out to reach everyone in this fashion.
>>
>> I tried to build Caffe on Monday myself but failed due to some linker
>> errors with ATLAS library. This morning I run into our own Arne who has
>> already built Caffe for himself on GPU2 and who was kind enough to share
>> his knowledge with me. I am also sharing his scripts with you. Long
>> story short it looks like OpenBLAS which is installed on all computing
>> nodes is the way for us to proceed. I will try to adjust
>> Makefile.config tonight and built Caffe on both machines. In the mean
>> time feel free to play at your own scratch space.
>>
>> Cheers,
>> Predrag
>>
>> This is setupAuton.sh
>> #!/bin/bash
>> hn=`hostname`
>> USE_GPU=0
>> if [ $hn = "ari.int.autonlab.org" ]; then
>>     ROOT_DIR="/home/scratch/suppe"
>> elif [ $hn = "gpu2.int.autonlab.org" ]; then
>>     ROOT_DIR="/home/scratch/suppe"
>>     USE_GPU=1
>> else
>>     echo "This build script is for the ARI and GPU2 servers."
>>     exit 1
>> fi
>>
>> INSTALL_DIR=$ROOT_DIR/private
>> BUILD_DIR=$ROOT_DIR/private/build
>>
>> This is buildCaffe.sh
>> #!/bin/bash
>> owd=`pwd`
>>
>> source ./setupAuton.sh
>>
>> if [ ! -d $BUILD_DIR ]; then
>>     mkdir -p $BUILD_DIR
>> fi
>> cd $BUILD_DIR
>> echo "Installing to: " $INSTALL_DIR
>>
>> temp=`pkg-config --cflags protobuf --silence-errors`
>> if [ $? -ne 0 ]; then
>>     echo "Building Google Protocol Buffers"
>>     git clone --branch v2.6.0 https://github.com/google/protobuf
>>     cd protobuf
>>     ./autogen.sh && ./configure --prefix=$INSTALL_DIR && make -j 10 &&
>> make install
>>     if [ $? -ne 0 ]; then
>>         exit 1
>>     fi
>>     ADD_PKG_PATH=$INSTALL_DIR/bin/pkgconfig
>>     PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$ADD_PKG_PATH
>> fi
>>
>> cd $BUILD_DIR
>> aclocal-1.14 --help >& /dev/null
>> if [ $? -ne 0 ]; then
>>     echo "Building Autotools"
>>     wget http://ftp.gnu.org/gnu/automake/automake-1.14.1.tar.gz
>>     tar -xzf automake-1.14.1.tar.gz
>>     rm automake-1.14.1.tar.gz
>>     cd automake-1.14.1
>>     ./configure --prefix=$INSTALL_DIR
>>     make -j 10 && make install
>>     if [ $? -ne 0 ]; then
>>         echo "Unable to build automake"
>>         exit 1
>>     fi
>>     ADDPATH=$ADDPATH:/$INSTALL_DIR/bin
>>     PATH=$PATH:$ADD_PATH
>> fi
>>
>> cd $BUILD_DIR
>> temp=`pkg-config --cflags libglog --silence-errors`
>> if [ $? -ne 0 ]; then
>>     echo "Building GLOG"
>>     wget http://google-glog.googlecode.com/files/glog-0.3.3.tar.gz
>>     tar -xzf glog-0.3.3.tar.gz
>>     rm glog-0.3.3.tar.gz
>>     cd glog-0.3.3
>>     ./configure --prefix=$INSTALL_DIR && make -j 10 && make install
>>     if [ $? -ne 0 ]; then
>>         exit 1
>>     fi
>>     ADD_PKG_PATH=$INSTALL_DIR/bin/pkgconfig
>>     PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$ADD_PKG_PATH
>> fi
>>
>>
>> cd $BUILD_DIR
>> temp=`pkg-config --cflags libgflags --silence-errors`
>> if [ $? -ne 0 ]; then
>>     echo "Building gflags"
>>     git clone --branch v2.0 https://github.com/gflags/gflags
>>     cd gflags
>>     ./configure --prefix=$INSTALL_DIR && make -j 10 && make install
>>     if [ $? -ne 0 ]; then
>>         exit 1
>>     fi
>>     ADD_PKG_PATH=$INSTALL_DIR/bin/pkgconfig
>>     PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$ADD_PKG_PATH
>> fi
>>
>> if [ ! -d $INSTALL_DIR/include/leveldb ]; then
>>     cd $BUILD_DIR
>>     git clone https://github.com/google/leveldb
>>     cd leveldb
>>     make -j 10
>>     if [ $? -ne 0 ]; then
>>         echo "LevelDB build failed"
>>         exit 1
>>     fi
>>     cp lib* $INSTALL_DIR/lib
>>     cp -rp include/leveldb $INSTALL_DIR/include
>> fi
>> cd $BUILD_DIR
>> if [ ! -e $INSTALL_DIR/lib/liblmdb.a ]; then
>>     git clone https://github.com/LMDB/lmdb
>>     cd lmdb/libraries/liblmdb
>>     make -j 20
>>     if [ $? -ne 0 ]; then
>>         echo "Error building LMDB"
>>         exit 1
>>     fi
>>     cp lmdb.h $INSTALL_DIR/include
>>     cp liblmdb* $INSTALL_DIR/lib
>> fi
>>
>>
>> ###########INSTALL HDF5##############
>> cd $BUILD_DIR
>> ./buildHDF5.sh $INSTALL_DIR
>> if [ $? -ne 0 ]; then
>>     exit 1
>> fi
>>
>> if [ ! -e $INSTALL_DIR/lib/libsnappy.a ]; then
>>     echo "Building libsnappy"
>>     wget --no-check-certificate
>> https://github.com/google/snappy/releases/download/1.1.3/snappy-1.1.3.tar.gz
>>     tar -xzpf snappy-1.1.3.tar.gz
>>     rm snappy*.tar.gz
>>     cd snappy-1.1.3
>>     ./configure --prefix=$INSTALL_DIR && make -j 10 && make install
>>     if [ $? -ne 0 ]; then
>>         echo "libsnappy build failed"
>>     exit 1
>>     fi
>> fi
>>
>>
>> if [ ! -d $BUILD_DIR/OpenBLAS ]; then
>>     cd $BUILD_DIR
>>     echo "Building OpenBlas"
>>     git clone https://github.com/xianyi/OpenBLAS
>>     cd OpenBLAS
>>     make TARGET=SANDYBRIDGE -j 20 && make install PREFIX=$INSTALL_DIR
>>     if [ $? -ne 0 ]; then
>>     exit 1
>>     fi
>> fi
>>
>> cd $BUILD_DIR
>> echo "Building Caffe"
>> if [ ! -e caffe ]; then
>> #    git clone ssh://suppe at alfa.ahs.ri.cmu.edu/home/suppe/repo/caffe
>>     git clone https://github.com/bvlc/caffe
>> fi
>> cd caffe
>> if [ $USE_GPU -eq 0 ]; then
>>     sed -e 's/# CPU_ONLY/CPU_ONLY/g' Makefile.config.example >
>> Makefile.config
>> else
>>     cp Makefile.config.example Makefile.config
>> fi
>> sed --in-place '\|^INCLUDE_DIRS|s|$| '$INSTALL_DIR'/include|'
>> Makefile.config
>> sed --in-place '\|^LIBRARY_DIRS|s|$| '$INSTALL_DIR'/lib|' Makefile.config
>> sed --in-place 's/BLAS := atlas/BLAS := open/g' Makefile.config
>> printf 'BLAS_INCLUDE := %s/include\n' $INSTALL_DIR >> Makefile.config
>> printf 'BLAS_LIB := %s/lib\n' $INSTALL_DIR >> Makefile.config
>>
>> make -j 100 && make distribute
>> if [ $? -ne 0 ]; then
>>     echo "Caffe build failed!"
>>     exit 1
>> fi
>>
>>
>>
>>
>> -------- Original Message --------
>> From: Arne Suppe <suppe at andrew.cmu.edu>
>> Subject: CAFFE Script
>> Date: Wed, 2 Dec 2015 10:38:43 -0500
>> To: Predrag Punosevac <predragp at cs.cmu.edu>
>>
>>
>> If you use the buildCaffe.sh script, you will need to update the
>> setupAuton.sh for your username.  In general, I use the Springdale version
>> of a package if its available, and only download and compile if its not.
>> You probably already installed versions of these packages.  Towards the
>> end, I build OpenBLAS and then used sed to update the Makefile.config.
>>
>> Hope this helps!
>> Arne
>>
>>
>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/autonlab-users/attachments/20151202/f2db3a0f/attachment.html>

From dsutherl at cs.cmu.edu  Wed Dec  2 17:09:06 2015
From: dsutherl at cs.cmu.edu (Dougal Sutherland)
Date: Wed, 02 Dec 2015 22:09:06 +0000
Subject: Building Caffe on GPU machines
In-Reply-To: <CA+HXLPD3KQ9+5reUgViE65_nAXn=HybktV7Uz2-nCgq=qiuQJg@mail.gmail.com>
References: <20151202191251.tjZUf2dxS1y3%predragp@cs.cmu.edu>
 <CA+HXLPDqhW0B-yHS2gnUrrFmzADOTLy0gAUUOXzOYESYp65Khw@mail.gmail.com>
 <CA+HXLPD3KQ9+5reUgViE65_nAXn=HybktV7Uz2-nCgq=qiuQJg@mail.gmail.com>
Message-ID: <CA+HXLPBCJPdirhOyiFfng7_cRvYD7vbUVr-4YQefZ1eRo-EO3w@mail.gmail.com>

Oops, sorry, one mistake above: the two   perl -ipe    (in the cudnn
section) should be   perl -i -pe.

On Wed, Dec 2, 2015 at 4:02 PM Dougal Sutherland <dsutherl at cs.cmu.edu>
wrote:

> Also: if you're running GPU stuff like caffe, nvidia-smi is the command to
> run to check which GPUs are available (htop or whatever won't show anything
> useful there). Unlike CPU jobs, where multiple people running stuff will
> just slow everything down, GPU jobs are often memory-bound and very likely
> to crash if multiple jobs ask for more memory than is available. Probably
> better to stick to one job per GPU unless you've coordinated with the
> person running the other job.
>
> On Wed, Dec 2, 2015 at 3:58 PM, Dougal Sutherland <dsutherl at cs.cmu.edu>
> wrote:
>
>> Hi all,
>>
>> FYI, almost all of the dependencies built in this script are actually
>> already installed on the server. This should work:
>>
>> *Base setup*
>> cd /home/scratch/$USER
>> git clone https://github.com/BLVC/caffe
>> cd caffe
>> cp Makefile.config.example Makefile.config
>> # do configuration changes here
>> # you'll need to do either openblas or atlas below
>> # you probably want cudnn as well
>> make -j23
>> make -j23 test
>> make runtest
>>
>>
>> *Using openblas*
>> This requires the openblas-devel package, which I just asked Predrag to
>> install. After he does, it should work with just:
>>
>> sed -i 's/BLAS := atlas/BLAS := open/' Makefile.config
>>
>>
>> *Using atlas*
>> Openblas is faster than atlas, but you'll probably be doing everything on
>> the GPU anyway, so it shouldn't really matter. Here's how to get Caffe to
>> work with red hat distributions' nonstandard atlas layout. (For some
>> reason, the Caffe developers rejected a pull request making this an option
>> in Makefile.config....)
>>
>> sed -i 's/LIBRARIES += cblas atlas/LIBRARIES += tatlas/' Makefile
>> sed -i 's|# BLAS_INCLUDE := /path/to/your/blas|BLAS_INCLUDE :=
>> /usr/include/atlas|' Makefile.config
>> sed -i 's|# BLAS_LIB := /path/to/your/blas|BLAS_LIB := /usr/lib64/atlas|'
>> Makefile.config
>>
>>
>> *Using cudnn*
>> If you want to use the cudnn library, which makes things faster
>> (sometimes substantially):
>>
>> Register as a developer at https://developer.nvidia.com/cudnn.
>> Wait a day for them to approve you. Or, just get the file from
>> ~dsutherl/cudnn-7.0-linux-x64-v3.0-prod.tgz.
>>
>> cd /home/scratch/$USER
>> tar xf ~dsutherl/cudnn-7.0-linux-x64-v3.0-prod.tgz
>> mv cuda cudnn
>>
>> cd caffe
>> sed -i 's/# USE_CUDNN := 1/USE_CUDNN := 1/' Makefile.config
>> perl -ipe 's|$| /home/scratch/'$USER'/cudnn/include| if /INCLUDE_DIRS
>> :=/' Makefile.config
>> perl -ipe 's|$| /home/scratch/'$USER'/cudnn/lib64| if /LIBRARY_DIRS :=/'
>> Makefile.config
>>
>> Then, whenever you run a caffe binary you'll have to have run
>> export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/home/scratch/$USER/cudnn/lib64"
>> in that shell (or set it wherever).
>>
>> *Python interface*
>> make py works. You'll need some requirements to import the package,
>> though, because several of the system-installed packages are too old. One
>> option is
>>
>> export PYTHONUSERBASE=/home/scratch/$USER/.local
>> pip install --user -r python/requirements.txt
>>
>> This will take a long time, since it has to compile scipy and a bunch of
>> other stuff. You'll also need to set the PYTHONUSERBASE variable before
>> launching any python process that needs to use this.
>>
>> *Matlab interface*
>> sed -i 's|# MATLAB_DIR := /usr/local|MATLAB_DIR
>> := /usr/local/MATLAB/R2015b|' Makefile.config
>> make matcaffe
>>
>> Haven't tested this at all, matlab is gross. :)
>>
>>
>> Hope that's helpful!
>> - Dougal
>>
>>
>> On Wed, Dec 2, 2015 at 2:13 PM Predrag Punosevac <predragp at cs.cmu.edu>
>> wrote:
>>
>>> Dear Autonians,
>>>
>>> I have being approached by several of you recently regarding
>>> availability of Caffe on GPU nodes. Instead of e-mailing individually
>>> people I opted out to reach everyone in this fashion.
>>>
>>> I tried to build Caffe on Monday myself but failed due to some linker
>>> errors with ATLAS library. This morning I run into our own Arne who has
>>> already built Caffe for himself on GPU2 and who was kind enough to share
>>> his knowledge with me. I am also sharing his scripts with you. Long
>>> story short it looks like OpenBLAS which is installed on all computing
>>> nodes is the way for us to proceed. I will try to adjust
>>> Makefile.config tonight and built Caffe on both machines. In the mean
>>> time feel free to play at your own scratch space.
>>>
>>> Cheers,
>>> Predrag
>>>
>>> This is setupAuton.sh
>>> #!/bin/bash
>>> hn=`hostname`
>>> USE_GPU=0
>>> if [ $hn = "ari.int.autonlab.org" ]; then
>>>     ROOT_DIR="/home/scratch/suppe"
>>> elif [ $hn = "gpu2.int.autonlab.org" ]; then
>>>     ROOT_DIR="/home/scratch/suppe"
>>>     USE_GPU=1
>>> else
>>>     echo "This build script is for the ARI and GPU2 servers."
>>>     exit 1
>>> fi
>>>
>>> INSTALL_DIR=$ROOT_DIR/private
>>> BUILD_DIR=$ROOT_DIR/private/build
>>>
>>> This is buildCaffe.sh
>>> #!/bin/bash
>>> owd=`pwd`
>>>
>>> source ./setupAuton.sh
>>>
>>> if [ ! -d $BUILD_DIR ]; then
>>>     mkdir -p $BUILD_DIR
>>> fi
>>> cd $BUILD_DIR
>>> echo "Installing to: " $INSTALL_DIR
>>>
>>> temp=`pkg-config --cflags protobuf --silence-errors`
>>> if [ $? -ne 0 ]; then
>>>     echo "Building Google Protocol Buffers"
>>>     git clone --branch v2.6.0 https://github.com/google/protobuf
>>>     cd protobuf
>>>     ./autogen.sh && ./configure --prefix=$INSTALL_DIR && make -j 10 &&
>>> make install
>>>     if [ $? -ne 0 ]; then
>>>         exit 1
>>>     fi
>>>     ADD_PKG_PATH=$INSTALL_DIR/bin/pkgconfig
>>>     PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$ADD_PKG_PATH
>>> fi
>>>
>>> cd $BUILD_DIR
>>> aclocal-1.14 --help >& /dev/null
>>> if [ $? -ne 0 ]; then
>>>     echo "Building Autotools"
>>>     wget http://ftp.gnu.org/gnu/automake/automake-1.14.1.tar.gz
>>>     tar -xzf automake-1.14.1.tar.gz
>>>     rm automake-1.14.1.tar.gz
>>>     cd automake-1.14.1
>>>     ./configure --prefix=$INSTALL_DIR
>>>     make -j 10 && make install
>>>     if [ $? -ne 0 ]; then
>>>         echo "Unable to build automake"
>>>         exit 1
>>>     fi
>>>     ADDPATH=$ADDPATH:/$INSTALL_DIR/bin
>>>     PATH=$PATH:$ADD_PATH
>>> fi
>>>
>>> cd $BUILD_DIR
>>> temp=`pkg-config --cflags libglog --silence-errors`
>>> if [ $? -ne 0 ]; then
>>>     echo "Building GLOG"
>>>     wget http://google-glog.googlecode.com/files/glog-0.3.3.tar.gz
>>>     tar -xzf glog-0.3.3.tar.gz
>>>     rm glog-0.3.3.tar.gz
>>>     cd glog-0.3.3
>>>     ./configure --prefix=$INSTALL_DIR && make -j 10 && make install
>>>     if [ $? -ne 0 ]; then
>>>         exit 1
>>>     fi
>>>     ADD_PKG_PATH=$INSTALL_DIR/bin/pkgconfig
>>>     PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$ADD_PKG_PATH
>>> fi
>>>
>>>
>>> cd $BUILD_DIR
>>> temp=`pkg-config --cflags libgflags --silence-errors`
>>> if [ $? -ne 0 ]; then
>>>     echo "Building gflags"
>>>     git clone --branch v2.0 https://github.com/gflags/gflags
>>>     cd gflags
>>>     ./configure --prefix=$INSTALL_DIR && make -j 10 && make install
>>>     if [ $? -ne 0 ]; then
>>>         exit 1
>>>     fi
>>>     ADD_PKG_PATH=$INSTALL_DIR/bin/pkgconfig
>>>     PKG_CONFIG_PATH=$PKG_CONFIG_PATH:$ADD_PKG_PATH
>>> fi
>>>
>>> if [ ! -d $INSTALL_DIR/include/leveldb ]; then
>>>     cd $BUILD_DIR
>>>     git clone https://github.com/google/leveldb
>>>     cd leveldb
>>>     make -j 10
>>>     if [ $? -ne 0 ]; then
>>>         echo "LevelDB build failed"
>>>         exit 1
>>>     fi
>>>     cp lib* $INSTALL_DIR/lib
>>>     cp -rp include/leveldb $INSTALL_DIR/include
>>> fi
>>> cd $BUILD_DIR
>>> if [ ! -e $INSTALL_DIR/lib/liblmdb.a ]; then
>>>     git clone https://github.com/LMDB/lmdb
>>>     cd lmdb/libraries/liblmdb
>>>     make -j 20
>>>     if [ $? -ne 0 ]; then
>>>         echo "Error building LMDB"
>>>         exit 1
>>>     fi
>>>     cp lmdb.h $INSTALL_DIR/include
>>>     cp liblmdb* $INSTALL_DIR/lib
>>> fi
>>>
>>>
>>> ###########INSTALL HDF5##############
>>> cd $BUILD_DIR
>>> ./buildHDF5.sh $INSTALL_DIR
>>> if [ $? -ne 0 ]; then
>>>     exit 1
>>> fi
>>>
>>> if [ ! -e $INSTALL_DIR/lib/libsnappy.a ]; then
>>>     echo "Building libsnappy"
>>>     wget --no-check-certificate
>>> https://github.com/google/snappy/releases/download/1.1.3/snappy-1.1.3.tar.gz
>>>     tar -xzpf snappy-1.1.3.tar.gz
>>>     rm snappy*.tar.gz
>>>     cd snappy-1.1.3
>>>     ./configure --prefix=$INSTALL_DIR && make -j 10 && make install
>>>     if [ $? -ne 0 ]; then
>>>         echo "libsnappy build failed"
>>>     exit 1
>>>     fi
>>> fi
>>>
>>>
>>> if [ ! -d $BUILD_DIR/OpenBLAS ]; then
>>>     cd $BUILD_DIR
>>>     echo "Building OpenBlas"
>>>     git clone https://github.com/xianyi/OpenBLAS
>>>     cd OpenBLAS
>>>     make TARGET=SANDYBRIDGE -j 20 && make install PREFIX=$INSTALL_DIR
>>>     if [ $? -ne 0 ]; then
>>>     exit 1
>>>     fi
>>> fi
>>>
>>> cd $BUILD_DIR
>>> echo "Building Caffe"
>>> if [ ! -e caffe ]; then
>>> #    git clone ssh://suppe at alfa.ahs.ri.cmu.edu/home/suppe/repo/caffe
>>>     git clone https://github.com/bvlc/caffe
>>> fi
>>> cd caffe
>>> if [ $USE_GPU -eq 0 ]; then
>>>     sed -e 's/# CPU_ONLY/CPU_ONLY/g' Makefile.config.example >
>>> Makefile.config
>>> else
>>>     cp Makefile.config.example Makefile.config
>>> fi
>>> sed --in-place '\|^INCLUDE_DIRS|s|$| '$INSTALL_DIR'/include|'
>>> Makefile.config
>>> sed --in-place '\|^LIBRARY_DIRS|s|$| '$INSTALL_DIR'/lib|' Makefile.config
>>> sed --in-place 's/BLAS := atlas/BLAS := open/g' Makefile.config
>>> printf 'BLAS_INCLUDE := %s/include\n' $INSTALL_DIR >> Makefile.config
>>> printf 'BLAS_LIB := %s/lib\n' $INSTALL_DIR >> Makefile.config
>>>
>>> make -j 100 && make distribute
>>> if [ $? -ne 0 ]; then
>>>     echo "Caffe build failed!"
>>>     exit 1
>>> fi
>>>
>>>
>>>
>>>
>>> -------- Original Message --------
>>> From: Arne Suppe <suppe at andrew.cmu.edu>
>>> Subject: CAFFE Script
>>> Date: Wed, 2 Dec 2015 10:38:43 -0500
>>> To: Predrag Punosevac <predragp at cs.cmu.edu>
>>>
>>>
>>> If you use the buildCaffe.sh script, you will need to update the
>>> setupAuton.sh for your username.  In general, I use the Springdale version
>>> of a package if its available, and only download and compile if its not.
>>> You probably already installed versions of these packages.  Towards the
>>> end, I build OpenBLAS and then used sed to update the Makefile.config.
>>>
>>> Hope this helps!
>>> Arne
>>>
>>>
>>>
>>>
>>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/autonlab-users/attachments/20151202/af02147b/attachment-0001.html>

From predragp at cs.cmu.edu  Tue Dec 22 12:02:41 2015
From: predragp at cs.cmu.edu (Predrag Punosevac)
Date: Tue, 22 Dec 2015 12:02:41 -0500
Subject: defected power supply
Message-ID: <20151222170241.BRqkGZiz668w%predragp@cs.cmu.edu>

Dear Autonians,

This is just a notice that I have fixed the defected power supply of our
DMZ firewall Horae. The failure has caused various network issues since
early morning including unavailability of the Auton Lab website.

Thanks,
Predrag