HDF5 on AutonLab resources

[Anthony Wertz]

George,

If you want to build specific dependencies yourself, you’ll have to do a little research, but most (if not all) build systems default to a system-visible install path (i.e., requires root) but allow specifying a custom (e.g., local) install path instead. For UNIX makefiles (one option for HDF5) it requires the —prefix directive. For CMAKE (an alternative, preferred (by me) option for HDF5) you use the CMAKE_INSTALL_PREFIX directive. For boost you have to double check, it uses it’s own build tool, but I think they directive is, again, —prefix. A common procedure is to install packages locally to ~/.local and add ~/.local/bin to your path and ~/.local/lib[64] to your LD_LIBRARY_PATH and LIBRARY_PATH environment variables. You may also want to look at PKG_CONFIG_PATH and MAN_PATH.

I routinely build specific versions of various packages (including both HDF5 and boost), but it can be cumbersome to get started if you’re not already pretty comfortable using C/C++ build systems and working out build dependencies. And note that building those packages may imply building some number of additional packages in order to get the required capabilities.

However, it is also worth noting that miniconda provides a lot of these dependencies already, including a newer version of HDF5. Predrag sent out an email earlier today or yesterday about where to find miniconda on the servers, I would google how to setup a conda environment and then just use `conda install hdf5`, the install caffe (noting that you’ll need to activate that conda environment to install and to use caffe).


- Anthony

> On 16Jun2020, at 22:41, Predrag Punosevac <predragp at andrew.cmu.edu> wrote:
> 
> George Stoica <gis at andrew.cmu.edu> wrote:
> 
>> Hi Predrag,
>> 
>> I hope everything is well.
>> 
>> I was wondering whether there might be an existing HDF5 installation path
>> on the autonlab resources? I have been trying to install caffe, and the
> 
> HDF5 had some nasty dependency issues. Package management is NP hard
> problem so that is expected from time to time on the large projects like
> RHEL. I removed it from our computing nodes as nobody needed it. I would
> be happy to install HDF5 if you tell me exactly what packages. It is
> actually very good and very useful data format. I like it a lot.
> I would install it on one of machines running RHEL 8.2. Eventually all
> computing nodes will run 8.xxx branch so there is no point looking back.
> 
> 
>> respective Makefile cannot find any HDF5 distribution. I have tried
>> installing the latest distribution myself but it seems like there may be a
>> roadblock with sudo privileges (at least based on the sources I have
>> found).
>> 
> 
> We use adhere to "The Principle of Least Privilege" around the Auton Lab
> as documented on our Wiki. No you nor for that matter most people with
> the lab accounts can't execute any commands which require elevated
> privileges.
> 
> 
>> Or do you know if there might be an existing caffe installation that may be
>> used? I ask this because there were other dependency issues such as a too
> 
> People did use Caffe in the past on our servers.
> 
> 
>> old version of BOOST (the server default is 1.53,whereas the MakeFile
>> expects at least 1.54). While I believe I've gotten past these problems, I
> 
> 
> There are multiple version of Boost C++ library installed on RHEL 7.8.
> RHEL 8.2 comes with quite new C++ compiler and Boost libraries. Not many
> people around the lab use C++ these days.
> 
> 
>> am not very familiar with solving these lower level dependency issues and
>> am concerned about potentially inadvertently screwing something up on the
>> server.
> 
> That is why you are not member of any sudoers group. The second rule in
> the lab is if you break things you get to keep all the pieces and you
> have to put them back together. Can you guess who broke the most things
> around here:-) Hint: He is just typing the replay to your email.
> 
> 
> Cheers,
> Predrag
> 
> 
>> 
>> Thanks very much,
>> George

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/autonlab-users/attachments/20200616/6bd3a731/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: Message signed with OpenPGP
URL: <http://mailman.srv.cs.cmu.edu/pipermail/autonlab-users/attachments/20200616/6bd3a731/attachment.sig>