HDF5 on AutonLab resources

[George Stoica]

Hi Anthony and Predrag,

Thanks very much for the advice and help.

I had used the -prefix flag to get the BOOST install working, but had some
additional trouble installing HDF5 using the same type of install. I didn't
know about these additional flags and path variables so this is very
helpful thank you! And additionally the hdf5 in conda. I'll take another
look into this.

Regarding an resource-wide installation of HDF5, it would be much
appreciated. I don't see any particular version restriction for me, so
whatever would work best for everyone.

Regarding the sudo privileges: I completely agree about (me) not having
these privileges and I apologize if my email came across in that way. I
only mentioned it because I wasn't sure how best to move forward. But both
the comments in these emails have been very helpful.

Thanks very much,
George

On Tue, Jun 16, 2020 at 8:10 PM Anthony Wertz <awertz at cmu.edu> wrote:

> George,
>
> If you want to build specific dependencies yourself, you’ll have to do a
> little research, but most (if not all) build systems default to a
> system-visible install path (i.e., requires root) but allow specifying a
> custom (e.g., local) install path instead. For UNIX makefiles (one option
> for HDF5) it requires the —prefix directive. For CMAKE (an alternative,
> preferred (by me) option for HDF5) you use the CMAKE_INSTALL_PREFIX
> directive. For boost you have to double check, it uses it’s own build tool,
> but I think they directive is, again, —prefix. A common procedure is to
> install packages locally to ~/.local and add ~/.local/bin to your path and
> ~/.local/lib[64] to your LD_LIBRARY_PATH and LIBRARY_PATH environment
> variables. You may also want to look at PKG_CONFIG_PATH and MAN_PATH.
>
> I routinely build specific versions of various packages (including both
> HDF5 and boost), but it can be cumbersome to get started if you’re not
> already pretty comfortable using C/C++ build systems and working out build
> dependencies. And note that building those packages may imply building some
> number of additional packages in order to get the required capabilities.
>
> *However, *it is also worth noting that miniconda provides a lot of these
> dependencies already, including a newer version of HDF5. Predrag sent out
> an email earlier today or yesterday about where to find miniconda on the
> servers, I would google how to setup a conda environment and then just use
> `conda install hdf5`, the install caffe (noting that you’ll need to
> activate that conda environment to install and to use caffe).
>
>
> - *Anthony*
>
> On 16Jun2020, at 22:41, Predrag Punosevac <predragp at andrew.cmu.edu> wrote:
>
> George Stoica <gis at andrew.cmu.edu> wrote:
>
> Hi Predrag,
>
> I hope everything is well.
>
> I was wondering whether there might be an existing HDF5 installation path
> on the autonlab resources? I have been trying to install caffe, and the
>
>
> HDF5 had some nasty dependency issues. Package management is NP hard
> problem so that is expected from time to time on the large projects like
> RHEL. I removed it from our computing nodes as nobody needed it. I would
> be happy to install HDF5 if you tell me exactly what packages. It is
> actually very good and very useful data format. I like it a lot.
> I would install it on one of machines running RHEL 8.2. Eventually all
> computing nodes will run 8.xxx branch so there is no point looking back.
>
>
> respective Makefile cannot find any HDF5 distribution. I have tried
> installing the latest distribution myself but it seems like there may be a
> roadblock with sudo privileges (at least based on the sources I have
> found).
>
>
> We use adhere to "The Principle of Least Privilege" around the Auton Lab
> as documented on our Wiki. No you nor for that matter most people with
> the lab accounts can't execute any commands which require elevated
> privileges.
>
>
> Or do you know if there might be an existing caffe installation that may be
> used? I ask this because there were other dependency issues such as a too
>
>
> People did use Caffe in the past on our servers.
>
>
> old version of BOOST (the server default is 1.53,whereas the MakeFile
> expects at least 1.54). While I believe I've gotten past these problems, I
>
>
>
> There are multiple version of Boost C++ library installed on RHEL 7.8.
> RHEL 8.2 comes with quite new C++ compiler and Boost libraries. Not many
> people around the lab use C++ these days.
>
>
> am not very familiar with solving these lower level dependency issues and
> am concerned about potentially inadvertently screwing something up on the
> server.
>
>
> That is why you are not member of any sudoers group. The second rule in
> the lab is if you break things you get to keep all the pieces and you
> have to put them back together. Can you guess who broke the most things
> around here:-) Hint: He is just typing the replay to your email.
>
>
> Cheers,
> Predrag
>
>
>
> Thanks very much,
> George
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/autonlab-users/attachments/20200616/2e454c88/attachment-0001.html>