Computing node resources

[Sibi Venkatesan]

Hi everyone,

I ran into an issue earlier today where I thought I was running some single
threaded python code but numpy defaults to multi-threaded for some linear
algebra. I think I might be the only one running into this problem (perhaps
because my numpy version is different or something). But I got around it by
setting the following environment variables when I wanted to force it to be
single threaded:

export MKL_NUM_THREADS=1
export NUMEXPR_NUM_THREADS=1
export OMP_NUM_THREADS=1

More info here:
http://stackoverflow.com/questions/17053671/python-how-do-you-stop-numpy-from-multithreading
http://stackoverflow.com/questions/30791550/limit-number-of-threads-in-numpy

Maybe there's a simpler way, or maybe no one else faces this problem. But
it did fix the issue for me.



On Wed, May 17, 2017 at 1:41 PM Benedikt Boecking <boecking at andrew.cmu.edu>
wrote:

> All,
>
> Right now there are over *80 threads* running on *lov4*, a machine that
> only has *64 cores*. The same happened on lov3 earlier today. I know that
> deadlines are approaching but please try to follow some reasonable person
> principles. Here is a non-exhaustive list of things you should do before
> running experiments on our servers:
>
> 1. Before starting a new job, check the amount of available memory and how
> many other jobs are currently running. The easiest way to do this is to use
> htop.
> 2. If a computing node is at its limit, check if any other nodes are
> underutilized (http://monit.autonlab.org:8080/status/hosts/)
> 3. “nice" your jobs if they require a lot of resources and will be running
> for a long time (https://en.wikipedia.org/wiki/Nice_(Unix))
> 4. Use a reasonable number of threads and limit excessive memory usage.
> 5. Close your jupyter notebooks, matlab sessions etc. that you don’t need
> anymore
> 6. Move files from the scratch to your home directory on zfsauton if you
> don’t need them anymore for your current experiments.
> 7. If you are using GPUs, use nvidia-smi to check utilization and make
> sure your code does not automatically allocate all GPUs and all GPU memory
> to your experiment.
>
> Please respond to this email if you have any additional recommendations
> for your fellow lab members.
>
> Best,
> Ben
>
>
>
> --

- Sibi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/autonlab-users/attachments/20170517/31c50c52/attachment.html>