high system cpu usage with recent numpy updates

[Simon Heath]

Hi all,

We've updated the /etc/profile default file on the compute nodes to have
the environment variable OMP_NUM_THREADS=1 by default.  This *should* make
things behave the way they behaved before the update.  If you want the
OpenMP library to do its threading for you then you can just override the
environment variable in your own ~/.profile file, or set it when you start
your script like Ben demonstrated.

NOTE that you will have to log out and log back in before this change takes
effect.  This includes any screen or tmux sessions you may have active!

Let Predrag or I know if this causes other problems.

Simon

On Mon, Apr 30, 2018 at 5:32 PM, David Bayani <dbayani at andrew.cmu.edu>
wrote:

> Thanks Ben. You beat me to the punch sending this out. I give credit to
> you for first suggesting updates to scipy were responsible. As Ben pointed
> out in person, this is expected to become more of an issue as the machines
> in the lab have their packages updated, motivating this group email so
> everyone knows now. Providing a bit more material below on the subject in
> case anyone is interested:
>
> The place this flag suggestion was initially found was:
>     https://stackoverflow.com/questions/22418634/numpy-
> openblas-set-maximum-number-of-threads
> There are some conversations and related links there that might be
> enlightening. Checking the release notes on the newer versions of scipy and
> numpy were also useful.
>
> For me, this became an issue when multiple threads were being spawn
> despite the fact that multithreading was not explicitly invoked in the code
> in question.
> Specifically, using only scipy sparse matrix libraries and looking at the
> column NLWP in htop,  some machines could be seen using one thread and
> others using as many threads as thread contexts.
> Machines that had single threads listed for the processes were using:
> >>> import scipy
> >>> scipy.__version__
> '0.12.1'
> >>> import numpy
> >>> numpy.__version__
> '1.7.1'
> and machines that invoked multiple threads were using:
> >>> import scipy
> >>> scipy.__version__
> '1.0.0'
> >>> import numpy
> >>> numpy.__version__
> '1.14.0'
> For the runs in question, life went much better after forcing the number
> of threads to one. As Ben said, using more threads does not imply that runs
> will go no slower.
>
> On Mon, Apr 30, 2018 at 3:59 PM, Matthew Barnes <mbarnes1 at andrew.cmu.edu>
> wrote:
>
>> This happens even with basic multiprocessing in Python, for example the
>> multiprocessing.Pool.map operation. Don't be like me and accidentally start
>> 2500 processes :)
>>
>> On Mon, Apr 30, 2018 at 3:53 PM Benedikt Boecking <
>> boecking at andrew.cmu.edu> wrote:
>>
>>> All,
>>>
>>> With newer versions of numpy (and maybe scipy) it is possible that some
>>> operations use all available CPUs by default (thanks to David Bayani for
>>> pointing this out). This can also happen if you use packages that rely on
>>> numpy and scipy such as statsmodels. On our servers this appears to be
>>> caused by the use of the open MP API.
>>>
>>> While automatic multi processing can be a great feature, it can cause
>>> trouble if it is combined with additional multi processing (e.g. your own
>>> use of the multiprocessing or joblib libraries) or when multiple users
>>> unwittingly spawn too many threads at the same time.
>>>
>>> If you want to control the number of threads used through open MP, use
>>> the OMP_NUM_THREADS environment variable  when you run your python code
>>> (with a reasonable number of threads):
>>>
>>> [user at server ~]$ OMP_NUM_THREADS=8 python yourscript.py
>>>
>>> Also, it is a great habit to run top or htop to monitor your resource
>>> consumption to make sure you aren’t inconveniencing other users of our
>>> lab’s resources.
>>>
>>> Best,
>>> Ben
>>>
>>
>


-- 
Simon Heath, Research Programmer and Analyst
Robotics Institute - Auton Lab
Carnegie Mellon University
sheath at andrew.cmu.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/autonlab-users/attachments/20180430/951c1553/attachment-0001.html>