high system cpu usage with recent numpy updates

[Matthew Barnes]

This happens even with basic multiprocessing in Python, for example the
multiprocessing.Pool.map operation. Don't be like me and accidentally start
2500 processes :)

On Mon, Apr 30, 2018 at 3:53 PM Benedikt Boecking <boecking at andrew.cmu.edu>
wrote:

> All,
>
> With newer versions of numpy (and maybe scipy) it is possible that some
> operations use all available CPUs by default (thanks to David Bayani for
> pointing this out). This can also happen if you use packages that rely on
> numpy and scipy such as statsmodels. On our servers this appears to be
> caused by the use of the open MP API.
>
> While automatic multi processing can be a great feature, it can cause
> trouble if it is combined with additional multi processing (e.g. your own
> use of the multiprocessing or joblib libraries) or when multiple users
> unwittingly spawn too many threads at the same time.
>
> If you want to control the number of threads used through open MP, use the
> OMP_NUM_THREADS environment variable  when you run your python code (with a
> reasonable number of threads):
>
> [user at server ~]$ OMP_NUM_THREADS=8 python yourscript.py
>
> Also, it is a great habit to run top or htop to monitor your resource
> consumption to make sure you aren’t inconveniencing other users of our
> lab’s resources.
>
> Best,
> Ben
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/autonlab-users/attachments/20180430/04ff92a3/attachment.html>