high system cpu usage with recent numpy updates

[Benedikt Boecking]

All,

With newer versions of numpy (and maybe scipy) it is possible that some operations use all available CPUs by default (thanks to David Bayani for pointing this out). This can also happen if you use packages that rely on numpy and scipy such as statsmodels. On our servers this appears to be caused by the use of the open MP API. 

While automatic multi processing can be a great feature, it can cause trouble if it is combined with additional multi processing (e.g. your own use of the multiprocessing or joblib libraries) or when multiple users unwittingly spawn too many threads at the same time. 

If you want to control the number of threads used through open MP, use the OMP_NUM_THREADS environment variable  when you run your python code (with a reasonable number of threads):

[user at server ~]$ OMP_NUM_THREADS=8 python yourscript.py 

Also, it is a great habit to run top or htop to monitor your resource consumption to make sure you aren’t inconveniencing other users of our lab’s resources.

Best,
Ben