Server load

[Benedikt Boecking]

Hi all,

We are currently running into problems on several servers due to some users spawning too many threads. On several compute nodes we have several thousand threads running in parallel. 

To be a good lab member please:
1. Use top/htop and nvidia-smi to monitor your resource usage and keep it reasonable (memory, cpu, and gpu usage). 

2.  Check that you aren’t using more threads than intended due to automatic multiprocessing. This happens for example with numpy and spicy linalg functions on the servers.You can control most of this behavior by setting the following variables before running scripts or notebooks, setting them to however many threads you want:

export MKL_NUM_THREADS=1
export NUMEXPR_NUM_THREADS=1
export OMP_NUM_THREADS=1

3. If you have (interactive) session open that you don’t need anymore please close them to free memory (in particular Matlab and jupyter notebooks)



Thanks,
Ben