<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">All,<div class=""><br class=""></div><div class="">Right now there are over <b class="">80 threads</b> running on <b class="">lov4</b>, a machine that only has <b class="">64 cores</b>. The same happened on lov3 earlier today. I know that deadlines are approaching but please try to follow some reasonable person principles. Here is a non-exhaustive list of things you should do before running experiments on our servers:</div><div class=""><br class=""></div><div class="">1. Before starting a new job, check the amount of available memory and how many other jobs are currently running. The easiest way to do this is to use htop. </div><div class="">2. If a computing node is at its limit, check if any other nodes are underutilized (<a href="http://monit.autonlab.org:8080/status/hosts/" class="">http://monit.autonlab.org:8080/status/hosts/</a>)</div><div class="">3. “nice" your jobs if they require a lot of resources and will be running for a long time (<a href="https://en.wikipedia.org/wiki/Nice_(Unix))" class="">https://en.wikipedia.org/wiki/Nice_(Unix))</a></div><div class="">4. Use a reasonable number of threads and limit excessive memory usage.</div><div class="">5. Close your jupyter notebooks, matlab sessions etc. that you don’t need anymore</div><div class="">6. Move files from the scratch to your home directory on zfsauton if you don’t need them anymore for your current experiments. </div><div class="">7. If you are using GPUs, use nvidia-smi to check utilization and make sure your code does not automatically allocate all GPUs and all GPU memory to your experiment.</div><div class=""><br class=""></div><div class="">Please respond to this email if you have any additional recommendations for your fellow lab members. </div><div class=""><br class=""></div><div class="">Best,</div><div class="">Ben</div><div class=""><br class=""></div><div class=""><div class="">
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;"><br class=""></div><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;"><br class=""></div></div></div></body></html>