Regarding GPU contention

[Sarveshwaran Jayaraman]

Hi All,


Following is snapshot of sample GPU node usage which is usually run by a single user. This pattern seems to have dramatically increased over the last couple of days.

|   0  GeForce RTX 208...  Off  | 00000000:18:00.0 Off |                  N/A |
| 35%   57C    P2    92W / 250W |   7500MiB / 11019MiB |     38%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce RTX 208...  Off  | 00000000:3B:00.0 Off |                  N/A |
| 38%   63C    P2   126W / 250W |   3738MiB / 11019MiB |     64%      Default |
+-------------------------------+----------------------+----------------------+
|   2  GeForce RTX 208...  Off  | 00000000:86:00.0 Off |                  N/A |
| 41%   66C    P2   137W / 250W |   3747MiB / 11019MiB |     55%      Default |
+-------------------------------+----------------------+----------------------+
|   3  GeForce RTX 208...  Off  | 00000000:AF:00.0 Off |                  N/A |
| 41%   67C    P2   140W / 250W |   3705MiB / 11019MiB |     54%      Default |

I usually need to load model backbone  for my experiments which requires around 10GB of RAM, but because of above usage pattern I am unable to really use any GPU. One way around this to increase percentage GPU usage and reduce number of GPUs used.

Please be mindful especially if you're running across multiple GPU and your process has GPU usage in bursts. Thanks for your help!


[1562005799537]<https://www.autonlab.org/>

        Sarvesh Jayaraman<https://www.linkedin.com/in/sarveshjayaraman/>
Sr. Research Analyst, Auton Lab
Carnegie Mellon University
Mob: +1-240-893-4287


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/autonlab-users/attachments/20200422/0936918d/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OutlookEmoji-156200579953713abd276-b334-4989-90ea-a7e5da57a386.png
Type: image/png
Size: 5461 bytes
Desc: OutlookEmoji-156200579953713abd276-b334-4989-90ea-a7e5da57a386.png
URL: <http://mailman.srv.cs.cmu.edu/pipermail/autonlab-users/attachments/20200422/0936918d/attachment.png>