Size limit of scratch directory

[Predrag Punosevac]

Hi Sebastian,

I am CC-ing users as your question and my answer will be of concern to others.

The sizes of scratch directories are only limited by the total
capacity of the HDD hosting the directory. Some of those HDDs are
small (256GB) as they are NVMe drivers. The others are not that small
2TB. Most computing nodes have free drive bays which will allow for
the expansion of the scratch capacity. I do have older HDDs in our
storage room which could be used for that. However, except in the
extreme cases of small 256 NVMe drives I was reluctant to consider
adding extra scratch space. Most often than not the scratch space is
filled because people don't clean after themselves. The system in
which I keep adding resources because people are lazy (for the lack of
better word) to pick up after themselves doesn't scale well. Thus in
the past we had to clean scratch on a few machines. IIRC this has
happened 3 times since I came to lab 8 years ago but it did happen.
facultiesCheers,
Predrag

On Wed, Jun 24, 2020 at 2:53 PM Sebastian Caldas Rivera <scaldas at cmu.edu> wrote:
>
> Hello Predrag,
>
> I hope everything is well with you and your family.
>
> I had a question about the maximum storage size of the scratch directory. I have been running into issues with my experiments on lov5 because of "No space left on device" errors. My scratch directory on that machine should be ~4GB. I tried copying some data into my scratch directory in another machine (lov3) to run experiments there but got the same error even while just copying the data.
>
> Are there any limits on the scratch directories I'm unaware of? Am I doing something wrong when it comes to handling my data?
>
> Thank you,
> Sebastian