GTA5 or Cityscapes dataset on Autonlab cluster?

[Predrag Punosevac]

Hitesh Arora <hiteshar at andrew.cmu.edu> wrote:

> Hi Autonians,
> 
> Has anyone already downloaded the GTA5 and/or Cityscapes datasets onto the
> Auton cluster, and willing to share. It would save a bit of time in
> downloading/transfer. Thanks!

Actually I have a better idea. Before we all went into hiding I fully
provisioned the newest file server bought by Dr. Jeff Schneider late
last year.  The server has for now only one ZFS pool due to the lack of
HDDs but six storage HDDs which are installed are 12 TB each. Thus the
ZFS pool I created has 48TB of useful space even before standard lz4
compression ZFS uses. For the first time we even have dedicated  2 NVMe
ZFS mirror slog for the pool.

https://www.ixsystems.com/blog/zfs-zil-and-slog-demystified/

although that should only impact write speed. 

I already remote replicated /zfsauton/data and /zfsauton/project data
sets onto the new file server but I am scared to pull the trigger and
change anything in the current NFS setup before we come out of hiding.
However, I think it would be a very good idea that we consolidate all
standard third party datasets people use in the Lab in a new ZFS
dataset/NFS share which will be accessible to all users across all
computing nodes. That, in theory, should off load some of the traffic
from other 2 NFS servers we use for home, data, and project directories.
That, in theory, should also partially solve the fact that scratch
directories are machine specific. Hopefully that would also help us use
less storage space as we will solve duplication problem.

What I need from users is the name of dataset you think we should share
with everyone and its current location. 

Most Kind Regards,
Predrag



> 
> Regards,
> Hitesh