GTA5 or Cityscapes dataset on Autonlab cluster?

[Hitesh Arora]

Hi Predrag,

That's a great idea.

Following are the couple dataset/binaries we could share. Would request
others to share other datasets.

1. CARLA binaries : /zfsauton2/home/hiteshar/CARLA_0.9.6
(These are currently being used by at least 6-8 people where each has their
own copies)

2. Cityscapes:
/zfsauton/project/public/deep_clustering/data/datasets/cityscapes
(Thanks to Vincent for sharing)

Thanks,
Hitesh

On Wed, Mar 25, 2020 at 6:46 PM Predrag Punosevac <predragp at andrew.cmu.edu>
wrote:

> Hitesh Arora <hiteshar at andrew.cmu.edu> wrote:
>
> > Hi Autonians,
> >
> > Has anyone already downloaded the GTA5 and/or Cityscapes datasets onto
> the
> > Auton cluster, and willing to share. It would save a bit of time in
> > downloading/transfer. Thanks!
>
> Actually I have a better idea. Before we all went into hiding I fully
> provisioned the newest file server bought by Dr. Jeff Schneider late
> last year.  The server has for now only one ZFS pool due to the lack of
> HDDs but six storage HDDs which are installed are 12 TB each. Thus the
> ZFS pool I created has 48TB of useful space even before standard lz4
> compression ZFS uses. For the first time we even have dedicated  2 NVMe
> ZFS mirror slog for the pool.
>
> https://www.ixsystems.com/blog/zfs-zil-and-slog-demystified/
>
> although that should only impact write speed.
>
> I already remote replicated /zfsauton/data and /zfsauton/project data
> sets onto the new file server but I am scared to pull the trigger and
> change anything in the current NFS setup before we come out of hiding.
> However, I think it would be a very good idea that we consolidate all
> standard third party datasets people use in the Lab in a new ZFS
> dataset/NFS share which will be accessible to all users across all
> computing nodes. That, in theory, should off load some of the traffic
> from other 2 NFS servers we use for home, data, and project directories.
> That, in theory, should also partially solve the fact that scratch
> directories are machine specific. Hopefully that would also help us use
> less storage space as we will solve duplication problem.
>
> What I need from users is the name of dataset you think we should share
> with everyone and its current location.
>
> Most Kind Regards,
> Predrag
>
>
>
> >
> > Regards,
> > Hitesh
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/autonlab-users/attachments/20200325/be2c51ad/attachment.html>