Neill file server practically killed

Mon May 22 12:24:04 EDT 2017

This message is of concern only for the members of Neill group.

I just got from a vocation to find out that one of the group
members have practically trashed the file server over the past 7 days.
A prime suspect is

5.8T    emcfowla

Namely over the past week somebody have created so much data
on the file server that ZFS pool is practically
100% in use (6.5 TB of compressed data). ZFS is not a normal file
system and once that full I will have very hard time to recover the
file server operation.

I need you to stop generating any data onto NFS right now as none of
you will be able to log into the system. It will take me a day or two
to develop a strategy how to deal with this (the fact that I am 
receiving
4 interns today is not helping). Server might be unusable for at least
a week.

One possible outcome will be that we will need to buy a new file server
to resume day to day operations while we clean and rebuilt this one.

Best,
Predrag

-------- Original Message --------
Subject: Autonlab-sysinfo Digest, Vol 34, Issue 51
Date: 2017-05-22 12:00
 From: autonlab-sysinfo-request at autonlab.org
To: autonlab-sysinfo at autonlab.org
Reply-To: autonlab-sysinfo at autonlab.org

Send Autonlab-sysinfo mailing list submissions to
	autonlab-sysinfo at autonlab.org

To subscribe or unsubscribe via the World Wide Web, visit
	https://mailman.srv.cs.cmu.edu/mailman/listinfo/autonlab-sysinfo
or, via email, send a message with subject or body 'help' to
	autonlab-sysinfo-request at autonlab.org

You can reach the person managing the list at
	autonlab-sysinfo-owner at autonlab.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Autonlab-sysinfo digest..."

Today's Topics:

    1. Critical Alerts (auton.sysnotify at gmail.com)
    2. Critical Alerts (auton.sysnotify at gmail.com)
    3. Critical Alerts (auton.sysnotify at gmail.com)
    4. Critical Alerts (auton.sysnotify at gmail.com)
    5. M/Monit report (ILIM: Monit instance changed monit on 22 May
       13:36:34 +0000) (auton.sysnotify at gmail.com)
    6. Critical Alerts (auton.sysnotify at gmail.com)

----------------------------------------------------------------------

Message: 1
Date: Mon, 22 May 2017 10:10:01 -0000
 From: auton.sysnotify at gmail.com
To: sysinfo at autonlab.org
Subject: Critical Alerts
Message-ID: <5922b8fa.83d4370a.3eeea.3018 at mx.google.com>
Content-Type: text/plain; charset="utf-8"

The capacity for the volume 'zfsneill' is currently at 92%, while the 
recommended value is below 80%.

------------------------------

Message: 2
Date: Mon, 22 May 2017 10:55:00 -0000
 From: auton.sysnotify at gmail.com
To: sysinfo at autonlab.org
Subject: Critical Alerts
Message-ID: <5922c385.c72c370a.2f0d0.2fec at mx.google.com>
Content-Type: text/plain; charset="utf-8"

The capacity for the volume 'zfsneill' is currently at 93%, while the 
recommended value is below 80%.

------------------------------

Message: 3
Date: Mon, 22 May 2017 11:45:00 -0000
 From: auton.sysnotify at gmail.com
To: sysinfo at autonlab.org
Subject: Critical Alerts
Message-ID: <5922cf3d.4e9f370a.98314.4357 at mx.google.com>
Content-Type: text/plain; charset="utf-8"

The capacity for the volume 'zfsneill' is currently at 94%, while the 
recommended value is below 80%.

------------------------------

Message: 4
Date: Mon, 22 May 2017 13:10:01 -0000
 From: auton.sysnotify at gmail.com
To: sysinfo at autonlab.org
Subject: Critical Alerts
Message-ID: <5922e32a.e332c80a.cf14b.05cc at mx.google.com>
Content-Type: text/plain; charset="utf-8"

The capacity for the volume 'zfsneill' is currently at 95%, while the 
recommended value is below 80%.

------------------------------

Message: 5
Date: Mon, 22 May 2017 13:53:36 +0000
 From: auton.sysnotify at gmail.com
To: sysinfo at autonlab.org
Subject: M/Monit report (ILIM: Monit instance changed monit on 22 May
	13:36:34 +0000)
Message-ID: <177C61CBCF6F4F16A56E7A5025779844 at monit.int.autonlab.org>
Content-Type: text/plain; charset="utf-8"

Date:        22 May 13:36:34 +0000
Host:        ILIM
Service:     monit
Action:      Start
Description: Monit 5.16 started

Your faithful employee,
M/Monit
-------------- next part --------------
An HTML attachment was scrubbed...
URL: 
<http://mailman.srv.cs.cmu.edu/pipermail/autonlab-sysinfo/attachments/20170522/5995afd9/attachment-0001.html>

------------------------------

Message: 6
Date: Mon, 22 May 2017 15:40:01 -0000
 From: auton.sysnotify at gmail.com
To: sysinfo at autonlab.org
Subject: Critical Alerts
Message-ID: <59230652.9163370a.9da54.5e52 at mx.google.com>
Content-Type: text/plain; charset="utf-8"

The capacity for the volume 'zfsneill' is currently at 96%, while the 
recommended value is below 80%.

------------------------------

Subject: Digest Footer

_______________________________________________
Autonlab-sysinfo mailing list
Autonlab-sysinfo at autonlab.org
https://mailman.srv.cs.cmu.edu/mailman/listinfo/autonlab-sysinfo

------------------------------

End of Autonlab-sysinfo Digest, Vol 34, Issue 51
************************************************