[auton-users] Auton System Status

[Michael J. Baysek]

  Hi Lab,

The system is mostly restored.  Some services I am still restoring, but 
primary services are available.

Yesterday evening, the power in the SCS machine room was severed.  I 
have been told that the outage occurred due to an incorrectly performed 
test on the new fire suppression system.  The test was performed by an 
external inspector.  The power interruption was long enough that all of 
the backup batteries went dead, and took the entire system (and Wean 
Hall 3611 SCS machine room) completely offline.  That said, you should 
know that any and all compute jobs that were running have been 
terminated since every last server went down.  It is likely that you 
will see (or have seen) a notice from SCS explaining the unexpected outage.

Normally, I receive a call from SCS when any of our public facing 
machines go down.  This time, however, I received no phone call, because 
they were in crisis mode themselves to get things like DNS, mail, and 
AFS restored.  In crisis mode, their policy is not to deal at all with 
'project servers' until the crisis is resolved.  They are still here in 
droves in the machine room finishing up their recovery.

Due to the entire machine room, including networking equipment losing 
the power, I received no notifications when anything went down.  
Additionally, I did not get my email until this morning, which is the 
first I had learned of the problem.  As a result of this incident, I 
will be setting up extensive off site monitoring, so that even if all 
else fails, I will still be notified of issues in a timely manner.

I recommend that everybody store my office number (412) 268-8939 and 
mobile number (412)-229-7356 in your cell phone.  Please feel welcome to 
call me if you see any extended unannounced, or unexpected down times.

Please don't call my mobile during off-hours for user requests and 
things that can wait until the next day - use email for that.  Please 
understand that there are just too many of you, and I'd be getting calls 
all the time at home and while out.

Again, system is 95% restored.  The 5% that is still to be done is not 
likely to affect most of you.

Cheers,

Mike