Athena is down

[Predrag Punosevac]

Predrag Punosevac <predragp at cs.cmu.edu> wrote:

> Predrag Punosevac <predragp at cs.cmu.edu> wrote:
> 
> > Dear Autonians,
> > 
> > I just got a message from Monit that Athena is down. Trying manually to
> > ping it didn't work. This is a very serious issue as Athena is our main
> > KVM host running typically more than 10 KVM guests. All Traffic Jam
> > project KVM instances are down and can't be recovered until I see what
> > is wrong with Athena.
> > 
> > I am looking into this right now.
> > 
> > Predrag
> 
> Dear Autonians,
> 
> Athena is up and running. I will proceed carefully with firing up
> virtual machines. Most importantly raid arrays look good at the moment.
> However the largest data pool/raid is still resyncing. It will take
> another couple hours to finish.
> 
> I will sent another update in about 2h.
> 
> Predrag

All virtual machines on Athena with exception of John Deere computing
node are up and running. Data pool is resync 65%. If you are in charge
with one of those virtual servers please take 10-15 minutes to make sure
that your services are running as expected. I am going through the log
files right now trying to understand what happened.

Predrag