Lop2 and Monit down + gateway questions

[Predrag Punosevac]

Nicholas Gisolfi <ngisolfi at cs.cmu.edu> wrote:

> Hi Predrag,
> 
> It looks like lop2 and monit are down.  There is a thread on the auton
> slack talking about alternate gateways...we found a list from an April 2020
> email of yours

Hi Autonians,


I figured out that it would be easier just to replay to all of you that
to replay to individual emails. 

This morning around 11:15 AM Kris Hutchings and I swapped two server
racks and this a bunch of electric work in the server room. This work
was planned since February and could no longer be postponed. Namely, I
claimed a bit more electricity for our lab which is necessary to bring
up 4 new CPU nodes which were collecting dust in the dungeon. I didn't
email about this as the two weeks in a row we had problems with people
coming up with the fever including your truly about 8 days ago so when I
saw Kris this morning in the server room we had to act at once.

I am in the process of restoring services from the most important once
(network, file servers) and hopefully bringing CPU nodes back before the
end of the day. It is possible that CPU work will have to be continued
tomorrow as there is just too much stuff down right now.

GPU computing should not be affected. Git and Monit appear to be down
but that is just a routing problem which I am fixing right now. 
lop2 is a virtual machine which will be restored hopefully shortly.

Predrag




> 
> lop2.autonlab.org
> lop1.autonlab.org
> lion.auton.cs.cmu.edu
> bash.autonlab.org
> 
> Ben (cc'ed) and I tried to log in via lion for the first time and it looks
> like our usual log-in credentials don't work.  I figured I should bring
> this to your attention, just in case this isn't expected behavior.
> 
> - Nick