bash.autonlab.org

[Predrag Punosevac]

deimos.autonlab.org is fully functional again. I am restarting OpenVPN
on your desktops right now. It will take about 30-40 min.

Sorry for all that nonsense.

Predrag

On Thu, Dec 5, 2019 at 11:05 PM Predrag Punosevac
<predragp at andrew.cmu.edu> wrote:
>
> Dear Autonians,
>
> I have performed tonight a routine patch update of our network
> servers. One of them deimos.autonlab.org is having trouble with a
> single 10 Gigabit interface. The machine is Supermicro Intel Xeon
> D-1541 network server which is our main VPN gateway. Without that
> server bash no longer can serve as a shell gateway and all Auton Lab
> desktops will be super slow as they will wait for primary DNS to time
> out.
>
> These servers have notoriously buggy firmware as previously noted by others
>
> https://tinkertry.com/how-to-work-around-intermittent-intel-x557-network-outages-on-12-core-xeon-d
>
> and reported by yours truly
>
> https://marc.info/?l=openbsd-misc&m=155789695701876&w=2
>
> A typical remedy is to try to cold rebooted server. I have done that
> using IPMI (yes I can power off and power on few selected servers from
> my home) but the temperature is still too high.  I will let server
> cool off over night and try to fire it up in the morning. During the
> night please use lop2.autonlab.org or lop1.autonlab.org if you have
> legacy account as gateways. In the case that network interface doesn't
> come in the morning I will have to plug the main network cable in one
> of reserved 1Gigabit interfaces as it is currently done with
> phobos.autonlab.org which had this problem ever since it was acquired.
> For that I will have to be in the machine room.
>
> I apologize for the inconvenience while people are getting ready for Vancuver.
>
> Best,
> Predrag