[auton-users] LOFTY and LOCK are dying painful dead

[predragp at andrew.cmu.edu]

Many of you have experienced sloooow DNS resolution and slooooow login
process on LOP1, LOP2 gateways and many computing nodes. If you check
M/Monit you will see that this is due to the fact that our old primary
domain controller LOFTY is dying a painful dead. Just check how much
memory is LOFTY swapping.

Lofty runs Master instance of Bind DNS as well as NIS server and even DHCP
server.

The good news is that we have new cluster of DNSs and fully functional
replacement for NIS (LDAP server). However connection computing nodes is
going to take a bit longer. Ten minutes ago I have edited DHCP server
configuration files on LOFTY trying to point computing nodes which has not
been rebuilt to new DNS clusters. However network connections will have to
be restarted to change to take the place and that one will not be forced
on nodes.

The nodes which need to be rebuilt also use LOCK as a default gateway and
LOCK is dying too. I am scared to massively switch nodes to  AREAS as a
gateway due to the fact that AREAS has very serious firewall rules and
might break some of the services that not documented.

The whole situation has been anticipated and the conversion will be
finished before LOFTY and LOCK die.


Predrag