From predragp at andrew.cmu.edu Tue Mar 4 23:40:25 2014 From: predragp at andrew.cmu.edu (predragp at andrew.cmu.edu) Date: Tue, 4 Mar 2014 23:40:25 -0500 Subject: [auton-users] /auton 97% full Message-ID: <68254a3164ec45d52d945aeafa92199f.squirrel@webmail.andrew.cmu.edu> There is only 300G left on the main file server. Could you please clean all unnecessary files until we release new file server? Most Kind Regards, Predrag From predragp at andrew.cmu.edu Tue Mar 11 14:25:15 2014 From: predragp at andrew.cmu.edu (predragp at andrew.cmu.edu) Date: Tue, 11 Mar 2014 14:25:15 -0400 Subject: [auton-users] Auton Lab Network Topology Message-ID: <6350334b6bce0afe8e58171126a83247.squirrel@webmail.andrew.cmu.edu> I just uploaded a figure of Auton Lab network topology which should help you navigate our network. Note you have to be on Auton Intranet in order to see the picture. Also note that for now connections and IP addresses are intensionally left out. Those will be published only if I get explicit permission from Lab directors. http://www.autonlab.org/auton_intranet/g2/21625.html Most Kind Regards, Predrag From predragp at andrew.cmu.edu Tue Mar 11 20:26:23 2014 From: predragp at andrew.cmu.edu (predragp at andrew.cmu.edu) Date: Tue, 11 Mar 2014 20:26:23 -0400 Subject: [auton-users] Important Network information Message-ID: Dear Autons, You might want to have the Auton Lab network topology map http://www.autonlab.org/auton_intranet/g2/21625.html while reading this e-mail. This is just a quick update on the progress of network update. A cluster of Unbounds (Unbound a validating, recursive, and caching DNS resolver) consisting of three machines Areas Atlas Horae is fully functional. However old Bind based Master & Slave (Lofty & Liar) DNS which is causing us so much pain is not decommissioned yet. It will be taken of line only once all machines in the lab are switched to DNS cluster. Areas is also designated as a new main firewall and gateway to the LAB while Horae is also DMZ firewall. Atlas main use is LDAP domain controller (to replace NIS) which is still not functional. I am in the process of killing DHCP server (runs on LOFTY) which was in the past used to dynamically assigned prescribed internal IP addresses to our server. All Auton Lab servers and virtual machines will have statically configured internal IP addresses within 24-36h. The following computing nodes are now fully switched to static IP addresses (listed next to the names), new gateway (Areas 192.168.6.2) and the above cluster of DNS servers. gaia 192.168.6.5 neill-zfs 192.168.6.59 low1 192.168.6.81 lov3 192.168.6.102 lov4 192.168.6.103 Within next 24 hours I will try to switch all other computing nodes. During this process you will notice at least two things 1. Until new OpenVPN server is up and running on Areas (hopefully next 36 hourse) the computing nodes will be reachable only through ssh gateway LOP1 and LOP2. You will not be able to ssh on them from your desktops because OpenVPN network is not cleared as safe. 2. I am filtering all network traffic on both external and internal interface in both directions in and out of Auton Lab LAN zone with very restrictive rules. Except ssh, http, and https you will not be able to reach outside world from computing nodes. Any other monkey business will have to be done in DMZ zone. Auton Lab gateways LOP1 & LOP2 will not be reconfigured because they act as proxy for TCWI instances, CVS and SVN proxy servers. Instead I will bring (as soon as LDAP is on) a new dedicated gateway (machine is already up and running). LOP1 & LOP2 as well as the rest of Auton Lab tangle will be dealt when the core network infrastructure is 100% functional. I apologize for any inconvenience. Unfortunately there is no easy way out of the mess and this has to be done better sooner than later. Most Kind Regards, Predrag Punosevac From predragp at andrew.cmu.edu Wed Mar 12 19:09:24 2014 From: predragp at andrew.cmu.edu (predragp at andrew.cmu.edu) Date: Wed, 12 Mar 2014 19:09:24 -0400 Subject: [auton-users] LOV4 back in business Message-ID: Dear Autonians, LOV4 is back in business after being frozen by runaway script. Unfortunately it had to be rebooted. I apologize for the inconvenience. Predrag From predragp at andrew.cmu.edu Thu Mar 13 22:08:50 2014 From: predragp at andrew.cmu.edu (predragp at andrew.cmu.edu) Date: Thu, 13 Mar 2014 22:08:50 -0400 Subject: [auton-users] New VPN server fully functional Message-ID: <72613282044fc4e954262fc4b1ec8a1d.squirrel@webmail.andrew.cmu.edu> Dear Autonians, If you are not using Auton Lab Linux based desktop you might stop reading this info now. If you care about security of our Lab you might want to continue reading even if you currently not using Linux based desktop. I have a good news. Our new OpenVPN server running on areas.autonlab.org is now fully functional. I tested both server and my own desktop machine as a client. These are the highlights: 1. Client and server run on machines as non privileged users in the chroot. 2. Certificates have proper permission. For example private keys are visible only by openvpn user. 3. Diffie hellman parameters 1024 bit keys (I can increase to 2048 if NSA type security is necessary) 4. Fully encrypted TLS connection 5. We use AES-256-CBC cryptography cipher (until now we used Blowfish) 6. I enabled HMAC firewall to help mitigate DoS attacks (ta.key) 7. The compression is enabled on all VPN links and finally 8. I am filtering in and out traffic on tun interface on areas.autonlab.org just like I am doing with any other physical interfaces. A casual computer user should notice change in the speed of DNS queries as soon as I turn on new VPNs on your machines. I have generated already certificates for all desktops. There is also one unpleasant discovery. Our NIS is using completely random ports to provide Linux computers with user credentials necessary for NFS (NFS itself uses ports 1111, 2049, 4000, 4001, 4002). This is completely unacceptable to me as such system can not be properly fire-walled and this is just additional reason that I speed up LDAP server (NIS replacement). Without LDAP I can not properly mount NFS on your desktops. Most Kind Regards, Predrag From predragp at andrew.cmu.edu Sun Mar 16 02:20:07 2014 From: predragp at andrew.cmu.edu (predragp at andrew.cmu.edu) Date: Sun, 16 Mar 2014 02:20:07 -0400 Subject: [auton-users] Autonlab machine spec page In-Reply-To: <53232049.6050203@cs.cmu.edu> References: <53232049.6050203@cs.cmu.edu> Message-ID: > Hi Dr. P^2, > > This is the page I was referring to: > http://www.autonlab.org/auton_intranet/computing/specs.html > Also, I noticed that the shared installation of matlab on our servers is > still R2013a and has licensing issues. > > Thanks, > TK > Dear Autonians, Per TK request I updated little bit Auton Intranet computing infrastructure documentation. As of today the following information is up to date. 1. Network topology (updated today again) http://www.autonlab.org/auton_intranet/computing/22083.html 2. Machine specifications http://www.autonlab.org/auton_intranet/computing/22086.html 3. OS Info http://www.autonlab.org/auton_intranet/computing/compute_os.html 4. Scientific software Info http://www.autonlab.org/auton_intranet/computing/22085.html Note that I added FAQ section to the list of installed software trying to address some commonly asked questions. Further I would like to address MATLAB issues mentioned in TKs e-mail. We do not have licensing issue with the MATLAB. We have the issue with obsolete network infrastructure which is currently being worked on. LOW1 has been switched to new gateway/firewall (Areas) while not being updated. As a consequence old shared installation of MATLAB 2013b on LOW1 which requires contact with the university licensing server is not usable at the moment. LOV3, LOV4, LOU1, LXV1, LXV2 are OK of course. At the moment only 5 out of 18 computing nodes are up to date but updating them now before I finish LDAP server on Atlas makes no sense because so much new things (new file servers release) depend on LDAP. I can assure you that once LDAP is finished I can update up to 3-4 computing nodes a day which means that for remaining 17-5=12 nodes (LOS1 is garbage and will not be updated) I need three working days. I hope that the answer to the second issue raised in TKs e-mail (obsolete MATLAB 2013b) is obvious. MATLAB 2014b has been released less than 7 days ago and updating MATLAB on obsolete and even non-obsolete computing nodes at the moment is low priority when there are so many more pressing issues. I do not want to put a time frame on MATLAB update but 2-3 weeks from now sounds like a good number. Please feel free to communicate any concerns you might have. I apologize for spotty documentation too in particularly regarding GUI access to computing nodes using X2Go client and LOP1 and LOP2 as proxies. I have everything on my radar screen but the finite amount of time to fix things. I didn't forget Munin metric monitoring. It is just put on hold until I finish more urgent things. Most Kind Regards, Predrag From predragp at andrew.cmu.edu Wed Mar 19 16:57:55 2014 From: predragp at andrew.cmu.edu (predragp at andrew.cmu.edu) Date: Wed, 19 Mar 2014 16:57:55 -0400 Subject: [auton-users] Important OpenVPN & DNS update Message-ID: Dear Autonians, Today at 4:00 PM EST old OpenVPN server on old firewall LOCK has been shutdown. That means that as of today 12 Linux desktops which I mange use new OpenVPN server running on Areas as well as new DNS servers. Due to the fact that LDAP is not available /auton shares are not usable. Also old computing nodes which are routed through LOCK has to be access through LOP1 and LOP2 since the firewall is preventing spoofing. However you should see the huge speed increase in name resolution. LOV3, LOV4, and LOW1 are available as usual. I am hopping to finish LDAP by the end of this week and have all computing nodes rebuild by next Wednesday (New file servers will become available as soon as I turn on LDAP). Most Kind Regards, Predrag P.S. As a bonus the software on all desktops is now updated, GCC 4.8.2 is available locally /opt/rh/devtoolset-2. I also updated local copies of MATLAB to 2014b on most desktops. From predragp at andrew.cmu.edu Wed Mar 26 15:21:50 2014 From: predragp at andrew.cmu.edu (predragp at andrew.cmu.edu) Date: Wed, 26 Mar 2014 15:21:50 -0400 Subject: [auton-users] SSH Security Info Message-ID: Dear Autonians, This is an important ssh info concerning overall lab network security which is related to today's maintenance work I performed on 14 Linux desktops. In a preparation to turn on LDAP I have restricted ssh access to Linux desktops only to desktops owners, auton-local, and backup accounts (backup doesn't have a real access it is only rsync command without ttu command). It occurred to me during the recent e-mail exchange with Jarod concerning new OpenVPN gateway that due to public IP address and LDAP enabled all Linux desktops could be used by anyone in the Lab as a ssh gateway to computing nodes but more importantly it could be used by anyone outside the lab to stage an attack to our internal network. By restricting ssh access on desktops to very few selected account I am practically eliminating large set of attack vectors. Yes We also run things as sshguard and fail2ban if people are curious. I would also like to add that unlike before your VPN tunnels can be only used for access to our internal network 192.168.6.0/24 and that VPN clients can't see each other. Finally I realize that some desktops potentially need to be accessed by additional users. For example it looks like Jarod have given permission to several people to access his desktop. If that is the case please let me know by an e-mail which users need to be able to have access to your desktop. By default those users should have only local accounts so they will not be in LDAP database and consequently unable to use your desktop to access internal network. Thank you for your attention! Predrag From predragp at andrew.cmu.edu Thu Mar 27 23:27:05 2014 From: predragp at andrew.cmu.edu (predragp at andrew.cmu.edu) Date: Thu, 27 Mar 2014 23:27:05 -0400 Subject: [auton-users] LDAP update Message-ID: <40e82fbc147751fb1e28030d19ea1d2d.squirrel@webmail.andrew.cmu.edu> Dear Autonians, LDAP server is ready. I have already created core users groups and user records for faculty. I have done everything from scratch since trying to extract info from NIS which has never been seriously cleaned would create garbage IMHO. The plan is to create record for remaining 50 users tomorrow and populate neill group first so that I could release their file server and rebuild their computing nodes over the weekend or possibly early next week. Releasing new file server and the rest of Auton Lab computing nodes will be slightly more chelenging due to existance of numerous project groups which have to be cleared by the Lab direstors before entered into LDAP. I apologize if this process looks painfully slow but I prefer to do things right instead of doing them fast. My surgery also have not help with the speed of the process. Most Kind Regards, Predrag Punosevac From predragp at andrew.cmu.edu Fri Mar 28 17:19:38 2014 From: predragp at andrew.cmu.edu (predragp at andrew.cmu.edu) Date: Fri, 28 Mar 2014 17:19:38 -0400 Subject: [auton-users] NEILL1 and NEILL2 down Message-ID: <25f6f217a8d9453092bb8b0d10a89ff9.squirrel@webmail.andrew.cmu.edu> I would like to take NEILL1 and NEILL2 down over the weekend in order to rebuild them and turn on LDAP. Dr. Neill could you please OK this? Predrag From predragp at andrew.cmu.edu Sat Mar 29 21:10:39 2014 From: predragp at andrew.cmu.edu (predragp at andrew.cmu.edu) Date: Sat, 29 Mar 2014 21:10:39 -0400 Subject: [auton-users] Monit aka. status page Message-ID: <26eb0c5bbba9f8044a7bed7ea05903c6.squirrel@webmail.andrew.cmu.edu> Dear Autonians, I would like to bring to your attention the fact that one of two lab monitoring systems Monit is now available WWW. You can see the real time status, load and many other useful parameters of selected computing nodes at http://monit.autonlab.org:8080 using username:auton password:Dr.Who I linked the webpage to the www.autonlab.org but you will need the username and password just to see the status. I hope you are having wonderful Saturday. Predrag