From dpelleg+ at cs.cmu.edu Fri Oct 11 09:14:55 2002 From: dpelleg+ at cs.cmu.edu (Dan Pelleg) Date: Fri, 11 Oct 2002 09:14:55 -0400 Subject: [auton-users] an interesting observation about scp Message-ID: <15782.52943.513109.796397@gargle.gargle.HOWL> Here's a way to make scp go faster. Maybe it's well-known, but I just found out about it. It'll significantly help on big transfers (say, 10MB or more). What you do, is simply use a less CPU-intensive encryption method. Obviously this only helps if the CPU is the bottleneck, but that is the case when you're copying on the CMU network and using sub-Ghz processors. A lightweight cipher to use is "blowfish". To use it you just type "scp -c blowfish" wherever you would type "scp". -- Dan Pelleg From dpelleg+ at cs.cmu.edu Fri Oct 11 09:30:49 2002 From: dpelleg+ at cs.cmu.edu (Dan Pelleg) Date: Fri, 11 Oct 2002 09:30:49 -0400 Subject: [auton-users] updated fwguide.txt Message-ID: <15782.53897.353668.251065@gargle.gargle.HOWL> I have updated the guide to using the lab machines through the firewall. It contains dos and donts and some explanations and tips which should help you get more work done. It's called "fwguide.txt" in the smallbutuseful CVS directory. If you don't have CVS access, ask me to mail you a copy. -- Dan Pelleg From dpelleg+ at cs.cmu.edu Fri Oct 11 09:54:11 2002 From: dpelleg+ at cs.cmu.edu (Dan Pelleg) Date: Fri, 11 Oct 2002 09:54:11 -0400 Subject: [auton-users] an interesting observation about scp In-Reply-To: <15782.53920.980628.766744@pad.boyan.com> References: <15782.52943.513109.796397@gargle.gargle.HOWL> <15782.53920.980628.766744@pad.boyan.com> Message-ID: <15782.55299.154833.513671@gargle.gargle.HOWL> Yeah, that's why I said files larger than 10MB. A careful modeling of Auton user's typing and thinking speeds shows this is the exact crossover point ;) But in any case, just drop this in the dotfile and forever forget about it: alias scp="scp -c blowfish" (in ~/.bashrc, for bash users) alias scp scp -c blowfish (in ~/.tcshrc or ~/.cshrc, for tcsh users) If you want to activate it just for specific hosts but not for others, you can drop a line to do just that in ~/.ssh/config. man ssh_config for details. J Boyan writes: > > Of course, when doing timing comparisons between the two methods, > don't forget to factor in the extra time spent typing '-c blowfish'. > And the time spent pondering whether to type it or not... > From nichol at cmu.edu Fri Oct 11 13:00:56 2002 From: nichol at cmu.edu (nichol) Date: Fri, 11 Oct 2002 13:00:56 -0400 (EDT) Subject: [auton-users] loki useage for the next week Message-ID: Dear AUTON users. Weng-keen will be done using with loki.auton.cs.cmu.edu tonight. After that I would like to run a bunch of npt timing tests (2 and 3pt) and would thus like exclusive access to the machine during this peroid. Could you please refrain from using this machine until next Friday? Cheers Bob From justin at boyan.com Fri Oct 11 09:31:12 2002 From: justin at boyan.com (J Boyan) Date: Fri, 11 Oct 2002 09:31:12 -0400 Subject: [auton-users] an interesting observation about scp In-Reply-To: <15782.52943.513109.796397@gargle.gargle.HOWL> References: <15782.52943.513109.796397@gargle.gargle.HOWL> Message-ID: <15782.53920.980628.766744@pad.boyan.com> Of course, when doing timing comparisons between the two methods, don't forget to factor in the extra time spent typing '-c blowfish'. And the time spent pondering whether to type it or not... - A lurker in Cambridge, MA Dan Pelleg writes: > > Here's a way to make scp go faster. Maybe it's well-known, but I just found > out about it. It'll significantly help on big transfers (say, 10MB or more). > > What you do, is simply use a less CPU-intensive encryption > method. Obviously this only helps if the CPU is the bottleneck, but that is > the case when you're copying on the CMU network and using sub-Ghz > processors. > > A lightweight cipher to use is "blowfish". To use it you just type > "scp -c blowfish" wherever you would type "scp". > > -- > > Dan Pelleg From komarek at andrew.cmu.edu Fri Oct 11 21:24:51 2002 From: komarek at andrew.cmu.edu (Paul Komarek) Date: Fri, 11 Oct 2002 21:24:51 -0400 (EDT) Subject: [auton-users] new functions Message-ID: ambs.[ch]: bool am_isinf( double val); TRUE iff val is inf. Written portably. bool am_isnan( double val); TRUE iff val is nan. Written portably. amiv.[ch]: string_array *mk_string_array_x( int size, ...) Makes string array of arbitrary size. char **mk_array_from_string_array( string_array *sa) Converts string array into an array of char pointers (like argv). dsut.[ch]: void copy_dyv_to_real_column( datset *ds, dyv *dv, int att); Copies dyv into a real-valued column of a datset. amdmex.[ch] void fprintf_subdym( FILE *f, char *pre, dyv *dv, dym *d, char *post, const ivec *goodrows, const ivec *goodcols) Prints part of a dym. -Paul Komarek From nichol at cmu.edu Thu Oct 17 21:31:55 2002 From: nichol at cmu.edu (nichol) Date: Thu, 17 Oct 2002 21:31:55 -0400 (EDT) Subject: [auton-users] npt & loki Message-ID: Dear All, loki is still crunching on the timing tests for our npt code. It is basically done everything except compute the number of triangles in our 10^7 dataset on the largest scales. So far, it has had 8000m of time! It is hard to see how much longer it needs, but for completeness it would be nice to have this data. ANdrew or Jeff, is there a way of getting some indication of how far along it is from the logfiles? Therefore, can I at least let it run on loki over this weekend (assuming no-one urgently needs loki)? Furthermore, would it be feasible to let it run until someone explicitly requests/needs loki, i.e. emails me saying "I need loki please stop your job"? Again, I expect this to be a one-off (well until we have 10^7 galaxies in the SDSS dataset). Thanks for your understanding and scream if anyone needs loki. In the meantime, please avoid using it so my timings are meaningful. Cheers Bob -- Bob Nichol 5000 Forbes Ave. Carnegie Mellon 412-268-8068 nichol at cmu.edu From dpelleg+ at cs.cmu.edu Fri Oct 18 07:40:03 2002 From: dpelleg+ at cs.cmu.edu (Dan Pelleg) Date: Fri, 18 Oct 2002 07:40:03 -0400 Subject: [auton-users] using SSH keys to avoid typing passwords Message-ID: <15791.62227.936120.725013@gs166.sp.cs.cmu.edu> This is a short guide to using SSH key pairs. If you do it right, it will let you log into many machines without typing (almost) any passwords. Unlike the way good ol' rsh used to do it, this is done securely with public-key cryptography and is similarly very convenient once you set it up. I'm posting this so we have it in the mailing-list archive and hopefully one day it'll find its way to the web docs. To start off, you need to generate a key, which is divided into two parts: private and public. You need to guard the private one very carefully - giving it to someone is like giving them your UNIX password. The public half, on the other hand, isn't as critical. What we'll do is use our primary work machine (say, your office desktop or your home machine) to generate a key pair. Then we put copies of the PUBLIC key on any machine we want to LOG IN TO. So it'll be, say, all of the lab's servers. After this one-time setup, every time you log in to your primary work machine, you can issue a command that says "I want to be able to use this private key here so I'll be granted access to any machine that has its public counterpart". You do this once (meaning once a day, if you log in every day). After doing that you'll be able to ssh or scp to any accepting machine and it'll work without asking for your password. Details: - To generate the key-pair (you only need to do this once, on your primary work machine): Type: ssh-keygen -t rsa Just hit "enter" when it asks which file to store it in. It will also ask for a password. I recommend choosing a good password and typing it in. However, it will not insist on it being non-empty. Empty passwords are a little less secure, and their corresponding keys can be stolen if the machine is broken into. So my advice is to use a meaningful password unless you are the only one with root/administrator access to this machine AND you make a continuous effort to keep it secure. Personally, I don't use empty passwords for remote shell logins. Also see a note below regarding AFS. When ssh-keygen is done it will save the private key in ~/.ssh/id_rsa and the public key in ~/.ssh/id_rsa.pub As we said, we need to put the PUBLIC key on the remote machine. Let's pretend it's lazy.auton.cs.cmu.edu: -type: scp .ssh/id_rsa.pub lazy.auton.cs.cmu.edu: (you will have to type your NIS password here like you always did - the thing is not working yet!) Now, to set things up on lazy: log into lazy (again, using your NIS password) and create the needed files: mkdir .ssh chmod 700 .ssh mv id_rsa.pub .ssh/authorized_keys2 chmod 600 .ssh/authorized_keys2 There, almost done. From this point on these are the steps you need to perform EVERY DAY: - you log into your primary work machine - type: eval `ssh-agent` note that these are *backticks* - type: ssh-add .ssh/id_rsa - type the password you gave to the key generator above (if it was empty, you won't need to) From nichol at cmu.edu Mon Oct 28 17:34:44 2002 From: nichol at cmu.edu (nichol) Date: Mon, 28 Oct 2002 17:34:44 -0500 (EST) Subject: [auton-users] loki and liver Message-ID: Dear All, Thanks for your patience over the timing experiments I have been running on loki. I have now finished these and return loki to you. Thanks again. I have a long npt code running on liver, but Im interested in the results more than the timings. So I have nice'ed the job so interactive jobs will run at higher priority. I expect this job to run for the remainder of this week. Cheers Bob -- Bob Nichol 5000 Forbes Ave. Carnegie Mellon 412-268-8068 nichol at cmu.edu From dpelleg+ at cs.cmu.edu Tue Oct 29 09:26:00 2002 From: dpelleg+ at cs.cmu.edu (Dan Pelleg) Date: Tue, 29 Oct 2002 09:26:00 -0500 Subject: [auton-users] handy shell aliases Message-ID: <15806.39544.450677.278473@gs166.sp.cs.cmu.edu> Here's a short guide to set up convenient shell aliases to reduce typing. I believe I've posted most of this before, but I'm doing it again because there are new people in the lab, and to make sure it's archived (btw, the list's archives are now linked from the lab's web page). Before we begin: if you don't know which shell you're using, then it's probably tcsh if you're on a SCS-facilitized machine, bash if you're on other GNU/Linux machines. Type "echo $SHELL" and look at the last component to find out. I'll provide instructions for both. *** First trick: tab-completion for host names. You probably know you can hit "tab" for the shell to auto-complete a file name you've started typing. We'll do the same for hostname so you can type something like: ssh lok and have it auto-complete to ssh loki.auton.cs.cmu.edu ==============================tcsh: add this to ~/.login: set hostnames = (gs166.sp.cs.cmu.edu loki.auton.cs.cmu.edu liver.auton.cs.cmu.edu) complete ssh 'p/*/$hostnames/' complete ping 'p/*/$hostnames/' complete xss 'p/*/$hostnames/' complete ftp 'p/1/$hostnames/' complete scp 'p/*/$hostnames/' complete traceroute 'p/*/$hostnames/' ======================================== ------------------------------bash: add this to ~/.bash_login: hostnames="{gs166,lamp,loop,limp,leem,ux7}.sp.cs.cmu.edu \ {limey,lazy,liver,loki,loon,lank,like}.auton.cs.cmu.edu" complete -W "$hostnames" ssh ping host xss ftp traceroute ---------------------------------------- *** Second trick: Add an alias "xss" that, once called with a host name: - starts up a new xterm window - sets the window's title to the name of the host - in that window, ssh-es to that host ==============================tcsh: add to ~/.login alias xss "xterm -T \!^ -e ssh -X \!^ &" ============================== ------------------------------bash: add to ~/.bash_login: function xss() { xterm -T $1 -e ssh -X $1 & } ---------------------------------------- *** saw-woman-in-half trick: Add tab-completion to CVS commands. So you can type "cvs co" and have it auto-complete to "cvs commit", saving you a total of THREE KEYSTROKES. ==============================tcsh complete cvs \ 'p/1/(add annotate checkout commit diff edit editors export\ history import init log login logout rdiff release remove rtag\ status tag unedit update watch watchers)/'\ 'n/{add,commit,log,diff,status}/f/'\ 'n/checkout/d/' ======================================== ------------------------------bash complete -F complete_cvs cvs # completion for "cvs" function complete_cvs() { # when we're called, we have COMP_WORDS the string array with # the current line in it, and COMP_CWORD pointing to the currently # edited (completing) word in it # we return the list of possible completions in the # array COMPREPLY # # the first word should be "cvs" # are we called to complete the second word? - suggest # the CVS commands if [[ ${COMP_CWORD} -eq 1 ]]; then COMPREPLY=( `compgen -W "add annotate checkout commit diff edit editors export\ history import init log login logout rdiff release remove rtag\ status tag unedit update watch watchers" ${COMP_WORDS[$COMP_CWORD]}` ) # are we called to complete the third word? - suggest # some files elif [[ ${COMP_CWORD} -eq 2 ]]; then # decide whether to suggest files or directories case ${COMP_WORDS[$COMP_CWORD]} in add | commit | log | diff | status ) list=f # files ;; checkout) list=d # directories ;; *) list=f # files ;; esac COMPREPLY=( `compgen -$list ${COMP_WORDS[$COMP_CWORD]}` ) fi } ---------------------------------------- Note I only said "saw in half". You didn't want your wife back in one piece, now did you? -- Dan Pelleg