From Connectionists-Request at CS.CMU.EDU Sun Nov 1 00:05:13 1992 From: Connectionists-Request at CS.CMU.EDU (Connectionists-Request@CS.CMU.EDU) Date: Sun, 01 Nov 92 00:05:13 EST Subject: Bi-monthly Reminder Message-ID: <11190.720594313@B.GP.CS.CMU.EDU> *** DO NOT FORWARD TO ANY OTHER LISTS *** This is an automatically posted bi-monthly reminder about how the CONNECTIONISTS list works and how to access various online resources. CONNECTIONISTS is not an edited forum like the Neuron Digest, or a free-for-all newsgroup like comp.ai.neural-nets. It's somewhere in between, relying on the self-restraint of its subscribers. Membership in CONNECTIONISTS is restricted to persons actively involved in neural net research. The following posting guidelines are designed to reduce the amount of irrelevant messages sent to the list. Before you post, please remember that this list is distributed to over a thousand busy people who don't want their time wasted on trivia. Also, many subscribers pay cash for each kbyte; they shouldn't be forced to pay for junk mail. Happy hacking. -- Dave Touretzky & David Redish --------------------------------------------------------------------- What to post to CONNECTIONISTS ------------------------------ - The list is primarily intended to support the discussion of technical issues relating to neural computation. - We encourage people to post the abstracts of their latest papers and tech reports. - Conferences and workshops may be announced on this list AT MOST twice: once to send out a call for papers, and once to remind non-authors about the registration deadline. A flood of repetitive announcements about the same conference is not welcome here. - Requests for ADDITIONAL references. This has been a particularly sensitive subject lately. Please try to (a) demonstrate that you have already pursued the quick, obvious routes to finding the information you desire, and (b) give people something back in return for bothering them. The easiest way to do both these things is to FIRST do the library work to find the basic references, then POST these as part of your query. Here's an example: WRONG WAY: "Can someone please mail me all references to cascade correlation?" RIGHT WAY: "I'm looking for references to work on cascade correlation. I've already read Fahlman's paper in NIPS 2, his NIPS 3 abstract, and found the code in the nn-bench archive. Is anyone aware of additional work with this algorithm? I'll summarize and post results to the list." - Announcements of job openings related to neural computation. - Short reviews of new text books related to neural computation. To send mail to everyone on the list, address it to Connectionists at CS.CMU.EDU ------------------------------------------------------------------- What NOT to post to CONNECTIONISTS: ----------------------------------- - Requests for addition to the list, change of address and other administrative matters should be sent to: "Connectionists-Request at cs.cmu.edu" (note the exact spelling: many "connectionists", one "request"). If you mention our mailing list to someone who may apply to be added to it, please make sure they use the above and NOT "Connectionists at cs.cmu.edu". - Requests for e-mail addresses of people who are believed to subscribe to CONNECTIONISTS should be sent to postmaster at appropriate-site. If the site address is unknown, send your request to Connectionists-Request at cs.cmu.edu and we'll do our best to help. A phone call to the appropriate institution may sometimes be simpler and faster. - Note that in many mail programs a reply to a message is automatically "CC"-ed to all the addresses on the "To" and "CC" lines of the original message. If the mailer you use has this property, please make sure your personal response (request for a Tech Report etc.) is NOT broadcast over the net. - Do NOT tell a friend about Connectionists at cs.cmu.edu. Tell him or her only about Connectionists-Request at cs.cmu.edu. This will save your friend from public embarrassment if she/he tries to subscribe. ------------------------------------------------------------------------------- The CONNECTIONISTS Archive: --------------------------- All e-mail messages sent to "Connectionists at cs.cmu.edu" starting 27-Feb-88 are now available for public perusal. A separate file exists for each month. The files' names are: arch.yymm where yymm stand for the obvious thing. Thus the earliest available data are in the file: arch.8802 Files ending with .Z are compressed using the standard unix compress program. To browse through these files (as well as through other files, see below) you must FTP them to your local machine. ------------------------------------------------------------------------------- How to FTP Files from the CONNECTIONISTS Archive ------------------------------------------------ 1. Open an FTP connection to host B.GP.CS.CMU.EDU (Internet address 128.2.242.8). 2. Login as user anonymous with password your username. 3. 'cd' directly to one of the following directories: /usr/connect/connectionists/archives /usr/connect/connectionists/bibliographies 4. The archives and bibliographies directories are the ONLY ones you can access. You can't even find out whether any other directories exist. If you are using the 'cd' command you must cd DIRECTLY into one of these two directories. Access will be denied to any others, including their parent directory. 5. The archives subdirectory contains back issues of the mailing list. Some bibliographies are in the bibliographies subdirectory. Problems? - contact us at "Connectionists-Request at cs.cmu.edu". ------------------------------------------------------------------------------- How to FTP Files from the Neuroprose Archive -------------------------------------------- Anonymous FTP on archive.cis.ohio-state.edu (128.146.8.52) pub/neuroprose directory This directory contains technical reports as a public service to the connectionist and neural network scientific community which has an organized mailing list (for info: connectionists-request at cs.cmu.edu) Researchers may place electronic versions of their preprints or articles in this directory, announce availability, and other interested researchers can rapidly retrieve and print the postscripts. This saves copying, postage and handling, by having the interested reader supply the paper. (Along this line, single spaced versions, if possible, will help!) To place a file, put it in the Inbox subdirectory, and send mail to pollack at cis.ohio-state.edu. Within a couple of days, I will move and protect it, and suggest a different name if necessary. When you announce a paper, you should consider whether (A) you want it automatically forwarded to other groups, like NEURON-DIGEST, (which gets posted to comp.ai.neural-networks) and if you want to provide (B) free or (C) prepaid hard copies for those unable to use FTP. If you do offer hard copies, be prepared for an onslaught. One author reported that when they allowed combination AB, the rattling around of their "free paper offer" on the worldwide data net generated over 2000 hardcopy requests! Experience dictates the preferred paradigm is to announce an FTP only version with a prominent "**DO NOT FORWARD TO OTHER GROUPS**" at the top of your announcement to the connectionist mailing list. Current naming convention is author.title.filetype[.Z] where title is enough to discriminate among the files of the same author. The filetype is usually "ps" for postscript, our desired universal printing format, but may be tex, which requires more local software than a spooler. Very large files (e.g. over 200k) must be squashed (with either a sigmoid function :) or the standard unix "compress" utility, which results in the .Z affix. To place or retrieve .Z files, make sure to issue the FTP command "BINARY" before transfering files. After retrieval, call the standard unix "uncompress" utility, which removes the .Z affix. An example of placing a file is attached as an appendix, and a shell script called Getps in the directory can perform the necessary retrival operations. For further questions contact: Jordan Pollack Assistant Professor CIS Dept/OSU Laboratory for AI Research 2036 Neil Ave Email: pollack at cis.ohio-state.edu Columbus, OH 43210 Phone: (614) 292-4890 Here is an example of naming and placing a file: gvax> cp i-was-right.txt.ps rosenblatt.reborn.ps gvax> compress rosenblatt.reborn.ps gvax> ftp archive.cis.ohio-state.edu Connected to archive.cis.ohio-state.edu. 220 archive.cis.ohio-state.edu FTP server ready. Name: anonymous 331 Guest login ok, send ident as password. Password:neuron 230 Guest login ok, access restrictions apply. ftp> binary 200 Type set to I. ftp> cd pub/neuroprose/Inbox 250 CWD command successful. ftp> put rosenblatt.reborn.ps.Z 200 PORT command successful. 150 Opening BINARY mode data connection for rosenblatt.reborn.ps.Z 226 Transfer complete. 100000 bytes sent in 3.14159 seconds ftp> quit 221 Goodbye. gvax> mail pollack at cis.ohio-state.edu Subject: file in Inbox. Jordan, I just placed the file rosenblatt.reborn.ps.Z in the Inbox. The INDEX sentence is "Boastful statements by the deceased leader of the neurocomputing field." Please let me know when it is ready to announce to Connectionists at cmu. BTW, I enjoyed reading your review of the new edition of Perceptrons! Frank ------------------------------------------------------------------------ How to FTP Files from the NN-Bench Collection --------------------------------------------- 1. Create an FTP connection from wherever you are to machine "pt.cs.cmu.edu" (128.2.254.155). 2. Log in as user "anonymous" with password your username. 3. Change remote directory to "/afs/cs/project/connect/bench". Any subdirectories of this one should also be accessible. Parent directories should not be. 4. At this point FTP should be able to get a listing of files in this directory and fetch the ones you want. Problems? - contact us at "nn-bench-request at cs.cmu.edu". From jose at tractatus.siemens.com Mon Nov 2 08:19:18 1992 From: jose at tractatus.siemens.com (Steve Hanson) Date: Mon, 2 Nov 1992 08:19:18 -0500 (EST) Subject: Registration Message-ID: <0exGfKC1GEMn8h8Uo_@tractatus> "Register early and often..." As an election day bonus for voting.. we have extended the PREREGISTRATION DEADLINE for NIPS*92 to the End of the WEEK (Nov. 6, 1992) Please send your completed registration form to NIPS*92 Registration Siemens Research Center 755 College Rd. East Princeton, NJ 08540 Stephen J. Hanson Learning Systems Department SIEMENS Research 755 College Rd. East Princeton, NJ 08540 From jose at tractatus.siemens.com Mon Nov 2 08:39:34 1992 From: jose at tractatus.siemens.com (Steve Hanson) Date: Mon, 2 Nov 1992 08:39:34 -0500 (EST) Subject: NIPS*92 Message-ID: NOTE that NIPS*92 is being held at new hotel for the first time this year. It will be Downtown CITY-CENTER MARIOTT in DENVER. This is a (as the name suggests) a centrally located hotel which we have a 72$ rate. You must Register this week in order to ensure this speical discount Rate. Please do so ASAP. To Make reservations call 303-297-1300 and be sure to mention you are with the NIPS*92 Group. Steve Hanson NIPS*92 General Chair Stephen J. Hanson Learning Systems Department SIEMENS Research 755 College Rd. East Princeton, NJ 08540 From sontag at control.rutgers.edu Mon Nov 2 05:49:15 1992 From: sontag at control.rutgers.edu (Eduardo Sontag) Date: Mon, 2 Nov 92 10:49:15 GMT Subject: "Neural networks with real weights: analog computational complexity" Message-ID: <9211021549.AA09420@control.rutgers.edu> Title: "Neural networks with real weights: analog computational complexity" Authors: Hava T. Siegelmann and Eduardo D. Sontag (Report SYCON 92-05, September 1992. 24 + i pp.) (Placed in neuroprose archive; filename: siegelmann.analog.ps.Z) Abstract: We pursue a particular approach to analog computation, based on dynamical systems of the type used in neural networks research. Our systems have a fixed structure, invariant in time, corresponding to an unchanging number of ``neurons''. If allowed exponential time for computation, they turn out to have unbounded power. However, under polynomial-time constraints there are limits on their capabilities, though being more powerful than Turing Machines. (A similar but more restricted model was shown to be polynomial-time equivalent to classical digital computation in previous work.) Moreover, there is a precise correspondence between nets and standard non-uniform circuits with equivalent resources, and as a consequence one has lower bound constraints on what they can compute. This relationship is perhaps surprising since our analog devices do not change in any manner with input size. We note that these networks are not likely to solve polynomially NP-hard problems, as the equality ``P = NP'' in our model implies the almost complete collapse of the standard polynomial hierarchy. In contrast to classical computational models, the models studied here exhibit at least some robustness with respect to noise and implementation errors. To obtain copies of this article: unix> ftp archive.cis.ohio-state.edu (or 128.146.8.52) Name : anonymous Password: ftp> cd pub/neuroprose ftp> binary ftp> get siegelmann.analog.ps.Z ftp> quit unix> uncompress siegelmann.analog.ps.Z unix> lpr -Pps siegelmann.analog.ps.Z (or however you print PostScript) (With many thanks to Jordan Pollack for providing this valuable service!) Please note: the file requires a fair amount of memory to print. If you have problems with FTP, I can e-mail you the postscript file; I cannot provide hardcopy, however. From nelsonde%avlab.dnet at aaunix.aa.wpafb.af.mil Mon Nov 2 15:06:16 1992 From: nelsonde%avlab.dnet at aaunix.aa.wpafb.af.mil (nelsonde%avlab.dnet@aaunix.aa.wpafb.af.mil) Date: Mon, 2 Nov 92 15:06:16 -0500 Subject: Four Papers Available Message-ID: <9211022006.AA06603@aaunix.aa.wpafb.af.mil> I N T E R O F F I C E M E M O R A N D U M Date: 02-Nov-1992 02:56pm EST From: DALE E. NELSON NELSONDE Dept: AAAT-1 Tel No: 57646 TO: Remote Addressee ( _AAUNIX::"CONNECTIONISTS at CS.CMU.EDU" ) Subject: Four Papers Available ********* DO NOT POST TO OTHER NETS ************* ********* DO NOT POST TO OTHER NETS ************* Prediction of Chaotic Time Series Using Cascade Correlation: Effects of Number of Inputs and Training Set Size Dale E. Nelson D. David Ensley Maj Steven K. Rogers, PhD ABSTRACT Most neural networks have been used for problems of classification. We have undertaken a study using neural networks to predict continuous valued functions which are aperiodic or chaotic. In addition, we are considering a relatively new class of neural networks, ontogenic neural networks. Ontogenic neural networks are networks which generate their own topology during training. Cascade Correlation2 is one such network. In this study we used the Cascade Correlation neural network to answer two questions regarding prediction. First, how do the number of inputs affect prediction accuracy. Second, how do the number of training exemplars affect prediction accuracy. For these experiments, the Mackey-Glass equation was used with a Tau value of 17 which yields a correlation dimension of 2.1. Takens' theorem7 for this data set states that the number of inputs to obtain a smooth mapping should be 3 to 5. We were experimentally able to verify this. Experiments were run varying the number of training exemplars from 50 to 450. The results showed that there is an overall trend towards lower predictive RMS error with a greater number of exemplars. However, there are good results obtained with only 50 exemplars which we are unable to explain at this time. In addition to these results, we discovered that the way in which predictive accuracy is generally represented, a graph of Mackey-Glass with the network output superimposed, can lead to erroneous conclusions! This paper is NOT available from Neuroprose. For paper copies send E-Mail with your mailing address to : nelsonde%avlab.dnet%aa.wpafb.af.mil DO NOT REPLY TO ENTIRE NETWORK...DO NOT USE REPLY MODE! ********* DO NOT POST TO OTHER NETS ************* ********* DO NOT POST TO OTHER NETS ************* ********* DO NOT POST TO OTHER NETS ************* ********* DO NOT POST TO OTHER NETS ************* A Taxonomy of Neural Network Optimality Dale E. Nelson Maj Steven K. Rogers, PhD ABSTRACT One of the long-standing problems with neural networks is how to decide on the correct topology for a given application. For many years the accepted approach was to use heuristics to "get close", then experiment to find the best topology. In recent years methodologies like the Abductory Inference Mechanism (AIM) from AbTech Corporation and Cascade Correlation from Carnegie Mellon University have emerged. These ontogenic (topology synthesizing) neural networks develop their topology by deciding when and what kind of nodes to add to the network during the training phase. Other methodologies examine the weights and try to "improve" by pruning some of the weights. This paper discusses the criteria which can be used to decide when one network topology is better than another. The taxonomy presented in this paper can be used to decide on methods for comparison of different neural network paradigms. Since the criteria for determining what is an optimum network is highly application specific, no attempt is made to propose the one right criteria. This taxonomy is a necessary step toward achieving robust ontogenic neural networks. This paper is NOT available from Neuroprose. For paper copies send E-Mail with your mailing address to : nelsonde%avlab.dnet%aa.wpafb.af.mil DO NOT REPLY TO ENTIRE NETWORK...DO NOT USE REPLY MODE! ********* DO NOT POST TO OTHER NETS ************* ********* DO NOT POST TO OTHER NETS ************* ********* DO NOT POST TO OTHER NETS ************* ********* DO NOT POST TO OTHER NETS ************* APPLYING CASCADE CORRELATION TO THE EXTRAPOLATION OF CHAOTIC TIME SERIES David Ensley Dale E. Nelson ABSTRACT Attempting to find near-optimal architectures, ontogenic neural networks develop their own architectures as they train. As part of a project entitled "Ontogenic Neural Networks for the Prediction of Chaotic Time Series," this paper presents findings of a ten-week research period on using the Cascade Correlation ontogenic neural network to extrapolate (predict) a chaotic time series generated from the Mackey-Glass equation. Truer, more informative measures of extrapolation accuracy than currently popular measures are presented. The effects of some network parameters on extrapolation accuracy were investigated. Sinusoidal activation functions turned out to be best for our data set. The best range for sigmoidal activation functions was [-1, +1]. One experiment demonstrates that extrapolation accuracy can be maximized by selecting the proper number of training exemplars. Though surprisingly good extrapolations have been obtained, there remain pitfalls. These pitfalls are discussed along with possible methods for avoiding them. This paper is NOT available from Neuroprose. For paper copies send E-Mail with your mailing address to : nelsonde%avlab.dnet%aa.wpafb.af.mil DO NOT REPLY TO ENTIRE NETWORK...DO NOT USE REPLY MODE! ********* DO NOT POST TO OTHER NETS ************* ********* DO NOT POST TO OTHER NETS ************* ********* DO NOT POST TO OTHER NETS ************* ********* DO NOT POST TO OTHER NETS ************* APPLYING THE ABDUCTORY INDUCTION MECHANISM (AIM) TO THE EXTRAPOLATION OF CHAOTIC TIME SERIES Dennis S. Buck Dale E. Nelson ABSTRACT This paper presents research done as part of a large effort to develop ontogenic (topology synthesizing) neural networks. One commerically available product, considered an ontogenic neural network, is the Abductory Induction Mechanism (AIM) program from AbTech Corporation of Charlottesville, Virginia. AIM creates a polynomial neural network of the third order during training. The methodology will discard any inputs it finds having a low relevance to predicting the training output. The depth and complexity of the network is controlled by a user-set Complexity Penalty Multiplier (CPM). This paper presents results of using AIM to predict the output of the Mackey-Glass equation. Comparisons are made based on the RMS error for an iterated prediction of 100 time steps beyond the training set. The data set was developed using a Tau value of 17 which yields a correlation dimension (an approximation of the fractal dimension) of 2.1. We explored the effect of different CPM values and found that a CPM value of 4.8 gives the best predictive results with the least computational complexity. We also conducted experiments using 2 to 10 inputs and 1 to 3 outputs. We found that AIM chose to use only 2 or 3 inputs, due to its ability to eliminate unnecessary inputs. This leads to the conclusion that Takens' theorem cannot be experimentally verified by this methodology! Our experiments showed that using 2 or 3 outputs, thus forcing the network to learn the first and second derivative of the equation, produced the best predictive results. We also discovered that the final network produced a predictive RMS error lower than the Cascade Correlation method with far less computational time. This paper is NOT available from Neuroprose. For paper copies send E-Mail with your mailing address to : nelsonde%avlab.dnet%aa.wpafb.af.mil DO NOT REPLY TO ENTIRE NETWORK...DO NOT USE REPLY MODE! ********* DO NOT POST TO OTHER NETS ************* ********* DO NOT POST TO OTHER NETS ************* From mozer at dendrite.cs.colorado.edu Mon Nov 2 16:35:14 1992 From: mozer at dendrite.cs.colorado.edu (Michael C. Mozer) Date: Mon, 2 Nov 1992 14:35:14 -0700 Subject: Connectionist Models Summer School 1993 Message-ID: <199211022135.AA27213@neuron.cs.colorado.edu> CALL FOR APPLICATIONS CONNECTIONIST MODELS SUMMER SCHOOL University of Colorado Boulder, Colorado June 21 - July 3, 1993 The University of Colorado will host the 1993 Connectionist Models Summer School from June 21 to July 3, 1993. The purpose of the summer school is to provide training to promising young researchers in connectionism (neural networks) by leaders of the field and to foster interdisciplinary collaboration. This will be the fourth such program in a series that was held at Carnegie-Mellon in 1986 and 1988 and at UC San Diego in 1990. Previous summer schools have been extremely successful and we look forward to the 1993 session with anticipation of another exciting event. The summer school will offer courses in many areas of connectionist modeling, with emphasis on artificial intelligence, cognitive science, cognitive neuroscience, theoretical foundations, and computational methods. Visiting faculty (see list of invited faculty below) will present daily lectures and tutorials, coordinate informal workshops, and lead small discussion groups. The summer school schedule is designed to allow for significant interaction among students and faculty. As in previous years, a proceedings of the summer school will be published. Applications will be considered only from graduate students currently enrolled in Ph.D. programs. About 50 students will be accepted. Admission is on a competitive basis. Tuition will be covered for all students, and we expect to have scholarships available to subsidize housing and meal costs, which will run approximately $300. Applications should include the following materials: * a one-page statement of purpose, explaining major areas of interest and prior background in connectionist modeling and neural networks; * a vita, including academic history, publications (if any), and a list of relevant courses taken with instructors' names and grades received; * two letters of recommendation from individuals familiar with the applicants' work; and * if room and board support is requested, a statement from the applicant describing potential sources of financial support available (department, advisor, etc.) and the estimated extent of need. We hope to have sufficient scholarship funds available to provide room and board to all accepted students regardless of financial need. Applications should be sent to: Connectionist Models Summer School c/o Institute of Cognitive Science Campus Box 344 University of Colorado Boulder, CO 80309 All application materials must be received by March 1, 1993. Decisions about acceptance and scholarship awards will be announced April 15. If you have additional questions, please write to the address above or send e-mail to "cmss at cs.colorado.edu". Organizing Committee Jeff Elman (UC San Diego) Mike Mozer (University of Colorado) Paul Smolensky (University of Colorado) Dave Touretzky (Carnegie-Mellon) Andreas Weigend (Xerox PARC and University of Colorado) Additional faculty will include: Andy Barto (University of Massachusetts, Amherst) Gail Carpenter (Boston University) Jack Cowan (University of Chicago) David Haussler (UC Santa Cruz) Geoff Hinton (University of Toronto) Mike Jordan (MIT) John Kruschke (Indiana University) Jay McClelland (Carnegie-Mellon) Steve Nowlan (Salk Institute) Dave Plaut (Carnegie-Mellon) Jordan Pollack (Ohio State) Dave Rumelhart (Stanford) Terry Sejnowski (UC San Diego and Salk Institute) From maresch at ox.ac.uk Mon Nov 2 05:25:28 1992 From: maresch at ox.ac.uk (Denis Mareschal) Date: Mon, 2 Nov 92 10:25:28 GMT Subject: No subject Message-ID: <27777.9211021025@black.ox.ac.uk> Hi, I'm interested in applications of Neural-networks to visual-tracking. In particular, the ability to predict or anticipate futur positions based on information about the current history of the trajectory as well as THE DEVELOPMENT of this ability. I've already found works by Pearlmutter (1989) and Dobnikar, Likar and Podbregar (1989) which deal with the explicit tracking of an object in 2-D space. However, most of the stuff I've turned up seems to be geared more towards modeling explicit physiological systems (E.g.: Krauzlis & Lisberger, 1989; Deno, Keller & Crandall, 1989). Does anyone know of further works that aren't necessarily related to physiological systems? Any help would be greatly appreciated and of course a list of responses will be compiled and posted if sufficient requests are made. Thanks a lot Cheers, Denis Mareschal Department of Psychology Oxford maresch at ox.ac.uk Deno, D. C., Keller, E. L., Crandall, W.F. (1989). Dynamical neural network organization of the visual pursuit system. IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, 36, pp. 85-92. Dobnikar, A., Likar, A., & Podbregar,D. (1989). Optimal visual tracking with artificial neural network. FIRST INTERNATIONAL CONFERENCE ON ARTIFICIAL NEURAL NETWORKS (Conf. publ. 313), London, IEE Krauzlis, R. J. & Lisberber, S.G. (1989). A control systems model of visual pursuit eye movements with realistic emergent properties. NEURAL COMPUTATION, 1, pp. 116-122. Pearlmutter, B. A. (1989). Learning state space trajectories in recurrent neural networks. In: INTERNATIONAL JOINT CONFERENCE ON NEURAL NETWORKS (Washington 1989), vol II, pp. 365-372. NY: IEEE. From jose at tractatus.siemens.com Mon Nov 2 09:22:14 1992 From: jose at tractatus.siemens.com (Steve Hanson) Date: Mon, 2 Nov 1992 09:22:14 -0500 (EST) Subject: Hotel reservation deadline for NIPS workshops In-Reply-To: <9210312353.AA05934@siemens.siemens.com> References: <9210312353.AA05934@siemens.siemens.com> Message-ID: Note that in the last 6 months that Mariott MARK Resort has been purchased by the Radisson and is now Called Radisson Vail Resort (same place, same facilties). If you have gotten reservations at the Mariott MARK Resort during this time under the NIPS*92 group, they will be honored by the Radisson. Everyone else who has yet to reserve a room at Vail, should call the Radisson as Gerry suggests ASAP. Steve NIPS*92 General Chair Stephen J. Hanson Learning Systems Department SIEMENS Research 755 College Rd. East Princeton, NJ 08540 From meyer at biologie.ens.fr Tue Nov 3 09:49:55 1992 From: meyer at biologie.ens.fr (Jean-Arcady MEYER) Date: Tue, 3 Nov 92 15:49:55 +0100 Subject: Adaptive Behavior - Table of Contents Message-ID: <9211031449.AA05812@wotan.ens.fr> The first issue of Adaptive Behavior was released in August 1992. The second is under press. For inquiries or paper submissions, please contact one of the editors: - Editor-in-Chief: Jean-Arcady Meyer, France - meyer at wotan.ens.fr - Associate Editors: Randall Beer, USA - beer at alpha.ces.cwru.edu Lashon Booker, USA - booker at starbase.mitre.org Jean-Louis Deneubourg, Belgium - sgoss at ulb.ac.be Janet Halperin, Canada - janh at zoo.utoronto.ca Pattie Maes, USA - pattie at media-lab.media.mit.edu Herbert Roitblat, USA - roitblat at uhunix.uhcc.hawaii.edu Ronald Williams, USA - rjw at corwin.ccs.northeastern.edu Stewart Wilson, USA - wilson at smith.rowland.com ============================================================================= ADAPTIVE BEHAVIOR 1:1 Table of Contents A Model of Primate Visual-Motor Conditional Learning by Andrew H. Fagg and Michael A. Arbib Postponed Conditioning: Testing a Hypothesis about Synaptic Strengthening by J. R. P. Halperin and D. W. Dunham The Evolution of Strategies for Multi-agent Environments By John J. Grefenstette Evolving Dynamical Neural Networks for Adaptive Behavior By Randall D. Beer and John C. Gallagher =============================================================================== ADAPTIVE BEHAVIOR 1:2 Table of Contents Adapted and Adaptive Properties in Neural Networks Responsible for Visual Pattern Discrimination. By J.-P. Ewert, T.W. Beneke, H. Buxbaum-Conradi, A. Dinges, S. Fingerling, M. Glagow, E. Schurg-Pfeiffer and W.W. Schwippert. Kinematic Model of a Stick Insect as an Example of a 6-legged Walking System. By U. Muller-Wilm, J. Dean, H. Cruse, H.J. Weidemann, J. Eltze and F. Pfeiffer. Evolution of Food Foraging Strategies for the Caribbean Anolis Lizard using Genetic Programming. By J.R. Koza, J.P. Rice and J. Roughgarden Behavior-based Robot Navigation for Extended Domains. By R.C. Arkin =============================================================================== From lvq at cochlea.hut.fi Tue Nov 3 12:29:14 1992 From: lvq at cochlea.hut.fi (LVQ_PAK) Date: Tue, 3 Nov 92 12:29:14 EET Subject: New version of Self-Organizing Maps PD program package Message-ID: <9211031029.AA03154@cochlea.hut.fi.hut.fi> ************************************************************************ * * * SOM_PAK * * * * The * * * * Self-Organizing Map * * * * Program Package * * * * Version 1.2 (November 2, 1992) * * * * Prepared by the * * SOM Programming Team of the * * Helsinki University of Technology * * Laboratory of Computer and Information Science * * Rakentajanaukio 2 C, SF-02150 Espoo * * FINLAND * * * * Copyright (c) 1992 * * * ************************************************************************ Some time ago we released the software package "LVQ_PAK" for the easy application of Learning Vector Quantization algorithms. Corresponding public-domain programs for the Self-Organizing Map (SOM) algorithms are now available via anonymous FTP on the Internet. "What does the Self-Organizing Map mean?", you may ask --- See the following reference, then: Teuvo Kohonen. The self-organizing map. Proceedings of the IEEE, 78(9):1464-1480, 1990. In short, Self-Organizing Map (SOM) defines a 'non-linear projection' of the probability density function of the high-dimensional input data onto the two-dimensional display. SOM places a number of reference vectors into an input data space to approximate to its data set in an ordered fashion. This package contains all the programs necessary for the application of Self-Organizing Map algorithms in an arbitrary complex data visualization task. This code is distributed without charge on an "as is" basis. There is no warranty of any kind by the authors or by Helsinki University of Technology. In the implementation of the SOM programs we have tried to use as simple code as possible. Therefore the programs are supposed to compile in various machines without any specific modifications made on the code. All programs have been written in ANSI C. The programs are available in two archive formats, one for the UNIX-environment, the other for MS-DOS. Both archives contain exactly the same files. These files can be accessed via FTP as follows: 1. Create an FTP connection from wherever you are to machine "cochlea.hut.fi". The internet address of this machine is 130.233.168.48, for those who need it. 2. Log in as user "anonymous" with your own e-mail address as password. 3. Change remote directory to "/pub/som_pak". 4. At this point FTP should be able to get a listing of files in this directory with DIR and fetch the ones you want with GET. (The exact FTP commands you use depend on your local FTP program.) Remember to use the binary transfer mode for compressed files. The som_pak program package includes the following files: - Documentation: README short description of the package and installation instructions som_doc.ps documentation in (c) PostScript format som_doc.ps.Z same as above but compressed som_doc.txt documentation in ASCII format - Source file archives (which contain the documentation, too): som_p1r2.exe Self-extracting MS-DOS archive file som_pak-1.2.tar UNIX tape archive file som_pak-1.2.tar.Z same as above but compressed An example of FTP access is given below unix> ftp cochlea.hut.fi (or 130.233.168.48) Name: anonymous Password: ftp> cd /pub/som_pak ftp> binary ftp> get som_pak-1.2.tar.Z ftp> quit unix> uncompress som_pak-1.2.tar.Z unix> tar xvfo som_pak-1.2.tar See file README for further installation instructions. All comments concerning this package should be addressed to som at cochlea.hut.fi. ************************************************************************ From C.Campbell at bristol.ac.uk Mon Nov 2 10:04:01 1992 From: C.Campbell at bristol.ac.uk (I C G Campbell) Date: 02 Nov 1992 15:04:01 +0000 (GMT) Subject: faculty position Message-ID: <16046.9211021504@irix.bristol.ac.uk> FACULTY POSITION UNIVERSITY OF BRISTOL, UNITED KINGDOM Department of Computer Science Applications are invited for a Lectureship in Computer Science now tenable. FURTHER PARTICULARS The Department is part of the Faculty of Engineering. It has a complement of eighteen full-time UFC-funded staff members, together with a further twelve full-time outside-funded staff, and two Visiting Industrial Professors: Professor J. M. Taylor (Director, Hewlett-Packard Research Laboratories, Bristol) and Professor I. M. Barron. There are three Professors in the Department: Professor M. H. Rogers, who is Head of Department, and Professors J. W. Lloyd and D. H. D. Warren. The Department has substantial research funding from ESPRIT, SERC, industry and government. The Department concentrates its research in three main areas: Logic Programming Parallel Computing Machine Intelligence although a number of other topics are also being pursued. For this appointment, we are looking for a strong candidate in any area of Computer Science, although preference will be given to candidates with research interests in Parallel Computing or Machine Intelligence. We are particularly looking for candidates whose interests will broaden and complement our current work in these areas. Current work in Parallel Computing covers a range of areas, including parallel logic programming systems and languages, memory organisation for multiprocessor architectures, shared data models for transputer-based systems, and parallel applications especially for engineering problems and computer graphics. We are seeking to broaden and strengthen this research. Candidates with a strong background in computer architecture would be particularly welcome. Current work in Machine Intelligence centres mainly on Computer Vision and Speech Processing. One major project in Computer Vision is the development of an autonomous road vehicle, based on real-time image analysis. Other research projects in Computer Vision include vehicle number plate decoding, aircraft engine inspection, and visual flow monitoring. Current work on Speech Processing within the Department concentrates on speech synthesis, but the Faculty supports a Centre for Communications Research within which there is a Speech Research Group incorporating researchers in most aspects of speech technology, including speech recognition, speech coding, speech perception, and the design of speech interfaces. There is an interest in neural network theory and neural computing elsewhere in the Faculty and we would welcome applications from candidates in this area. The Department has a flourishing undergraduate and post-graduate teaching programme and participates in degree programmes in the Engineering, Science and Arts Faculties. These programmes lead to B.Sc. degrees in Computer Science, and Computer Science with Mathematics, a B.Eng. in Computer Systems Engineering, a B.A. in Computer Science and a Modern Language, and M.Sc. degrees in Computer Science, Foundations of Artificial Intelligence, and Information Engineering. The salary will be within the Lecturer Scale and the initial placement will depend on age, qualifications and experience. The closing date for applications is 27th November 1992. Further particulars may be obtained from the Head of the Computer Science Department (tel: 0272-303584; or e-mail: barbara at bristol.uk.ac.compsci). From reggia at cs.UMD.EDU Tue Nov 3 12:36:59 1992 From: reggia at cs.UMD.EDU (James A. Reggia) Date: Tue, 3 Nov 92 12:36:59 -0500 Subject: Fellowship Position: Neural Computation in Neurology Message-ID: <9211031736.AA26595@avion.cs.UMD.EDU> Research Training Fellowship in Neural Modelling Available (MD Degree required) The Clinical Stroke Research Center at the University of Maryland School of Medicine will offer two Junior Javits research fellowships starting July 1, 1993. One of these positions provides research training in the use of neural networks in cerebrovascular disease. A clinical back- ground (MD degree and specialization in neurology) is required. The Fellowship is for two years and is research intensive, but would also usually involve some clinical work in the Stroke Center. There is substantial flexibility in the details of the research training and research work. The first year salary is anticipated to be $33,000 plus fringe benefits. To apply send a letter and curriculum vitae to Dr. Thomas Price Director, Clinical Stroke Research Center University of Maryland Hospital 22 South Greene Street Baltimore, MD 21201 Questions about the research program can be sent to: Jim Reggia reggia at cs.umd.edu From beer at ICSI.Berkeley.EDU Tue Nov 3 13:47:35 1992 From: beer at ICSI.Berkeley.EDU (Joachim Beer) Date: Tue, 03 Nov 92 10:47:35 PST Subject: workshop announcement Message-ID: <9211031847.AA13155@icsib11.ICSI.Berkeley.EDU> *************************************************** * Workshop on Software & Programming Issues for * * Connectionist Supercomputers * *************************************************** April 19-20, 1993 at International Computer Science Institute (ICSI) 1947 Center Street Berkeley, CA 94704 Sponsored by: Adaptive Solutions, Inc. ICSI Siemens AG The goal of this workshop is to bring together connectionist researchers to address software and programming issues in the framework of large scale connectionist systems. Scope and technical theme of the workshop is outlined below. Due to space considerations the workshop will be by invitation only. Interested parties are encouraged to submit a one-page proposal outlinig their work in this area by January 31. Submissions should be send to ICSI at the address above or by e-mail to beer at icsi.berkeley.edu The increased importance of ANNs for elucidating deep conceptual questions in artificial intelligence and their potential for attacking real world problems warrant the design and construction of connectionist supercomputers. Several research labs have undertaken to develop such machines. These machines will allow researchers to investigate and apply ANNs on a scale which up to now was not computationally feasible. As with other parallel hardware, the main problem is adequate software for connectionist supercomputers. Most "solutions" offer isolated instances which deal only with a limited class of particular ANN algorithms rather than providing a comprehensive programming model for this new paradigm. This approach was acceptable for small and structurally simple ANNs. However, to fully utilize the emerging connectionist supercomputers an expressive, clean, and flexible software environment is called for. This is being recognized by the developers of the connectionist supercomputers, and an intergral part of these projects is the development of an appropriate software environment. While each connectionist supercomputer project has unique goals and possibly a focus on particular application areas, it would nevertheless be very fruitful to compare how the fundamental software questions that everybody in this field faces are being approached. The following (incomplete) list outlines some of the issues: * Embedding connectionist systems in traditional software environments, eg. client/server models vs. integrated "seamless" environments. * ANN description languages * Handling of sparse and irregular nets * Facilities for mapping nets onto the underlying architecture * Handling of complete applications including embedded non-connectionist instructions * Should there be a machine independent intermediate language? What would be the disadvantages? * Software issues for dedicated embedded ANNs vs. "general purpose" connectionist supercomputers. * Graphical user interfaces for ANN systems * System support for high I/O rates (while this is a general question in comp. sci. there are nevertheless some unique problems for ANN systems in dealing with large external data sets). From crr at cogsci.psych.utah.edu Wed Nov 4 16:35:40 1992 From: crr at cogsci.psych.utah.edu (crr@cogsci.psych.utah.edu) Date: Wed, 04 Nov 92 14:35:40 -0700 Subject: paper in neuroprose / change of e-mail address announcement Message-ID: <9211042135.AA08788@cogsci.psych.utah.edu> The following paper has been placed in the Neuroprose archives at Ohio State (filename: rosenberg.scintigrams.ps.Z). Ftp instructions follow the abstract. A Neural Network that Learns to Interpret Myocardial Planar Thallium Scintigrams (This paper is to appear in Neural Computation.) Charles Rosenberg, PhD Department of Computer Science Hebrew University Jerusalem, Israel [For current address, see below] Jacob Erel, MD Department of Cardiology Sapir Medical Center - Meir General Hospital Kfar Saba, Israel Henri Atlan, MD, PhD Department of Biophysics and Nuclear Medicine Hadassah Medical Center Jerusalem, Israel ABSTRACT The planar thallium-201 ($^{201}$Tl) myocardial perfusion scintigram is a widely used diagnostic technique for detecting and estimating the risk of coronary artery disease. Interpretation is currently based on visual scoring of myocardial defects combined with image quantitation and is known to have a significant subjective component. Neural networks learned to interpret thallium scintigrams as determined by both individual and multiple (consensus) expert ratings. Four different types of networks were explored: single-layer, two-layer back-propagation (BP), BP with weight smoothing, and two-layer radial basis function (RBF). The RBF network was found to yield the best performance (94.8\% generalization by region) and compares favorably with human experts. We conclude that this network is a valuable clinical tool that can be used as a reference ``diagnostic support system'' to help reduce inter- and intra- observer variability. This system is now being further developed to include other variables that are expected to improve the final clinical diagnosis. ----------------------------------------------------- FTP INSTRUCTIONS the usual way: unix> ftp archive.cis.ohio-state.edu (or 128.146.8.52) Name: anonymous Password: ftp> cd pub/neuroprose ftp> binary ftp> get rosenberg.scintigrams.ps.Z ftp> quit unix> uncompress rosenberg.scintigrams.ps.Z unix> lpr rosenberg.scintigrams.ps Current address: Dr. Charles Rosenberg GRECC 182 VA Medical Center 500 Foothill Dr. Salt Lake City, UT 84148 (801) 582-1565 ext. 2458 FAX: (801) 583-7338 crr at cogsci.psych.utah.edu From wray at ptolemy.arc.nasa.gov Wed Nov 4 17:34:55 1992 From: wray at ptolemy.arc.nasa.gov (Wray Buntine) Date: Wed, 4 Nov 92 14:34:55 PST Subject: CP for AI and Stats Workshop Message-ID: <9211042234.AA23199@ptolemy.arc.nasa.gov> NOTE: This second call for participation contains a list of papers and posters being presented. ;;;-------------------------- Cut here ----------------------------------- 2nd Call for Participants and Schedule for Fourth International Workshop on Artificial Intelligence and Statistics January 3-6, 1993 Ft. Lauderdale, Florida PURPOSE: This is the fourth in a series of workshops which has brought together researchers in Artificial Intelligence and in Statistics to discuss problems of mutual interest. The result has been an unqualified success. The exchange has broadened research in both fields and has strongly encouraged interdisciplinary work. This workshop will have as its primary theme: ``Selecting models from data'' FORMAT: Approximately 60 papers by leading researchers in Artificial Intelligence and Statistics have been selected for presentation. To encourage interaction and a broad exchange of ideas, the presentations will be limited to 20 discussion papers in single session meetings over the three days. Focussed poster sessions, each with a short presentation, provide the means for presenting and discussing the remaining 40 research papers. Attendance at the workshop is *not* limited. The three days of research presentations will be preceded by a day of tutorials. These are intended to expose researchers in each field to the methodology used in the other field. LANGUAGE: The language will be English. FORMAT: One day of tutorials and three days of focussed poster sessions, presentations and panels. The presentations are scheduled in the mornings and evenings, leaving the afternoons free for discussions in more relaxed environments. SCHEDULE: Sun: Jan. 3rd. -------------- Sunday is scheduled for tutorials. There are 4 -- at most two can be attended without conflict. AI for statisticians Morning: Doug Fisher -- Intro. to learning including neural networks Afternoon: Judea Pearl -- Graphical models, causal reasoning, and qualitative decision making. Statistics for AI Morning: Wray Buntine -- Introduction to Statistics and Decision Analysis Afternoon: Daryl Pregibon -- Overview of Statistical Models Mon: Jan. 4th. --------------- 8:30--10:00 1st. Session---Model Selection Peter Cheeseman--Introduction: "Overview of Model Selection" Beat E. Neuenschwander, Bernard D. Flury, "Principal Components and Model Selection". Cullen Schaffer, "Selecting a Classification Method by Cross-Validation". Stanley Sclove, "Small-Sample and Large-Sample Statistical Model Selection Criteria". -------------------------------------------------------- 10:00--10:30 break -------------------------------------------------------- 10:30--12:00 2nd. Session---Model Comparison C. Feng, A. Sutherland, R. King, S. Muggleton, R. Henery, Comparison of Classification Algorithms in Machine Learning, Statistics, and Neural Networks (DRAFT). Richard D. De Veaux, "A Tale of Two Nonparametric Estimation Schemes: MARS and Neural Networks". Christopher de Vaney, "A Support Architecture for Statistical Meta-Information with Knowledge-Based Extensions". + Discussion (speakers and audience) --------------------------------------------------------- Lunch (provided) --------------------------------------------------------- 1:30--3:00 1st panel--Alternative Approaches to Model Selection Panel Moderator: Wayne Oldford --------------------------------------------------------- 3:00--3:30 break --------------------------------------------------------- 3:30--5:00 3rd. Session---Statistics in AI Nathaniel G. Martin, James F. Allen, "Statistical Probabilities for Planning". Arcot Rajasekar, "On Closures in Knowledge Base Systems". Steffen L. Lauritzen, B. Thiesson, DJ Spiegelhalter, "Diagnostic Systems Created by Model Selection Methods-A Case Study". Vladimir Cherkassky, "Statistical and Neural Network Techniques For Nonparametric Regression". -------------------------------------------------------- -------------------------------------------------------- Tue: Jan. 5th. -------------- 8:30--10:00 4th Session---Causal Models Floriana Esposito, Donato Malerba, Giovanni Semeraro, "Comparison of Statistical Methods for Inferring Causation". J. Pearl and N. Wermuth, "When Do Association Graphs have Causal Explanations". Richard Scheines, "Inferring Causal Structure Among Unmeasured Variables". + Invited speaker -------------------------------------------------------- 10:00--10:30 break -------------------------------------------------------- 10:30--12:00 5th Session---Very Short "poster" presentations -------------------------------------------------------- break--rest of afternoon off -------------------------------------------------------- 6:00 -7:30 buffet supper (provided) 7:30 -8:40 1st poster session (see list of posters at end) 8:50 -10:00 2nd poster session (preceded by 10 minute changeover) -------------------------------------------------------- -------------------------------------------------------- Wed: Jan. 6th. -------------- 8:30--10:00 6th Session---Influence Diagrams and Probabilistic Networks Remco R Bourckaert, "Conditional Dependence in Probabilistic Networks". Geoffrey Rutledge MD, Ross Shachter, "A Method for the Dynamic Selection of Models Under Time Constraints". Gregory M. Provan, "Diagnosis Over Time Using Temporal Influence Diagrams". + Discussion (speakers and audience) -------------------------------------------------------- 10:00--10:30 break -------------------------------------------------------- 10:30--12:00 7th Session---AI in Statistics R. W. Oldford, D. G. Anglin, "Modelling Response Models in Software". D. J. Hand, "Statistical Strategy: Step 1". David Draper, "Assessment and Propagation of Model Uncertainty". Debby Keen, Arcot Rajasekar, "Reasoning With Inductive Dependencies" --------------------------------------------------------- Lunch (provided) --------------------------------------------------------- 1:30--3:00 2nd panel 3:00--Business meeting -----------------------Posters-------------------------------- Russell G. Almond, "An Ontology for Graphical Models". D.L. Banks, R.A. Maxion, "Comparative Evaluation of New Wave Methods for Model Selection". Raj Bhatnagar, Laveen N Kanal, "Models from Data for Various Types of Reasoning". Djamel Bouchaffra, Jacques Rouault, "Different ways of capturing the observations in a nonstationary hidden Markov model: application to the problem of Morphological Ambiguities". Victor L. Brailovsky, "Model selection by perturbing data set (extended abstract)". Carla E. Brodley, Paul Utgoff, "Dynamic Recursive Model Class Selection for Classifier Construction Extended Abstract". W. Buntine, "On Generic Priors in Learning". Paul R. Cohen, "Path Analysis Models of an Autonomous Agent in a Complex Environment". Sally Jo Cunningham, Paul Denize, "A Tool for Model Genertion and Knowledge Acquisition". Luc Devroye, Oliver Kamoun, "Probabilistic Min-Max Trees". E. Diday, P. Brito and E. Mfoumoune, "Modelling Probabilistic Data by Conceptual Pyramidal Clustering". Kris Dockx, James Lutsko, "SA/GA: Survival of the Fittest in Alaska". Zbigniew Duszak, Jerzy Grzymala-Busse, Waldemar W. Koczkoda, "Rule Induction Based on Statistics and Rough Set Theory". J. J. Faraway, "Choise of Order in Regression Strategy". Karina Gibert, "Combining a Knowledge-based System and a Clustering Method For an Inductive Construction of Models". Scott D. Goodwin, Eric Neufeld, Andre Trudel, "Extrapolating Definite Integral Information". Jonathan Gratch, Gerald DeJong, "Rational Learning: Finding a Balance Between Utility and Efficiency". A. K. Gupta, "Information Theoretic Approach to Some Multivariate Tests of Homogeneity". Paula Hietala, "Statistical Reasoning to Enhance User Modelling in Consulting Systems". Adele Howe, Paul R. Cohen, "Detecting and Explaining Dependencies in Execution Traces". Sung-Ho Kim, "On Combining Conditional Influence Diagrams". Willi Klosgen, "Discovery in Databases". G. J. Knafl, A. Semrl, "Software Reliability Expert (SRX)". Bing Leng, Bruce Buchanan, "Using Knowledge-Assisted Discriminant Analysis to Generate New Comparative Terms for Symblic Learner". James F. Lutsko, Bart Kuijpers, "Simulated Annealing in the Construction of Near-Optimal Decision Trees". Yong Ma, David Wilkins, John S. Chandler, "An Extended Bayesian Belief Function Approach to Handle Noise in Inductive Learning". Izhar Matzkevich, Bruce Abramson, "Towards Prior Compromise in Belief Networks (Extended Abstract)". Johnathan Oliver, "Decision Graphs - An Extension of Decision Trees". Egmar Rodel, "A Knowledge Based System for Testing Bivariate Dependence". A.R. Runnaalls, "Global vs Local Sampling Procedures for Inference on Directed Graphs". David Russell, "Statistical Inferencing in a Real-Time Heuristic Controller". Geoffrey Rutledge MD, Ross Shachter, "A Method for the Dynamic Selection of Models Under Time Constraints". Steven Salzberg, David Aha, "Learning to Catch: Applying Nearest Neighbor algorithms to Dynamic Control Tasks". D. Moreira dos Santos, "Selecting a Frailty Model for Longitudinal Breast Cancer Data". Glenn Shafer, "Recursion in Join Trees". P. Shenoy, "Searching For Alternative Representation of Data: A Case for Tetrad". Hidetoshi Shimodaira, "A New Criterion for Selecting Models from Partially Observed Data". P. Smyth, "The Nature of Class Labels in Supervised Learning". Peter Spirtes, Clark Glymour, "Inference, Intervention and Prediction". Marco Valtora, R. Mechling, "PaCCIN: A parallel Constructor of Markov Networks". Aaron Wallack, Ed Nicolson, "Optimal Design of Reflective Sensors Using Probabilistic Analysis". Bradley Whitehall, David Sirag, "Clustering of Smybolically Described Events for Prediction of Numeric Attributes". Nevin Lianwen Zhang, Runping Qi, David Poole, "Minizing Decision Table Sizes in Stepwise-Decomposable Influence Diagrams". Ping Zhang, "On the Choise of Penalty term in Generalized FPE criterion". PROGRAM COMMITTEE: General Chair: R.W. Oldford U. of Waterloo, Canada Programme Chair: P. Cheeseman NASA (Ames), USA Members: W. Buntine NASA (Ames), USA Wm. Dumouchel BBN, USA D.J. Hand Open University, UK W.A. Gale AT&T Bell Labs, USA H. Lenz Free University, Germany D. Lubinsky AT&T Bell Labs, USA M. Deutsch-McLeish U. of Guelph, Canada E. Neufeld U. of Saskatchewan, Canada J. Pearl UCLA, USA D. Pregibon AT&T Bell Labs, USA P. Shenoy U. of Kansas, USA P. Smythe JPL, USA SPONSORS: Society for Artificial Intelligence And Statistics International Association for Statistical Computing REGISTRATION: All fees paid: Before Dec 1, 1992 After Dec 1, 1992 Scientific programme: $225 $275 Full-time Students $135 $175 - Registration fee includes three continental breakfasts and two lunches supplied at the workshop site. - Students must supply proof of full-time student status (at the workshop) to be eligible for reduced rates. A REGISTRATION FORM APPEARS AT THE END OF THIS MESSAGE. TUTORIALS: There are four three hour tutorials planned. Two introducing statistical methodology to AI researchers and two introducing AI methodology to statistical researchers. Before Dec 1, 1992 After Dec 1, 1992 Per Tutorial $65 $75 Full-time Students $40 $45 The tutorials are introductions to the following topics: 1. Learning, including a discussion of neural networks. Speaker: Doug Fisher, Vanderbilt University Orientation: AI for statisticians 2. Graphical models, causal reasoning, and qualitative decision making. Speaker: Judea Pearl, UCLA Orientation: AI for statisticians. 3. Overview of statistical models. Emphasis on generalised linear and additive models. Speaker: Daryl Pregibon, AT&T Bell Labs Orientation: Statistics for AI researchers. 4. Introduction to Statistics. General introduction to statistical topics Speaker: Wray Buntine, NASA Ames Orientation: Statistics for AI researchers. Please indicate which tutorial(s) you are registering for. PAYMENT OF FEES: All workshop fees are payable by cheque or money order in U.S. dollars (drawn on a U.S. bank) to the Society for Artificial Intelligence and Statistics. Send cheque or money order to: R.W. Oldford Chair, 4th Int'l Workshop on A.I. & Stats. Dept. of Statistics & Actuarial Science University of Waterloo Waterloo, Ontario N2L 3G1 CANADA NOTE: ACCOMODATIONS MUST BE ARRANGED DIRECTLY WITH THE HOTEL. ACCOMODATION: We have arranged for a block of rooms to be available to participants at the Workshop site hotel for $85 per night (single or double + tax). Arrangements must be made directly with the hotel. Please mention the Workshop on all communications. Rates are available Jan 1 to Jan 10 (if booked before Dec 17, 1992). Pier 66 Resort and Marina 2301 S.E. 17th Street Causeway Ft. Lauderdale, Florida 33316 (305) 525 6666 (800) 327 3796 (USA only) (800) 432 1956 (Florida only) Fax: (305) 728 3551 Telex: 441-650 REGISTRATION FORM: 4th International Workshop on AI and Statistics January 3-6, 1993 Ft. Lauderdale, Florida Name: _______________________________ Affiliation: _______________________________ Address: _____________________________________________ _____________________________________________ _____________________________________________ _____________________________________________ e-mail: _____________________________________________ Fax: ___________________________ Phone: ___________________________ Scientific Programme Registration ...................... US$___________ Tutorial 1. Learning ................................... US$___________ Tutorial 2. Causal Reasoning ........................... US$___________ Tutorial 3. Statistical Models ......................... US$___________ Tutorial 4. Introduction to Statistics ................. US$___________ _______________________________________________________________________ Total Payment .......................................... US$___________ From gmk at osprey.siemens.com Wed Nov 4 12:33:46 1992 From: gmk at osprey.siemens.com (Gary M. Kuhn) Date: Wed, 4 Nov 92 12:33:46 EST Subject: Call for Papers, NNSP'93 Message-ID: <9211041733.AA00365@osprey.siemens.com> CALL FOR PAPERS _______________ 1993 IEEE Workshop on Neural Networks for Signal Processing September 7-9, 1993 Baltimore, MD, USA Sponsored by the IEEE Technical Committee on Neural Networks in cooperation with the IEEE Neural Networks Council The third of a series of IEEE workshops on Neural Networks for Signal Processing will be held at the Maritime Institute of Technology and Graduate Studies, Linthicum, Maryland, USA, in September of 1993. Papers are solicited for, but not limited to, the following topics: 1. Applications: Image processing and understanding, speech recognition, communications, sensor fusion, medical diagnoses, nonlinear adaptive filtering and other general signal processing and pattern recognition topics. 2. Theory: Neural network system theory, identification and spectral estimation, and learning theory and algorithms. 3. Implementation: Digital, analog, and hybrid technologies and system development. Prospective authors are invited to submit 4 copies of extended summaries of no more than 6 pages. The top of the first page of the summary should include a title, authors' names, affiliations, address, telephone and fax numbers and email address if any. Camera-ready full papers of accepted proposals will be published in a hard-bound volume by IEEE and distributed at the workshop. Due to workshop facility constraints, attendance will be limited with priority given to those who submit written technical contributions. For further information, please contact Karin Cermele at the NNSP'93 Princeton office, (Tel.) +1 609 734 3383, (Fax) +1 609 734 6565, (e-mail) kic at learning.siemens.com. PLEASE SEND PAPER SUBMISSIONS TO: _______________ NNSP'93 Siemens Corporate Research 755 College Road East Princeton, NJ 08540 USA SCHEDULE _______________ Submission of extended summary: February 15 Notification of acceptance: April 19 Submission of photo-ready paper: June 1 Advanced registration, before: June 1 WORKSHOP COMMITTEE _______________ General Chairs Gary Kuhn Barbara Yoon Siemens Corporate Research DARPA-MTO 755 College Road East 3701 N. Fairfax Dr. Princeton, NJ 08540, USA Arlington, VA 22203-1714 USA gmk at learning.siemens.com byoon at a.darpa.mil Program Chair Proceedings Chair Rama Chellappa Candace Kamm Dept. of Electrical Engineering Box 1910 University of Maryland Bellcore, 445 South Street College Park, MD 20742, USA Morristown, NJ 07962, USA chella at eng.umd.edu cak at bellcore.com Finance Chair Raymond Watrous Siemens Corporate Research 755 College Road East Princeton, NJ 08540, USA watrous at learning.siemens.com Program Committee Joshua Alspector John Makhoul Les Atlas B.S. Manjunath Charles Bachmann Tomaso Poggio Gerard Chollet Jose Principe Frank Fallside Ulrich Ramacher Lee Giles Noboru Sonehara S.J. Hanson Eduardo Sontag Y.H. Hu J.A.A. Sorensen B.H. Juang Yoh'ichi Tohkura Shigeru Katagiri Christoph von der Malsburg S.Y. Kung Christian Wellekens Yann LeCun From cohn at psyche.mit.edu Wed Nov 4 15:52:45 1992 From: cohn at psyche.mit.edu (David Cohn) Date: Wed, 4 Nov 92 15:52:45 EST Subject: Post-NIPS Robot Learning workshop program Message-ID: <9211042052.AA08017@psyche.mit.edu> ___________________________________________________________________________ PROGRAM FOR THE POST-NIPS WORKSHOP "ROBOT LEARNING" Vail, Colorado, Dec 5th, 1992 NIPS=92 Workshop: Robot Learning ================= Intended Audience: Connectionists and Non-Connectionists in Robotics, ================== Control, and Active Learning Organizers: =========== Sebastian Thrun (CMU) Tom Mitchell (CMU) David Cohn (MIT) thrun at cs.cmu.edu mitchell at cs.cmu.edu cohn at psyche.mit.edu Program: ======== Robot learning has grasped the attention of many researchers over the past few years. Previous robotics research has demonstrated the difficulty of manually encoding sufficiently accurate models of the robot and its environment to succeed at complex tasks. Recently a wide variety of learning techniques ranging from statistical calibration techniques to neural networks and reinforcement learning have been applied to problems of perception, modeling and control. Robot learning is characterized by sensor noise, control error, dynamically changing environments and the opportunity for learning by experimentation. This workshop will provide a forum for researchers active in the area of robot learning and related fields. It will include informal tutorials and presentations of recent results, given by experts in this field, as well as significant time for open discussion. Problems to be considered include: How can current learning robot techniques scale to more complex domains, characterized by massive sensor input, complex causal interactions, and long time scales? How can previously acquired knowledge accelerate subsequent learning? What representations are appropriate and how can they be learned? Although each session has listed "speakers," the intent is that each speaker will not simply present their own work, but will introduce their work interactively, as a launching point for group discussion on their chosen area. After all speakers have finished, the remaining time will be used to discuss at length issues that the group feels need most urgently to be addressed. Below, we have listed the tentative agenda, which is followed by brief abstracts of each author's topic. For those who wish to get a head start on the workshop, we have included a list of references and/or recommended readings, some of which are available by anonymous ftp. ===================================================================== ===================================================================== AGENDA ===================================================================== ===================================================================== SESSION ONE (early morning session), 7:30 - 9:30: ------------------------------------------------- TITLE: "Robot learning: scaling up and state of the art" Keynote speaker: Chris Atkeson (30 min) "Paradigms for Robot Learning" Speakers: Steve Hanson (15 min) (title to be announced) Satinder Singh (15 min) Behavior-Based Reinforcement Learning Andrew W. Moore(15 min) The Parti-Game Algorithm for Variable Resolution Reinforcement Learning Richard Yee (15 min) Building Abstractions to Accelerate Weak Learners SESSION TWO (apres-ski session), 4:30 - 6:30: --------------------------------------------- PANEL: "Robot learning: Where are the new ideas coming from?" Keynote speaker: Andy Barto (30 min) Speakers: Tom Mitchell (10 min each) Chris Atkeson Dean Pomerleau Steve Suddarth ===================================================================== ===================================================================== ABSTRACTS ===================================================================== Session 1: Scaling up and the state of the art When: Saturday, Dec 5, 7:30-9:30 a.m. ===================================================================== ===================================================================== Keynote: Chris Atkeson (cga at ai.mit.edu) Title: Paradigms for Robot Learning Abstract: This talk will survey a variety of robot learning tasks and learning paradigms to perform those tasks. The tasks include pattern classification, regression/function approximation, root finding, function optimization, designing feedback controllers, trajectory following, stochastic modeling, stochastic control, and strategy generation. Given this wide range of tasks it seems reasonable to ask if there is any commonality among them, or any way in which solving one task might make other tasks easier to perform. In our own work we have typically taken an indirect approach: our learning algorithms explicitly form models, and then solve the problem using algorithms that assume complete knowledge. It is not at all clear which learning tasks are best dealt with using an indirect approach, and which are handled better with a direct approach in which the control strategy is learned directly. Nor is it clear how to cope with uncertainty and incomplete knowledge, either by modeling it explicitly, using stochastic models, or using game theory and assuming a malevolent world. I hope to provoke a discussion on these issues. ====================================================================== Presenter: Satinder Pal Singh (singh at cs.umass.edu) Title: Behavior-Based Reinforcement Learning Abstract: Control architectures based on reinforcement learning have been successfully applied to agents/robots that use their repertoire of primitive control actions to achieve goals in an external environment. The optimal policy for any goal is a state-dependent composition of the given "primitive" policies (a primitive policy "A" assigns action A to every state). In that sense, the primitive policies form the "basis" set from which optimal solutions can be "composed". I argue that reinforcement learning can be greatly accelerated by redefining the basis set of policies available to the agent. These redefined basis policies should correspond to "behaviors" that are useful across the set of tasks faced by the agent. Behavior-based RL, i.e., the application of RL to behavior-based robotics (ref Brooks), has several advantages: it can drastically reduce the effective dimensionality of the action space, it provides a framework for incorporating prior knowledge into RL architectures, it provides a technique for achieving transfer of learning, and finally by restricting the rules of composition and the types of behaviors it may become possible to perform "robust" reinforcement learning. I will provide examples from my own work and that of others to illustrate these ideas. (Refs 4, 5, 6) ====================================================================== Presenter: Andrew W. Moore (awm at ai.mit.edu) Title The Parti-Game Algorithm for Variable Resolution Reinforcement Learning Can we efficiently learn in continuous state-spaces, while requiring only relatively few real-world experienvces during the learning stage? Dividing a continuous state-space into a fine grid can mean a tragically large number of unnecessary experiences, while a coarse grid or parametric representation can become stuck. This talk overviews a new algorithm which, in real time, tries to adaptively alter the resolution of a state space partitioning to be coarse where it can and fine where it must to be if it is to avoid becoming stuck. The key idea turns out to be the treatment of the problem as a game instead of a Markov decision task. Possible prior reading: Ref 7 (Overview of some other uses of kd-trees in Machine learning) Ref 8 (A non-real-time algorithm which uses a different partitioning strategy) Ref 9 (A search control technique which Parti-Game uses) Refs 9, 10 ====================================================================== Presenter: Richard Yee, (yee at cs.umass.edu) Title: Building Abstractions to Accelerate Weak Learners Abstract: Learning methods based on dynamic programming (DP) are promising approaches to the problem of controlling dynamical systems. Practical DP-based learning will require function approximation methods that are well-suited for learning optimal value functions, which map system states into numeric estimates of utility. Such approximation problems are generally characterized by non-stationary, dependent training data and, in many cases, little prospect for incorporating strong {\em a priori\/} learning biases. Consequently. this talk considers learning approaches that begin weakly (e.g., using rote memorization) but strengthen their learning biases as experiences accrue. Abstracting from stored experiences should accelerate learning by improving generalization. Bootstrapping such abstraction processes (cf.\ "hypothesis boosting") might be a practical means for scaling DP-based learning across a wide variety of applications. (Refs 1, 2, 3, 4) ===================================================================== Session 2: Where are the new ideas coming from? When: Saturday, Dec 5, 4:30-6:30 p.m. ===================================================================== ===================================================================== Keynote: Andrew G. Barto (barto at cs.umass.edu) Title: Reinforcement Learning Theory Although reinforcement learning is being studied more widely than ever before, especially methods based on approximating dynamic programming (DP), its theoretical foundations are not yet highly developed. In this talk, I discuss what I percieve to be the current state and the missing links in this theory. This topic raises such questions as the following: Just what is DP-based reinforcement learning from a mathematical perspective? What is the relationship between DP-based reinforcement learning and other methods for approximating DP? What theoretical justification exists for combining function approximation methods (such as artificial neural networks) with DP-based learning? What kinds of problems are best suited to DP-based reinforcement learning? Is theory important? ===================================================================== Presenter: Dean Pomerleau Title: Combining artificial neural networks and symbolic processing for autonomous robot guidance Artificial neural networks are capable of performing the reactive aspects of autonomous driving, such as staying on the road and avoiding obstacles. This talk describes an efficient technique for training individual networks to perform these reactive driving tasks. But driving requires more than a collection of isolated capabilities. To achieve true autonomy, a system must determine which capabilities should be employed in the current situation to achieve its objectives. Such goal directed behavior is difficult to implement in an entirely connectionist system. This talk describes a rule-based technique for combining multiple artificial neural networks with map-based symbolic reasoning to achieve high level behaviors. The resulting system is not only able to stay on the road, it is able follow a route to a predetermined destination, turning appropriately at intersections and stopping when it has reached its goal. (Refs 11, 12, 13, 14, 15) ===================================================================== ===================================================================== References ===================================================================== ===================================================================== (#1) Yee, Richard, "Abstraction in Control Learning", Department of Computer and Information Science, University of Massachusetts, Amherst, MA 01003, COINS Technical Report 92-16, March 1992. anonymous ftp: envy.cs.umass.edu:pub/yee.abstrn.ps.Z (#2) Barto, Andrew G. and Richard S. Sutton and Christopher J. C. H. Watkins, Sequential decision problems and neural networks, in Advances in Neural Information Processing Systems 2, 1990, Touretzky, D. S., ed. (#3) Barto, Andrew G. and Richard S. Sutton and Christopher J. C. H. Watkins", Learning and Sequential Decision Making, in Learning and Computational Neuroscience: Foundations of Adaptive Networks, 1990. anonymous ftp: archive.cis.ohio-state.edu:pub/neuroprose/barto.sequential_decisions.ps.Z (#4) Barto, Andrew G. and Steven J. Bradtke and Satinder Pal Singh, Real-time learning and control using asynchronous dynamic programming, Computer and Information Science, University of Massachusetts, Amherst, MA 01003, COINS Technical Report TR-91-57, August 1991. anonymous ftp: archive.cis.ohio-state.edu:pub/neuroprose/barto.realtime-dp.ps.Z (#5) Singh, S.P.," Transfer of Learning by Composing Solutions for Elemental Sequential Tasks, Machine Learning, 8:(3/4):323-339, May 1992. anonymous ftp: envy.cs.umass.edu:pub/singh-compose.ps.Z (#6) Singh, S.P., "Scaling reinforcement learning algorithms by learning variable temporal resolution models, Proceedings of the Ninth Machine Learning Conference, D. Sleeman and P. Edwards, eds., July 1992. anonymous ftp: envy.cs.umass.edu:pub/singh-scaling.ps.Z (#7) S. M. Omohundro, Efficient Algorithms with Neural Network Behaviour, Journal of Complex Systems, Vol 1, No 2, pp 273-347, 1987. (#8) A. W. Moore, Variable Resolution Dynamic Programming: Efficiently Learning Action Maps in Multivariate Real-valued State-spaces, in "Machine Learning: Proceedings of the Eighth International Workshop", edited by Birnbaum, L. and Collins, G., published by Morgan Kaufman. June 1991. (#9) A. W. Moore and C. G. Atkeson, Memory-based Reinforcement Learning: Converging with Less Data and Less Real Time, 1992. See the NIPS92 talk or else preprints available by request to awm at ai.mit.edu (#10) J. Peng and R. J. Williams, Efficient Search Control in Dyna, College of Computer Science, Northeastern University, March, 1992 (#11) Pomerleau, D.A., Gowdy, J., Thorpe, C.E. (1991) Combining artificial neural networks and symbolic processing for autonomous robot guidance. In {\it Engineering Applications of Artificial Intelligence, 4:4} pp. 279-285. (#12) Pomerleau, D.A. (1991) Efficient Training of Artificial Neural Networks for Autonomous Navigation. In {\it Neural Computation 3:1} pp. 88-97. (#13) Touretzky, D.S., Pomerleau, D.A. (1989) What's hidden in the hidden units? {\it BYTE 14(8)}, pp. 227-233. (#14) Pomerleau, D.A. (1991) Rapidly Adapting Artificial Neural Networks for Autonomous Navigation. In {\it Advances in Neural Information Processing Systems 3}, R.P. Lippmann, J.E. Moody, and D.S. Touretzky (ed.), Morgan Kaufmann, pp. 429-435. (#15) Pomerleau, D.A. (1989) ALVINN: An Autonomous Land Vehicle In a Neural Network. In {\it Advances in Neural Information Processing Systems 1}, D.S. Touretzky (ed.), Morgan Kaufmann, pp. 305-313. From warthman at garnet.berkeley.edu Thu Nov 5 20:54:17 1992 From: warthman at garnet.berkeley.edu (warthman@garnet.berkeley.edu) Date: Thu, 5 Nov 92 17:54:17 -0800 Subject: Audio Synthesizer Message-ID: <9211060154.AA24538@garnet.berkeley.edu> ********************** News Release ************************ November 5, 1992 ************************************************************ Neural-Network Audio Synthesizer Debuts at Paris Opera House ************************************************************ Palo Alto, California -- The old Opera House in Paris, France, will feature five performances by the Merce Cunningham Dance Company, November 12 to 17, in which a new type of audio synthesizer based on an artificial neural network will be used to generate electronic music. The synthesizer's musical accompaniment was composed and will be performed by David Tudor and his dance company colleague, Takehisa Kosugi. The audio synthesizer is built around an integrated-circuit chip from Intel Corporation in Santa Clara, California. The chip, called the Intel 80170NX electrically trainable analog neural network (ETANN), simulates the function of nerve cells in a biological brain. A remarkable range of audio effects can be generated with the electronic synthesizer -- from unique space-age and science-fiction sounds to passages that sound very much like birds, heart beats, porpoises, engines, and acoustical, percussion or string musical instruments. Sounds are generated internally by the synthesizer. External inputs such as voice, music, or random sounds can optionally be used to enrich or control the internally generated sounds. In addition to generating outputs to multiple audio speakers, the synthesizer can simultaneously drive oscilloscopes or other visual devices. The neural network chip's software consists of numeric values representing interconnection strengths between inputs and outputs -- a configuration analogous to the excitatory or inhibitory strengths of synapse connections between biological nerve cells. The artificial neurons can be connected in loops, using the programmable interconnection strengths, or they can be connected outside the chip with cables and feedback circuits. Audio oscillations occur as a result of delay in the feedback paths and thermal noise in the neural network chip. The sounds are generally rich because of the complexity of the circuitry. The concept for the synthesizer evolved from a project begun in 1989 by Forrest Warthman and David Tudor. The synthesizer was designed and built by Warthman; Mark Thorson, a hardware designer and associate editor of Microprocessor Report; and Mark Holler, Intel's program manager for neural network products. John Cage visited the design group in Palo Alto a few months before his passing away at the age of 79 this year. His observations on the synthesizer's role in musical composition and dance performance contributed to its current design. A description of the synthesizer's architecture and circuitry will appear in the February 1993 issue of Dr. Dobb's Journal. From Dave_Touretzky at DST.BOLTZ.CS.CMU.EDU Thu Nov 5 22:25:21 1992 From: Dave_Touretzky at DST.BOLTZ.CS.CMU.EDU (Dave_Touretzky@DST.BOLTZ.CS.CMU.EDU) Date: Thu, 05 Nov 92 22:25:21 EST Subject: a couple of administrative matters Message-ID: <22435.721020321@DST.BOLTZ.CS.CMU.EDU> #1. Due to the increasing size of the CONNECTIONISTS list, we are no longer able to wade through hundreds returned fail messages (most of which are spurious) to determine who is and is not receiving mail successfully. We will now ask you to contact us if you stop receiving mail. Connectionists sends out more than a message a week, so if you do not receive mail for ONE WEEK, send mail to Connectionists-Request at cs.cmu.edu with your full name, address, and where you are receiving mail. This last we need in case you are on a local redistribution list. We also ask that if your account is about to expire or you are moving, please contact us at the above address so that we can remove or update your entry. Thank you David Redish and Dave Touretzky Connectionists-Request at cs.cmu.edu ================================================================ #2. Do you wonder what happened to all the people who used to post here asking for references on random topics? The're still with us, but the list is now FILTERED and those messages are killed before they make it out to our readership. Please don't send requests for references to this list; they will be rejected. There is one exception to the above rule. If you have already compiled a substantial bibliography on some topic, you may post it along with a request for *additional* references. (People are encouraged to post useful bibliographies even if they're not looking for additions.) But if you simply can't resist the urge to post a request for references, a plea for free software, or a really elementary question about neural nets, then the folks over on comp.ai.neural-nets are the ones you should be pestering. Subscribe today! From MARCHESF%PACEVM.BITNET at BITNET.CC.CMU.EDU Sat Nov 7 16:28:14 1992 From: MARCHESF%PACEVM.BITNET at BITNET.CC.CMU.EDU (Dr. Francis T. Marchese) Date: 07 Nov 1992 16:28:14 -0500 (EST) Subject: Call for Participation Message-ID: <01GQVKHN9OAA9AMLQA@BITNET.CC.CMU.EDU> *** Call For Participation *** Conference on Understanding Images Sponsored By NYC ACM/SIGGRAPH and Pace University's School of Computer Science and Information Systems To Be Held at: Pace University New York City, New York May 21-22,1993 Artists, designers, scientists, engineers and educators share the problem of moving information from one mind to another. Traditionally, they have used pictures, words, demonstrations, music and dance to communicate imagery. However, expressing complex notions such as God and infinity or a seemingly well defined concept such as a flower can present challenges which far exceed their technical skills. The explosive use of computers as visualization and expression tools has compounded this problem. In hypermedia, multimedia and virtual reality systems vast amounts of information confront the observer or participant. Wading through a multitude of simultaneous images and sounds in possibly unfamiliar representations, a confounded user asks: What does it all mean? Since image construction, transmission, reception, decipherment and ultimate understanding are complex tasks strongly influenced by physiology, education and culture; and since electronic media radically amplify each processing step, then we, as electronic communicators, must determine the fundamental paradigms for composing imagery for understanding. Therefore, the purpose of this conference is to bring together a breadth of disciplines, including, but not limited to, the physical, biological and computational sciences, technology, art, psychology, philosophy and education, in order to define and discuss the issues essential to image understanding within the computer graphics context. To this end we seek proposals for individual presentations, panel discussions, static displays, interactive environments, performances and beyond. Submissions: Contributors are requested to submit a one page proposal by January 15, 1993. Accepted presentations will be included in the proceedings. Direct all inquires and submissions to: Professor Francis T. Marchese Department of Computer Science Pace University New York, NY 10038 USA Email: MARCHESF at PACEVM.Bitnet Phone: 212-346-1803 Fax: 212-346-1933 From jbower at cns.caltech.edu Mon Nov 9 12:54:23 1992 From: jbower at cns.caltech.edu (Jim Bower) Date: Mon, 9 Nov 92 09:54:23 PST Subject: extended claims Message-ID: <9211091754.AA17739@smaug.cns.caltech.edu> From carsten at thep.lu.se Tue Nov 10 09:01:40 1992 From: carsten at thep.lu.se (carsten@thep.lu.se) Date: Tue, 10 Nov 92 15:01:40 +0100 Subject: Postdoc Position in Lund Message-ID: <9211101401.AA19495@dacke.thep.lu.se> A two year postdoc position will be available within the Complex Systems group at the Department of Theoretical Physics, University of Lund, Sweden, starting September 1st 1993. The major research area of the group is Artificial Neural Networks with tails into chaos and difficult computational problems in general. Although some application studies occur, algorithmic development is the focus in particular within the following areas: * Using Feed-back ANN for finding good solutions to combinatorial optimization problems; knapsacks, scheduling, track-finding. * Time-series prediction. * Robust multi-layer perceptron updating procedures including noise. * Deformable template methods -- robust statistics. * Configurational Chemistry -- Polymers, Proteins ... * Application work within the domain of experimental physics, in particular in connection with the upcoming SSC/LHC experiments. Lund University is the largest campus in Scandinavia located in a picturesque 1000 year old city (100k inhabitants). Lund is strategically well located in the south of Sweden with 1.5 hrs commuting distance to Copenhagen (Denmark). The candidate should have a PhD in a relevant field, which need not be Physics/Theoretical Physics. Applications and three letters of recommendation should be sent to (not later than December 15): Carsten Peterson Department of Theoretical Physics University of Lund Solvegatan 14A S-223 62 Lund Sweden or Bo S\"{o}derberg Department of Theoretical Physics University of Lund Solvegatan 14A S-223 62 Lund Sweden From weber at forwiss.tu-muenchen.de Wed Nov 11 04:51:41 1992 From: weber at forwiss.tu-muenchen.de (Walter Weber) Date: Wed, 11 Nov 1992 10:51:41 +0100 Subject: Generalization ability of a BPTT-net Message-ID: <9211110951.AA17888@forwiss.tu-muenchen.de> Dear Connectionists, let me first briefly introduce my work: I'm just writing a masters' thesis at the Technical University of Munich and i'm trying to deal with neural control. My goal is to let a neural controller learn how to pull the gas- and brake-pedals of a car in order to move in appropriate distance to a leading car. Input signals are speed (v), dv/dt, distance (d) and dd/dt (all inputs are coded in [0;1]) and there is only 1 output which varies from 0 to 1. (control signals for the brake are coded in [0;0.5], signals for the gas-pedal are coded in [0.5;1] to get a continuous trajectory for the control signals over time). The training trajectories are from a PID-controller which was implemented at BMW for the PROMETHEUS project. I've done several aproaches to build up a net that could solve this problem (Elman-net, Jordan-net, fully recurrent NN), but the most promising approach seems to be the BPTT-algorithm applied to a three-layer recurrent network with input connected to a fully recurrent hidden layer and that hidden layer connected to the output. The activation-function i use is a sigmoidal function with an offset, the output produced varies in [0;1]. The learning ability of this BPTT-net is quite well (even very well if i use teacher-forcing), but if i want the network to predict the remaining part of a given trajectory after it used the first part (mostly about 70 - 80%) as training data, the following problems occur: 1. The net cannot perform peaks (short and strong pulls of the pedals) even if such data was learned well during training. (My idea about that was: the use of teacher forcing only makes training better but does not influence the generalization ability in a positive way, so the net can only perform what it was able to learn without teacher forcing. And only teacher forcing enabled the net to learn such peaks). 2. If a part of the test trajectory looks similar to a part of the training trajectory with the only difference is a offset between the two trajectory-parts (e.g. the part of the test trajectory varies between 0.6 and 0.8, the part of the training trajectory varies between 0.4 and 0.6) the network produces an output - in trying to approximate the test data - which would fit exactly the training data, but is wrong for the test data (with the error is the offset between the two trajectory-parts). How can i get rid of these offsets in the test data? For my english is not very well, i hope one can understand the problems i tried to describe above. And if anybody has an answer to my questions, i would be very glad if i got answers. So thank you and bye, --- Walter Weber =================================================================== | weber at forwiss.tu-muenchen.de Walter Weber | | FORWISS - TU Muenchen | | Tel.: 089/48095-229 Orleanstr. 34 | | -231 D-8000 Muenchen 80 | =================================================================== From craign at SIMPLEMIND.UROL.BCM.TMC.EDU Tue Nov 10 10:36:36 1992 From: craign at SIMPLEMIND.UROL.BCM.TMC.EDU (Craig Stuart Niederberger) Date: Tue, 10 Nov 92 09:36:36 CST Subject: Neural computing ideas and biological terms Message-ID: <9211101536.AA13052@SIMPLEMIND.UROL.BCM.TMC.EDU.noname> With regards to Jim Bower's complaint: > >From a recent posting: > > "The audio synthesizer is built around an integrated-circuit > chip from Intel Corporation in Santa Clara, California. The > chip, called the Intel 80170NX electrically trainable analog > neural network (ETANN), simulates the function of nerve > cells in a biological brain." > > Unlikely in that we don't yet know how nerve cells in a biological > brain function. Is it really necessary many years (now) > into neural net research to continue to lean on the brain for > moral support? > > Sorry to retorically beat a dead horse, but statements like this > are annoying to those of us whose primary interest is to understand > how the brain works. They also still occur far to frequently > especially in association with products. > Technically, I agree that a rather large schism exists between the unknowns of neurophysiology and the creative endeavors of neural computing. However, I disagree with the contention that "leaning on the brain for moral support" is necessarily bad. I have been applying neural computational models to analyze clinical data, and have found it at times difficult to communicate the significance of these models to my frequently non-mathematically oriented colleagues. Often, I have resorted to explanations that are couched in biological terms rather than mathematical ones, with the opinion that it is better to communicate something rather than nothing at all. Perhaps the focus should be to try to bring more investigators into the fold. Not only do new investigators yield new research and new ideas, but funding from these quarters will follow as well. Craig Niederberger ___________________________________________________________________________ | | | Craig Niederberger M.D. Internet EMAIL: craign at mbcr.bcm.tmc.edu | | Department of Urology, Room 440E US Phone: 713-798-7267 | | Baylor College of Medicine FAX: 713-798-5577 | | One Baylor Plaza [o o] [+ +] [o o] [+ +] [o o] | | Houston, Texas 77030-3498 USA [<-->] [ -- ] [ == ] [<-->] [ ^^ ] | |___________________________________________________________________________| From neuron at cattell.psych.upenn.edu Thu Nov 12 18:29:37 1992 From: neuron at cattell.psych.upenn.edu (Neuron-Digest Moderator, Peter Marvit) Date: Thu, 12 Nov 92 18:29:37 EST Subject: Neuron Digest Message-ID: <232.721610977@cattell.psych.upenn.edu> Dear Connectionists, A note here to [re-]introduce the moderated forum/mailing list called "Neuron Digest" and supply ftp/archive information (which apparently is still incorrect on some lists-of-lists). As the bi-monthly reminder of the Connectionists list notes, Neuron Digest is a moderated forum aimed at sophisticated but general audience. There is no restriction on being a subscriber to the Digest. Connectionists will recognize considerable overlap with respect to Paper and Conference announcements between this mailing list and the Digest. However, Neuron Digest tends to have a bit more free-wheeling discussions and questions (plus occasional editorial comments by the moderator). Especially, there tends to be more neophyte "looking for references" queries. On the other hand, being moderated, Neuron Digest is not as timely as other fora. An average of one to two issues come out per week. For the forseeable future, the Digest will be gatewayed (one-way) to the USENET group comp.ai.neural-nets. While you current Connectionists may not wish to add to your e-mailboxes, feel free to access the back issues (with indices coming by the New Year, hopefully) or mention the Digest to your students. Appended is the standard "welcome" blurb for Neuron Digest (liberally copied from Ken Laws). Archive information is at the end. I would be happy to answer any questions (between running experiments, of course). : Peter Marvit, Neuron Digest Moderator : : Email: : : Courtesy of the Psychology Department, University of Pennsylvania : : 3815 Walnut St., Philadelphia, PA 19104 w:215/898-9208 h:215/387-6433 : ------------------------------ CUT HERE ------------------------------- Internet: NEURON at cattell.psych.upenn.edu Neuron-Digest is a list (in digest form) dealing with all aspects of neural networks (and any type of network or neuromorphic system), especially: NATURAL SYSTEMS Software Simulations Neurobiology Hardware Neuroscience Digital ARTIFICIAL SYSTEMS Analog Neural Networks Optical Algorithms Cellular Automatons Some key words which may stir up some further interest include: Hebbian Systems Widrow-Hoff Algorithm Perceptron Threshold Logic Holography Content Addressable Memories Lyapunov Stability Criterion Navier-Stokes Equation Annealing Spin Glasses Locally Couples Systems Globally Coupled Systems Dynamical Systems (Adaptive) Control Theory Back-Propagation Generalized Delta Rule Pattern Recognition Vision Systems Parallel Distributed Processing Connectionism Any contribution in these areas is accepted. Any of the following are reasonable: Abstracts Reviews Lab Descriptions Research Overviews Work Planned or in Progress Half-Baked Ideas Conference Announcements Conference Reports Bibliographies History Connectionism Puzzles and Unsolved Problems Anecdotes, Jokes, and Poems Queries and Requests Address Changes (Bindings) Archived files/messages will be available with anonymous ftp from the machine cattell.psych.upenn.edu (130.91.68.31) in the directory pub/Neuron-Digest. That directory contains back issues with the names vol-nn-no-mm (e.g., vol-3-no-02). I'm also collecting simulation software in pub/Neuron-Software. Contributions are welcome. All requests to be added to or deleted from this list, problems, questions, etc., should be sent to neuron-request at cattell.psych.upenn.edu. Moderator: Peter Marvit ------------------------------ CUT HERE ------------------------------- From dhw at santafe.edu Thu Nov 12 22:41:16 1992 From: dhw at santafe.edu (dhw@santafe.edu) Date: Thu, 12 Nov 92 20:41:16 MST Subject: No subject Message-ID: <9211130341.AA03280@zia> Concerning the recent comments by Jim Bower and Craig Niederberger: Justifying a particular technique solely by an analogy (rather than with a close scrutinization of the task at hand) is dubious in the best of circumstances. Using such a technique when the analogy is in fact poor is engineering by hope. Maybe fun. Might work. Might (!) even give insight. But not science. (At least, not science unless the posited "insight" somehow provides after-the-fact justification for the technique, or at least gives some reason to believe the technique is a reasonable heuristic.) David Wolpert dhw at santafe.edu From yirgan at dendrite.cs.colorado.edu Thu Nov 12 12:01:19 1992 From: yirgan at dendrite.cs.colorado.edu (Juergen Schmidhuber) Date: Thu, 12 Nov 1992 10:01:19 -0700 Subject: new papers Message-ID: <199211121701.AA16099@thalamus.cs.Colorado.EDU> The following papers are now available: ------------------------------------------------------------------- DISCOVERING PREDICTABLE CLASSIFICATIONS (Technical Report CU-CS-626-92) .. Jurgen Schmidhuber, University of Colorado .. .. Daniel Prelinger, Technische Universitat Munchen ABSTRACT: Prediction problems are among the most common learning problems for neural networks (e.g. in the context of time series prediction, control, etc.). With many such problems, however, perfect prediction is inherently impossible. For such cases we present novel unsupervised systems that learn to classify patterns such that the classifications are predictable while still being as specific as possible. The approach can be related to the IMAX method of Hinton, Becker and Zemel (1989, 1991). Experiments include Becker's and Hinton's stereo task, which can be solved more readily by our system. ------------------------------------------------------------------- PLANNING SIMPLE TRAJECTORIES USING NEURAL SUBGOAL GENERATORS (for SAB92) .. Jurgen Schmidhuber, University of Colorado .. .. Reiner Wahnsiedler, Technische Universitat Munchen ABSTRACT: We consider the problem of reaching a given goal state from a given start state by letting an `animat' produce a sequence of actions in an environment with multiple obstacles. Simple trajectory planning tasks are solved with the help of `neural' gradient-based algorithms for learning without a teacher to generate sequences of appropriate subgoals in response to novel start/goal combinations. ------------------------------------------------------------------- STEPS TOWARDS `SELF-REFERENTIAL' LEARNING: A THOUGHT EXPERIMENT (Technical Report CU-CS-627-91) .. Jurgen Schmidhuber, University of Colorado ABSTRACT: A major difference between human learning and machine learning is that humans can reflect about their own learning behavior and adapt it to typical learning tasks in a given environment. To make some initial theoretical steps toward `intro- spective' machine learning, I present - as a thought experiment - a `self-referential' recurrent neural network which can run and actively modify its own weight change algorithm. Due to the generality of the architecture, there are no theoretical limits to the sophistication of the modified weight change algorithms running on the network (except for unavoidable pre-wired time and storage constraints). In theory, the network's weight matrix can learn not only to change itself, but it can also learn the way it changes itself, and the way it changes the way it changes itself --- and so on ad infinitum. No endless recursion is involved, however. For one variant of the architecture, I present a simple but general initial reinforcement learning algorithm. For another variant, I derive a more complex exact gradient-based algorithm for supervised sequence learning. A disadvantage of the latter algorithm is its computational complexity per time step which is independent of sequence length and equals O(n_conn^2 log n_conn), where n_conn is the number of connections. Another disadvantage is the high number of local minima of the unusually complex error surface. The purpose of my thought experiment, however, is not to come up with the most efficient or most practical `introspective' or `self-referential' weight change algorithm, but to show that such algorithms are possible at all. --------------------------------------------------------------------- To obtain copies, do: unix> ftp archive.cis.ohio-state.edu Name: anonymous Password: (your email address) ftp> binary ftp> cd pub/neuroprose ftp> get schmidhuber.predclass.ps.Z ftp> get schmidhuber.subgoals.ps.Z ftp> get schmidhuber.selfref.ps.Z ftp> bye unix> uncompress schmidhuber.predclass.ps.Z unix> uncompress schmidhuber.subgoals.ps.Z unix> uncompress schmidhuber.selfref.ps.Z unix> lpr schmidhuber.predclass.ps unix> lpr schmidhuber.subgoals.ps unix> lpr schmidhuber.selfref.ps --------------------------------------------------------------------- Sorry, no hardcopies. (Except maybe in very special urgent cases). .. Jurgen Address until December 17, 1992: .. Jurgen Schmidhuber Department of Computer Science University of Colorado Campus Box 430 Boulder, CO 80309, USA email: yirgan at cs.colorado.edu Address after December 17, 1992: .. Jurgen Schmidhuber .. Institut fur Informatik .. .. Technische Universitat Munchen .. Arcisstr. 21, 8000 Munchen 2, Germany email: schmidhu at informatik.tu-muenchen.de From lba at sara.inesc.pt Fri Nov 13 07:28:43 1992 From: lba at sara.inesc.pt (Luis B. Almeida) Date: Fri, 13 Nov 92 11:28:43 -0100 Subject: Generalization ability of a BPTT-net In-Reply-To: Walter Weber's message of Wed, 11 Nov 1992 10:51:41 +0100 <9211110951.AA17888@forwiss.tu-muenchen.de> Message-ID: <9211131228.AA23127@sara.inesc.pt> [To the "Connectionists" moderator: I am sending the following response to Walter Weber directly, I don't know if you would consider it appropriate to also publish it in the Connectionists]. I would like to make two suggestions, concerning the first problem: a) Teacher forcing, though often very useful, does not necessarily perform descent (and therefore minimization) on the objective function. Why not use the weights obtained with teacher forcing as an initialization for a second training stage, which would use normal BPTT without teacher forcing? b) Using a sigmoid on the output unit means that, in order to produce peaks (values close to 0 or to 1), the sum at the input of that unit must become relatively large, in absolute value. The net might perform better if you remove the sigmoid from the output unit, which will then become linear. I didn't fully understand your second problem. What are the inputs to the net, in the [.6, .8] case, and in the [.4, .6] case? are they the same? Are they similar in some way? Luis B. Almeida INESC Phone: +351-1-544607 Apartado 10105 Fax: +351-1-525843 P-1017 Lisboa Codex Portugal lba at inesc.pt lba at inesc.uucp (if you have access to uucp) From mav at cs.uq.oz.au Wed Nov 11 21:38:59 1992 From: mav at cs.uq.oz.au (Simon Dennis) Date: Thu, 12 Nov 92 12:38:59 +1000 Subject: TR: What does the human memory environment look like? Message-ID: <9211120239.AA07755@uqcspe.cs.uq.oz.au> The following technical report is available for anonymous ftp. What does the environment look like? Setting the scene for interactive models of human memory Simon Dennis Department of Computer Science University of Queensland Janet Wiles Departments of Computer Science and Psychology University of Queensland Michael Humphreys Department of Psychology University of Queensland Abstract We set the scene for a class of interactive models of human memory in which performance is dependent both on the structure of the environment and the structure of the model's mechanism. Such a system is capable of learning representations, control processes and decision criterion in order to successfully interact with its environment. That the (observable) environment is responsible for performance in interactive models allows the elimination of assumptions which are embedded in the mechanism of current models. Interactive models also offer the possibility of addressing the development of the mechanisms of memory which are currently poorly understood. Analyses of the relevant environments of four touchstone phenomena: the list strength effect in recognition; the crossed associates paradigm; the ABABr paradigm and the word frequency effect in recognition were performed to establish the context in which interactive accounts of these phenomena must be set. The rational model of human memory (Anderson & Milson, 1989) is advocated as a model of environmental contingencies and hence of interest to the interactive modelers. It is found to be consistent with empirical environmental data in the case of the list strength effect and, with some modification, in the case of the word frequency effect in recognition. While it proved difficult to analyze the relevant environment in the cued recall paradigms, the rational model was found to be consistent with experimental evidence. The issues involved in the process of environmental analysis were explored. Conclusions Our major purpose has been to set the scene for interactive models of human memory and we have done this in three ways. Firstly, we addressed the philosophical issue of how a representation attains its meaning. We argued that models which have as their basis the physical symbol system hypothesis, will encounter difficulties with the symbol grounding problem, and will find it difficult to give an account of meaning attainment. Interactive models provided a way of avoiding the problem. Secondly, we set the psychological modeling scene by outlining the advantages of interactive models as models of human memory. Not only do interactive models provide a much needed link to the developmental literature, but they also allow mechanistic accounts to shed some of their assumptions onto the observable environment. Thirdly, we have started the task of analyzing the environment. The environmental analyses which have been conducted are encouraging especially for the recognition paradigms. An environmental analysis of the effect of repetition suggests that not only is the main effect of repetition on performance accuracy accounted for by a simple interactivist account, but the lack of a list strength effect in recognition is also a natural consequence of the environmental contingencies. The word frequency effect in recognition was also found to mirror environmental statistics while the joint information paradigms proved difficult to analyze. Rational analysis was successful with only minor modification in all cases examined. Given these results there is reason to suppose that the environmental approach has a valuable contribution to make to the understanding of human memory. In conclusion then we would like to draw out two important implications. Firstly, we reiterate \citeA{Anderson90}, in suggesting that memory researchers should divert some of their effort to the empirical study of the environment. Secondly, we propose that interactive models of memory will be in the best position to take advantage of such research. Ftp instructions: To retrieve the technical report ftp to exstream.cs.uq.oz.au, cd pub/TECHREPORTS/department, change to binary mode and get TR0249.ps.Z. Example: $ ftp exstream.cs.uq.oz.au Connected to exstream.cs.uq.oz.au. 220 exstream FTP server (Version 6.12 Fri May 8 16:33:17 EST 1992) ready. Name (exstream.cs.uq.oz.au:mav): anonymous 331 Guest login ok, send e-mail address as password. Password: 230- Welcome to ftp.cs.uq.oz.au 230-This is the University of Queensland Computer Science Anonymous FTP server. 230-For people outside of the department, please restrict your usage to outside 230-of the hours 8am to 6pm. 230- 230-The local time is Thu Nov 12 11:18:01 1992 230- 230 Guest login ok, access restrictions apply. ftp> cd pub/TECHREPORTS/department 250 CWD command successful. ftp> bin 200 Type set to I. ftp> get TR0249.ps.Z 200 PORT command successful. 150 Opening BINARY mode data connection for TR0249.ps.Z (174928 bytes). 226 Transfer complete. local: TR0249.ps.Z remote: TR0249.ps.Z 174928 bytes received in 1.4 seconds (1.2e+02 Kbytes/s) ftp> quit 221 Goodbye. $ Please address requests for hard copies to: Simon Dennis Department of Computer Science University of Queensland 4072 Australia ------------------------------------------------------------------------------- Simon Dennis Address: Department of Computer Science Email: mav at cs.uq.oz.au University of Queensland QLD 4072 Australia ------------------------------------------------------------------------------- From regier at ICSI.Berkeley.EDU Fri Nov 13 21:25:35 1992 From: regier at ICSI.Berkeley.EDU (Terry Regier) Date: Fri, 13 Nov 92 18:25:35 PST Subject: TR available Message-ID: <9211140225.AA19513@icsib4.ICSI.Berkeley.EDU> The technical report version of my dissertation is now available by ftp. Ftp instructions follow the abstract. The Acquisition of Lexical Semantics for Spatial Terms: A Connectionist Model of Perceptual Categorization Terry Regier UC Berkeley ICSI technical report TR-92-062 This thesis describes a connectionist model which learns to perceive spatial events and relations in simple movies of 2-dimensional objects, so as to name the events and relations as a speaker of a particular natural language would. Thus, the model learns perceptually grounded semantics for natural language spatial terms. Natural languages differ -- sometimes dramatically -- in the ways in which they structure space. The aim here has been to have the model be able to perform this learning task for words from any natural language, and to have learning take place in the absence of explicit negative evidence, in order to rule out ad hoc solutions and to approximate the conditions under which children learn. The central focus of this thesis is a connectionist system which has succeeded in learning spatial terms from a number of different languages. The design and construction of this system have resulted in several technical contributions. The first is a very simple but effective means of learning without explicit negative evidence. This thesis also presents the notion of partially-structured connectionism, a marriage of structured and unstructured network design techniques capturing the best of each paradigm. Finally, the thesis introduces the idea of learning within highly specialized structural devices. Scientifically, the primary result of the work described here is a computational model of the acquisition of visually grounded semantics. This model successfully learns words for spatial events and relations from a range of languages with widely differing spatial systems, including English, Mixtec (a Mexican Indian language), German, Bengali, and Russian. And perhaps most importantly, the model does more than just recapitulate the data; it also generates a number of falsifiable linguistic predictions regarding the sorts of semantic features, and combinations of features, one might expect to find in lexemes for spatial events and relations in the world's natural languages. To ftp: % ftp icsi-ftp.berkeley.edu Name: anonymous Password: [your e-mail address] ftp> binary ftp> cd pub/techreports ftp> get tr-92-062.ps.Z ftp> quit % uncompress tr-92-062.ps.Z % lpr -P[your-postscript-printer] tr-92-062.ps -- ---------------------------------------------------------------------- Terry Regier Computer Science, UC Berkeley regier at icsi.Berkeley.EDU International Computer Science Institute From SANTINI at INGFI1.CINECA.IT Fri Nov 13 11:50:00 1992 From: SANTINI at INGFI1.CINECA.IT (SANTINI@INGFI1.CINECA.IT) Date: 13 Nov 1992 16:50 +0000 (N) Subject: Two reports available Message-ID: <4812@INGFI1.CINECA.IT> The following two technical reports have been posted in the directory /neural/papers of the ftp server aguirre.ingfi1.cineca.it (150.217.11.13): AN ALGORITHM FOR TRAINING NEURAL NETWORKS WITH ARBITRARY FEEDBACK STRUCTURE S. Santini(*) A. Del Bimbo(*) R. Jain(+) (*) Dipartimento di sistemi e Informatica, Universita' di Firenze, Firenze, Italy (+) Artificial Intelligence Lab., The University of Michigan, Ann Arbor, MI Abstract In this report, we consider multi-layer discrete-time dynamic networks with multiple unit-delay feedback paths between layers. A block notation is introduced for this class of networks which allows a great flexibility in the description of the network architecture and permits a unified treatment of static and dynamic networks. Networks are defined by recursively arranging blocks. Network blocks which satisfy certain *trainability* conditions can be embedded into other blocks through a set of *elementary connections*, so that the overall network still satisfies the same trainability conditions. The problem of training such a network is thus reduced to the definition of an algorithm ensuring the trainability of a block, assuming that all the embedded blocks are trainable. An algorithm is presented which is a block-matrix version of Forward Propagation, and is based on Werbos' ordered derivatives. This report is in the file: santini.feedback_NN_algorithm.ps.Z ----------------------------------------------------------------------- SYSTEMS IN WHICH NOBODY KNOWS THE BIG PICTURE S. Santini A. Del Bimbo Dipartimento di sistemi e Informatica, Universita' di Firenze, Firenze, Italy Abstract There is an understatement in engineering activities: if you have a problem, create a hierarchy. Hierarchical and centralized schemes are considered just ``the way things ought to be done''. In this paper, we briefly introduce the connectionist point of view, in which problems are solved by emergent computation, arising from local interaction of units, without any centralized control. In these systems, there is no knowledge of the overall problem, and in no place there is enough intelligence to ``understand'' the problem. Yet, this kind of (dis)organization can actually solve problems and -- more important -- its properties can be mathematically analyzed. We present an example: a Neural Gas network that finds the minimum path between two points in an area were obstacles are present. We show that this global problem can be solved without any global organization. Moreover, we proof global properties of the emergent computation. This is in the file: santini.big_picture.ps.Z ------------------------------------------------------------------------ HOW TO GET THE FILES: --------------------- % ftp aguirre.ingfi1.cineca.it (or 150.217.11.13) Connected to aguirre.ingfi1.cineca.it 220 aguirre FTP server (SunOS 4.1) ready. Name: anonymous 331 Guest login ok, send ident as password. Password: 230 Guest login ok, access restrictions apply. ftp> cd neural/papers ftp binary ftp> get santini.feedback_NN_algorithm.ps.Z ftp> get santini.feedback_NN_algorithm.ps.Z ftp> quit % uncompress santini.feedback_NN_algorithm.ps.Z % lpr santini.feedback_NN_algorithm.ps % uncompress santini.feedback_NN_algorithm.ps.Z % lpr santini.feedback_NN_algorithm.ps For any problem in retrieving files, or for any discussion and comment about their content, contact me at: santini at ingfi1.cineca.it Simone Santini From koza at CS.Stanford.EDU Sun Nov 15 19:43:15 1992 From: koza at CS.Stanford.EDU (John Koza) Date: Sun, 15 Nov 92 16:43:15 PST Subject: New Book and Videotape on Genetic Programming Message-ID: BOOK AND VIDEOTAPE ON GENETIC PROGRAMMING A new book and a one-hour videotape (in VHS NTSC, PAL, and SECAM formats) on genetic programming are now available from the MIT Press. NEW BOOK... GENETIC PROGRAMMING: ON THE PROGRAMMING OF COMPUTERS BY MEANS OF NATURAL SELECTION by John R. Koza, Stanford University The recently developed genetic programming paradigm provides a way to genetically breed a computer program to solve a wide variety of problems. Genetic programming starts with a population of randomly created computer programs and iteratively applies the Darwinian reproduction operation and the genetic crossover (sexual recombination) operation in order to breed better individual programs. The book describes and illustrates genetic programming with 81 examples from various fields. 840 pages. 270 Illustrations. ISBN 0-262-11170-5. Contents... 1 Introduction and Overview 2 Pervasiveness of the Problem of Program Induction 3 Introduction to Genetic Algorithms 4 The Representation Problem for Genetic Algorithms 5 Overview of Genetic Programming 6 Detailed Description of Genetic Programming 7 Four Introductory Examples of Genetic Programming 8 Amount of Processing Required to Solve a Problem 9 Nonrandomness of Genetic Programming 10 Symbolic Regression - Error-Driven Evolution 11 Control - Cost-Driven Evolution 12 Evolution of Emergent Behavior 13 Evolution of Subsumption 14 Entropy-Driven Evolution 15 Evolution of Strategy 16 Co-Evolution 17 Evolution of Classification 18 Iteration, Recursion, and Setting 19 Evolution of Constrained Syntactic Structures 20 Evolution of Building Blocks 21 Evolution of Hierarchies of Building Blocks 22 Parallelization of Genetic Programming 23 Ruggedness of Genetic Programming 24 Extraneous Variables and Functions 25 Operational Issues 26 Review of Genetic Programming 27 Comparison with Other Paradigms 28 Spontaneous Emergence of Self-Replicating and Self-Improving Computer Programs 29 Conclusions Appendices contain simple software in Common LISP for implementing experiments in genetic programming. ONE-HOUR VIDEOTAPE... GENETIC PROGRAMMING: THE MOVIE by John R. Koza and James P. Rice, Stanford University The one-hour videotape (in VHS NTSC, PAL, and SECAM formats) provides a general introduction to genetic programming and a visualization of actual computer runs for 22 of the problems discussed in the book GENETIC PROGRAMMING: ON THE PROGRAMMING OF COMPUTER BY MEANS OF NATURAL SELECTION. The problems include symbolic regression, the intertwined spirals, the artificial ant, the truck backer upper, broom balancing, wall following, box moving, the discrete pursuer-evader game, the differential pursuer- evader game, inverse kinematics for controlling a robot arm, emergent collecting behavior, emergent central place foraging, the integer randomizer, the one-dimensional cellular automaton randomizer, the two-dimensional cellular automaton randomizer, task prioritization (Pac Man), programmatic image compression, solving numeric equations for a numeric root, optimization of lizard foraging, Boolean function learning for the 11-multiplexer, co- evolution of game-playing strategies, and hierarchical automatic function definition as applied to learning the Boolean even-11- parity function. ---------------------------ORDER FORM---------------------- PHONE: 800-326-4471 TOLL-FREE or 617-625-8569 MAIL: The MIT Press, 55 Hayward Street, Cambridge, MA 02142 FAX: 617-625-9080 Please send ____ copies of the book GENETIC PROGRAMMING: ON THE PROGRAMMING OF COMPUTERS BY MEANS OF NATURAL SELECTION by John R. Koza (KOZGII) (ISBN 0-262-11170-5) @ $55.00. ____ copies of the one-hour videotape GENETIC PROGRAMMING: THE MOVIE by John R. Koza and James P. Rice in VHS NTSC format (KOZGVV) (ISBN 0-262-61084-1) @$34.95 ____ copies of the videotape in PAL format (KOZGPV) (ISBN 0-262- 61087-6) @$44.95 ____ copies of the videotape in SECAM format (KOZGSV) (ISBN 0- 262-61088-4) @44.95. Name __________________________________ Address_________________________________ City____________________________________ State_________________Zip________________ Country_________________________________ Phone Number ___________________________ $ _______ Total $ _______ Shipping and Handling ($3 per item. Outside U.S. and Canada, add $6 per item for surface rate or $22 per item for airmail) $ _______ Canada - Add 7% GST $ _______ Total due MIT Press __ Payment attached (check payable to The MIT Press in U.S. funds) __ Please charge to my VISA or MASTERCARD credit card Number ________________________________ Credit Card Expires _________________________________ Signature ________________________________ From eichler at pi18.arc.umn.edu Fri Nov 13 15:06:44 1992 From: eichler at pi18.arc.umn.edu (Rogene Eichler) Date: Fri, 13 Nov 92 14:06:44 CST Subject: Neural computing ideas and biological terms Message-ID: <9211132006.AA10502@pi18.arc.umn.edu> > Technically, I agree that a rather large schism exists between the > unknowns of neurophysiology and the creative endeavors of neural > computing. However, I disagree with the contention that "leaning > on the brain for moral support" is necessarily bad. I have been applying > neural computational models to analyze clinical data, and have found it > at times difficult to communicate the significance of these models to my > frequently non-mathematically oriented colleagues. Often, I have resorted > to explanations that are couched in biological terms rather than mathematical > ones, with the opinion that it is better to communicate something rather > than nothing at all. > > Craig Niederberger I would have to agree with Craig that it is necessary to frame one's model in biological terms when speaking to biologists. (Most cringe at the sight of a mathematical equation.) It is difficult to keep an audience's attention if you are speaking in a foreign tongue. It is your gain and their loss, to speak both languages. However, I grow tired of defending the validity of models to biologists who do not seem satisfied with any model that does not capture every last nuiance of complexity or that does not explain every last experimental finding. Modeling the brain will provide valuable insights into how we process information and how we can exploit those rules for artificial systems. But they do not need to duplicate every last brain dynamic to be useful or valid. And when modelers continue to make the claim 'just like the brain' for the sake of convincing you of the validity of the model, they are reinforcing the claim that the brain is the ONLY measure. It seems time for network modelers to develop a new set of measures, or perhaps some confidence in the significance of stand-alone models. That is my interpretation of what "leaning on the brain for moral support" is really about. - Rogene From N.E.Sharkey at dcs.ex.ac.uk Mon Nov 16 06:36:21 1992 From: N.E.Sharkey at dcs.ex.ac.uk (Noel Sharkey) Date: Mon, 16 Nov 92 11:36:21 GMT Subject: 4 research posts Message-ID: <1630.9211161136@propus.dcs.exeter.ac.uk> Research Posts at Centre for Connection Science Department of Computer Science University of Exeter UK Four new research posts will be available (expected start January 1st, 1993) at the Centre for Connection Science, Department of Computer Science, as part of a 3-year research project funded by the SERC/DTI and led by Noel Sharkey and Derek Partridge. The project will investigate the reliability of software systems implemented as neural nets using the multiversion programming strategy. Two of the posts will at the post-doctoral level (grade 1A). The ideal applicant will be proficient in both neural computing and software engineering (although training in one or the other may be given). In addition, there is a requirement for at least one of the successful applicants to work on the formal analysis of network implementation as a paradigm for reliable software. The other two posts will be for Research Assistants/Experimental Officers at grade 1b. One of these will be required to have a high level of proficiency in C programming and general computing skills. The other will be part-time, and preference will be given to an applicant with good mathematical and engineering skills (particulary control systems). For more information please contact Lyn Shackelton by email (lyn at dcs.ex.ac.uk or by telephone (0392-264066 mornings 10.00-2.00). From sofge at ai.mit.edu Mon Nov 16 19:25:25 1992 From: sofge at ai.mit.edu (Donald Sofge) Date: Mon, 16 Nov 92 19:25:25 EST Subject: Book Announcement Message-ID: <9211170025.AA02693@rice-chex> <<<<<---------------------- New Book Announcement ---------------------->>>> HANDBOOK OF INTELLIGENT CONTROL: Neural, Fuzzy, and Adaptive Approaches Edited by David A. White and Donald A. Sofge Handbook of Intelligent Control provides a hands-on approach to integrating intelligent control approaches with existing control architectures for the nonlinear control of complex multivariate systems. It is an attempt by leading industry and academic researchers to present the current state-of-the-art developments in intelligent control in a highly readable, often tutorial format such that many of the techniques described within may readily implemented by the reader. The goals of this approach are: % To provide firm mathematical and theoretical foundations for current intelligent control methods % To demonstrate how these methods may be effectively combined with existing control practices % To provide overviews and extensive bibliographic references to much current work in this field % To provide examples of real-world applications in robotics, aerospace, chemical engineering, and manufacturing which have served as a driving force behind new innovations in intelligent control. Discussions concerning these applications are provided for two main reasons: to demonstrate how existing intelligent control techniques may be applied to real-world problems, and to provide challenging real-world problems which serve as impetus for new innovations in intelligent control and for which intelligent control solutions may provide the only solutions. This book is an outgrowth of three major workshops held under the National Science Foundation Intelligent Control Initiative. The chapters included in this volume are implementations and extensions of creative new ideas discussed at these workshops. The application and integration of neural, fuzzy, and adaptive methods into "real-world" engineering problems makes this an ideal book for practicing engineers, as well as graduate and academic researchers. This volume contains the following chapters: Foreword - Paul Werbos, Elbert Marsh, Kishan Baheti, Maria Burka, and Howard Moraff Editors' Preface Donald Sofge and David White Introduction to Intelligent Control 1 Intelligent Control: An Overview and Evaluation Karl strom and Thomas McAvoy 2 An Introduction to Connectionist Learning Control Systems Walter Baker and Jay Farrell 3 Neurocontrol and Supervised Learning: An Overview and Evaluation Paul Werbos Conventional Control and Intelligent Approaches 4 Fuzzy Logic in Control Engineering Reza Langari, Hamid Berenji, and Lotfi Zadeh 5 Adaptive Control of Dynamical Systems Using Neural Networks K.S. Narendra 6 Optimal Control: A Foundation for Intelligent Control David White and Michael Jordan 7 Development and Application of CMAC Neural Network-based Control Gordan Kraft, Tom Miller, and D. Dietz Applications of Intelligent Control 8 Artificial Neural Networks in Manufacturing and Process Control Judy Franklin and David White 9 Applied Learning-Optimal Control for Manufacturing Donald Sofge and David White 10 Neural Networks, System Identification, and Control in the Chemical Process Industries Paul Werbos, Thomas McAvoy, and Ted Su 11 Flight, Propulsion, and Thermal Control of Advanced Aircraft and Hypersonic Vehicles David White, Albion Bowers, Ken Iliff, Greg Noffz, Mark Gonda, and John Menousek Advances in System Identification, Optimization and Learning Control Theory 12 Reinforcement Learning and Adaptive Critic Methods Andrew Barto 13 Approximate Dynamic Programming for Real-time Control and Neural Modeling Paul Werbos 14 The Role of Exploration in Learning Control Sebastian Thrun Publication Date: September 1992 568 pages Price: $59.95 Available from: Van Nostrand Reinhold VNR Order Processing P.O. Box 668 Florence, Kentucky 41022-0668 Call Toll-free: 1 (800) 926-2665 From N.E.Sharkey at dcs.ex.ac.uk Mon Nov 16 08:19:32 1992 From: N.E.Sharkey at dcs.ex.ac.uk (Noel Sharkey) Date: Mon, 16 Nov 92 13:19:32 GMT Subject: TR available Message-ID: <1692.9211161319@propus.dcs.exeter.ac.uk> TECH REPORT AVAILABLE: Computer Science TR R257 ADAPTIVE GENERALISATION AND THE TRANSFER OF KNOWLEDGE Noel E. Sharkey and Amanda J.C. Sharkey Center for Connection Science, University of Exeter Adaptive generalisation is the ability to use prior knowledge in the performance of novel tasks. Thus, if we are to model intelligent behaviour with neural nets, they must be able to generalise across task domains. Our objective is to elucidate the aetiology of transfer of information between connectionist nets. First, a method is described that provides a standardised score for the quantification of how much task structure a net has extracted, and to what degree knowledge has been transferred between tasks. This method is then applied in three simulation studies to examine Input-to-Hidden (IH) and Hidden-to-Output (HO) decision hyperplanes as determinants of transfer effects. In the first study, positive transfer is demonstrated between functions that require the vertices of their input spaces to be divided similarly, and negative transfer between functions that require decision regions of different shapes. In the other two studies, input and output similarity are varied independently in a series of paired associate learning tasks. Further explanation of transfer effects is provided through the use of a new technique that permits better visualisation of the entire computational space by showing both the relative position of inputs in Hidden Unit space, and the HO decision regions implemented by a set of weights. * This research was supported by an award from the Economic and Social Research Council, Grant No R000233441. An earlier version of this paper appears in the Proceedings of the Second Irish Neural Networks Conference, Belfast 1992. The current version will appear in an AI review special issue on Connectionism. For a postscript version, email: ptsec at uk.ac.exeter.dcs Some hardmail copies are available from the same source. Our hardmail address Mrs June Stevens, Dept. Computer Science University of Exeter Exeter EX4 4PT Devon U.K. From kolen-j at cis.ohio-state.edu Wed Nov 18 08:27:32 1992 From: kolen-j at cis.ohio-state.edu (john kolen) Date: Wed, 18 Nov 92 08:27:32 -0500 Subject: Neural computing ideas and biological terms In-Reply-To: Rogene Eichler's message of Fri, 13 Nov 92 14:06:44 CST <9211132006.AA10502@pi18.arc.umn.edu> Message-ID: <9211181327.AA07750@pons.cis.ohio-state.edu> Rogene Eichler writes However, I grow tired of defending the validity of models to biologists who do not seem satisfied with any model that does not capture every last nuiance of complexity or that does not explain every last experimental finding. Modeling the brain will provide valuable insights into how we process information and how we can exploit those rules for artificial systems. But they do not need to duplicate every last brain dynamic to be useful or valid. This is especially true if it is NOT the case that the details of brain function are the roots of brain behavior. These minute details may be washed out by dynamical principles which have their own behavioral "chemistry". For a mathematical example, look at the universality of symbol dynamics in unimodal iterated mapping (that's just a single bump, like the logistic function). As long as the mappings meet some fairly general qualifications, the iterated systems based on those mappings share the qualitative behavior, namely the bifurcation structure, regardless of the quantitative differences between the individual mappings. John Kolen From Robert.Kentridge at durham.ac.uk Wed Nov 18 12:15:13 1992 From: Robert.Kentridge at durham.ac.uk (Robert.Kentridge@durham.ac.uk) Date: Wed, 18 Nov 92 17:15:13 GMT Subject: Neural computing ideas and biological terms In-Reply-To: <9211132006.AA10502@pi18.arc.umn.edu>; from "Rogene Eichler" at Nov 13, 92 2:06 pm Message-ID: <27835.9211181715@deneb.dur.ac.uk> Rogene Eichler writes: > However, I grow tired of defending the validity of models to biologists > who do not seem satisfied with any model that does not capture every last > nuiance of complexity or that does not explain every last experimental > finding. Modeling the brain will provide valuable insights into how we > process information and how we can exploit those rules for artificial > systems. But they do not need to duplicate every last brain dynamic to > be useful or valid. And when modelers continue to make the claim 'just > like the brain' for the sake of convincing you of the validity of the > model, they are reinforcing the claim that the brain is the ONLY measure. I think a distinction can be drawn here between models which are simplifications of known biology but which don't include biological impossibilities and models which are simple and biologically impossible. Of course, this distinction might be a little in the eye of the beholder, for example I'd argue that, in a network, single compartment neurons are acceptable simplifications which still allow one to draw some conclusions about information processing in biological neural networks, but I know people who would disagree. On the other hand its pretty hard to argue that back-prop is any kind of simplification of biology. Of course, this assumes that your interest is in finding out about biology. If you just want to use networks in their own right then fine (but be wary of leaning on biology too much to justifiy their design!) cheers, bob -- Dr. R.W. Kentridge phone: +44 91 374 2621 Psychology Dept., email: robert.kentridge at durham.ac.uk University of Durham, Durham DH1 3LE, U.K. From sam at vaxserv.sarnoff.com Thu Nov 19 11:24:12 1992 From: sam at vaxserv.sarnoff.com (Scott A. Markel x2683) Date: Thu, 19 Nov 92 11:24:12 EST Subject: NIPS 92 Workshop on Training Issues Message-ID: <9211191624.AA16897@sarnoff.sarnoff.com> **************************** NIPS 92 Workshop **************************** "Computational Issues in Neural Network Training" or Why is Back-Propagation Still So Popular? ******************************************************************************* Roger Crane and I are are leading a NIPS '92 workshop on "Computational Issues in Neural Network Training". Our workshop will be on Saturday, 5 December, the second of two days of workshops in Vail. The discussion will focus on optimization techniques currently used by neural net researchers, and include some other techniques that are available. Back- propagation is still the optimization technique of choice even though there are obvious problems in training with BP: speed, convergence, ... . Several innovative algorithms have been proposed by the neural net community to improve upon BP, e.g., Scott Fahlman's QuickProp. We feel that there are classical optimization techniques that are superior to back-propagation. In fact, gradient descent (BP) fell out of favor with the mathematical optimization folks way backin the 60's! So why is BP still so popular? Topics along these lines include: * Why are classical methods generally ignored? * Computational speed * Convergence criteria (or lack thereof!) Broader issues to be discussed include: * Local minima * Selection of starting points * Conditioning (for higher order methods) * Characterization of the error surface If you would like to present something on any of these or similar topics, please contact me by e-mail and we can discuss details. Workshops are scheduled for a total of four hours. We're allowing for approxi- mately 8 presentations of 10-20 minutes each, since we want to make sure that ample time is reserved for discussion and informal presentations. We will encourage (incite) lively audience participation. By the way, none of the NIPS workshops are limited to presenters only. People who want to show up and just listen are more than welcome. Scott Markel Computational Science Research smarkel at sarnoff.com David Sarnoff Research Center Tel. 609-734-2683 CN 5300 FAX 609-734-2662 Princeton, NJ 08543-5300 From N.E.Sharkey at dcs.ex.ac.uk Fri Nov 20 08:56:14 1992 From: N.E.Sharkey at dcs.ex.ac.uk (Noel Sharkey) Date: Fri, 20 Nov 92 13:56:14 GMT Subject: workshop in UK. Message-ID: <2308.9211201356@propus.dcs.exeter.ac.uk> ******************* CALL FOR DISCUSSION ABSTRACTS ************************* WORKSHOP ON CONNECTIONISM, COGNITION AND A NEW AI A workshop at the 9th Biennial Conference on Artificial Intelligence (AISB-93) at the University of Birmingham, England, during 29th March - 2nd April 1993, organised by the Society for the Study of Artificial Intelligence and Simulation of Behaviour (SSAISB). A number of recent developments in Connectionist Research have strong implications for the future of AI and the study of Cognition. Among the most important are developments in Learning, Representation, and Productivity (or Generalisation). The aim of the workshop would be to focus on how these developments may change the way we look at AI and the study of Cognition. Our goal is to have a lively and invigorating debate on the state-of-the-art. SUGGESTED TOPICS INCLUDE (BUT ARE NOT RESTRICTED TO). * Connectionist representation * Generalisation and Transfer of Knowledge * Learning Machines and models of human deveopmental. * Symbolic Learning versus Connectionist learning * Advantages of Connectionist/Symbolic hybrids. * Modelling Cognitive Neuropsychology * Connectionist modelling of Creativity and music (or other arts). DEADLINE FOR SUBMISSION: 15th December, 1992 ORGANISER Noel Sharkey Centre for Connection Science, Computer Science, Exeter. COMMITTEE Andy Clark (Sussex). Glyn Humphries (Birmingham) Kim Plunkett (Oxford) Chris Thornton (Sussex) WORKSHOP ENTRANCE: Attendance at the workshop will be limited to 50 or 60 places, so please LET US KNOW AS SOON AS POSSIBLE IF YOU ARE PLANNING TO ATTEND, and to which of the following categories you belong. DISCUSSION PAPERS Acceptance of discussion papers will be decided on the basis of extended abstracts (try to keep them under 500 words please) clearly specifying a 15 to 20 minute discussion topic for oral presentation. There will also be a small number of invited contributors. ORDINARY PARTICIPANTS A limited number places will be available for participants who wish to sit in on the discussion but do not wish to present a paper. But please get in early with a short note saying what is your purpose in attending. Please send submissions to Noel E. Sharkey, Centre for Connection Science Dept. Computer Science University of Exeter Exeter EX4 4PT Devon U.K. or email noel at uk.ac.exeter.dcs From ingber at alumni.cco.caltech.edu Sun Nov 22 23:33:27 1992 From: ingber at alumni.cco.caltech.edu (Lester Ingber) Date: Sun, 22 Nov 1992 20:33:27 -0800 Subject: Very Fast Simulated Reannealing code available for beta testing Message-ID: <9211230433.AA16677@alumni.cco.caltech.edu> VERY FAST SIMULATED REANNEALING (VFSR) (C) Lester Ingber ingber at alumni.caltech.edu and Bruce Rosen rosen at ringer.cs.utsa.edu 1. License and Availability 1.1. GNU Copyleft License This Very Fast Simulated Reannealing (VFSR) code is being made available under a GNU COPYING-LIB "copyleft" license, and is owned jointly by Lester Ingber and Bruce Rosen[1]. Please read the copy of this license contained in this directory. 1.2. NETLIB Electronic Availability of VFSR You can obtain our code from NETLIB. This can be done interactively, or you can obtain it by electronic mail request. 1.2.1. Interactive From your local machine login to research.att.com: local% ftp research.att.com Name (research.att.com:your_login_name): netlib Password: [type in your_login_name or anything] ftp> cd opt ftp> binary ftp> get vfsr.Z ftp> quit After `uncompress vfsr.Z' read the header of vfsr for simple directions on obtaining your source files. For example, on most machines, after `sh vfsr' they will reside in a VFSR directory. 1.2.2. Electronic Mail Request Send the following one-line electronic mail request send vfsr from opt [For general NETLIB info, just use: send index] to one of the NETLIB sites: netlib at research.att.com (AT&T Bell Labs, NJ, USA) [most recent version] netlib at ornl.gov (Oak Ridge Natl Lab, TN, USA) netlib at ukc.ac.uk (U Kent, UK) netlib at nac.no (Oslo, Norway) netlib at cs.uow.edu.au (U Wollongong, NSW, Australia) 2. Background and Context VFSR was developed in 1987 to deal with the necessity of performing adaptive global optimization on multivariate nonlinear stochastic systems[2]. VFSR was recoded and applied to several complex systems, in combat analysis[3], finance[4], and neuro- science[5]. A comparison has shown VFSR to be superior to a standard genetic algorithm simulation on a suite of standard test problems[6], and VFSR has been examined in the context of a review of methods of simulated annealing[7]. A project comparing standard Boltzmann annealing with "fast" Cauchy annealing with VFSR has concluded that VFSR is a superior algorithm[8]. A paper has indicated how this technique can be enhanced by combining it with some other powerful algorithms[9]. 2.1. Efficiency Versus Necessity VFSR is not necessarily an "efficient" code. For example, if you know that your cost function to be optimized is something close to a parabola, then a simple gradient Newton search method most likely would be faster than VFSR. VFSR is believed to be faster and more robust than other simulated annealing techniques for most complex problems with multiple local optima; again, be careful to note that some problems are best treated by other algorithms. If you do not know much about the structure of your system, and especially if it has complex constraints, and you need to search for a global optimum, then we heartily recommend our VFSR code to you. 2.2. Outline of Use Set up the VFSR interface: Your program should be divided into two basic modules. (1) The user calling procedure, contain- ing the cost function to be minimized (or its negative if you require a global maximum), here is contained in user.c and user.h. (2) The VFSR optimization procedure, here is contained in vfsr.c and vfsr.h. Furthermore, there are some options to explore in the Makefile. We assume there will be no confusion over the standard uses of the term "parameter" in different con- texts, e.g., as an element passed by a subroutine or as a physi- cal coefficient in a cost function. In VFSR/TESTS we have included some user_out files from some sample runs, containing timed runs on a Sun4c/4.1.3 (SPARC-2) using compilers cc, acc and gcc-2.3.1, and on a Dec5100/Ultrix-4.2 using compilers cc and gcc-2.2.2. No attempt was made to optimize the use of any of these compilers, so that the runs do not really signify any testing of these compilers or architectures; rather they are meant to be used as a guide to determine what you might expect on your own machine. 3. Makefile This file was generated using `make doc'. The Makefile con- tains some options for formatting this file differently, includ- ing the PostScript version README.ps and the text version README. Since complex problems by their nature are often quite unique, it is unlikely that our default parameters are just right for your problem. However, our experience has shown that if you a priori do not have any reason to determine your own parameters, then you might do just fine using our defaults, and we recommend using them as a first-order guess. Most of our defaults can be changed simply by uncommenting lines in the Makefile. Remember to recompile the entire code every time you change any options. Depending on how you integrate VFSR into your own user modules, you may wish to modify this Makefile or at least use some of these options in your own compilation procedures. Read through all the options in the Makefile. As the com- ments therein suggest, it may be necessary to change some of them on some systems. Here are just a couple of examples you might consider: 3.1. SMALL_FLOAT For example, on one convex running our test problem in user.c the SMALL_FLOAT default was too small and the code crashed. A larger value was found to give reasonable results. The reason is that the fat tail of VFSR, associated with high parameter temperatures, is very important for searching the breadth of the ranges especially in the initial stages of search. However, the parameter temperatures require small values at the final stages of the search to converge to the best solution, albeit this is reached very quickly given the exponential sched- ule proven in the referenced publications to be permissible with VFSR. Note that our test problem in user.c is a particularly nasty one, with 1E20 local minima and requiring VFSR to search over many orders of magnitude of the cost function before cor- rectly finding the global minimum. In VFSR/TESTS We have included vfsr_out files comparing results using SMALL_FLOAT=1.0E-16, SMALL_FLOAT=1.0E-18 (the default), and SMALL_FLOAT=1.0E-20. Although the same final results were achieved, the intermediate calculations differ some- what. 3.2. HAVE_ANSI As another example, setting HAVE_ANSI=FALSE will permit you to use an older K&R C compiler. This option can be used if you do not have an ANSI compiler, overriding the default HAVE_ANSI=TRUE. 4. User Module We have set up this module as user.c and user.h. You may wish to combine them into one file, or you may wish to use our VFSR module as one component of a library required for a large project. 4.1. int main(int argc, char **argv) In main, set up your initializations and calling statements to vfsr. In the files user.c and user.h, we have provided a sam- ple program, as well as a sample cost function for your conve- nience. If you do not intend to pass parameters into main, then you can just declare it as main() without the argc and argv argu- ments. 4.2. void user_initialize_parameters() Before calling vfsr, the user must allocate storage and ini- tialize some of the passed parameters. A sample procedure is provided as a template. In this procedure the user should allo- cate storage for the passed arrays and define the minimum and maximum values. Below, we detail all the parameters which must be initialized. If your arrays are of size 1, still use them as arrays as described in user.c. 4.3. double user_cost_function(double *x, int *valid_flag) You can give any name to user_cost_function as long as you pass this name to vfsr. x (or whatever name you pass to vfsr) is an array of doubles representing a set of parameters to evaluate, and valid_flag (or whatever name you pass to vfsr) is the address of an integer. In user_cost_function, *valid_flag should be set to FALSE (0) if the parameters violate a set of user defined con- straints (e.g., as defined by a set of boundary conditions) or TRUE (1) if the parameters represent a valid state. If *valid_flag is FALSE, no acceptance test will be attempted, and a new set of trial parameters will be generated. The function returns a real value which VFSR will minimize. 4.4. double user_random_generator() A random number generator function must be passed next. It may be as simple as one of the UNIX random number generators (e.g. drand48), or may be user defined, but it should return a real value within [0,1) and not take any parameters. We have provided a good random number generator, randflt, and its auxil- iary routines with the code in the file user module. 4.5. void initialize_rng() Most random number generators should be "warmed-up" by call- ing a set of dummy random numbers. 4.6. void print_time(char *message) As a convenience, we have included this subroutine, and its auxiliary routine aux_print_time, to keep track of the time spent during optimization. It takes as its only parameter a string which will be printed. We have given an example in user_cost_function to illustrate how print_time may be called periodically every set number of calls by defining PRINT_FREQUENCY in user.h. 4.7. vfsr( user_cost_function, user_random_generator, number_parameters, parameter_type, parameter_initial_final, final_cost, parameter_minimum, parameter_maximum, tangents, curvature); This is the form of the call to vfsr from user.c. 4.8. void vfsr( double (*user_cost_function) (), double (*user_random_generator) (), int number_parameters, int *parameter_type, double *parameter_initial_final, double final_cost, double *parameter_minimum, double *parameter_maximum, double *tangents, double *curvature) This is how vfsr is defined in the VFSR module, contained in vfsr.c and vfsr.h. Each parameter is described below as it must be passed to this module from the user module. 4.8.1. double (*user_cost_function) () The parameter (*user_cost_function*) () is a pointer to the cost function that you defined in your user module. 4.8.2. double (*user_random_generator) () As discussed above, a pointer to the random number generator function, defined in the user module, must be passed next. 4.8.3. int number_parameters An integer containing the dimensionality of the state space is passed next. Each of the arrays that follow are to be of the size number_parameters. 4.8.4. int *parameter_type This integer array is passed next. Each element of this array (each flag) is either REAL_TYPE (0) (indicating the parame- ter is a real value) or INTEGER_TYPE (1) (indicating the parame- ter can take on only integer values). 4.8.5. double *parameter_initial_final Next, an array of doubles is passed. Initially, this array holds the set of starting parameters which should satisfy any constraints or boundary conditions. Upon return from the VFSR procedure, the array will contain the best set of parameters found by vfsr to minimize the user's cost function. Experience shows that any guesses within the acceptable ranges should suf- fice, since initially the system is at high annealing temperature and VFSR samples the breadth of the ranges. 4.8.6. double final_cost This double should be defined in the calling program. Upon return from the vfsr call, it will be the minimum cost value found by vfsr. 4.8.7. double *parameter_minimum 4.8.8. double *parameter_maximum These two arrays of doubles should also be passed. Since VFSR works only on bounded search spaces, these arrays should contain the minimum and maximum values each parameter can attain. If you aren't sure, try a factor of 10 or 100 times any reason- able values. The exponential temperature annealing schedule should quickly sharpen the search down to the most important region. 4.8.9. double *tangents 4.8.10. double *curvature These two arrays of doubles should be passed last. On return from vfsr, for real parameters, they contain the first and second derivatives of the cost function with respect to its parameters. These can be useful for determining the value of your fit. In this implementation of VFSR, the tangents are used to determine the relative reannealing among parameters. 5. Bug Reports While we do not have time to help you solve your own appli- cations, we do want VFSR to be helpful to a large community. Therefore, we welcome your bug reports and constructive critiques regarding our code. "Flames" will be rapidly quenched. References 1. L. Ingber and B. Rosen, "vfsr," Very Fast Simulated Rean- nealing (VFSR) Source Code, NETLIB Electronic Ftp Archive, netlib at research.att.com (1992). 2. L. Ingber, "Very fast simulated re-annealing," Mathl. Com- put. Modelling, 8, 12, pp. 967-973 (1989). 3. L. Ingber, H. Fujio, and M.F. Wehner, "Mathematical compari- son of combat computer models to exercise data," Mathl. Com- put. Modelling, 1, 15, pp. 65-90 (1991). 4. L. Ingber, "Statistical mechanical aids to calculating term structure models," Phys. Rev. A, 12, 42, pp. 7057-7064 (1990). 5. L. Ingber, "Statistical mechanics of neocortical interac- tions: A scaling paradigm applied to electroencephalogra- phy," Phys. Rev. A, 6, 44, pp. 4017-4060 (1991). 6. L. Ingber and B. Rosen, "Genetic algorithms and very fast simulated reannealing: A comparison," Mathl. Comput. Mod- elling, 11, 16, pp. 87-100 (1992). 7. L. Ingber, "Simulated annealing: Practice versus theory," Statistics Comput., p. (to be published) (1993). 8. B. Rosen, "Function optimization based on advanced simulated annealing," Report, University of Texas, San Antonio, TX (1992). 9. L. Ingber, "Generic mesoscopic neural networks based on sta- tistical mechanics of neocortical interactions," Phys. Rev. A, 4, 45, pp. R2183-R2186 (1992). [*] Some (p)reprints can be obtained via anonymous ftp from ftp.umiacs.umd.edu [128.8.120.23] in the pub/ingber direc- tory. | Prof. Lester Ingber ingber at alumni.caltech.edu # | P.O. Box 857 # | McLean, VA 22101 703-848-1859 = [10ATT]0-700-L-INGBER # From cateau at tkyux.phys.s.u-tokyo.ac.jp Mon Nov 23 21:26:06 1992 From: cateau at tkyux.phys.s.u-tokyo.ac.jp (Hideyuki Cateau) Date: Tue, 24 Nov 92 11:26:06 +0900 Subject: Neural computing ideas and biological terms Message-ID: <9211240226.AA04316@tkyux.phys.s.u-tokyo.ac.jp> In reply to the following discussion: >Rogene Eichler writes > However, I grow tired of defending the validity of models to biologists > who do not seem satisfied with any model that does not capture every last > nuiance of complexity or that does not explain every last experimental > finding. Modeling the brain will provide valuable insights into how we > process information and how we can exploit those rules for artificial > systems. But they do not need to duplicate every last brain dynamic to > be useful or valid. > >This is especially true if it is NOT the case that the details of brain >function are the roots of brain behavior. These minute details may be >washed out by dynamical principles which have their own behavioral >"chemistry". For a mathematical example, look at the universality of >symbol dynamics in unimodal iterated mapping (that's just a single bump, >like the logistic function). As long as the mappings meet some fairly >general qualifications, the iterated systems based on those mappings share >the qualitative behavior, namely the bifurcation structure, regardless of >the quantitative differences between the individual mappings. > >John Kolen I agree to Dr.Kolen. I would like the connectionists to pay attention to the possibility that neural network models can explain not only the qualitative aspects of our brain but also the "quantitative" one of it. In fact, I and my collaborators have found that the learning pace of the back propagation modeland the human brain are subject to a same power law with nearly equal values of the exponent. This is reported in "Power law in the human memory and in the neural network model, H.Cateau, T.Nakajima, H.Nunokawa and N.Fuchikami", which is placed in the neuroprose as a file cateau.power.tar.Z. Let us denote the time which is spent when one memorize M items by t(M). As M increases the learning pace slows down as you usually experience. A psychologist Foucault (M.Foucaut, Annee Psychol.19 (1913)218) found experimentally that this slowing down behavior is described by a following power law: t(M)= const*M^D where D is a constant. He expecially claimed that D=2. We have performed the same experiment by ourself and found that the power law is true with high statistical confidence and that D is between 1 and 2. Then we examined whether or not the back propagation(BP)network has the same property when it memorize some items. The answer was yes. The BP network is subject to the power law with a fairly nice precision. Furthermore the observed value of the exponent was two up to errors. All connectionists can easily check this interesting property by themselves and convince themselves that the fitting of the data to the above law is very good. When we make the BP memorize several items, the memories embedded in the connection weights interfere each other. Thus the slowing down of the learning is expected to occur also fo the BP. This is a qualitative expectation. But above result shows that the similarity is not only qualitative but also quantitative. I think this shows that the BP model, although it is too simple, surely simulate some essential feature of the real brain and that the studies of the neural network model cast a light on a secret of our brain. When we discuss whether or not the neural network model can explain some experimental results of the brain, we usually have, in our mind, physiological experiments. However, there are also many psychological experiments for the human brain. Many of such results are scientifically reliable because the statistical significance of the results were strictly checked. I belive that it is really meaningful as a study of the brain that we examine whether the existing neural network models can explain the many other psychological experiments. Hideyuki Cateau Particle theory group, Department of Physics,University of Tokyo,7-3-1, Hongo,Bunkyoku,113 Japan e-mail:cateau at star.phys.metro-u.ac.jp From lina at mimosa.physio.nwu.edu Mon Nov 23 17:55:19 1992 From: lina at mimosa.physio.nwu.edu (Lina Massone) Date: Mon, 23 Nov 92 16:55:19 CST Subject: paper available Message-ID: <9211232255.AA00498@mimosa.physio.nwu.edu> ************************************************* PLEASE DO NOT FORWARD TO OTHER BOARDS ************************************************ The following paper is available. A VELOCITY-BASED MODEL FOR CONTROL OF OCULAR SACCADES Lina L.E. Massone Department of Physiology Department of Electrical Eng. and Comp. Sci. Northwestern University This paper presents a computational closed-loop model of the saccadic system based on the experimental observation by Munoz, Pellisson and Guitton [1991] that the neural activity on the collicular motor map shifts, during eye movements, towards the rostral area of the superior colliculus. This assumption, together with other assumptions on how the colliculus projects to the burst cells in the brainstem and on the architecture of the network that translates the collicular signal into actual eye movements, results in a system that can: (1) perform a spatio-temporal transformation between a stimulation site on the collicular motor map and an eye movement, (2) spontaneously produce oblique saccades whose trajectories are, as dictated by experimental data, curved when the horizontal and vertical components of the motor error are unequal and straight when the horizontal and vertical components of the motor error are equal, (3) automatically maintain the eye position in the orbit at the end of a saccade by exploiting the internal dynamic of the network, (4) continuously produce efferent copies of the movements without the need for reset signals, (5) reproduce the outcome of the lidocaine experiment by Lee, Roher and Sparks [1988] without assuming a population averaging criterion to combine the activity of collicular cells. This model was developed as part of a theoretical study on the common properties of the eye and arm control systems and on the hypothetical role that the tecto- reticulo-spinal tract might play in the control of arm movements. Munoz, Pellison, Guitton [1991] Movement of neural activity on the superior colliculus motor map during gaze shifts, Science, 251, 1358-1360. Lee, Roher, Sparks [1988] Population coding of saccadic eye movements by neurons in the superior colliculus, Nature, 332, 357-360. A poster will be at NIPS. Email requests to: lina at mimosa.physio.nwu.edu From jlm at crab.psy.cmu.edu Tue Nov 24 09:38:22 1992 From: jlm at crab.psy.cmu.edu (James L. McClelland) Date: Tue, 24 Nov 92 09:38:22 EST Subject: cognition and biology Message-ID: <9211241438.AA18676@crab.psy.cmu.edu.noname> Hideyuki Cateau writes: > I believe that it is really meaningful as a study of the brain that we > examine whether the existing neural network models can explain the many > other psychological experiments. I hope by now nearly everyone agrees that it is very valuable to try to understand which robust phenomena of human cognition depend on which properties of the underlying mechanisms. Sometimes very abstract and general features that connectionist systems share with other systems are doing the work; other times it is going to turn out to be specific features not shared by a wide range of abstract models. The power law appears to be a case of the former, since it has probably been accounted for by more psychological models than any other phenomenon. There are clear cases in which listening to the brain has made a difference to our understanding at the abstract level. The phrase listening to the brain actually comes from a paper of Terry Sejnowski's in which he pointed out the stochastic character of neural activity. This observation contributed importantly to the development of the Boltzmann machine. More recently I have found that another robust regularity of psychological performance, called the independence law, arises from neural network models with bi-directional (symmetric) connections. This does not occur if the network uses the deterministic activation function of McClelland and Rumelhart's interactive activation model but it does occur if the network uses any of a wide range of stochastic activation functions, including the Boltzmann machine activation function and various continuous diffusion functions. These kinds of discoveries make it clear that abstraction is of the essence of understanding, but they also make it clear that it is important to abstract the right things. To me this argues forcefully for an interactive style of research, in which both details and abstractions matter. -- Jay McClelland From zipser at cogsci.UCSD.EDU Tue Nov 24 11:15:27 1992 From: zipser at cogsci.UCSD.EDU (David Zipser) Date: Tue, 24 Nov 1992 08:15:27 -0800 Subject: Neural computing ideas and biological terms Message-ID: <9211241618.AA29605@cogsci.UCSD.EDU> Some of you interested in realistic neural network models of the nervous system may want to look at a recent paper: Zipser, D. (1992). Identification models of the nervous system. Neuroscience, 47, 853-862. David Zipser From eichler at pi18.arc.umn.edu Tue Nov 24 12:54:34 1992 From: eichler at pi18.arc.umn.edu (Rogene Eichler) Date: Tue, 24 Nov 92 11:54:34 CST Subject: networks and biology Message-ID: <9211241754.AA06345@pi18.arc.umn.edu> > I agree to Dr.Kolen. I would like the connectionists to pay attention to > the possibility that neural network models can explain not only the > qualitative aspects of our brain but also the "quantitative" one of it. > > In fact, I and my collaborators have found that the learning pace of the > back propagation modeland the human brain are subject to a same power law > with nearly equal values of the exponent. This is reported in > "Power law in the human memory and in the neural network model, > H.Cateau, T.Nakajima, H.Nunokawa and N.Fuchikami", which is > placed in the neuroprose as a file cateau.power.tar.Z. The results of your work sound very exciting, indeed. But it is important not to get trapped in HUGE statements like ' neural network models can explain not only the qualitative aspects of our brain but also the "quantitative" one of it.' You are basing your statement on the ability of a subset of network models to explain a very small subset of the behaviors that are observable and testable by somewhat similar criteria. Furthermore, it could be argued that the criteria you are using for your comparison is qualitative in nature because of the testing methods employed to measure human performance in some cognitive tasks. Your work has shown that complex network systems can demonstrate similar emergent properties. That statement, supported by the performance measures you cited, is very powerful. But you have substituted one black box for another- nothing can be said quantitatively about how or where brain function occurs. - Rogene From rroberts at pstar.psy.du.edu Tue Nov 24 14:15:01 1992 From: rroberts at pstar.psy.du.edu (rroberts@pstar.psy.du.edu) Date: Tue Nov 24 12:15:01 MST 1992 Subject: No subject Message-ID: From floreen at cs.Helsinki.FI Wed Nov 25 08:09:29 1992 From: floreen at cs.Helsinki.FI (Patrik Floreen) Date: Wed, 25 Nov 92 15:09:29 +0200 Subject: 3 reports available Message-ID: <9211251309.AA20947@hydra.Helsinki.FI> The following 3 reports are now available: "Attraction Radii in Binary Hopfield Nets are Hard to Compute" by Patrik Floreen and Pekka Orponen, University of Helsinki. The name of the file is floreen.attrrad.ps.Z "Neural Networks and Complexity Theory" by Pekka Orponen, University of Helsinki. The name of the file is orponen.nncomp.ps.Z "On the Computational Power of Discrete Hopfield Nets" by Pekka Orponen, University of Helsinki. The name of the file is orponen.hoppow.ps.Z ------------ Abstracts of the papers: "Attraction Radii in Binary Hopfield Nets are Hard to Compute" ABSTRACT: We prove that it is an NP-hard problem to determine the attraction radius of a stable vector in a binary Hopfield memory network, and even that the attraction radius is hard to approximate. Under synchronous updating, the problems are already NP-hard for two-step attraction radii; direct (one-step) attraction radii can be computed in polynomial time. "Neural Networks and Complexity Theory" ABSTRACT: We survey some of the central results in the complexity theory of discrete neural networks, with pointers to the literature. "On the Computational Power of Discrete Hopfield Nets" ABSTRACT: We prove that polynomial size discrete synchronous Hopfield networks with hidden units compute exactly the class of Boolean functions PSPACE/poly, i.e., the same functions as are computed by polynomial space-bounded nonuniform Turing machines. As a corollary to the construction, we observe also that networks with polynomially bounded interconnection weights compute exactly the class of functions P/poly. --------------------------------------------------------------------------- To obtain copies of the postscript files, please use Jordan Pollack's service: Example: unix> ftp cheops.cis.ohio-state.edu # (or ftp 128.146.8.62) Name (cheops.cis.ohio-state.edu:): anonymous Password (cheops.cis.ohio-state.edu:anonymous): ftp> cd pub/neuroprose ftp> binary ftp> get (remote-file) floreen.attrrad.ps.Z (local-file) foo.ps.Z ftp> quit unix> uncompress foo.ps unix> lpr -P(your_local_postscript_printer) foo.ps Likewise for orponen.hoppow.ps.Z and orponen.nncomp.ps.Z ---------------------------------------------------------------------------- If you have any difficulties with the above, please send e-mail to floreen at cs.helsinki.fi. DO NOT "reply" to this message, please. From efiesler at idiap.ch Wed Nov 25 07:42:47 1992 From: efiesler at idiap.ch (E. Fiesler) Date: Wed, 25 Nov 92 13:42:47 +0100 Subject: Paper & software available on modified cascor. Message-ID: <9211251242.AA08738@idiap.ch> Paper available ---------------- The following paper has been published in Proc. Neuro-Nimes '92, Nimes, France, November 1992, pp. 455-466. This paper is available at the IDIAP ftp site. Instructions for obtaining a copy of this paper are given at the end of this message. ----------------------------------------------------------------------- Variations on the Cascade-Correlation Learning Architecture for Fast Convergence in Robot Control Natalio Simon Henk Corporaal Eugene Kerckhoffs Delft University of Technology, The Netherlands Abstract -------- Most applications of Neural Networks in Control Systems use a version of the Back-Propagation algorithm for training. Learning in these networks is generally a slow and very time consuming process. Cascade-Correlation is a supervised learning algorithm that automatically determines the size and topology of the network and is quicker than back-propagation in learning for several benchmarks. We present a modified version of the Cascade-Correlation learning algorithm, which is used to implement the inverse kinematic transformations of a robot arm controller with two and three degrees of freedom. This new version shows faster convergence than the original and scales better to bigger training sets and lower tolerances. ========================================================================= Public Domain Code available ---------------------------- The code of the modified cascade-correlation learning architecture, presented in the above report, is also available at the IDIAP ftp site. Instructions for obtaining a copy of this software are given at the end of this message. A description of the code follows: /********************************************************************************/ /* C implementation of the Modified Cascade-Correlation learning algorithm */ /* */ /* Modified by: N. Simon */ /* Department of Electrical Engineering */ /* Computer Architecture and Digital Systems */ /* Delft University of Technology */ /* 2600 GA Delft, The Netherlands */ /* */ /* E-mail: natalio at zen.et.tudelft.nl */ /* */ /* */ /* This code is a modification of the original code written by R. Scott */ /* Crowder of Carnegie Mellon University (version 1.32). */ /* That code is a port to C from the original Common Lisp implementation */ /* written by Scott E. Fahlman. (Version dated June 1 1990.) */ /* *//* */ /* For an explanation of the original algorithm, see "The */ /* Cascade-Correlation Learning Architecture" by Scott E. Fahlman and */ /* Christian Lebiere in D. S. Touretzky (ed.), "Advances in Neural */ /* Information Processing Systems 2", Morgan Kaufmann, 1990. A somewhat */ /* longer version is available as CMU Computer Science Tech Report */ /* CMU-CS-90-100. */ /* */ /* For an explanation of the Modified Cascade-Correlation learning */ /* see "Variations on the Cascade-Correlation Learning Architecture for */ /* Fast Convergence in Robot Control" by N. Simon, H. Corporaal and */ /* E. Kerckhoffs, in Proc. Neuro-Nimes '92, Nimes, France, 1992, */ /* pp. 455-466. */ /********************************************************************************/ Instructions for obtaining a copy of the paper: unix> ftp Maya.IDIAP.CH (or: ftp 192.33.221.1) login: anonymous password: ftp> cd pub/papers/neural ftp> binary ftp> get simon.variations.ps.Z ftp> bye unix> zcat simon.variations.ps.Z | lpr (or however you uncompress and print a postscript file) Instructions for obtaining a copy of the software: unix> ftp Maya.IDIAP.CH (or: ftp 192.33.221.1) login: anonymous password: ftp> cd pub/software/neural ftp> binary ftp> get mcascor.c.Z ftp> bye unix> uncompress mcascor.c.Z E. Fiesler IDIAP From tomy at maxwell.ee.washington.edu Wed Nov 25 19:03:28 1992 From: tomy at maxwell.ee.washington.edu (Thomas A. Tucker) Date: Wed, 25 Nov 92 16:03:28 PST Subject: NIPS 92 Young Scientists' Network Announcement Message-ID: <9211260003.AA23313@shockley.ee.washington.edu> ANNOUNCING AN INFORMAL NIPS92 PRESENCE -- The Young Scientists' Network -- The YSN is a mailing list dedicated to the "discussion of issues involving the employment of scientists, especially those just beginning their careers." YSN started about two years ago when a group of physics PhD's noted a common thread among their difficulties in securing academic and industrial employment -- but since then scientists in other fields have reported similar situations. Our conclusion has been that there's a glut of scientists, not the shortage that had been predicted for years. Consequently, many well-trained persons are finding themselves under- or un-employed, and, contrary to traditional attitudes, this poor employment record is NOT a reflection of individual abilities so much as it is an indictment of the priorities of American science. The discussions on YSN have identified many contributors to this phenomenon: universities producing too many graduates, insubstantial R&D investments by US companies, research funds drying up, the growing tendency of universities to create non-tenure-track positions when facing budget constraints, and a host of less tangible but undeniably real interpersonal and political pressures. We'd like to offer the opportunity to learn about and get involved with the Young Scientists' Network next week at NIPS. With luck, we'll be able to set a meeting time in Denver by next Monday, and this information should be available when you check in. Our concrete goals: -- an opportunity to learn more about the aims and means of the Young Scientists' Network, with email contact information and sample YSN Digests. -- solicitation of ideas that might be presented in a national legislative or administrative forum (Congress or the NSF, for example), to help foster economic opportunities for research. -- compiling a directory of academic and industrial organizations which might offer potential employment (the responses we receive, the more complete the directory). -- compiling a possible roster of sources for funding. -- compiling a list of graduate programs offering study in computational neuroscience or other connectionist fields with an emphasis on interdisciplinary work. -- we hope to make available survey information which will help to document the nature and magnitude of YSN members' concerns. Of course, we're open to any suggestions you may wish to put forward, and we'll be available for casual discussion throughout the conference. We look forward to seeing you there, Thomas A. Tucker (tomy at ee.washington.edu) Pamela A. Abshire (pa11722 at medtronic.com) From jbower at cns.caltech.edu Wed Nov 25 17:58:25 1992 From: jbower at cns.caltech.edu (Jim Bower) Date: Wed, 25 Nov 92 14:58:25 PST Subject: Brains and Braun Message-ID: <9211252258.AA21496@smaug.cns.caltech.edu> With respect to the recent Biology / abstract modeling discussion. I would like to suggest that the distance between Cognitive Psychology and the details of brain circuitry might very well be as substantial as the distance between connectionist models and brain structure. Accordingly, relations between abstract descriptions of brain function and the performance of abstract networks does not necessarily make me, as a computational neurobiologist, more comfortable with either. Jim Bower From jbower at cns.caltech.edu Wed Nov 25 21:59:54 1992 From: jbower at cns.caltech.edu (Jim Bower) Date: Wed, 25 Nov 92 18:59:54 PST Subject: Details, who needs them?? Message-ID: <9211260259.AA21750@smaug.cns.caltech.edu> >I grow tired of defending the validity of models to biologists > who do not seem satisfied with any model that does not capture >every last nuance of complexity or that does not explain every last >experimental finding. In response to this and several similar statements, I have to say that in now many years of building biologically motivated neural models, and attending neural modeling meetings of all sorts, I have never yet met such a neurobiologist. I have of course met many who object to the kind of "brain-hype" that originally prompted my remarks. However, there can be no question that the issue of the level of detail necessary to account for brain function, or to do something really interesting with neural networks is a subject of active debate. With respect to neural networks, I would point out that this question has been around from the begining of neural network research. Further, not that long ago, many believed, and argued loudly that simple networks could do everything. Several of us said that if that were true, the brain would be simple and because it is not, it is likely that artificial networks would have to get more complex to do anything real or even very interesting. As we head to the NIPS meeting, it is fairly clear that the simple neural networks have not done very well evolutionarily. Further, the derivatives are clear. With respect to the detail necessary to understand the brain, this is also an area of active debate in the young field of computational neuroscience. However, from our own work and that of others, maybe it is time to make the statement that it appears as though the details might matter a great deal. For example, through the interaction of realistic models and experimental work, we have recently stumbled across a regulatory mechanism in the olfactory cerebral cortex that may be involved in switching the network from a learning to a recall state. If correct, this switching mechanism serves to reduce the possibility of the corruption of new memories with old memories. While it would be inappropriate to describe the results in this forum in detail, it turns out that the mechanism bears a resemblance to an approach used by Kohonen to avoid the same problem. Further, when the more elaborate details of the biologically derived mechanism are placed in a Kohonen associative memory, the performance of the original Kohonen net is improved. In this case, however, the connection to Kohonen's work was made only after we performed the biological experiments. This is not because we did not know Kohonen's work, but because the basic mechanism was so unbiological that it would have made little sense to specifically look for it in the network. The biological modeling now done, we can see that Kohonen's approach appears as a minimal implementation of a much more sophisticated, complicated, and apparently more effective memory regulation mechanism. While it is not the common practice on this network or in this field to point out ones own shortcomings, it turns out that we did not know, prior to doing the realistic modeling, which biological details might matter the most. Once they were discovered, it was fairly trivial to modify an abstract model to include them. The point here is that only through paying close attention to the biological details was this mechanism discovered. From this and a few other examples in the new and growing field of computational neuroscience, it may very well be that we will actually have to pay very close attention to the structure of the nervous system if we are going to learn anything new about how machines like the brain work. I acknowledge that this may or may not be eventually relevant to neural networks and connectionism as I am yet to be convinced that these are particularly good models for whatever type of computational object the brain is. However, if there is some connection, it might be necessary to have those interested in advancing the state of artificial networks seek more information about neurobiology than they can obtain at their favorite annual neural network meeting, from a basic neurobiology textbook, or from some "overview" published by some certified leader of the field. Who knows, it might even be necessary to learn how to use an electrode. Jim Bower For those interested in a very general overview of the work described above, I have placed the following review article in neuroprose: "The Modulation of Learning State in a Biological Associative Memory: An in vitro, in vivo, and in computo Study of Object Recognition in Mammalian Olfactory Cortex." James M. Bower To retrieve from neuroprose: unix> ftp cheops.cis.ohio-state.edu Name (cheops.cis.ohio-state.edu:becker): anonymous Password: (use your email address) ftp> cd pub/neuroprose ftp> get bower.ACH.asci.Z 200 PORT command successful. 150 Opening BINARY mode data connection for bower.ACH.asci.Z 226 Transfer complete. ###### bytes received in ## seconds (## Kbytes/s) ftp> quit From cateau at tkyux.phys.s.u-tokyo.ac.jp Thu Nov 26 00:16:29 1992 From: cateau at tkyux.phys.s.u-tokyo.ac.jp (Hideyuki Cateau) Date: Thu, 26 Nov 92 14:16:29 +0900 Subject: cognition and biology Message-ID: <9211260516.AA05354@tkyux.phys.s.u-tokyo.ac.jp> Jay McClelland writes: >Sometimes very >abstract and general features that connectionist systems share with >other systems are doing the work; other times it is going to turn out >to be specific features not shared by a wide range of abstract models. >The power law appears to be a case of the former, since it has >probably been accounted for by more psychological models than any >other phenomenon. I have two things which I would like to say in relation to his comments. First, since I am not a psychologist, I have asked many psychologists about the power law which I am interested in. Then I knew that the power laws are frequently found in psychological experiments such as Stevens' law. I also knew there are various psychological models which derive some of the power laws. But up to now, I have never found in literatures or heard from psychologist, that there is a non-neural-network-based psychological model which explains exactly the same experiment in question. If anyone know such work, please tell it to me. It is very intriguing to me to examine which model is better. Second, I am actually a physisist majoring in an elementary particle physics. Particle physisists generally believe that all the phenomena occuring in this world must be explained, after all, from fundamental laws of the elemenary particles, because this world consists of the elementary particles. In just the same way, I believe that every intellectual phenomenum of our brain is derived from th activities of the neurons of which our brain consists. So I have a tendency to prefer the neural-network-based model to other psychological models. This is the reason why I think my work is meaningful although there might be other psychological models which also explain the the experiment in question. Of course it is wrong to say that one way of thinking is correct and another way is incorrect. Both non-neural-network-based way and neural- network-based way will be useful for our understanding of nature. In a community of particle physics, the two different stand points are clearly separated. Those who on the former stand point are called phenomenologists, while those who on the latter stand point are called theoretical theorists. The former people are trying to find a simple law which reproduces experimental facts based on some assumptipons, but not so serious about why such law holds. The latter people are trying to derive the fundamental laws of physics from the first principle, but they frequently fall into the study of the toy models which are only of academic interest. Anyway, both ways of thinking is necessary for understanding of nature. Hideyuki Cateau From cateau at tkyux.phys.s.u-tokyo.ac.jp Thu Nov 26 07:14:43 1992 From: cateau at tkyux.phys.s.u-tokyo.ac.jp (Hideyuki Cateau) Date: Thu, 26 Nov 92 21:14:43 +0900 Subject: Neural computing ideas ... Message-ID: <9211261214.AA01048@tkyux.phys.s.u-tokyo.ac.jp> Bard Ermentrout writes: >Bacterial cultures in a nutrient depleted medium with inhibited >motility produce fractal colonies that have the same fractal >dimension as diffusion-limited aggregation processes. However the >mechanism for one is totally different from the other. All Hopf >bifurcations from rest scale in the smae fashion. All of these >examples are consequences of the fact that many mathematical >phenomena obey fixed scaling laws. But to say that because backprop >and humnan memory learn with the same scaling laws implies that backprop >has something to do with real learning is at best specious. Just becausee >two phenomena scale the same way doenst mean that the mechanisms are >identical. It is easy to find good examples just like it is easy to find bad ones. Ising model of the magnetism is really simpified model. No one might be able to predict this model is a very good abstraction of the real complex material. However, the scaling behavior in the vicinity of the critical point of metal is correctly understood by Ising model. The model of the superconductivity proposed by B.C.S. was also a good model. It deepen the understanding of the superconductivity of the real material. There are countless number of good examples as well as bad examples. So it does not seem to make sense to judge that our proposal is a bad one being based on the fact that there are many bad examples in the world. As he said it is true that there are various kinds of scaling behavior in nature. There is a potential danger that we are incorrectly convinced that two of them are derived from the same mechanism just because the values of the exponent coincide each other. But we are not only based on the accordance of the exponents but also based on other circumstances in which we are working. As I have noted in the original mail, the reason for the slowing down of the learning pace is considered to be an interfarence between different items which are memorized in the brain. Back prop slows down by the same reason. The configuration of the connection weights Wij which is good for one vector to memorize is not good for another vector to memorize in general. Back prop must find out common grounds to memorize serveral items. It costs time and slows down the memory. At least for us, under these circumstances, it seems natural to expect an analogy between the two slowing downs. Then we performed a simulation to obtain a plausible result. Watching one aspect of the matter is always dangerous. Taking a very simple example, we use a trigonometric function when we measure the hieght of a tall tree. On the other hand, we also have a trigonometric function when we solve the wave eqution. Of course it is unnatural to expect some deep connection between the two situations. But for our case it is natural at least to us. We agree that the evidences might still be not enough. Our work is on the process of exploring more detailed ananlogy between the two systems, the brain and the back prop. If the result of our simulation were negative, the exploring program had quickly reached the end. Our conclusion in that case would have been that there were at least one evidence which indicated back prop was not a good model for our brain. But, at this level of exploration, we think we have got a plausible evidence and it is worth reporting. In the future, we might find an aspect which is not common between the back prop and brain. To reach such conclusion is also meaningful because our purpose is not to prove that the back prop is the best model of the brain but to know to what extent the back prop is a good model of our brain. We think that it is the starting point of the understanding of the brain based on the neural network models. H.Cateau From jlm at crab.psy.cmu.edu Thu Nov 26 08:45:44 1992 From: jlm at crab.psy.cmu.edu (James L. McClelland) Date: Thu, 26 Nov 92 08:45:44 EST Subject: independence law Message-ID: <9211261345.AA27057@crab.psy.cmu.edu.noname> Several readers of connectionists have asked for clarification of the independence law and/or pointers to publications. The independence law (I call it that) is a finding in the literature on the effect of context on perception of near-threshold stimuli (Morton, Psychological Review, 1968). It arises in studies in which one manipulates contextual support for a particular response (e.g. by providing a context such as 'I like coffee with cream and _____' vs 'The next word will be ______'). The '_____' represents the near-threshold stimulus. The context manipulation increases p = the probability of choosing some response of interest (e.g. 'sugar') relative to the neutral condition. Now, the independence law is the finding that context exterts the same effect on the variable log(p/(1-p)), independent of the exact nature of the stimulus. Of course, one needs a marginal stimulus to avoid p = 1 or 0. It had been claimed (Massaro, Cognitive Psychology, 1989) that the independence law is inconsistent with symmetrically connected networks, but this claim was based on simulations of the (non-stochastic) interactive activation model of McClelland and Rumelhart (Psych Review, 1981). I found, though, (McClelland, Cognitive Psychology, 1991) that in fact stochastic, symmetrically connected networks can -- indeed, must -- adhere to the independence law if they adhere to what I called the structural independence constraint on architecture. The constraint is roughly the following: a) the network be structured so that there are three separate pools of units: one for receiving and representing the stimulus input, one for recieving and representing the context, and one for combining these influences to represent the set of possible response alternatives (e.g., words). b) there can be no connections between any of the units in the stimulus input part of the network and any of the units in the context part of the network. In my 91 paper the result was developed for Boltzmann machines using localist representations of letters or words. In recent work with Javier Movellan (not yet ready for circulation) we are establishing considerable generality to these results and relating the findings to other literatures. It turns out that what I called structural independence in a stochastic symmetric network is tantamount to adherence to an assumption called 'conditional independence' that is used in pattern recognition in order to select the response with the maximum a posteriori probability (MAP) given context and stimulus information. We will announce this paper when it is ready on connectionists. From cateau at tkyux.phys.s.u-tokyo.ac.jp Thu Nov 26 08:44:11 1992 From: cateau at tkyux.phys.s.u-tokyo.ac.jp (Hideyuki Cateau) Date: Thu, 26 Nov 92 22:44:11 +0900 Subject: networks and biology Message-ID: <9211261344.AA11468@tkyux.phys.s.u-tokyo.ac.jp> Rogene Eichler writes: >................. >................. >You are basing your statement on the ability of a subset of network models >to explain a very small subset of the behaviors that are observable and >testable by somewhat similar criteria. Furthermore, it could be argued that >the criteria you are using for your comparison is qualitative in nature >because of the testing methods employed to measure human performance in >some cognitive tasks. > >Your work has shown that complex network systems can demonstrate similar >emergent properties. That statement, supported by the performance measures >you cited, is very powerful. But you have substituted one black box for >another- nothing can be said quantitatively about how or where brain function >occurs. I agree with the first statement of the upper paragraph above. We should examine various kinds of neural network model to see the universality of the power law, although we have checked it for various parameters of the back prop model. He further says that "... explain a very small subset of the behaviors ...". He is completely correct. But I am not so serious about the point. At the first sight, the back prop model is too simple a model to be regarded as model of the brain. I never think that the back prop can model the whole behavior of our complex brain. Only one point of similarity between the back prop and the brain is surprising to me, and I thought it is worth reporting. I know that it is not the first dicovery of the similiarity between neural network models and the real brain. I argued that the novelty of our result is that our result is quantitative instead of qualitative. At this point our opinions disagree with each other. He writes: >... >the criteria you are using for your comparison is qualitative in nature >because of the testing methods employed to measure human performance in >some cognitive tasks. >... First of all, I do not think that measuring the human cognitive tasks are always qualitative. I am a physisist. So, when I was not familiar with works of psychologists, I certainly thought that most psychological experiments would be only qualitative. I believed that they had in general low reproducibility, compared with physical experiments. But I changed my idea after I read many psychological papers and I myself performed a psycho- logical experiment of the power law in question. The power law was very stable. The value of the exponent varies depending on persons or other factors, but the value always fell within the range between 1 and 2. This is a definite quantitative fact. The exponent for back prop was two up to errors, as I wrote in the original mail. This is nothing but a quatitative accordance. If the exponent for back prop had been observed to be 10, for example, we would have concluded that the behaviors of the brain and back prop qualitatively coincided, but the accordance were not quantitative. I agree that our result have not opened a black box, which he mentioned in his last paragraph. It would be a very long way to the point when we open the black box when we finally see the secret of the brain. What we could do now is to hit the black box and carefully hear the sound it emits, to tumble it down and observe the reaction etc. We, in some sense, hit the black box by the psychological experiment and got the power law as reaction of it. On the other hand we verified that it was the same reaction when we hit the back prop. It is encouraging to the neural networkers who believe that the study of the neural network model is useful for the study of the brain. H.Cateau From schwaber at eplrx7.es.duPont.com Wed Nov 25 10:11:01 1992 From: schwaber at eplrx7.es.duPont.com (Jim Schwaber) Date: Wed, 25 Nov 92 10:11:01 EST Subject: NIPS workshop - REAL biological computation Message-ID: <9211251511.AA12325@eplrx7.es.duPont.com> -----------NIPS 92 WORKSHOP---------------------- Real Applications of Real Biological Circuits or "If back-prop is not enough how will we get more?" or "Is anybody really getting anywhere with biology?" --------------------------------------------------- When: Friday, Dec. 4th ==== Intended Audience: Those interested in detailed biological modeling. ================== Those interested in nonlinear control. Those interested in neuronal signal processing. Those interested in connecting the above. Organizers: =========== Richard Granger Jim Schwaber granger at ics.uci.edu schwaber at eplrx7.es.dupont.com Agenda: ======= Morning Session, 7:30 - 9:30, Brain Control Systems and Chemical --------------- Process Control Jim Schwaber Brainstem reflexes as adaptive controllers Dupont Babatunde Ogunnaike Reverse engineering brain control systems DuPont Frank Doyle Neurons as nonlinear systems for control Purdue John Hopfield Discussant Caltech Afternoon Session, 4:30 - 6:30, Real biological modeling, nonlinear ----------------- systems and signal processing Richard Granger Signal processing in real neural systems: is UC Irvine it applicable? Gary Green The single neuron as a nonlinear system - its Newcastle Volterra kernels as described by neural networks. Program: ======== We anticipate that the topic will generate several points of view. Thus, presenters will restrict themselves to a very, very few slides intended to make a point for discussion. Given that there now are concrete examples of taking biological principles to application, we expect the discussion will center more on how, and at what level, rather than whether "reverse engineering the brain" is useful. Granger (UC Irvine): ------- The architectures, performance rules and learning rules of most artificial neural networks are at odds with the anatomy and physiology of real biological neural circuitry. For example, mammalian telencephelon (forebrain) is characterized by extremely sparse connectivity (~1-5%), almost entirely lacks dense recurrent connections, and has extensive lateral local circuit connections; inhibition is delayed-onset and relatively long-lasting (100s of milliseconds) compared to rapid-onset brief excitation (10s of milliseconds), and they are not interchangeable. Excitatory connections learn, but there is very little evidence for plasticity in inhibitory connections. Real synaptic plasticity rules are sensitive to temporal information, are not Hebbian, and do not contain "supervision" signals in any form related to those common in ANNs. These discrepancies between natural and artificial NNs raise the question of whether such biological details are largely extraneous to the behavioral and computational utility of neural circuitry, or whether such properties may yield novel rules that confer useful computational abilities to networks that use them. In this workshop we will explicitly analyze the power and utility of a range of novel algorithms derived from detailed biology, and illustrate specific industrial applicatons of these algorithms in the fields of process control and signal processing. Ogunnaike (DuPont): ----------- REVERSE ENGINEERING BRAIN CONTROL SYSTEMS: EXPLORING THE POTENTIAL FOR APPLICATIONS IN CHEMICAL PROCESS CONTROL. ===================================================================== The main motivation for our efforts lies in the simple fact that there are remarkable analogies between the human body and the chemical process plant. Furthermore, it is known that while the brain has been quite successful in performing its task as the central supervisor of intricate control systems operating under conditions which leave very little margin for error, the control computer in the chemical process plant has not been so successful. We have been concerned with seeking answers to the following question: ``Is it possible to ``reverse engineer'' a biological control system and use the understanding to develop novel approaches to chemical process control systems design and analysis?'' Our discussion will provide an overview of the tentative answers we have to date. We will first provide a brief summary of the salient features and main problems of chemical process control; we will then introduce the biological control system under study (the baroreceptor vagal reflex); finally we will present an actual industrial process whose main features indicate that it may benefit from the knowledge garnered from the neurobiological studies. Doyle (Purdue): ------ We are focusing our research on two levels: 1) Neuron level: investigating novel building blocks for process modeling applications which are motivated by realistic biological neurons. 2) Network Level: looking for novel approaches to nonlinear dynamic scheduling algorithms for process control and modeling (again, motivated by biological signal processing in the baroreceptor reflex). Green (Newcastle): ------- I would love to tell the NIPS people about Volterra series, especially as we have now made a connection between neural networks, Volterra series and the differential geometric representation of networks. This allows us to say why one, two or more layers are necessary for a particular analytic problem. We can also say how to invert nets which are homeomorphic in their mappings. More importantly for us biologists we can turn the state equations of membrane currents, using neural networks into approximate Volterra kernels which I think (!) helps understand the dynamics. This gives a solution to the differential equations, albeit an approximate one in practical terms. The equations are invertible and therefore allow a formal link between current clamp and voltage clamp at the equation level. The method we have used to do this is of interest to chem. eng. people because we can use the same concepts in non-linear control. It appears at first glance that we can link the everyday use of neural networks to well established theory through a study of tangent spaces of networks. We construct a state space model of a plant, calculate the differential of the rate of change of output with respect to the input. Calculate the same for a neural network. Compare coefficients. The solution to the set of simultaneous equations for the coefficents produces a network which is formally equivalent to the solution of the original differential equation which defined the state equations. We will be making the claim that analytic solutions of non-linear differential equations is possible using neural networks for some problems. For all other problems an approximate solution is possible but the architecture that must be used can be defined. Last I'll show how this is related to the old techniques using Volterra series and why the kernels and inverse transforms can be directly extracted from networks. I think it is a new method of solving what is a very old problem. All in 20 minutes ! From zhang at psych.lsa.umich.edu Fri Nov 27 10:12:04 1992 From: zhang at psych.lsa.umich.edu (zhang@psych.lsa.umich.edu) Date: Fri, 27 Nov 92 10:12:04 EST Subject: No subject Message-ID: <9211271512.AA00495@parvo.psych.lsa.umich.edu> Position in Cognitive Psychology University of Michigan The University of Michigan Department of Psychology invites applications for a tenure-track position in the area of Cognition, beginning September 1, 1993. The appointment will most likely be made at the Assistant Professor level, but it is possible at any rank. We seek candidates with primary interests and technical skills in cognitive psychology. Our primary goal is to hire an outstanding cognitive psychologist, and thus we will look at candidates with any specific research interest. We have a preference for candidates interested in higher mental processes or for candidates with computational modeling skills (including connectionism). Responsibilities include graduate and undergraduate teaching, as well as research and research supervision. Send curriculum vitae, letters of reference, copies of recent publications, and a statement of research and teaching interests no later than January 8, 1993 to: Gary Olson, Chair, Cognitive Processes Search Committee, Department of Psychology, University of Michigan, 330 Packard Road, Ann Arbor, Michigan 48104. The University of Michigan is an Equal Opportunity/Affirmative Action employer. From zhang at psych.lsa.umich.edu Sun Nov 29 17:47:23 1992 From: zhang at psych.lsa.umich.edu (zhang@psych.lsa.umich.edu) Date: Sun, 29 Nov 92 17:47:23 EST Subject: No subject Message-ID: <9211292247.AA00606@parvo.psych.lsa.umich.edu> Position in Cognitive Psychology University of Michigan The University of Michigan Department of Psychology invites applications for a tenure-track position in the area of Cognition, beginning September 1, 1993. The appointment will most likely be made at the Assistant Professor level, but it is possible at any rank. We seek candidates with primary interests and technical skills in cognitive psychology. Our primary goal is to hire an outstanding cognitive psychologist, and thus we will look at candidates with any specific research interest. We have a preference for candidates interested in higher mental processes or for candidates with computational modeling skills (including connectionism). Responsibilities include graduate and undergraduate teaching, as well as research and research supervision. Send curriculum vitae, letters of reference, copies of recent publications, and a statement of research and teaching interests no later than January 8, 1993 to: Gary Olson, Chair, Cognitive Processes Search Committee, Department of Psychology, University of Michigan, 330 Packard Road, Ann Arbor, Michigan 48104. The University of Michigan is an Equal Opportunity/Affirmative Action employer. From ken at cns.caltech.edu Sun Nov 29 09:29:29 1992 From: ken at cns.caltech.edu (Ken Miller) Date: Sun, 29 Nov 92 06:29:29 PST Subject: No subject Message-ID: <9211291429.AA06827@zenon.cns.caltech.edu> POSTDOCTORAL POSITIONS COMPUTATIONAL NEUROSCIENCE UNIVERSITY OF CALIFORNIA, SAN FRANCISCO I will soon be beginning a new lab at UCSF, and anticipate several positions for postdocs beginning in 1993 and 1994 (prospective graduate students are also encouraged to apply to the UCSF Neuroscience Program). The lab will focus on understanding both development and mature processing in the cerebral cortex. Theoretical, computational, and experimental approaches will be taken. Candidates should have skills relevant to one or more of those approaches. The most important criteria are demonstrated scientific ability and creativity, and a deep interest in grappling with the details of neurobiology and the brain. Past work has focused on modeling of development in visual cortex under Hebbian and similar ``correlation-based" rules of synaptic plasticity. The goal has been to understand these rules in a general way that allows experimental predictions to be made. Models have been formulated for the development of ocular dominance and orientation columns. A few references are listed below. Future work of the lab will extend the developmental modeling, and will also take various approaches to understanding mature cortical function. These will include detailed biophysical modeling of visual cortical networks, many-cell recording from visual cortex, and use of a number of theoretical methods to guide and interpret this recording. There will also be opportunities for theoretical forays in new directions, in particular in collaborations with the other Neuroscientists at UCSF. Facilities to develop new experimental directions that are relevant to the lab's program, for example slice studies and use of optical methods, will also exist. I will be part of the Keck Center for Systems Neuroscience at UCSF, which will be a very interactive environment for Systems Neurobiology. Other members will include: * Alan Basbaum (pain systems); * Allison Doupe (song learning in songbirds); * Steve Lisberger (oculomotor system); * Michael Merzenich (adult cortical plasticity); * Christof Schreiner (auditory system); * Michael Stryker (visual system, development and plasticity); Closely related faculty members include Roger Nicoll (hippocampus, LTP); Rob Malenka (hippocampus, LTP); Howard Fields (pain systems); and Henry Ralston (spinal cord and thalamus). Please send a letter describing your interests and a C.V., and arrange to have three letters of recommendation sent to Ken Miller Division of Biology 216-76 Caltech Pasadena, CA 91125 ken at cns.caltech.edu Some References: Miller, K.D. (1992). ``Models of Activity-Dependent Neural Development." Seminars in the Neurosciences, 4:61-73. Miller, K.D. (1992). ``Development of Orientation Columns Via Competition Between ON- and OFF-Center Inputs." NeuroReport 3:73-76. MacKay, D.J.C. and K.D. Miller (1990). ``Analysis of Linsker's simulations of Hebbian rules," Neural Computation 2:169-182. Miller, K.D. (1990). ``Correlation-based mechanisms of neural development," in Neuroscience and Connectionist Theory, M.A. Gluck and D.E. Rumelhart, Eds. (Lawrence Erlbaum Associates, Hillsdale NJ), pp. 267-353. Miller, K.D., J.B. Keller and M.P. Stryker (1989). ``Ocular dominance column development: analysis and simulation," Science 245:605-615. Miller, K.D., B. Chapman and M.P. Stryker (1989). ``Responses of cells in cat visual cortex depend on NMDA receptors," Proc. Nat. Acad. Sci. USA 86:5183-5187. From moody at chianti.cse.ogi.edu Sat Nov 28 17:35:24 1992 From: moody at chianti.cse.ogi.edu (John Moody) Date: Sat, 28 Nov 92 14:35:24 -0800 Subject: NATO ASI on Statistics and Neural Networks Message-ID: <9211282235.AA08058@chianti.cse.ogi.edu> NATO Advanced Studies Institute (ASI) on Statistics and Neural Networks June 21 - July 2, 1993, Les Arcs, France Directors: Professor Vladimir Cherkassky Department of Electrical Eng. University of Minnesota, Minneapolis, MN 55455 tel.(612) 625-9597 fax (612) 625- 4583 email: cherkass at ee.umn.edu Professor Jerome H. Friedman Statistics Department Stanford University Stanford, CA 94309 tel (415 )723-9329 fax(415) 926-3329 email: jhf at playfair.stanford.edu Professor Harry Wechsler Computer Science Department George Mason University Fairfax VA 22030 tel (703) 993-1533 fax (703) 993-1521 email: wechsler at gmuvax2.gmu.edu List of invited lecturers: I. Alexander, L. Almeida, A. Barron, A. Buja, E. Bienenstock, G. Carpenter, V. Cherkassky, T. Hastie, F. Fogelman, J. Friedman, H. Freeman, F. Girosi, S. Grossberg, J. Kittler, R. Lippmann, J. Moody, G. Palm, R. Tibshirani, H. Wechsler, C. Wellekens. Objective, Agenda and Participants: Nonparametric estimation is a problem of fundamental importance for many applications involving pattern classification and discrimination. This problem has been addressed in Statistics, Pattern Recognition, Chaotic Systems Theory, and more recently in Artificial Neural Network (ANN) research. This ASI will bring together leading researchers from these fields to present an up-to-date review of the current state-of-the art, to identify fundamental concepts and trends for future development, to assess the relative advantages and limitations of statistical vs neural network techniques for various pattern recognition applications, and to develop a coherent framework for the joint study of Statistics and ANNs. Topics range from theoretical modeling and adaptive computational methods to empirical comparisons between statistical and neural network techniques. Lectures will be presented in a tutorial manner to benefit the participants of ASI. A two-week programme is planned, complete with lectures, industrial/government sessions, poster sessions and social events. It is expected that over seventy students (which can be researchers or practitioners at the post-graduate or graduate level) will attend, drawn from each NATO country and from Central and Eastern Europe. The proceedings of ASI will be published by Springer-Verlag. Applications: Applications for participation at the ASI are sought. Prospective students, industrial or government participants should send a brief statement of what they intend to accomplish and what form their participation would take. Each application should include a curriculum vitae, with a brief summary of relevant scientific or professional accomplishments, and a documented statement of financial need (if funds are applied for). Optionally, applications may include a one page summary for making a short presentation at the poster session. Poster presentations focusing on comparative evaluation of statistical and neural network methods and application studies are especially sought. For junior applicants, support letters from senior members of the professional community familiar with the applicant's work would strengthen the application. Prospective participants from Greece, Portugal and Turkey are especially encouraged to apply. Costs and Funding: The estimated cost of hotel accommodations and meals for the two-week duration of the ASI is US$1,600. In addition, participants from industry will be charged an industrial registration fee, not to exceed US$1,000. Participants representing industrial sponsors will be exempt from the fee. We intend to subsidize costs of participants to the maximum extent possible by available funding. Prospective participants should also seek support from their national scientific funding agencies. The agencies, such as the American NSF or the German DFG, may provide some ASI travel funds upon the recommendation of an ASI director. Additional funds exist for students from Greece, Portugal and Turkey. We are also seeking additional sponsorship of ASI. Every sponsor will be fully acknowledged at the ASI site as well as in the printed proceedings. Correspondence and Registration: Applications should be forwarded to Dr. Cherkassky at the above address. Applications arriving after March 1, 1993 may not be considered. All approved applicants will be informed of the exact registration arrangements. Informal email inquiries can be addressed to Dr. Cherkassky at nato_asi at ee.umn.edu. From jbower at cns.caltech.edu Mon Nov 30 20:00:30 1992 From: jbower at cns.caltech.edu (Jim Bower) Date: Mon, 30 Nov 92 17:00:30 PST Subject: paper Message-ID: <9212010100.AA25727@smaug.cns.caltech.edu> Sorry, the paper I refered to in my previous posting is now in neuroprose and renamed: bower.ach.asc.Z Jim Bower From ingber at alumni.cco.caltech.edu Mon Nov 30 10:17:21 1992 From: ingber at alumni.cco.caltech.edu (Lester Ingber) Date: Mon, 30 Nov 1992 07:17:21 -0800 Subject: Very Fast Simulated Reannealing version 6.20 Message-ID: <9211301517.AA01764@alumni.cco.caltech.edu> VERY FAST SIMULATED REANNEALING (VFSR) (C) Lester Ingber ingber at alumni.caltech.edu and Bruce Rosen rosen at ringer.cs.utsa.edu The good news is that the people who have gotten our beta version of VFSR to work on their applications are very pleased. The bad news is that because of some blunders made in the process of making the code user-friendly, the code has to be modified to use as a standalone function call. This bug is corrected and some other fixes/changes are made in version v6.20. This version is now updated in netlib at research.att.com. It will eventually find its way into the other NETLIB archives. To access the new version: Interactive local% ftp research.att.com Name (research.att.com:your_login_name): netlib Password: [type in your_login_name or anything] ftp> cd opt ftp> binary ftp> get vfsr.Z ftp> quit local% uncompress vfsr.Z local% sh vfsr Electronic Mail Request local% mail netlib at research.att.com [mail netlib at ornl.gov] [mail netlib at ukc.ac.uk] [mail netlib at nac.no] [mail netlib at cs.uow.edu.au] send vfsr from opt ^D [or however you send mail] Lester || Prof. Lester Ingber ingber at alumni.caltech.edu || || P.O. Box 857 || || McLean, VA 22101 703-848-1859 = [10ATT]0-700-L-INGBER || From BARNEV%BGEARN.BITNET at BITNET.CC.CMU.EDU Mon Nov 2 13:03:24 1992 From: BARNEV%BGEARN.BITNET at BITNET.CC.CMU.EDU (IT&P) Date: Mon, 02 Nov 92 13:03:24 BG Subject: Conference Announcement Message-ID: ----------------------------Original message---------------------------- DEAR COLLEAGUE, I would like to invite you to the 18. issue of the International INFORMATION TECHNOLOGIES AND PROGRAMMING conference to be held in Sofia, the capital of Bulgaria from 27 June to 4 July 1993. MAIN TOPICS: - Information Technologies and Telecommunications in Business and Public Administration - Hypertext and Multimedia Systems - Graphical Methods for Scientific and Technical Computing PROGRAMME COMMITTEE: L. AIELLO (Universita di Roma 'La Sapienza', Italy) M. Mac an AIRCHINNIGH (Trinity College, Dublin, Ireland) P. BARNEV (Institute of Mathematics, Sofia, Bulgaria) - chairman J. HOFFER (Indiana University, Bloomington, USA) S. KERPEDJIEV (Institute of Mathematics, Sofia, Bulgaria) B. KOKINOV (Institute of Mathematics, Sofia, Bulgaria) V. KOTOV (Institute of Informatics Systems, Novosibirsk, Russia) N. SPYRATOS (Universite de Paris-Sud, Paris, France) N. STREITZ (Gesellschaft fuer Mathematik und Daterverarbeitung, Darmstadt, Germany) C. THANOS (Instituto de Elaborazione della Informazione, Pisa, Italy) T. VAMOS (Hungarian Academy of Sciences, Budapest, Hungary) The deadline for submitting papers is February15, 1993. Electronic submission is welcome but the final camera-ready form should be in hard-copy form. All papers will be reviewed by members of the International Programme Committee. As a member of the Program Committee I would like to encourage the submission of connectionist papers. To receive more information, please contact me or Prof. Barnev (barnev at bgearn). Sincerely yours, Boicho Kokinov From Connectionists-Request at CS.CMU.EDU Sun Nov 1 00:05:13 1992 From: Connectionists-Request at CS.CMU.EDU (Connectionists-Request@CS.CMU.EDU) Date: Sun, 01 Nov 92 00:05:13 EST Subject: Bi-monthly Reminder Message-ID: <11190.720594313@B.GP.CS.CMU.EDU> *** DO NOT FORWARD TO ANY OTHER LISTS *** This is an automatically posted bi-monthly reminder about how the CONNECTIONISTS list works and how to access various online resources. CONNECTIONISTS is not an edited forum like the Neuron Digest, or a free-for-all newsgroup like comp.ai.neural-nets. It's somewhere in between, relying on the self-restraint of its subscribers. Membership in CONNECTIONISTS is restricted to persons actively involved in neural net research. The following posting guidelines are designed to reduce the amount of irrelevant messages sent to the list. Before you post, please remember that this list is distributed to over a thousand busy people who don't want their time wasted on trivia. Also, many subscribers pay cash for each kbyte; they shouldn't be forced to pay for junk mail. Happy hacking. -- Dave Touretzky & David Redish --------------------------------------------------------------------- What to post to CONNECTIONISTS ------------------------------ - The list is primarily intended to support the discussion of technical issues relating to neural computation. - We encourage people to post the abstracts of their latest papers and tech reports. - Conferences and workshops may be announced on this list AT MOST twice: once to send out a call for papers, and once to remind non-authors about the registration deadline. A flood of repetitive announcements about the same conference is not welcome here. - Requests for ADDITIONAL references. This has been a particularly sensitive subject lately. Please try to (a) demonstrate that you have already pursued the quick, obvious routes to finding the information you desire, and (b) give people something back in return for bothering them. The easiest way to do both these things is to FIRST do the library work to find the basic references, then POST these as part of your query. Here's an example: WRONG WAY: "Can someone please mail me all references to cascade correlation?" RIGHT WAY: "I'm looking for references to work on cascade correlation. I've already read Fahlman's paper in NIPS 2, his NIPS 3 abstract, and found the code in the nn-bench archive. Is anyone aware of additional work with this algorithm? I'll summarize and post results to the list." - Announcements of job openings related to neural computation. - Short reviews of new text books related to neural computation. To send mail to everyone on the list, address it to Connectionists at CS.CMU.EDU ------------------------------------------------------------------- What NOT to post to CONNECTIONISTS: ----------------------------------- - Requests for addition to the list, change of address and other administrative matters should be sent to: "Connectionists-Request at cs.cmu.edu" (note the exact spelling: many "connectionists", one "request"). If you mention our mailing list to someone who may apply to be added to it, please make sure they use the above and NOT "Connectionists at cs.cmu.edu". - Requests for e-mail addresses of people who are believed to subscribe to CONNECTIONISTS should be sent to postmaster at appropriate-site. If the site address is unknown, send your request to Connectionists-Request at cs.cmu.edu and we'll do our best to help. A phone call to the appropriate institution may sometimes be simpler and faster. - Note that in many mail programs a reply to a message is automatically "CC"-ed to all the addresses on the "To" and "CC" lines of the original message. If the mailer you use has this property, please make sure your personal response (request for a Tech Report etc.) is NOT broadcast over the net. - Do NOT tell a friend about Connectionists at cs.cmu.edu. Tell him or her only about Connectionists-Request at cs.cmu.edu. This will save your friend from public embarrassment if she/he tries to subscribe. ------------------------------------------------------------------------------- The CONNECTIONISTS Archive: --------------------------- All e-mail messages sent to "Connectionists at cs.cmu.edu" starting 27-Feb-88 are now available for public perusal. A separate file exists for each month. The files' names are: arch.yymm where yymm stand for the obvious thing. Thus the earliest available data are in the file: arch.8802 Files ending with .Z are compressed using the standard unix compress program. To browse through these files (as well as through other files, see below) you must FTP them to your local machine. ------------------------------------------------------------------------------- How to FTP Files from the CONNECTIONISTS Archive ------------------------------------------------ 1. Open an FTP connection to host B.GP.CS.CMU.EDU (Internet address 128.2.242.8). 2. Login as user anonymous with password your username. 3. 'cd' directly to one of the following directories: /usr/connect/connectionists/archives /usr/connect/connectionists/bibliographies 4. The archives and bibliographies directories are the ONLY ones you can access. You can't even find out whether any other directories exist. If you are using the 'cd' command you must cd DIRECTLY into one of these two directories. Access will be denied to any others, including their parent directory. 5. The archives subdirectory contains back issues of the mailing list. Some bibliographies are in the bibliographies subdirectory. Problems? - contact us at "Connectionists-Request at cs.cmu.edu". ------------------------------------------------------------------------------- How to FTP Files from the Neuroprose Archive -------------------------------------------- Anonymous FTP on archive.cis.ohio-state.edu (128.146.8.52) pub/neuroprose directory This directory contains technical reports as a public service to the connectionist and neural network scientific community which has an organized mailing list (for info: connectionists-request at cs.cmu.edu) Researchers may place electronic versions of their preprints or articles in this directory, announce availability, and other interested researchers can rapidly retrieve and print the postscripts. This saves copying, postage and handling, by having the interested reader supply the paper. (Along this line, single spaced versions, if possible, will help!) To place a file, put it in the Inbox subdirectory, and send mail to pollack at cis.ohio-state.edu. Within a couple of days, I will move and protect it, and suggest a different name if necessary. When you announce a paper, you should consider whether (A) you want it automatically forwarded to other groups, like NEURON-DIGEST, (which gets posted to comp.ai.neural-networks) and if you want to provide (B) free or (C) prepaid hard copies for those unable to use FTP. If you do offer hard copies, be prepared for an onslaught. One author reported that when they allowed combination AB, the rattling around of their "free paper offer" on the worldwide data net generated over 2000 hardcopy requests! Experience dictates the preferred paradigm is to announce an FTP only version with a prominent "**DO NOT FORWARD TO OTHER GROUPS**" at the top of your announcement to the connectionist mailing list. Current naming convention is author.title.filetype[.Z] where title is enough to discriminate among the files of the same author. The filetype is usually "ps" for postscript, our desired universal printing format, but may be tex, which requires more local software than a spooler. Very large files (e.g. over 200k) must be squashed (with either a sigmoid function :) or the standard unix "compress" utility, which results in the .Z affix. To place or retrieve .Z files, make sure to issue the FTP command "BINARY" before transfering files. After retrieval, call the standard unix "uncompress" utility, which removes the .Z affix. An example of placing a file is attached as an appendix, and a shell script called Getps in the directory can perform the necessary retrival operations. For further questions contact: Jordan Pollack Assistant Professor CIS Dept/OSU Laboratory for AI Research 2036 Neil Ave Email: pollack at cis.ohio-state.edu Columbus, OH 43210 Phone: (614) 292-4890 Here is an example of naming and placing a file: gvax> cp i-was-right.txt.ps rosenblatt.reborn.ps gvax> compress rosenblatt.reborn.ps gvax> ftp archive.cis.ohio-state.edu Connected to archive.cis.ohio-state.edu. 220 archive.cis.ohio-state.edu FTP server ready. Name: anonymous 331 Guest login ok, send ident as password. Password:neuron 230 Guest login ok, access restrictions apply. ftp> binary 200 Type set to I. ftp> cd pub/neuroprose/Inbox 250 CWD command successful. ftp> put rosenblatt.reborn.ps.Z 200 PORT command successful. 150 Opening BINARY mode data connection for rosenblatt.reborn.ps.Z 226 Transfer complete. 100000 bytes sent in 3.14159 seconds ftp> quit 221 Goodbye. gvax> mail pollack at cis.ohio-state.edu Subject: file in Inbox. Jordan, I just placed the file rosenblatt.reborn.ps.Z in the Inbox. The INDEX sentence is "Boastful statements by the deceased leader of the neurocomputing field." Please let me know when it is ready to announce to Connectionists at cmu. BTW, I enjoyed reading your review of the new edition of Perceptrons! Frank ------------------------------------------------------------------------ How to FTP Files from the NN-Bench Collection --------------------------------------------- 1. Create an FTP connection from wherever you are to machine "pt.cs.cmu.edu" (128.2.254.155). 2. Log in as user "anonymous" with password your username. 3. Change remote directory to "/afs/cs/project/connect/bench". Any subdirectories of this one should also be accessible. Parent directories should not be. 4. At this point FTP should be able to get a listing of files in this directory and fetch the ones you want. Problems? - contact us at "nn-bench-request at cs.cmu.edu". From jose at tractatus.siemens.com Mon Nov 2 08:19:18 1992 From: jose at tractatus.siemens.com (Steve Hanson) Date: Mon, 2 Nov 1992 08:19:18 -0500 (EST) Subject: Registration Message-ID: <0exGfKC1GEMn8h8Uo_@tractatus> "Register early and often..." As an election day bonus for voting.. we have extended the PREREGISTRATION DEADLINE for NIPS*92 to the End of the WEEK (Nov. 6, 1992) Please send your completed registration form to NIPS*92 Registration Siemens Research Center 755 College Rd. East Princeton, NJ 08540 Stephen J. Hanson Learning Systems Department SIEMENS Research 755 College Rd. East Princeton, NJ 08540 From jose at tractatus.siemens.com Mon Nov 2 08:39:34 1992 From: jose at tractatus.siemens.com (Steve Hanson) Date: Mon, 2 Nov 1992 08:39:34 -0500 (EST) Subject: NIPS*92 Message-ID: NOTE that NIPS*92 is being held at new hotel for the first time this year. It will be Downtown CITY-CENTER MARIOTT in DENVER. This is a (as the name suggests) a centrally located hotel which we have a 72$ rate. You must Register this week in order to ensure this speical discount Rate. Please do so ASAP. To Make reservations call 303-297-1300 and be sure to mention you are with the NIPS*92 Group. Steve Hanson NIPS*92 General Chair Stephen J. Hanson Learning Systems Department SIEMENS Research 755 College Rd. East Princeton, NJ 08540 From sontag at control.rutgers.edu Mon Nov 2 05:49:15 1992 From: sontag at control.rutgers.edu (Eduardo Sontag) Date: Mon, 2 Nov 92 10:49:15 GMT Subject: "Neural networks with real weights: analog computational complexity" Message-ID: <9211021549.AA09420@control.rutgers.edu> Title: "Neural networks with real weights: analog computational complexity" Authors: Hava T. Siegelmann and Eduardo D. Sontag (Report SYCON 92-05, September 1992. 24 + i pp.) (Placed in neuroprose archive; filename: siegelmann.analog.ps.Z) Abstract: We pursue a particular approach to analog computation, based on dynamical systems of the type used in neural networks research. Our systems have a fixed structure, invariant in time, corresponding to an unchanging number of ``neurons''. If allowed exponential time for computation, they turn out to have unbounded power. However, under polynomial-time constraints there are limits on their capabilities, though being more powerful than Turing Machines. (A similar but more restricted model was shown to be polynomial-time equivalent to classical digital computation in previous work.) Moreover, there is a precise correspondence between nets and standard non-uniform circuits with equivalent resources, and as a consequence one has lower bound constraints on what they can compute. This relationship is perhaps surprising since our analog devices do not change in any manner with input size. We note that these networks are not likely to solve polynomially NP-hard problems, as the equality ``P = NP'' in our model implies the almost complete collapse of the standard polynomial hierarchy. In contrast to classical computational models, the models studied here exhibit at least some robustness with respect to noise and implementation errors. To obtain copies of this article: unix> ftp archive.cis.ohio-state.edu (or 128.146.8.52) Name : anonymous Password: ftp> cd pub/neuroprose ftp> binary ftp> get siegelmann.analog.ps.Z ftp> quit unix> uncompress siegelmann.analog.ps.Z unix> lpr -Pps siegelmann.analog.ps.Z (or however you print PostScript) (With many thanks to Jordan Pollack for providing this valuable service!) Please note: the file requires a fair amount of memory to print. If you have problems with FTP, I can e-mail you the postscript file; I cannot provide hardcopy, however. From nelsonde%avlab.dnet at aaunix.aa.wpafb.af.mil Mon Nov 2 15:06:16 1992 From: nelsonde%avlab.dnet at aaunix.aa.wpafb.af.mil (nelsonde%avlab.dnet@aaunix.aa.wpafb.af.mil) Date: Mon, 2 Nov 92 15:06:16 -0500 Subject: Four Papers Available Message-ID: <9211022006.AA06603@aaunix.aa.wpafb.af.mil> I N T E R O F F I C E M E M O R A N D U M Date: 02-Nov-1992 02:56pm EST From: DALE E. NELSON NELSONDE Dept: AAAT-1 Tel No: 57646 TO: Remote Addressee ( _AAUNIX::"CONNECTIONISTS at CS.CMU.EDU" ) Subject: Four Papers Available ********* DO NOT POST TO OTHER NETS ************* ********* DO NOT POST TO OTHER NETS ************* Prediction of Chaotic Time Series Using Cascade Correlation: Effects of Number of Inputs and Training Set Size Dale E. Nelson D. David Ensley Maj Steven K. Rogers, PhD ABSTRACT Most neural networks have been used for problems of classification. We have undertaken a study using neural networks to predict continuous valued functions which are aperiodic or chaotic. In addition, we are considering a relatively new class of neural networks, ontogenic neural networks. Ontogenic neural networks are networks which generate their own topology during training. Cascade Correlation2 is one such network. In this study we used the Cascade Correlation neural network to answer two questions regarding prediction. First, how do the number of inputs affect prediction accuracy. Second, how do the number of training exemplars affect prediction accuracy. For these experiments, the Mackey-Glass equation was used with a Tau value of 17 which yields a correlation dimension of 2.1. Takens' theorem7 for this data set states that the number of inputs to obtain a smooth mapping should be 3 to 5. We were experimentally able to verify this. Experiments were run varying the number of training exemplars from 50 to 450. The results showed that there is an overall trend towards lower predictive RMS error with a greater number of exemplars. However, there are good results obtained with only 50 exemplars which we are unable to explain at this time. In addition to these results, we discovered that the way in which predictive accuracy is generally represented, a graph of Mackey-Glass with the network output superimposed, can lead to erroneous conclusions! This paper is NOT available from Neuroprose. For paper copies send E-Mail with your mailing address to : nelsonde%avlab.dnet%aa.wpafb.af.mil DO NOT REPLY TO ENTIRE NETWORK...DO NOT USE REPLY MODE! ********* DO NOT POST TO OTHER NETS ************* ********* DO NOT POST TO OTHER NETS ************* ********* DO NOT POST TO OTHER NETS ************* ********* DO NOT POST TO OTHER NETS ************* A Taxonomy of Neural Network Optimality Dale E. Nelson Maj Steven K. Rogers, PhD ABSTRACT One of the long-standing problems with neural networks is how to decide on the correct topology for a given application. For many years the accepted approach was to use heuristics to "get close", then experiment to find the best topology. In recent years methodologies like the Abductory Inference Mechanism (AIM) from AbTech Corporation and Cascade Correlation from Carnegie Mellon University have emerged. These ontogenic (topology synthesizing) neural networks develop their topology by deciding when and what kind of nodes to add to the network during the training phase. Other methodologies examine the weights and try to "improve" by pruning some of the weights. This paper discusses the criteria which can be used to decide when one network topology is better than another. The taxonomy presented in this paper can be used to decide on methods for comparison of different neural network paradigms. Since the criteria for determining what is an optimum network is highly application specific, no attempt is made to propose the one right criteria. This taxonomy is a necessary step toward achieving robust ontogenic neural networks. This paper is NOT available from Neuroprose. For paper copies send E-Mail with your mailing address to : nelsonde%avlab.dnet%aa.wpafb.af.mil DO NOT REPLY TO ENTIRE NETWORK...DO NOT USE REPLY MODE! ********* DO NOT POST TO OTHER NETS ************* ********* DO NOT POST TO OTHER NETS ************* ********* DO NOT POST TO OTHER NETS ************* ********* DO NOT POST TO OTHER NETS ************* APPLYING CASCADE CORRELATION TO THE EXTRAPOLATION OF CHAOTIC TIME SERIES David Ensley Dale E. Nelson ABSTRACT Attempting to find near-optimal architectures, ontogenic neural networks develop their own architectures as they train. As part of a project entitled "Ontogenic Neural Networks for the Prediction of Chaotic Time Series," this paper presents findings of a ten-week research period on using the Cascade Correlation ontogenic neural network to extrapolate (predict) a chaotic time series generated from the Mackey-Glass equation. Truer, more informative measures of extrapolation accuracy than currently popular measures are presented. The effects of some network parameters on extrapolation accuracy were investigated. Sinusoidal activation functions turned out to be best for our data set. The best range for sigmoidal activation functions was [-1, +1]. One experiment demonstrates that extrapolation accuracy can be maximized by selecting the proper number of training exemplars. Though surprisingly good extrapolations have been obtained, there remain pitfalls. These pitfalls are discussed along with possible methods for avoiding them. This paper is NOT available from Neuroprose. For paper copies send E-Mail with your mailing address to : nelsonde%avlab.dnet%aa.wpafb.af.mil DO NOT REPLY TO ENTIRE NETWORK...DO NOT USE REPLY MODE! ********* DO NOT POST TO OTHER NETS ************* ********* DO NOT POST TO OTHER NETS ************* ********* DO NOT POST TO OTHER NETS ************* ********* DO NOT POST TO OTHER NETS ************* APPLYING THE ABDUCTORY INDUCTION MECHANISM (AIM) TO THE EXTRAPOLATION OF CHAOTIC TIME SERIES Dennis S. Buck Dale E. Nelson ABSTRACT This paper presents research done as part of a large effort to develop ontogenic (topology synthesizing) neural networks. One commerically available product, considered an ontogenic neural network, is the Abductory Induction Mechanism (AIM) program from AbTech Corporation of Charlottesville, Virginia. AIM creates a polynomial neural network of the third order during training. The methodology will discard any inputs it finds having a low relevance to predicting the training output. The depth and complexity of the network is controlled by a user-set Complexity Penalty Multiplier (CPM). This paper presents results of using AIM to predict the output of the Mackey-Glass equation. Comparisons are made based on the RMS error for an iterated prediction of 100 time steps beyond the training set. The data set was developed using a Tau value of 17 which yields a correlation dimension (an approximation of the fractal dimension) of 2.1. We explored the effect of different CPM values and found that a CPM value of 4.8 gives the best predictive results with the least computational complexity. We also conducted experiments using 2 to 10 inputs and 1 to 3 outputs. We found that AIM chose to use only 2 or 3 inputs, due to its ability to eliminate unnecessary inputs. This leads to the conclusion that Takens' theorem cannot be experimentally verified by this methodology! Our experiments showed that using 2 or 3 outputs, thus forcing the network to learn the first and second derivative of the equation, produced the best predictive results. We also discovered that the final network produced a predictive RMS error lower than the Cascade Correlation method with far less computational time. This paper is NOT available from Neuroprose. For paper copies send E-Mail with your mailing address to : nelsonde%avlab.dnet%aa.wpafb.af.mil DO NOT REPLY TO ENTIRE NETWORK...DO NOT USE REPLY MODE! ********* DO NOT POST TO OTHER NETS ************* ********* DO NOT POST TO OTHER NETS ************* From mozer at dendrite.cs.colorado.edu Mon Nov 2 16:35:14 1992 From: mozer at dendrite.cs.colorado.edu (Michael C. Mozer) Date: Mon, 2 Nov 1992 14:35:14 -0700 Subject: Connectionist Models Summer School 1993 Message-ID: <199211022135.AA27213@neuron.cs.colorado.edu> CALL FOR APPLICATIONS CONNECTIONIST MODELS SUMMER SCHOOL University of Colorado Boulder, Colorado June 21 - July 3, 1993 The University of Colorado will host the 1993 Connectionist Models Summer School from June 21 to July 3, 1993. The purpose of the summer school is to provide training to promising young researchers in connectionism (neural networks) by leaders of the field and to foster interdisciplinary collaboration. This will be the fourth such program in a series that was held at Carnegie-Mellon in 1986 and 1988 and at UC San Diego in 1990. Previous summer schools have been extremely successful and we look forward to the 1993 session with anticipation of another exciting event. The summer school will offer courses in many areas of connectionist modeling, with emphasis on artificial intelligence, cognitive science, cognitive neuroscience, theoretical foundations, and computational methods. Visiting faculty (see list of invited faculty below) will present daily lectures and tutorials, coordinate informal workshops, and lead small discussion groups. The summer school schedule is designed to allow for significant interaction among students and faculty. As in previous years, a proceedings of the summer school will be published. Applications will be considered only from graduate students currently enrolled in Ph.D. programs. About 50 students will be accepted. Admission is on a competitive basis. Tuition will be covered for all students, and we expect to have scholarships available to subsidize housing and meal costs, which will run approximately $300. Applications should include the following materials: * a one-page statement of purpose, explaining major areas of interest and prior background in connectionist modeling and neural networks; * a vita, including academic history, publications (if any), and a list of relevant courses taken with instructors' names and grades received; * two letters of recommendation from individuals familiar with the applicants' work; and * if room and board support is requested, a statement from the applicant describing potential sources of financial support available (department, advisor, etc.) and the estimated extent of need. We hope to have sufficient scholarship funds available to provide room and board to all accepted students regardless of financial need. Applications should be sent to: Connectionist Models Summer School c/o Institute of Cognitive Science Campus Box 344 University of Colorado Boulder, CO 80309 All application materials must be received by March 1, 1993. Decisions about acceptance and scholarship awards will be announced April 15. If you have additional questions, please write to the address above or send e-mail to "cmss at cs.colorado.edu". Organizing Committee Jeff Elman (UC San Diego) Mike Mozer (University of Colorado) Paul Smolensky (University of Colorado) Dave Touretzky (Carnegie-Mellon) Andreas Weigend (Xerox PARC and University of Colorado) Additional faculty will include: Andy Barto (University of Massachusetts, Amherst) Gail Carpenter (Boston University) Jack Cowan (University of Chicago) David Haussler (UC Santa Cruz) Geoff Hinton (University of Toronto) Mike Jordan (MIT) John Kruschke (Indiana University) Jay McClelland (Carnegie-Mellon) Steve Nowlan (Salk Institute) Dave Plaut (Carnegie-Mellon) Jordan Pollack (Ohio State) Dave Rumelhart (Stanford) Terry Sejnowski (UC San Diego and Salk Institute) From maresch at ox.ac.uk Mon Nov 2 05:25:28 1992 From: maresch at ox.ac.uk (Denis Mareschal) Date: Mon, 2 Nov 92 10:25:28 GMT Subject: No subject Message-ID: <27777.9211021025@black.ox.ac.uk> Hi, I'm interested in applications of Neural-networks to visual-tracking. In particular, the ability to predict or anticipate futur positions based on information about the current history of the trajectory as well as THE DEVELOPMENT of this ability. I've already found works by Pearlmutter (1989) and Dobnikar, Likar and Podbregar (1989) which deal with the explicit tracking of an object in 2-D space. However, most of the stuff I've turned up seems to be geared more towards modeling explicit physiological systems (E.g.: Krauzlis & Lisberger, 1989; Deno, Keller & Crandall, 1989). Does anyone know of further works that aren't necessarily related to physiological systems? Any help would be greatly appreciated and of course a list of responses will be compiled and posted if sufficient requests are made. Thanks a lot Cheers, Denis Mareschal Department of Psychology Oxford maresch at ox.ac.uk Deno, D. C., Keller, E. L., Crandall, W.F. (1989). Dynamical neural network organization of the visual pursuit system. IEEE TRANSACTIONS ON BIOMEDICAL ENGINEERING, 36, pp. 85-92. Dobnikar, A., Likar, A., & Podbregar,D. (1989). Optimal visual tracking with artificial neural network. FIRST INTERNATIONAL CONFERENCE ON ARTIFICIAL NEURAL NETWORKS (Conf. publ. 313), London, IEE Krauzlis, R. J. & Lisberber, S.G. (1989). A control systems model of visual pursuit eye movements with realistic emergent properties. NEURAL COMPUTATION, 1, pp. 116-122. Pearlmutter, B. A. (1989). Learning state space trajectories in recurrent neural networks. In: INTERNATIONAL JOINT CONFERENCE ON NEURAL NETWORKS (Washington 1989), vol II, pp. 365-372. NY: IEEE. From jose at tractatus.siemens.com Mon Nov 2 09:22:14 1992 From: jose at tractatus.siemens.com (Steve Hanson) Date: Mon, 2 Nov 1992 09:22:14 -0500 (EST) Subject: Hotel reservation deadline for NIPS workshops In-Reply-To: <9210312353.AA05934@siemens.siemens.com> References: <9210312353.AA05934@siemens.siemens.com> Message-ID: Note that in the last 6 months that Mariott MARK Resort has been purchased by the Radisson and is now Called Radisson Vail Resort (same place, same facilties). If you have gotten reservations at the Mariott MARK Resort during this time under the NIPS*92 group, they will be honored by the Radisson. Everyone else who has yet to reserve a room at Vail, should call the Radisson as Gerry suggests ASAP. Steve NIPS*92 General Chair Stephen J. Hanson Learning Systems Department SIEMENS Research 755 College Rd. East Princeton, NJ 08540 From meyer at biologie.ens.fr Tue Nov 3 09:49:55 1992 From: meyer at biologie.ens.fr (Jean-Arcady MEYER) Date: Tue, 3 Nov 92 15:49:55 +0100 Subject: Adaptive Behavior - Table of Contents Message-ID: <9211031449.AA05812@wotan.ens.fr> The first issue of Adaptive Behavior was released in August 1992. The second is under press. For inquiries or paper submissions, please contact one of the editors: - Editor-in-Chief: Jean-Arcady Meyer, France - meyer at wotan.ens.fr - Associate Editors: Randall Beer, USA - beer at alpha.ces.cwru.edu Lashon Booker, USA - booker at starbase.mitre.org Jean-Louis Deneubourg, Belgium - sgoss at ulb.ac.be Janet Halperin, Canada - janh at zoo.utoronto.ca Pattie Maes, USA - pattie at media-lab.media.mit.edu Herbert Roitblat, USA - roitblat at uhunix.uhcc.hawaii.edu Ronald Williams, USA - rjw at corwin.ccs.northeastern.edu Stewart Wilson, USA - wilson at smith.rowland.com ============================================================================= ADAPTIVE BEHAVIOR 1:1 Table of Contents A Model of Primate Visual-Motor Conditional Learning by Andrew H. Fagg and Michael A. Arbib Postponed Conditioning: Testing a Hypothesis about Synaptic Strengthening by J. R. P. Halperin and D. W. Dunham The Evolution of Strategies for Multi-agent Environments By John J. Grefenstette Evolving Dynamical Neural Networks for Adaptive Behavior By Randall D. Beer and John C. Gallagher =============================================================================== ADAPTIVE BEHAVIOR 1:2 Table of Contents Adapted and Adaptive Properties in Neural Networks Responsible for Visual Pattern Discrimination. By J.-P. Ewert, T.W. Beneke, H. Buxbaum-Conradi, A. Dinges, S. Fingerling, M. Glagow, E. Schurg-Pfeiffer and W.W. Schwippert. Kinematic Model of a Stick Insect as an Example of a 6-legged Walking System. By U. Muller-Wilm, J. Dean, H. Cruse, H.J. Weidemann, J. Eltze and F. Pfeiffer. Evolution of Food Foraging Strategies for the Caribbean Anolis Lizard using Genetic Programming. By J.R. Koza, J.P. Rice and J. Roughgarden Behavior-based Robot Navigation for Extended Domains. By R.C. Arkin =============================================================================== From lvq at cochlea.hut.fi Tue Nov 3 12:29:14 1992 From: lvq at cochlea.hut.fi (LVQ_PAK) Date: Tue, 3 Nov 92 12:29:14 EET Subject: New version of Self-Organizing Maps PD program package Message-ID: <9211031029.AA03154@cochlea.hut.fi.hut.fi> ************************************************************************ * * * SOM_PAK * * * * The * * * * Self-Organizing Map * * * * Program Package * * * * Version 1.2 (November 2, 1992) * * * * Prepared by the * * SOM Programming Team of the * * Helsinki University of Technology * * Laboratory of Computer and Information Science * * Rakentajanaukio 2 C, SF-02150 Espoo * * FINLAND * * * * Copyright (c) 1992 * * * ************************************************************************ Some time ago we released the software package "LVQ_PAK" for the easy application of Learning Vector Quantization algorithms. Corresponding public-domain programs for the Self-Organizing Map (SOM) algorithms are now available via anonymous FTP on the Internet. "What does the Self-Organizing Map mean?", you may ask --- See the following reference, then: Teuvo Kohonen. The self-organizing map. Proceedings of the IEEE, 78(9):1464-1480, 1990. In short, Self-Organizing Map (SOM) defines a 'non-linear projection' of the probability density function of the high-dimensional input data onto the two-dimensional display. SOM places a number of reference vectors into an input data space to approximate to its data set in an ordered fashion. This package contains all the programs necessary for the application of Self-Organizing Map algorithms in an arbitrary complex data visualization task. This code is distributed without charge on an "as is" basis. There is no warranty of any kind by the authors or by Helsinki University of Technology. In the implementation of the SOM programs we have tried to use as simple code as possible. Therefore the programs are supposed to compile in various machines without any specific modifications made on the code. All programs have been written in ANSI C. The programs are available in two archive formats, one for the UNIX-environment, the other for MS-DOS. Both archives contain exactly the same files. These files can be accessed via FTP as follows: 1. Create an FTP connection from wherever you are to machine "cochlea.hut.fi". The internet address of this machine is 130.233.168.48, for those who need it. 2. Log in as user "anonymous" with your own e-mail address as password. 3. Change remote directory to "/pub/som_pak". 4. At this point FTP should be able to get a listing of files in this directory with DIR and fetch the ones you want with GET. (The exact FTP commands you use depend on your local FTP program.) Remember to use the binary transfer mode for compressed files. The som_pak program package includes the following files: - Documentation: README short description of the package and installation instructions som_doc.ps documentation in (c) PostScript format som_doc.ps.Z same as above but compressed som_doc.txt documentation in ASCII format - Source file archives (which contain the documentation, too): som_p1r2.exe Self-extracting MS-DOS archive file som_pak-1.2.tar UNIX tape archive file som_pak-1.2.tar.Z same as above but compressed An example of FTP access is given below unix> ftp cochlea.hut.fi (or 130.233.168.48) Name: anonymous Password: ftp> cd /pub/som_pak ftp> binary ftp> get som_pak-1.2.tar.Z ftp> quit unix> uncompress som_pak-1.2.tar.Z unix> tar xvfo som_pak-1.2.tar See file README for further installation instructions. All comments concerning this package should be addressed to som at cochlea.hut.fi. ************************************************************************ From C.Campbell at bristol.ac.uk Mon Nov 2 10:04:01 1992 From: C.Campbell at bristol.ac.uk (I C G Campbell) Date: 02 Nov 1992 15:04:01 +0000 (GMT) Subject: faculty position Message-ID: <16046.9211021504@irix.bristol.ac.uk> FACULTY POSITION UNIVERSITY OF BRISTOL, UNITED KINGDOM Department of Computer Science Applications are invited for a Lectureship in Computer Science now tenable. FURTHER PARTICULARS The Department is part of the Faculty of Engineering. It has a complement of eighteen full-time UFC-funded staff members, together with a further twelve full-time outside-funded staff, and two Visiting Industrial Professors: Professor J. M. Taylor (Director, Hewlett-Packard Research Laboratories, Bristol) and Professor I. M. Barron. There are three Professors in the Department: Professor M. H. Rogers, who is Head of Department, and Professors J. W. Lloyd and D. H. D. Warren. The Department has substantial research funding from ESPRIT, SERC, industry and government. The Department concentrates its research in three main areas: Logic Programming Parallel Computing Machine Intelligence although a number of other topics are also being pursued. For this appointment, we are looking for a strong candidate in any area of Computer Science, although preference will be given to candidates with research interests in Parallel Computing or Machine Intelligence. We are particularly looking for candidates whose interests will broaden and complement our current work in these areas. Current work in Parallel Computing covers a range of areas, including parallel logic programming systems and languages, memory organisation for multiprocessor architectures, shared data models for transputer-based systems, and parallel applications especially for engineering problems and computer graphics. We are seeking to broaden and strengthen this research. Candidates with a strong background in computer architecture would be particularly welcome. Current work in Machine Intelligence centres mainly on Computer Vision and Speech Processing. One major project in Computer Vision is the development of an autonomous road vehicle, based on real-time image analysis. Other research projects in Computer Vision include vehicle number plate decoding, aircraft engine inspection, and visual flow monitoring. Current work on Speech Processing within the Department concentrates on speech synthesis, but the Faculty supports a Centre for Communications Research within which there is a Speech Research Group incorporating researchers in most aspects of speech technology, including speech recognition, speech coding, speech perception, and the design of speech interfaces. There is an interest in neural network theory and neural computing elsewhere in the Faculty and we would welcome applications from candidates in this area. The Department has a flourishing undergraduate and post-graduate teaching programme and participates in degree programmes in the Engineering, Science and Arts Faculties. These programmes lead to B.Sc. degrees in Computer Science, and Computer Science with Mathematics, a B.Eng. in Computer Systems Engineering, a B.A. in Computer Science and a Modern Language, and M.Sc. degrees in Computer Science, Foundations of Artificial Intelligence, and Information Engineering. The salary will be within the Lecturer Scale and the initial placement will depend on age, qualifications and experience. The closing date for applications is 27th November 1992. Further particulars may be obtained from the Head of the Computer Science Department (tel: 0272-303584; or e-mail: barbara at bristol.uk.ac.compsci). From reggia at cs.UMD.EDU Tue Nov 3 12:36:59 1992 From: reggia at cs.UMD.EDU (James A. Reggia) Date: Tue, 3 Nov 92 12:36:59 -0500 Subject: Fellowship Position: Neural Computation in Neurology Message-ID: <9211031736.AA26595@avion.cs.UMD.EDU> Research Training Fellowship in Neural Modelling Available (MD Degree required) The Clinical Stroke Research Center at the University of Maryland School of Medicine will offer two Junior Javits research fellowships starting July 1, 1993. One of these positions provides research training in the use of neural networks in cerebrovascular disease. A clinical back- ground (MD degree and specialization in neurology) is required. The Fellowship is for two years and is research intensive, but would also usually involve some clinical work in the Stroke Center. There is substantial flexibility in the details of the research training and research work. The first year salary is anticipated to be $33,000 plus fringe benefits. To apply send a letter and curriculum vitae to Dr. Thomas Price Director, Clinical Stroke Research Center University of Maryland Hospital 22 South Greene Street Baltimore, MD 21201 Questions about the research program can be sent to: Jim Reggia reggia at cs.umd.edu From beer at ICSI.Berkeley.EDU Tue Nov 3 13:47:35 1992 From: beer at ICSI.Berkeley.EDU (Joachim Beer) Date: Tue, 03 Nov 92 10:47:35 PST Subject: workshop announcement Message-ID: <9211031847.AA13155@icsib11.ICSI.Berkeley.EDU> *************************************************** * Workshop on Software & Programming Issues for * * Connectionist Supercomputers * *************************************************** April 19-20, 1993 at International Computer Science Institute (ICSI) 1947 Center Street Berkeley, CA 94704 Sponsored by: Adaptive Solutions, Inc. ICSI Siemens AG The goal of this workshop is to bring together connectionist researchers to address software and programming issues in the framework of large scale connectionist systems. Scope and technical theme of the workshop is outlined below. Due to space considerations the workshop will be by invitation only. Interested parties are encouraged to submit a one-page proposal outlinig their work in this area by January 31. Submissions should be send to ICSI at the address above or by e-mail to beer at icsi.berkeley.edu The increased importance of ANNs for elucidating deep conceptual questions in artificial intelligence and their potential for attacking real world problems warrant the design and construction of connectionist supercomputers. Several research labs have undertaken to develop such machines. These machines will allow researchers to investigate and apply ANNs on a scale which up to now was not computationally feasible. As with other parallel hardware, the main problem is adequate software for connectionist supercomputers. Most "solutions" offer isolated instances which deal only with a limited class of particular ANN algorithms rather than providing a comprehensive programming model for this new paradigm. This approach was acceptable for small and structurally simple ANNs. However, to fully utilize the emerging connectionist supercomputers an expressive, clean, and flexible software environment is called for. This is being recognized by the developers of the connectionist supercomputers, and an intergral part of these projects is the development of an appropriate software environment. While each connectionist supercomputer project has unique goals and possibly a focus on particular application areas, it would nevertheless be very fruitful to compare how the fundamental software questions that everybody in this field faces are being approached. The following (incomplete) list outlines some of the issues: * Embedding connectionist systems in traditional software environments, eg. client/server models vs. integrated "seamless" environments. * ANN description languages * Handling of sparse and irregular nets * Facilities for mapping nets onto the underlying architecture * Handling of complete applications including embedded non-connectionist instructions * Should there be a machine independent intermediate language? What would be the disadvantages? * Software issues for dedicated embedded ANNs vs. "general purpose" connectionist supercomputers. * Graphical user interfaces for ANN systems * System support for high I/O rates (while this is a general question in comp. sci. there are nevertheless some unique problems for ANN systems in dealing with large external data sets). From crr at cogsci.psych.utah.edu Wed Nov 4 16:35:40 1992 From: crr at cogsci.psych.utah.edu (crr@cogsci.psych.utah.edu) Date: Wed, 04 Nov 92 14:35:40 -0700 Subject: paper in neuroprose / change of e-mail address announcement Message-ID: <9211042135.AA08788@cogsci.psych.utah.edu> The following paper has been placed in the Neuroprose archives at Ohio State (filename: rosenberg.scintigrams.ps.Z). Ftp instructions follow the abstract. A Neural Network that Learns to Interpret Myocardial Planar Thallium Scintigrams (This paper is to appear in Neural Computation.) Charles Rosenberg, PhD Department of Computer Science Hebrew University Jerusalem, Israel [For current address, see below] Jacob Erel, MD Department of Cardiology Sapir Medical Center - Meir General Hospital Kfar Saba, Israel Henri Atlan, MD, PhD Department of Biophysics and Nuclear Medicine Hadassah Medical Center Jerusalem, Israel ABSTRACT The planar thallium-201 ($^{201}$Tl) myocardial perfusion scintigram is a widely used diagnostic technique for detecting and estimating the risk of coronary artery disease. Interpretation is currently based on visual scoring of myocardial defects combined with image quantitation and is known to have a significant subjective component. Neural networks learned to interpret thallium scintigrams as determined by both individual and multiple (consensus) expert ratings. Four different types of networks were explored: single-layer, two-layer back-propagation (BP), BP with weight smoothing, and two-layer radial basis function (RBF). The RBF network was found to yield the best performance (94.8\% generalization by region) and compares favorably with human experts. We conclude that this network is a valuable clinical tool that can be used as a reference ``diagnostic support system'' to help reduce inter- and intra- observer variability. This system is now being further developed to include other variables that are expected to improve the final clinical diagnosis. ----------------------------------------------------- FTP INSTRUCTIONS the usual way: unix> ftp archive.cis.ohio-state.edu (or 128.146.8.52) Name: anonymous Password: ftp> cd pub/neuroprose ftp> binary ftp> get rosenberg.scintigrams.ps.Z ftp> quit unix> uncompress rosenberg.scintigrams.ps.Z unix> lpr rosenberg.scintigrams.ps Current address: Dr. Charles Rosenberg GRECC 182 VA Medical Center 500 Foothill Dr. Salt Lake City, UT 84148 (801) 582-1565 ext. 2458 FAX: (801) 583-7338 crr at cogsci.psych.utah.edu From wray at ptolemy.arc.nasa.gov Wed Nov 4 17:34:55 1992 From: wray at ptolemy.arc.nasa.gov (Wray Buntine) Date: Wed, 4 Nov 92 14:34:55 PST Subject: CP for AI and Stats Workshop Message-ID: <9211042234.AA23199@ptolemy.arc.nasa.gov> NOTE: This second call for participation contains a list of papers and posters being presented. ;;;-------------------------- Cut here ----------------------------------- 2nd Call for Participants and Schedule for Fourth International Workshop on Artificial Intelligence and Statistics January 3-6, 1993 Ft. Lauderdale, Florida PURPOSE: This is the fourth in a series of workshops which has brought together researchers in Artificial Intelligence and in Statistics to discuss problems of mutual interest. The result has been an unqualified success. The exchange has broadened research in both fields and has strongly encouraged interdisciplinary work. This workshop will have as its primary theme: ``Selecting models from data'' FORMAT: Approximately 60 papers by leading researchers in Artificial Intelligence and Statistics have been selected for presentation. To encourage interaction and a broad exchange of ideas, the presentations will be limited to 20 discussion papers in single session meetings over the three days. Focussed poster sessions, each with a short presentation, provide the means for presenting and discussing the remaining 40 research papers. Attendance at the workshop is *not* limited. The three days of research presentations will be preceded by a day of tutorials. These are intended to expose researchers in each field to the methodology used in the other field. LANGUAGE: The language will be English. FORMAT: One day of tutorials and three days of focussed poster sessions, presentations and panels. The presentations are scheduled in the mornings and evenings, leaving the afternoons free for discussions in more relaxed environments. SCHEDULE: Sun: Jan. 3rd. -------------- Sunday is scheduled for tutorials. There are 4 -- at most two can be attended without conflict. AI for statisticians Morning: Doug Fisher -- Intro. to learning including neural networks Afternoon: Judea Pearl -- Graphical models, causal reasoning, and qualitative decision making. Statistics for AI Morning: Wray Buntine -- Introduction to Statistics and Decision Analysis Afternoon: Daryl Pregibon -- Overview of Statistical Models Mon: Jan. 4th. --------------- 8:30--10:00 1st. Session---Model Selection Peter Cheeseman--Introduction: "Overview of Model Selection" Beat E. Neuenschwander, Bernard D. Flury, "Principal Components and Model Selection". Cullen Schaffer, "Selecting a Classification Method by Cross-Validation". Stanley Sclove, "Small-Sample and Large-Sample Statistical Model Selection Criteria". -------------------------------------------------------- 10:00--10:30 break -------------------------------------------------------- 10:30--12:00 2nd. Session---Model Comparison C. Feng, A. Sutherland, R. King, S. Muggleton, R. Henery, Comparison of Classification Algorithms in Machine Learning, Statistics, and Neural Networks (DRAFT). Richard D. De Veaux, "A Tale of Two Nonparametric Estimation Schemes: MARS and Neural Networks". Christopher de Vaney, "A Support Architecture for Statistical Meta-Information with Knowledge-Based Extensions". + Discussion (speakers and audience) --------------------------------------------------------- Lunch (provided) --------------------------------------------------------- 1:30--3:00 1st panel--Alternative Approaches to Model Selection Panel Moderator: Wayne Oldford --------------------------------------------------------- 3:00--3:30 break --------------------------------------------------------- 3:30--5:00 3rd. Session---Statistics in AI Nathaniel G. Martin, James F. Allen, "Statistical Probabilities for Planning". Arcot Rajasekar, "On Closures in Knowledge Base Systems". Steffen L. Lauritzen, B. Thiesson, DJ Spiegelhalter, "Diagnostic Systems Created by Model Selection Methods-A Case Study". Vladimir Cherkassky, "Statistical and Neural Network Techniques For Nonparametric Regression". -------------------------------------------------------- -------------------------------------------------------- Tue: Jan. 5th. -------------- 8:30--10:00 4th Session---Causal Models Floriana Esposito, Donato Malerba, Giovanni Semeraro, "Comparison of Statistical Methods for Inferring Causation". J. Pearl and N. Wermuth, "When Do Association Graphs have Causal Explanations". Richard Scheines, "Inferring Causal Structure Among Unmeasured Variables". + Invited speaker -------------------------------------------------------- 10:00--10:30 break -------------------------------------------------------- 10:30--12:00 5th Session---Very Short "poster" presentations -------------------------------------------------------- break--rest of afternoon off -------------------------------------------------------- 6:00 -7:30 buffet supper (provided) 7:30 -8:40 1st poster session (see list of posters at end) 8:50 -10:00 2nd poster session (preceded by 10 minute changeover) -------------------------------------------------------- -------------------------------------------------------- Wed: Jan. 6th. -------------- 8:30--10:00 6th Session---Influence Diagrams and Probabilistic Networks Remco R Bourckaert, "Conditional Dependence in Probabilistic Networks". Geoffrey Rutledge MD, Ross Shachter, "A Method for the Dynamic Selection of Models Under Time Constraints". Gregory M. Provan, "Diagnosis Over Time Using Temporal Influence Diagrams". + Discussion (speakers and audience) -------------------------------------------------------- 10:00--10:30 break -------------------------------------------------------- 10:30--12:00 7th Session---AI in Statistics R. W. Oldford, D. G. Anglin, "Modelling Response Models in Software". D. J. Hand, "Statistical Strategy: Step 1". David Draper, "Assessment and Propagation of Model Uncertainty". Debby Keen, Arcot Rajasekar, "Reasoning With Inductive Dependencies" --------------------------------------------------------- Lunch (provided) --------------------------------------------------------- 1:30--3:00 2nd panel 3:00--Business meeting -----------------------Posters-------------------------------- Russell G. Almond, "An Ontology for Graphical Models". D.L. Banks, R.A. Maxion, "Comparative Evaluation of New Wave Methods for Model Selection". Raj Bhatnagar, Laveen N Kanal, "Models from Data for Various Types of Reasoning". Djamel Bouchaffra, Jacques Rouault, "Different ways of capturing the observations in a nonstationary hidden Markov model: application to the problem of Morphological Ambiguities". Victor L. Brailovsky, "Model selection by perturbing data set (extended abstract)". Carla E. Brodley, Paul Utgoff, "Dynamic Recursive Model Class Selection for Classifier Construction Extended Abstract". W. Buntine, "On Generic Priors in Learning". Paul R. Cohen, "Path Analysis Models of an Autonomous Agent in a Complex Environment". Sally Jo Cunningham, Paul Denize, "A Tool for Model Genertion and Knowledge Acquisition". Luc Devroye, Oliver Kamoun, "Probabilistic Min-Max Trees". E. Diday, P. Brito and E. Mfoumoune, "Modelling Probabilistic Data by Conceptual Pyramidal Clustering". Kris Dockx, James Lutsko, "SA/GA: Survival of the Fittest in Alaska". Zbigniew Duszak, Jerzy Grzymala-Busse, Waldemar W. Koczkoda, "Rule Induction Based on Statistics and Rough Set Theory". J. J. Faraway, "Choise of Order in Regression Strategy". Karina Gibert, "Combining a Knowledge-based System and a Clustering Method For an Inductive Construction of Models". Scott D. Goodwin, Eric Neufeld, Andre Trudel, "Extrapolating Definite Integral Information". Jonathan Gratch, Gerald DeJong, "Rational Learning: Finding a Balance Between Utility and Efficiency". A. K. Gupta, "Information Theoretic Approach to Some Multivariate Tests of Homogeneity". Paula Hietala, "Statistical Reasoning to Enhance User Modelling in Consulting Systems". Adele Howe, Paul R. Cohen, "Detecting and Explaining Dependencies in Execution Traces". Sung-Ho Kim, "On Combining Conditional Influence Diagrams". Willi Klosgen, "Discovery in Databases". G. J. Knafl, A. Semrl, "Software Reliability Expert (SRX)". Bing Leng, Bruce Buchanan, "Using Knowledge-Assisted Discriminant Analysis to Generate New Comparative Terms for Symblic Learner". James F. Lutsko, Bart Kuijpers, "Simulated Annealing in the Construction of Near-Optimal Decision Trees". Yong Ma, David Wilkins, John S. Chandler, "An Extended Bayesian Belief Function Approach to Handle Noise in Inductive Learning". Izhar Matzkevich, Bruce Abramson, "Towards Prior Compromise in Belief Networks (Extended Abstract)". Johnathan Oliver, "Decision Graphs - An Extension of Decision Trees". Egmar Rodel, "A Knowledge Based System for Testing Bivariate Dependence". A.R. Runnaalls, "Global vs Local Sampling Procedures for Inference on Directed Graphs". David Russell, "Statistical Inferencing in a Real-Time Heuristic Controller". Geoffrey Rutledge MD, Ross Shachter, "A Method for the Dynamic Selection of Models Under Time Constraints". Steven Salzberg, David Aha, "Learning to Catch: Applying Nearest Neighbor algorithms to Dynamic Control Tasks". D. Moreira dos Santos, "Selecting a Frailty Model for Longitudinal Breast Cancer Data". Glenn Shafer, "Recursion in Join Trees". P. Shenoy, "Searching For Alternative Representation of Data: A Case for Tetrad". Hidetoshi Shimodaira, "A New Criterion for Selecting Models from Partially Observed Data". P. Smyth, "The Nature of Class Labels in Supervised Learning". Peter Spirtes, Clark Glymour, "Inference, Intervention and Prediction". Marco Valtora, R. Mechling, "PaCCIN: A parallel Constructor of Markov Networks". Aaron Wallack, Ed Nicolson, "Optimal Design of Reflective Sensors Using Probabilistic Analysis". Bradley Whitehall, David Sirag, "Clustering of Smybolically Described Events for Prediction of Numeric Attributes". Nevin Lianwen Zhang, Runping Qi, David Poole, "Minizing Decision Table Sizes in Stepwise-Decomposable Influence Diagrams". Ping Zhang, "On the Choise of Penalty term in Generalized FPE criterion". PROGRAM COMMITTEE: General Chair: R.W. Oldford U. of Waterloo, Canada Programme Chair: P. Cheeseman NASA (Ames), USA Members: W. Buntine NASA (Ames), USA Wm. Dumouchel BBN, USA D.J. Hand Open University, UK W.A. Gale AT&T Bell Labs, USA H. Lenz Free University, Germany D. Lubinsky AT&T Bell Labs, USA M. Deutsch-McLeish U. of Guelph, Canada E. Neufeld U. of Saskatchewan, Canada J. Pearl UCLA, USA D. Pregibon AT&T Bell Labs, USA P. Shenoy U. of Kansas, USA P. Smythe JPL, USA SPONSORS: Society for Artificial Intelligence And Statistics International Association for Statistical Computing REGISTRATION: All fees paid: Before Dec 1, 1992 After Dec 1, 1992 Scientific programme: $225 $275 Full-time Students $135 $175 - Registration fee includes three continental breakfasts and two lunches supplied at the workshop site. - Students must supply proof of full-time student status (at the workshop) to be eligible for reduced rates. A REGISTRATION FORM APPEARS AT THE END OF THIS MESSAGE. TUTORIALS: There are four three hour tutorials planned. Two introducing statistical methodology to AI researchers and two introducing AI methodology to statistical researchers. Before Dec 1, 1992 After Dec 1, 1992 Per Tutorial $65 $75 Full-time Students $40 $45 The tutorials are introductions to the following topics: 1. Learning, including a discussion of neural networks. Speaker: Doug Fisher, Vanderbilt University Orientation: AI for statisticians 2. Graphical models, causal reasoning, and qualitative decision making. Speaker: Judea Pearl, UCLA Orientation: AI for statisticians. 3. Overview of statistical models. Emphasis on generalised linear and additive models. Speaker: Daryl Pregibon, AT&T Bell Labs Orientation: Statistics for AI researchers. 4. Introduction to Statistics. General introduction to statistical topics Speaker: Wray Buntine, NASA Ames Orientation: Statistics for AI researchers. Please indicate which tutorial(s) you are registering for. PAYMENT OF FEES: All workshop fees are payable by cheque or money order in U.S. dollars (drawn on a U.S. bank) to the Society for Artificial Intelligence and Statistics. Send cheque or money order to: R.W. Oldford Chair, 4th Int'l Workshop on A.I. & Stats. Dept. of Statistics & Actuarial Science University of Waterloo Waterloo, Ontario N2L 3G1 CANADA NOTE: ACCOMODATIONS MUST BE ARRANGED DIRECTLY WITH THE HOTEL. ACCOMODATION: We have arranged for a block of rooms to be available to participants at the Workshop site hotel for $85 per night (single or double + tax). Arrangements must be made directly with the hotel. Please mention the Workshop on all communications. Rates are available Jan 1 to Jan 10 (if booked before Dec 17, 1992). Pier 66 Resort and Marina 2301 S.E. 17th Street Causeway Ft. Lauderdale, Florida 33316 (305) 525 6666 (800) 327 3796 (USA only) (800) 432 1956 (Florida only) Fax: (305) 728 3551 Telex: 441-650 REGISTRATION FORM: 4th International Workshop on AI and Statistics January 3-6, 1993 Ft. Lauderdale, Florida Name: _______________________________ Affiliation: _______________________________ Address: _____________________________________________ _____________________________________________ _____________________________________________ _____________________________________________ e-mail: _____________________________________________ Fax: ___________________________ Phone: ___________________________ Scientific Programme Registration ...................... US$___________ Tutorial 1. Learning ................................... US$___________ Tutorial 2. Causal Reasoning ........................... US$___________ Tutorial 3. Statistical Models ......................... US$___________ Tutorial 4. Introduction to Statistics ................. US$___________ _______________________________________________________________________ Total Payment .......................................... US$___________ From gmk at osprey.siemens.com Wed Nov 4 12:33:46 1992 From: gmk at osprey.siemens.com (Gary M. Kuhn) Date: Wed, 4 Nov 92 12:33:46 EST Subject: Call for Papers, NNSP'93 Message-ID: <9211041733.AA00365@osprey.siemens.com> CALL FOR PAPERS _______________ 1993 IEEE Workshop on Neural Networks for Signal Processing September 7-9, 1993 Baltimore, MD, USA Sponsored by the IEEE Technical Committee on Neural Networks in cooperation with the IEEE Neural Networks Council The third of a series of IEEE workshops on Neural Networks for Signal Processing will be held at the Maritime Institute of Technology and Graduate Studies, Linthicum, Maryland, USA, in September of 1993. Papers are solicited for, but not limited to, the following topics: 1. Applications: Image processing and understanding, speech recognition, communications, sensor fusion, medical diagnoses, nonlinear adaptive filtering and other general signal processing and pattern recognition topics. 2. Theory: Neural network system theory, identification and spectral estimation, and learning theory and algorithms. 3. Implementation: Digital, analog, and hybrid technologies and system development. Prospective authors are invited to submit 4 copies of extended summaries of no more than 6 pages. The top of the first page of the summary should include a title, authors' names, affiliations, address, telephone and fax numbers and email address if any. Camera-ready full papers of accepted proposals will be published in a hard-bound volume by IEEE and distributed at the workshop. Due to workshop facility constraints, attendance will be limited with priority given to those who submit written technical contributions. For further information, please contact Karin Cermele at the NNSP'93 Princeton office, (Tel.) +1 609 734 3383, (Fax) +1 609 734 6565, (e-mail) kic at learning.siemens.com. PLEASE SEND PAPER SUBMISSIONS TO: _______________ NNSP'93 Siemens Corporate Research 755 College Road East Princeton, NJ 08540 USA SCHEDULE _______________ Submission of extended summary: February 15 Notification of acceptance: April 19 Submission of photo-ready paper: June 1 Advanced registration, before: June 1 WORKSHOP COMMITTEE _______________ General Chairs Gary Kuhn Barbara Yoon Siemens Corporate Research DARPA-MTO 755 College Road East 3701 N. Fairfax Dr. Princeton, NJ 08540, USA Arlington, VA 22203-1714 USA gmk at learning.siemens.com byoon at a.darpa.mil Program Chair Proceedings Chair Rama Chellappa Candace Kamm Dept. of Electrical Engineering Box 1910 University of Maryland Bellcore, 445 South Street College Park, MD 20742, USA Morristown, NJ 07962, USA chella at eng.umd.edu cak at bellcore.com Finance Chair Raymond Watrous Siemens Corporate Research 755 College Road East Princeton, NJ 08540, USA watrous at learning.siemens.com Program Committee Joshua Alspector John Makhoul Les Atlas B.S. Manjunath Charles Bachmann Tomaso Poggio Gerard Chollet Jose Principe Frank Fallside Ulrich Ramacher Lee Giles Noboru Sonehara S.J. Hanson Eduardo Sontag Y.H. Hu J.A.A. Sorensen B.H. Juang Yoh'ichi Tohkura Shigeru Katagiri Christoph von der Malsburg S.Y. Kung Christian Wellekens Yann LeCun From cohn at psyche.mit.edu Wed Nov 4 15:52:45 1992 From: cohn at psyche.mit.edu (David Cohn) Date: Wed, 4 Nov 92 15:52:45 EST Subject: Post-NIPS Robot Learning workshop program Message-ID: <9211042052.AA08017@psyche.mit.edu> ___________________________________________________________________________ PROGRAM FOR THE POST-NIPS WORKSHOP "ROBOT LEARNING" Vail, Colorado, Dec 5th, 1992 NIPS=92 Workshop: Robot Learning ================= Intended Audience: Connectionists and Non-Connectionists in Robotics, ================== Control, and Active Learning Organizers: =========== Sebastian Thrun (CMU) Tom Mitchell (CMU) David Cohn (MIT) thrun at cs.cmu.edu mitchell at cs.cmu.edu cohn at psyche.mit.edu Program: ======== Robot learning has grasped the attention of many researchers over the past few years. Previous robotics research has demonstrated the difficulty of manually encoding sufficiently accurate models of the robot and its environment to succeed at complex tasks. Recently a wide variety of learning techniques ranging from statistical calibration techniques to neural networks and reinforcement learning have been applied to problems of perception, modeling and control. Robot learning is characterized by sensor noise, control error, dynamically changing environments and the opportunity for learning by experimentation. This workshop will provide a forum for researchers active in the area of robot learning and related fields. It will include informal tutorials and presentations of recent results, given by experts in this field, as well as significant time for open discussion. Problems to be considered include: How can current learning robot techniques scale to more complex domains, characterized by massive sensor input, complex causal interactions, and long time scales? How can previously acquired knowledge accelerate subsequent learning? What representations are appropriate and how can they be learned? Although each session has listed "speakers," the intent is that each speaker will not simply present their own work, but will introduce their work interactively, as a launching point for group discussion on their chosen area. After all speakers have finished, the remaining time will be used to discuss at length issues that the group feels need most urgently to be addressed. Below, we have listed the tentative agenda, which is followed by brief abstracts of each author's topic. For those who wish to get a head start on the workshop, we have included a list of references and/or recommended readings, some of which are available by anonymous ftp. ===================================================================== ===================================================================== AGENDA ===================================================================== ===================================================================== SESSION ONE (early morning session), 7:30 - 9:30: ------------------------------------------------- TITLE: "Robot learning: scaling up and state of the art" Keynote speaker: Chris Atkeson (30 min) "Paradigms for Robot Learning" Speakers: Steve Hanson (15 min) (title to be announced) Satinder Singh (15 min) Behavior-Based Reinforcement Learning Andrew W. Moore(15 min) The Parti-Game Algorithm for Variable Resolution Reinforcement Learning Richard Yee (15 min) Building Abstractions to Accelerate Weak Learners SESSION TWO (apres-ski session), 4:30 - 6:30: --------------------------------------------- PANEL: "Robot learning: Where are the new ideas coming from?" Keynote speaker: Andy Barto (30 min) Speakers: Tom Mitchell (10 min each) Chris Atkeson Dean Pomerleau Steve Suddarth ===================================================================== ===================================================================== ABSTRACTS ===================================================================== Session 1: Scaling up and the state of the art When: Saturday, Dec 5, 7:30-9:30 a.m. ===================================================================== ===================================================================== Keynote: Chris Atkeson (cga at ai.mit.edu) Title: Paradigms for Robot Learning Abstract: This talk will survey a variety of robot learning tasks and learning paradigms to perform those tasks. The tasks include pattern classification, regression/function approximation, root finding, function optimization, designing feedback controllers, trajectory following, stochastic modeling, stochastic control, and strategy generation. Given this wide range of tasks it seems reasonable to ask if there is any commonality among them, or any way in which solving one task might make other tasks easier to perform. In our own work we have typically taken an indirect approach: our learning algorithms explicitly form models, and then solve the problem using algorithms that assume complete knowledge. It is not at all clear which learning tasks are best dealt with using an indirect approach, and which are handled better with a direct approach in which the control strategy is learned directly. Nor is it clear how to cope with uncertainty and incomplete knowledge, either by modeling it explicitly, using stochastic models, or using game theory and assuming a malevolent world. I hope to provoke a discussion on these issues. ====================================================================== Presenter: Satinder Pal Singh (singh at cs.umass.edu) Title: Behavior-Based Reinforcement Learning Abstract: Control architectures based on reinforcement learning have been successfully applied to agents/robots that use their repertoire of primitive control actions to achieve goals in an external environment. The optimal policy for any goal is a state-dependent composition of the given "primitive" policies (a primitive policy "A" assigns action A to every state). In that sense, the primitive policies form the "basis" set from which optimal solutions can be "composed". I argue that reinforcement learning can be greatly accelerated by redefining the basis set of policies available to the agent. These redefined basis policies should correspond to "behaviors" that are useful across the set of tasks faced by the agent. Behavior-based RL, i.e., the application of RL to behavior-based robotics (ref Brooks), has several advantages: it can drastically reduce the effective dimensionality of the action space, it provides a framework for incorporating prior knowledge into RL architectures, it provides a technique for achieving transfer of learning, and finally by restricting the rules of composition and the types of behaviors it may become possible to perform "robust" reinforcement learning. I will provide examples from my own work and that of others to illustrate these ideas. (Refs 4, 5, 6) ====================================================================== Presenter: Andrew W. Moore (awm at ai.mit.edu) Title The Parti-Game Algorithm for Variable Resolution Reinforcement Learning Can we efficiently learn in continuous state-spaces, while requiring only relatively few real-world experienvces during the learning stage? Dividing a continuous state-space into a fine grid can mean a tragically large number of unnecessary experiences, while a coarse grid or parametric representation can become stuck. This talk overviews a new algorithm which, in real time, tries to adaptively alter the resolution of a state space partitioning to be coarse where it can and fine where it must to be if it is to avoid becoming stuck. The key idea turns out to be the treatment of the problem as a game instead of a Markov decision task. Possible prior reading: Ref 7 (Overview of some other uses of kd-trees in Machine learning) Ref 8 (A non-real-time algorithm which uses a different partitioning strategy) Ref 9 (A search control technique which Parti-Game uses) Refs 9, 10 ====================================================================== Presenter: Richard Yee, (yee at cs.umass.edu) Title: Building Abstractions to Accelerate Weak Learners Abstract: Learning methods based on dynamic programming (DP) are promising approaches to the problem of controlling dynamical systems. Practical DP-based learning will require function approximation methods that are well-suited for learning optimal value functions, which map system states into numeric estimates of utility. Such approximation problems are generally characterized by non-stationary, dependent training data and, in many cases, little prospect for incorporating strong {\em a priori\/} learning biases. Consequently. this talk considers learning approaches that begin weakly (e.g., using rote memorization) but strengthen their learning biases as experiences accrue. Abstracting from stored experiences should accelerate learning by improving generalization. Bootstrapping such abstraction processes (cf.\ "hypothesis boosting") might be a practical means for scaling DP-based learning across a wide variety of applications. (Refs 1, 2, 3, 4) ===================================================================== Session 2: Where are the new ideas coming from? When: Saturday, Dec 5, 4:30-6:30 p.m. ===================================================================== ===================================================================== Keynote: Andrew G. Barto (barto at cs.umass.edu) Title: Reinforcement Learning Theory Although reinforcement learning is being studied more widely than ever before, especially methods based on approximating dynamic programming (DP), its theoretical foundations are not yet highly developed. In this talk, I discuss what I percieve to be the current state and the missing links in this theory. This topic raises such questions as the following: Just what is DP-based reinforcement learning from a mathematical perspective? What is the relationship between DP-based reinforcement learning and other methods for approximating DP? What theoretical justification exists for combining function approximation methods (such as artificial neural networks) with DP-based learning? What kinds of problems are best suited to DP-based reinforcement learning? Is theory important? ===================================================================== Presenter: Dean Pomerleau Title: Combining artificial neural networks and symbolic processing for autonomous robot guidance Artificial neural networks are capable of performing the reactive aspects of autonomous driving, such as staying on the road and avoiding obstacles. This talk describes an efficient technique for training individual networks to perform these reactive driving tasks. But driving requires more than a collection of isolated capabilities. To achieve true autonomy, a system must determine which capabilities should be employed in the current situation to achieve its objectives. Such goal directed behavior is difficult to implement in an entirely connectionist system. This talk describes a rule-based technique for combining multiple artificial neural networks with map-based symbolic reasoning to achieve high level behaviors. The resulting system is not only able to stay on the road, it is able follow a route to a predetermined destination, turning appropriately at intersections and stopping when it has reached its goal. (Refs 11, 12, 13, 14, 15) ===================================================================== ===================================================================== References ===================================================================== ===================================================================== (#1) Yee, Richard, "Abstraction in Control Learning", Department of Computer and Information Science, University of Massachusetts, Amherst, MA 01003, COINS Technical Report 92-16, March 1992. anonymous ftp: envy.cs.umass.edu:pub/yee.abstrn.ps.Z (#2) Barto, Andrew G. and Richard S. Sutton and Christopher J. C. H. Watkins, Sequential decision problems and neural networks, in Advances in Neural Information Processing Systems 2, 1990, Touretzky, D. S., ed. (#3) Barto, Andrew G. and Richard S. Sutton and Christopher J. C. H. Watkins", Learning and Sequential Decision Making, in Learning and Computational Neuroscience: Foundations of Adaptive Networks, 1990. anonymous ftp: archive.cis.ohio-state.edu:pub/neuroprose/barto.sequential_decisions.ps.Z (#4) Barto, Andrew G. and Steven J. Bradtke and Satinder Pal Singh, Real-time learning and control using asynchronous dynamic programming, Computer and Information Science, University of Massachusetts, Amherst, MA 01003, COINS Technical Report TR-91-57, August 1991. anonymous ftp: archive.cis.ohio-state.edu:pub/neuroprose/barto.realtime-dp.ps.Z (#5) Singh, S.P.," Transfer of Learning by Composing Solutions for Elemental Sequential Tasks, Machine Learning, 8:(3/4):323-339, May 1992. anonymous ftp: envy.cs.umass.edu:pub/singh-compose.ps.Z (#6) Singh, S.P., "Scaling reinforcement learning algorithms by learning variable temporal resolution models, Proceedings of the Ninth Machine Learning Conference, D. Sleeman and P. Edwards, eds., July 1992. anonymous ftp: envy.cs.umass.edu:pub/singh-scaling.ps.Z (#7) S. M. Omohundro, Efficient Algorithms with Neural Network Behaviour, Journal of Complex Systems, Vol 1, No 2, pp 273-347, 1987. (#8) A. W. Moore, Variable Resolution Dynamic Programming: Efficiently Learning Action Maps in Multivariate Real-valued State-spaces, in "Machine Learning: Proceedings of the Eighth International Workshop", edited by Birnbaum, L. and Collins, G., published by Morgan Kaufman. June 1991. (#9) A. W. Moore and C. G. Atkeson, Memory-based Reinforcement Learning: Converging with Less Data and Less Real Time, 1992. See the NIPS92 talk or else preprints available by request to awm at ai.mit.edu (#10) J. Peng and R. J. Williams, Efficient Search Control in Dyna, College of Computer Science, Northeastern University, March, 1992 (#11) Pomerleau, D.A., Gowdy, J., Thorpe, C.E. (1991) Combining artificial neural networks and symbolic processing for autonomous robot guidance. In {\it Engineering Applications of Artificial Intelligence, 4:4} pp. 279-285. (#12) Pomerleau, D.A. (1991) Efficient Training of Artificial Neural Networks for Autonomous Navigation. In {\it Neural Computation 3:1} pp. 88-97. (#13) Touretzky, D.S., Pomerleau, D.A. (1989) What's hidden in the hidden units? {\it BYTE 14(8)}, pp. 227-233. (#14) Pomerleau, D.A. (1991) Rapidly Adapting Artificial Neural Networks for Autonomous Navigation. In {\it Advances in Neural Information Processing Systems 3}, R.P. Lippmann, J.E. Moody, and D.S. Touretzky (ed.), Morgan Kaufmann, pp. 429-435. (#15) Pomerleau, D.A. (1989) ALVINN: An Autonomous Land Vehicle In a Neural Network. In {\it Advances in Neural Information Processing Systems 1}, D.S. Touretzky (ed.), Morgan Kaufmann, pp. 305-313. From warthman at garnet.berkeley.edu Thu Nov 5 20:54:17 1992 From: warthman at garnet.berkeley.edu (warthman@garnet.berkeley.edu) Date: Thu, 5 Nov 92 17:54:17 -0800 Subject: Audio Synthesizer Message-ID: <9211060154.AA24538@garnet.berkeley.edu> ********************** News Release ************************ November 5, 1992 ************************************************************ Neural-Network Audio Synthesizer Debuts at Paris Opera House ************************************************************ Palo Alto, California -- The old Opera House in Paris, France, will feature five performances by the Merce Cunningham Dance Company, November 12 to 17, in which a new type of audio synthesizer based on an artificial neural network will be used to generate electronic music. The synthesizer's musical accompaniment was composed and will be performed by David Tudor and his dance company colleague, Takehisa Kosugi. The audio synthesizer is built around an integrated-circuit chip from Intel Corporation in Santa Clara, California. The chip, called the Intel 80170NX electrically trainable analog neural network (ETANN), simulates the function of nerve cells in a biological brain. A remarkable range of audio effects can be generated with the electronic synthesizer -- from unique space-age and science-fiction sounds to passages that sound very much like birds, heart beats, porpoises, engines, and acoustical, percussion or string musical instruments. Sounds are generated internally by the synthesizer. External inputs such as voice, music, or random sounds can optionally be used to enrich or control the internally generated sounds. In addition to generating outputs to multiple audio speakers, the synthesizer can simultaneously drive oscilloscopes or other visual devices. The neural network chip's software consists of numeric values representing interconnection strengths between inputs and outputs -- a configuration analogous to the excitatory or inhibitory strengths of synapse connections between biological nerve cells. The artificial neurons can be connected in loops, using the programmable interconnection strengths, or they can be connected outside the chip with cables and feedback circuits. Audio oscillations occur as a result of delay in the feedback paths and thermal noise in the neural network chip. The sounds are generally rich because of the complexity of the circuitry. The concept for the synthesizer evolved from a project begun in 1989 by Forrest Warthman and David Tudor. The synthesizer was designed and built by Warthman; Mark Thorson, a hardware designer and associate editor of Microprocessor Report; and Mark Holler, Intel's program manager for neural network products. John Cage visited the design group in Palo Alto a few months before his passing away at the age of 79 this year. His observations on the synthesizer's role in musical composition and dance performance contributed to its current design. A description of the synthesizer's architecture and circuitry will appear in the February 1993 issue of Dr. Dobb's Journal. From Dave_Touretzky at DST.BOLTZ.CS.CMU.EDU Thu Nov 5 22:25:21 1992 From: Dave_Touretzky at DST.BOLTZ.CS.CMU.EDU (Dave_Touretzky@DST.BOLTZ.CS.CMU.EDU) Date: Thu, 05 Nov 92 22:25:21 EST Subject: a couple of administrative matters Message-ID: <22435.721020321@DST.BOLTZ.CS.CMU.EDU> #1. Due to the increasing size of the CONNECTIONISTS list, we are no longer able to wade through hundreds returned fail messages (most of which are spurious) to determine who is and is not receiving mail successfully. We will now ask you to contact us if you stop receiving mail. Connectionists sends out more than a message a week, so if you do not receive mail for ONE WEEK, send mail to Connectionists-Request at cs.cmu.edu with your full name, address, and where you are receiving mail. This last we need in case you are on a local redistribution list. We also ask that if your account is about to expire or you are moving, please contact us at the above address so that we can remove or update your entry. Thank you David Redish and Dave Touretzky Connectionists-Request at cs.cmu.edu ================================================================ #2. Do you wonder what happened to all the people who used to post here asking for references on random topics? The're still with us, but the list is now FILTERED and those messages are killed before they make it out to our readership. Please don't send requests for references to this list; they will be rejected. There is one exception to the above rule. If you have already compiled a substantial bibliography on some topic, you may post it along with a request for *additional* references. (People are encouraged to post useful bibliographies even if they're not looking for additions.) But if you simply can't resist the urge to post a request for references, a plea for free software, or a really elementary question about neural nets, then the folks over on comp.ai.neural-nets are the ones you should be pestering. Subscribe today! From MARCHESF%PACEVM.BITNET at BITNET.CC.CMU.EDU Sat Nov 7 16:28:14 1992 From: MARCHESF%PACEVM.BITNET at BITNET.CC.CMU.EDU (Dr. Francis T. Marchese) Date: 07 Nov 1992 16:28:14 -0500 (EST) Subject: Call for Participation Message-ID: <01GQVKHN9OAA9AMLQA@BITNET.CC.CMU.EDU> *** Call For Participation *** Conference on Understanding Images Sponsored By NYC ACM/SIGGRAPH and Pace University's School of Computer Science and Information Systems To Be Held at: Pace University New York City, New York May 21-22,1993 Artists, designers, scientists, engineers and educators share the problem of moving information from one mind to another. Traditionally, they have used pictures, words, demonstrations, music and dance to communicate imagery. However, expressing complex notions such as God and infinity or a seemingly well defined concept such as a flower can present challenges which far exceed their technical skills. The explosive use of computers as visualization and expression tools has compounded this problem. In hypermedia, multimedia and virtual reality systems vast amounts of information confront the observer or participant. Wading through a multitude of simultaneous images and sounds in possibly unfamiliar representations, a confounded user asks: What does it all mean? Since image construction, transmission, reception, decipherment and ultimate understanding are complex tasks strongly influenced by physiology, education and culture; and since electronic media radically amplify each processing step, then we, as electronic communicators, must determine the fundamental paradigms for composing imagery for understanding. Therefore, the purpose of this conference is to bring together a breadth of disciplines, including, but not limited to, the physical, biological and computational sciences, technology, art, psychology, philosophy and education, in order to define and discuss the issues essential to image understanding within the computer graphics context. To this end we seek proposals for individual presentations, panel discussions, static displays, interactive environments, performances and beyond. Submissions: Contributors are requested to submit a one page proposal by January 15, 1993. Accepted presentations will be included in the proceedings. Direct all inquires and submissions to: Professor Francis T. Marchese Department of Computer Science Pace University New York, NY 10038 USA Email: MARCHESF at PACEVM.Bitnet Phone: 212-346-1803 Fax: 212-346-1933 From jbower at cns.caltech.edu Mon Nov 9 12:54:23 1992 From: jbower at cns.caltech.edu (Jim Bower) Date: Mon, 9 Nov 92 09:54:23 PST Subject: extended claims Message-ID: <9211091754.AA17739@smaug.cns.caltech.edu> From carsten at thep.lu.se Tue Nov 10 09:01:40 1992 From: carsten at thep.lu.se (carsten@thep.lu.se) Date: Tue, 10 Nov 92 15:01:40 +0100 Subject: Postdoc Position in Lund Message-ID: <9211101401.AA19495@dacke.thep.lu.se> A two year postdoc position will be available within the Complex Systems group at the Department of Theoretical Physics, University of Lund, Sweden, starting September 1st 1993. The major research area of the group is Artificial Neural Networks with tails into chaos and difficult computational problems in general. Although some application studies occur, algorithmic development is the focus in particular within the following areas: * Using Feed-back ANN for finding good solutions to combinatorial optimization problems; knapsacks, scheduling, track-finding. * Time-series prediction. * Robust multi-layer perceptron updating procedures including noise. * Deformable template methods -- robust statistics. * Configurational Chemistry -- Polymers, Proteins ... * Application work within the domain of experimental physics, in particular in connection with the upcoming SSC/LHC experiments. Lund University is the largest campus in Scandinavia located in a picturesque 1000 year old city (100k inhabitants). Lund is strategically well located in the south of Sweden with 1.5 hrs commuting distance to Copenhagen (Denmark). The candidate should have a PhD in a relevant field, which need not be Physics/Theoretical Physics. Applications and three letters of recommendation should be sent to (not later than December 15): Carsten Peterson Department of Theoretical Physics University of Lund Solvegatan 14A S-223 62 Lund Sweden or Bo S\"{o}derberg Department of Theoretical Physics University of Lund Solvegatan 14A S-223 62 Lund Sweden From weber at forwiss.tu-muenchen.de Wed Nov 11 04:51:41 1992 From: weber at forwiss.tu-muenchen.de (Walter Weber) Date: Wed, 11 Nov 1992 10:51:41 +0100 Subject: Generalization ability of a BPTT-net Message-ID: <9211110951.AA17888@forwiss.tu-muenchen.de> Dear Connectionists, let me first briefly introduce my work: I'm just writing a masters' thesis at the Technical University of Munich and i'm trying to deal with neural control. My goal is to let a neural controller learn how to pull the gas- and brake-pedals of a car in order to move in appropriate distance to a leading car. Input signals are speed (v), dv/dt, distance (d) and dd/dt (all inputs are coded in [0;1]) and there is only 1 output which varies from 0 to 1. (control signals for the brake are coded in [0;0.5], signals for the gas-pedal are coded in [0.5;1] to get a continuous trajectory for the control signals over time). The training trajectories are from a PID-controller which was implemented at BMW for the PROMETHEUS project. I've done several aproaches to build up a net that could solve this problem (Elman-net, Jordan-net, fully recurrent NN), but the most promising approach seems to be the BPTT-algorithm applied to a three-layer recurrent network with input connected to a fully recurrent hidden layer and that hidden layer connected to the output. The activation-function i use is a sigmoidal function with an offset, the output produced varies in [0;1]. The learning ability of this BPTT-net is quite well (even very well if i use teacher-forcing), but if i want the network to predict the remaining part of a given trajectory after it used the first part (mostly about 70 - 80%) as training data, the following problems occur: 1. The net cannot perform peaks (short and strong pulls of the pedals) even if such data was learned well during training. (My idea about that was: the use of teacher forcing only makes training better but does not influence the generalization ability in a positive way, so the net can only perform what it was able to learn without teacher forcing. And only teacher forcing enabled the net to learn such peaks). 2. If a part of the test trajectory looks similar to a part of the training trajectory with the only difference is a offset between the two trajectory-parts (e.g. the part of the test trajectory varies between 0.6 and 0.8, the part of the training trajectory varies between 0.4 and 0.6) the network produces an output - in trying to approximate the test data - which would fit exactly the training data, but is wrong for the test data (with the error is the offset between the two trajectory-parts). How can i get rid of these offsets in the test data? For my english is not very well, i hope one can understand the problems i tried to describe above. And if anybody has an answer to my questions, i would be very glad if i got answers. So thank you and bye, --- Walter Weber =================================================================== | weber at forwiss.tu-muenchen.de Walter Weber | | FORWISS - TU Muenchen | | Tel.: 089/48095-229 Orleanstr. 34 | | -231 D-8000 Muenchen 80 | =================================================================== From craign at SIMPLEMIND.UROL.BCM.TMC.EDU Tue Nov 10 10:36:36 1992 From: craign at SIMPLEMIND.UROL.BCM.TMC.EDU (Craig Stuart Niederberger) Date: Tue, 10 Nov 92 09:36:36 CST Subject: Neural computing ideas and biological terms Message-ID: <9211101536.AA13052@SIMPLEMIND.UROL.BCM.TMC.EDU.noname> With regards to Jim Bower's complaint: > >From a recent posting: > > "The audio synthesizer is built around an integrated-circuit > chip from Intel Corporation in Santa Clara, California. The > chip, called the Intel 80170NX electrically trainable analog > neural network (ETANN), simulates the function of nerve > cells in a biological brain." > > Unlikely in that we don't yet know how nerve cells in a biological > brain function. Is it really necessary many years (now) > into neural net research to continue to lean on the brain for > moral support? > > Sorry to retorically beat a dead horse, but statements like this > are annoying to those of us whose primary interest is to understand > how the brain works. They also still occur far to frequently > especially in association with products. > Technically, I agree that a rather large schism exists between the unknowns of neurophysiology and the creative endeavors of neural computing. However, I disagree with the contention that "leaning on the brain for moral support" is necessarily bad. I have been applying neural computational models to analyze clinical data, and have found it at times difficult to communicate the significance of these models to my frequently non-mathematically oriented colleagues. Often, I have resorted to explanations that are couched in biological terms rather than mathematical ones, with the opinion that it is better to communicate something rather than nothing at all. Perhaps the focus should be to try to bring more investigators into the fold. Not only do new investigators yield new research and new ideas, but funding from these quarters will follow as well. Craig Niederberger ___________________________________________________________________________ | | | Craig Niederberger M.D. Internet EMAIL: craign at mbcr.bcm.tmc.edu | | Department of Urology, Room 440E US Phone: 713-798-7267 | | Baylor College of Medicine FAX: 713-798-5577 | | One Baylor Plaza [o o] [+ +] [o o] [+ +] [o o] | | Houston, Texas 77030-3498 USA [<-->] [ -- ] [ == ] [<-->] [ ^^ ] | |___________________________________________________________________________| From neuron at cattell.psych.upenn.edu Thu Nov 12 18:29:37 1992 From: neuron at cattell.psych.upenn.edu (Neuron-Digest Moderator, Peter Marvit) Date: Thu, 12 Nov 92 18:29:37 EST Subject: Neuron Digest Message-ID: <232.721610977@cattell.psych.upenn.edu> Dear Connectionists, A note here to [re-]introduce the moderated forum/mailing list called "Neuron Digest" and supply ftp/archive information (which apparently is still incorrect on some lists-of-lists). As the bi-monthly reminder of the Connectionists list notes, Neuron Digest is a moderated forum aimed at sophisticated but general audience. There is no restriction on being a subscriber to the Digest. Connectionists will recognize considerable overlap with respect to Paper and Conference announcements between this mailing list and the Digest. However, Neuron Digest tends to have a bit more free-wheeling discussions and questions (plus occasional editorial comments by the moderator). Especially, there tends to be more neophyte "looking for references" queries. On the other hand, being moderated, Neuron Digest is not as timely as other fora. An average of one to two issues come out per week. For the forseeable future, the Digest will be gatewayed (one-way) to the USENET group comp.ai.neural-nets. While you current Connectionists may not wish to add to your e-mailboxes, feel free to access the back issues (with indices coming by the New Year, hopefully) or mention the Digest to your students. Appended is the standard "welcome" blurb for Neuron Digest (liberally copied from Ken Laws). Archive information is at the end. I would be happy to answer any questions (between running experiments, of course). : Peter Marvit, Neuron Digest Moderator : : Email: : : Courtesy of the Psychology Department, University of Pennsylvania : : 3815 Walnut St., Philadelphia, PA 19104 w:215/898-9208 h:215/387-6433 : ------------------------------ CUT HERE ------------------------------- Internet: NEURON at cattell.psych.upenn.edu Neuron-Digest is a list (in digest form) dealing with all aspects of neural networks (and any type of network or neuromorphic system), especially: NATURAL SYSTEMS Software Simulations Neurobiology Hardware Neuroscience Digital ARTIFICIAL SYSTEMS Analog Neural Networks Optical Algorithms Cellular Automatons Some key words which may stir up some further interest include: Hebbian Systems Widrow-Hoff Algorithm Perceptron Threshold Logic Holography Content Addressable Memories Lyapunov Stability Criterion Navier-Stokes Equation Annealing Spin Glasses Locally Couples Systems Globally Coupled Systems Dynamical Systems (Adaptive) Control Theory Back-Propagation Generalized Delta Rule Pattern Recognition Vision Systems Parallel Distributed Processing Connectionism Any contribution in these areas is accepted. Any of the following are reasonable: Abstracts Reviews Lab Descriptions Research Overviews Work Planned or in Progress Half-Baked Ideas Conference Announcements Conference Reports Bibliographies History Connectionism Puzzles and Unsolved Problems Anecdotes, Jokes, and Poems Queries and Requests Address Changes (Bindings) Archived files/messages will be available with anonymous ftp from the machine cattell.psych.upenn.edu (130.91.68.31) in the directory pub/Neuron-Digest. That directory contains back issues with the names vol-nn-no-mm (e.g., vol-3-no-02). I'm also collecting simulation software in pub/Neuron-Software. Contributions are welcome. All requests to be added to or deleted from this list, problems, questions, etc., should be sent to neuron-request at cattell.psych.upenn.edu. Moderator: Peter Marvit ------------------------------ CUT HERE ------------------------------- From dhw at santafe.edu Thu Nov 12 22:41:16 1992 From: dhw at santafe.edu (dhw@santafe.edu) Date: Thu, 12 Nov 92 20:41:16 MST Subject: No subject Message-ID: <9211130341.AA03280@zia> Concerning the recent comments by Jim Bower and Craig Niederberger: Justifying a particular technique solely by an analogy (rather than with a close scrutinization of the task at hand) is dubious in the best of circumstances. Using such a technique when the analogy is in fact poor is engineering by hope. Maybe fun. Might work. Might (!) even give insight. But not science. (At least, not science unless the posited "insight" somehow provides after-the-fact justification for the technique, or at least gives some reason to believe the technique is a reasonable heuristic.) David Wolpert dhw at santafe.edu From yirgan at dendrite.cs.colorado.edu Thu Nov 12 12:01:19 1992 From: yirgan at dendrite.cs.colorado.edu (Juergen Schmidhuber) Date: Thu, 12 Nov 1992 10:01:19 -0700 Subject: new papers Message-ID: <199211121701.AA16099@thalamus.cs.Colorado.EDU> The following papers are now available: ------------------------------------------------------------------- DISCOVERING PREDICTABLE CLASSIFICATIONS (Technical Report CU-CS-626-92) .. Jurgen Schmidhuber, University of Colorado .. .. Daniel Prelinger, Technische Universitat Munchen ABSTRACT: Prediction problems are among the most common learning problems for neural networks (e.g. in the context of time series prediction, control, etc.). With many such problems, however, perfect prediction is inherently impossible. For such cases we present novel unsupervised systems that learn to classify patterns such that the classifications are predictable while still being as specific as possible. The approach can be related to the IMAX method of Hinton, Becker and Zemel (1989, 1991). Experiments include Becker's and Hinton's stereo task, which can be solved more readily by our system. ------------------------------------------------------------------- PLANNING SIMPLE TRAJECTORIES USING NEURAL SUBGOAL GENERATORS (for SAB92) .. Jurgen Schmidhuber, University of Colorado .. .. Reiner Wahnsiedler, Technische Universitat Munchen ABSTRACT: We consider the problem of reaching a given goal state from a given start state by letting an `animat' produce a sequence of actions in an environment with multiple obstacles. Simple trajectory planning tasks are solved with the help of `neural' gradient-based algorithms for learning without a teacher to generate sequences of appropriate subgoals in response to novel start/goal combinations. ------------------------------------------------------------------- STEPS TOWARDS `SELF-REFERENTIAL' LEARNING: A THOUGHT EXPERIMENT (Technical Report CU-CS-627-91) .. Jurgen Schmidhuber, University of Colorado ABSTRACT: A major difference between human learning and machine learning is that humans can reflect about their own learning behavior and adapt it to typical learning tasks in a given environment. To make some initial theoretical steps toward `intro- spective' machine learning, I present - as a thought experiment - a `self-referential' recurrent neural network which can run and actively modify its own weight change algorithm. Due to the generality of the architecture, there are no theoretical limits to the sophistication of the modified weight change algorithms running on the network (except for unavoidable pre-wired time and storage constraints). In theory, the network's weight matrix can learn not only to change itself, but it can also learn the way it changes itself, and the way it changes the way it changes itself --- and so on ad infinitum. No endless recursion is involved, however. For one variant of the architecture, I present a simple but general initial reinforcement learning algorithm. For another variant, I derive a more complex exact gradient-based algorithm for supervised sequence learning. A disadvantage of the latter algorithm is its computational complexity per time step which is independent of sequence length and equals O(n_conn^2 log n_conn), where n_conn is the number of connections. Another disadvantage is the high number of local minima of the unusually complex error surface. The purpose of my thought experiment, however, is not to come up with the most efficient or most practical `introspective' or `self-referential' weight change algorithm, but to show that such algorithms are possible at all. --------------------------------------------------------------------- To obtain copies, do: unix> ftp archive.cis.ohio-state.edu Name: anonymous Password: (your email address) ftp> binary ftp> cd pub/neuroprose ftp> get schmidhuber.predclass.ps.Z ftp> get schmidhuber.subgoals.ps.Z ftp> get schmidhuber.selfref.ps.Z ftp> bye unix> uncompress schmidhuber.predclass.ps.Z unix> uncompress schmidhuber.subgoals.ps.Z unix> uncompress schmidhuber.selfref.ps.Z unix> lpr schmidhuber.predclass.ps unix> lpr schmidhuber.subgoals.ps unix> lpr schmidhuber.selfref.ps --------------------------------------------------------------------- Sorry, no hardcopies. (Except maybe in very special urgent cases). .. Jurgen Address until December 17, 1992: .. Jurgen Schmidhuber Department of Computer Science University of Colorado Campus Box 430 Boulder, CO 80309, USA email: yirgan at cs.colorado.edu Address after December 17, 1992: .. Jurgen Schmidhuber .. Institut fur Informatik .. .. Technische Universitat Munchen .. Arcisstr. 21, 8000 Munchen 2, Germany email: schmidhu at informatik.tu-muenchen.de From lba at sara.inesc.pt Fri Nov 13 07:28:43 1992 From: lba at sara.inesc.pt (Luis B. Almeida) Date: Fri, 13 Nov 92 11:28:43 -0100 Subject: Generalization ability of a BPTT-net In-Reply-To: Walter Weber's message of Wed, 11 Nov 1992 10:51:41 +0100 <9211110951.AA17888@forwiss.tu-muenchen.de> Message-ID: <9211131228.AA23127@sara.inesc.pt> [To the "Connectionists" moderator: I am sending the following response to Walter Weber directly, I don't know if you would consider it appropriate to also publish it in the Connectionists]. I would like to make two suggestions, concerning the first problem: a) Teacher forcing, though often very useful, does not necessarily perform descent (and therefore minimization) on the objective function. Why not use the weights obtained with teacher forcing as an initialization for a second training stage, which would use normal BPTT without teacher forcing? b) Using a sigmoid on the output unit means that, in order to produce peaks (values close to 0 or to 1), the sum at the input of that unit must become relatively large, in absolute value. The net might perform better if you remove the sigmoid from the output unit, which will then become linear. I didn't fully understand your second problem. What are the inputs to the net, in the [.6, .8] case, and in the [.4, .6] case? are they the same? Are they similar in some way? Luis B. Almeida INESC Phone: +351-1-544607 Apartado 10105 Fax: +351-1-525843 P-1017 Lisboa Codex Portugal lba at inesc.pt lba at inesc.uucp (if you have access to uucp) From mav at cs.uq.oz.au Wed Nov 11 21:38:59 1992 From: mav at cs.uq.oz.au (Simon Dennis) Date: Thu, 12 Nov 92 12:38:59 +1000 Subject: TR: What does the human memory environment look like? Message-ID: <9211120239.AA07755@uqcspe.cs.uq.oz.au> The following technical report is available for anonymous ftp. What does the environment look like? Setting the scene for interactive models of human memory Simon Dennis Department of Computer Science University of Queensland Janet Wiles Departments of Computer Science and Psychology University of Queensland Michael Humphreys Department of Psychology University of Queensland Abstract We set the scene for a class of interactive models of human memory in which performance is dependent both on the structure of the environment and the structure of the model's mechanism. Such a system is capable of learning representations, control processes and decision criterion in order to successfully interact with its environment. That the (observable) environment is responsible for performance in interactive models allows the elimination of assumptions which are embedded in the mechanism of current models. Interactive models also offer the possibility of addressing the development of the mechanisms of memory which are currently poorly understood. Analyses of the relevant environments of four touchstone phenomena: the list strength effect in recognition; the crossed associates paradigm; the ABABr paradigm and the word frequency effect in recognition were performed to establish the context in which interactive accounts of these phenomena must be set. The rational model of human memory (Anderson & Milson, 1989) is advocated as a model of environmental contingencies and hence of interest to the interactive modelers. It is found to be consistent with empirical environmental data in the case of the list strength effect and, with some modification, in the case of the word frequency effect in recognition. While it proved difficult to analyze the relevant environment in the cued recall paradigms, the rational model was found to be consistent with experimental evidence. The issues involved in the process of environmental analysis were explored. Conclusions Our major purpose has been to set the scene for interactive models of human memory and we have done this in three ways. Firstly, we addressed the philosophical issue of how a representation attains its meaning. We argued that models which have as their basis the physical symbol system hypothesis, will encounter difficulties with the symbol grounding problem, and will find it difficult to give an account of meaning attainment. Interactive models provided a way of avoiding the problem. Secondly, we set the psychological modeling scene by outlining the advantages of interactive models as models of human memory. Not only do interactive models provide a much needed link to the developmental literature, but they also allow mechanistic accounts to shed some of their assumptions onto the observable environment. Thirdly, we have started the task of analyzing the environment. The environmental analyses which have been conducted are encouraging especially for the recognition paradigms. An environmental analysis of the effect of repetition suggests that not only is the main effect of repetition on performance accuracy accounted for by a simple interactivist account, but the lack of a list strength effect in recognition is also a natural consequence of the environmental contingencies. The word frequency effect in recognition was also found to mirror environmental statistics while the joint information paradigms proved difficult to analyze. Rational analysis was successful with only minor modification in all cases examined. Given these results there is reason to suppose that the environmental approach has a valuable contribution to make to the understanding of human memory. In conclusion then we would like to draw out two important implications. Firstly, we reiterate \citeA{Anderson90}, in suggesting that memory researchers should divert some of their effort to the empirical study of the environment. Secondly, we propose that interactive models of memory will be in the best position to take advantage of such research. Ftp instructions: To retrieve the technical report ftp to exstream.cs.uq.oz.au, cd pub/TECHREPORTS/department, change to binary mode and get TR0249.ps.Z. Example: $ ftp exstream.cs.uq.oz.au Connected to exstream.cs.uq.oz.au. 220 exstream FTP server (Version 6.12 Fri May 8 16:33:17 EST 1992) ready. Name (exstream.cs.uq.oz.au:mav): anonymous 331 Guest login ok, send e-mail address as password. Password: 230- Welcome to ftp.cs.uq.oz.au 230-This is the University of Queensland Computer Science Anonymous FTP server. 230-For people outside of the department, please restrict your usage to outside 230-of the hours 8am to 6pm. 230- 230-The local time is Thu Nov 12 11:18:01 1992 230- 230 Guest login ok, access restrictions apply. ftp> cd pub/TECHREPORTS/department 250 CWD command successful. ftp> bin 200 Type set to I. ftp> get TR0249.ps.Z 200 PORT command successful. 150 Opening BINARY mode data connection for TR0249.ps.Z (174928 bytes). 226 Transfer complete. local: TR0249.ps.Z remote: TR0249.ps.Z 174928 bytes received in 1.4 seconds (1.2e+02 Kbytes/s) ftp> quit 221 Goodbye. $ Please address requests for hard copies to: Simon Dennis Department of Computer Science University of Queensland 4072 Australia ------------------------------------------------------------------------------- Simon Dennis Address: Department of Computer Science Email: mav at cs.uq.oz.au University of Queensland QLD 4072 Australia ------------------------------------------------------------------------------- From regier at ICSI.Berkeley.EDU Fri Nov 13 21:25:35 1992 From: regier at ICSI.Berkeley.EDU (Terry Regier) Date: Fri, 13 Nov 92 18:25:35 PST Subject: TR available Message-ID: <9211140225.AA19513@icsib4.ICSI.Berkeley.EDU> The technical report version of my dissertation is now available by ftp. Ftp instructions follow the abstract. The Acquisition of Lexical Semantics for Spatial Terms: A Connectionist Model of Perceptual Categorization Terry Regier UC Berkeley ICSI technical report TR-92-062 This thesis describes a connectionist model which learns to perceive spatial events and relations in simple movies of 2-dimensional objects, so as to name the events and relations as a speaker of a particular natural language would. Thus, the model learns perceptually grounded semantics for natural language spatial terms. Natural languages differ -- sometimes dramatically -- in the ways in which they structure space. The aim here has been to have the model be able to perform this learning task for words from any natural language, and to have learning take place in the absence of explicit negative evidence, in order to rule out ad hoc solutions and to approximate the conditions under which children learn. The central focus of this thesis is a connectionist system which has succeeded in learning spatial terms from a number of different languages. The design and construction of this system have resulted in several technical contributions. The first is a very simple but effective means of learning without explicit negative evidence. This thesis also presents the notion of partially-structured connectionism, a marriage of structured and unstructured network design techniques capturing the best of each paradigm. Finally, the thesis introduces the idea of learning within highly specialized structural devices. Scientifically, the primary result of the work described here is a computational model of the acquisition of visually grounded semantics. This model successfully learns words for spatial events and relations from a range of languages with widely differing spatial systems, including English, Mixtec (a Mexican Indian language), German, Bengali, and Russian. And perhaps most importantly, the model does more than just recapitulate the data; it also generates a number of falsifiable linguistic predictions regarding the sorts of semantic features, and combinations of features, one might expect to find in lexemes for spatial events and relations in the world's natural languages. To ftp: % ftp icsi-ftp.berkeley.edu Name: anonymous Password: [your e-mail address] ftp> binary ftp> cd pub/techreports ftp> get tr-92-062.ps.Z ftp> quit % uncompress tr-92-062.ps.Z % lpr -P[your-postscript-printer] tr-92-062.ps -- ---------------------------------------------------------------------- Terry Regier Computer Science, UC Berkeley regier at icsi.Berkeley.EDU International Computer Science Institute From SANTINI at INGFI1.CINECA.IT Fri Nov 13 11:50:00 1992 From: SANTINI at INGFI1.CINECA.IT (SANTINI@INGFI1.CINECA.IT) Date: 13 Nov 1992 16:50 +0000 (N) Subject: Two reports available Message-ID: <4812@INGFI1.CINECA.IT> The following two technical reports have been posted in the directory /neural/papers of the ftp server aguirre.ingfi1.cineca.it (150.217.11.13): AN ALGORITHM FOR TRAINING NEURAL NETWORKS WITH ARBITRARY FEEDBACK STRUCTURE S. Santini(*) A. Del Bimbo(*) R. Jain(+) (*) Dipartimento di sistemi e Informatica, Universita' di Firenze, Firenze, Italy (+) Artificial Intelligence Lab., The University of Michigan, Ann Arbor, MI Abstract In this report, we consider multi-layer discrete-time dynamic networks with multiple unit-delay feedback paths between layers. A block notation is introduced for this class of networks which allows a great flexibility in the description of the network architecture and permits a unified treatment of static and dynamic networks. Networks are defined by recursively arranging blocks. Network blocks which satisfy certain *trainability* conditions can be embedded into other blocks through a set of *elementary connections*, so that the overall network still satisfies the same trainability conditions. The problem of training such a network is thus reduced to the definition of an algorithm ensuring the trainability of a block, assuming that all the embedded blocks are trainable. An algorithm is presented which is a block-matrix version of Forward Propagation, and is based on Werbos' ordered derivatives. This report is in the file: santini.feedback_NN_algorithm.ps.Z ----------------------------------------------------------------------- SYSTEMS IN WHICH NOBODY KNOWS THE BIG PICTURE S. Santini A. Del Bimbo Dipartimento di sistemi e Informatica, Universita' di Firenze, Firenze, Italy Abstract There is an understatement in engineering activities: if you have a problem, create a hierarchy. Hierarchical and centralized schemes are considered just ``the way things ought to be done''. In this paper, we briefly introduce the connectionist point of view, in which problems are solved by emergent computation, arising from local interaction of units, without any centralized control. In these systems, there is no knowledge of the overall problem, and in no place there is enough intelligence to ``understand'' the problem. Yet, this kind of (dis)organization can actually solve problems and -- more important -- its properties can be mathematically analyzed. We present an example: a Neural Gas network that finds the minimum path between two points in an area were obstacles are present. We show that this global problem can be solved without any global organization. Moreover, we proof global properties of the emergent computation. This is in the file: santini.big_picture.ps.Z ------------------------------------------------------------------------ HOW TO GET THE FILES: --------------------- % ftp aguirre.ingfi1.cineca.it (or 150.217.11.13) Connected to aguirre.ingfi1.cineca.it 220 aguirre FTP server (SunOS 4.1) ready. Name: anonymous 331 Guest login ok, send ident as password. Password: 230 Guest login ok, access restrictions apply. ftp> cd neural/papers ftp binary ftp> get santini.feedback_NN_algorithm.ps.Z ftp> get santini.feedback_NN_algorithm.ps.Z ftp> quit % uncompress santini.feedback_NN_algorithm.ps.Z % lpr santini.feedback_NN_algorithm.ps % uncompress santini.feedback_NN_algorithm.ps.Z % lpr santini.feedback_NN_algorithm.ps For any problem in retrieving files, or for any discussion and comment about their content, contact me at: santini at ingfi1.cineca.it Simone Santini From koza at CS.Stanford.EDU Sun Nov 15 19:43:15 1992 From: koza at CS.Stanford.EDU (John Koza) Date: Sun, 15 Nov 92 16:43:15 PST Subject: New Book and Videotape on Genetic Programming Message-ID: BOOK AND VIDEOTAPE ON GENETIC PROGRAMMING A new book and a one-hour videotape (in VHS NTSC, PAL, and SECAM formats) on genetic programming are now available from the MIT Press. NEW BOOK... GENETIC PROGRAMMING: ON THE PROGRAMMING OF COMPUTERS BY MEANS OF NATURAL SELECTION by John R. Koza, Stanford University The recently developed genetic programming paradigm provides a way to genetically breed a computer program to solve a wide variety of problems. Genetic programming starts with a population of randomly created computer programs and iteratively applies the Darwinian reproduction operation and the genetic crossover (sexual recombination) operation in order to breed better individual programs. The book describes and illustrates genetic programming with 81 examples from various fields. 840 pages. 270 Illustrations. ISBN 0-262-11170-5. Contents... 1 Introduction and Overview 2 Pervasiveness of the Problem of Program Induction 3 Introduction to Genetic Algorithms 4 The Representation Problem for Genetic Algorithms 5 Overview of Genetic Programming 6 Detailed Description of Genetic Programming 7 Four Introductory Examples of Genetic Programming 8 Amount of Processing Required to Solve a Problem 9 Nonrandomness of Genetic Programming 10 Symbolic Regression - Error-Driven Evolution 11 Control - Cost-Driven Evolution 12 Evolution of Emergent Behavior 13 Evolution of Subsumption 14 Entropy-Driven Evolution 15 Evolution of Strategy 16 Co-Evolution 17 Evolution of Classification 18 Iteration, Recursion, and Setting 19 Evolution of Constrained Syntactic Structures 20 Evolution of Building Blocks 21 Evolution of Hierarchies of Building Blocks 22 Parallelization of Genetic Programming 23 Ruggedness of Genetic Programming 24 Extraneous Variables and Functions 25 Operational Issues 26 Review of Genetic Programming 27 Comparison with Other Paradigms 28 Spontaneous Emergence of Self-Replicating and Self-Improving Computer Programs 29 Conclusions Appendices contain simple software in Common LISP for implementing experiments in genetic programming. ONE-HOUR VIDEOTAPE... GENETIC PROGRAMMING: THE MOVIE by John R. Koza and James P. Rice, Stanford University The one-hour videotape (in VHS NTSC, PAL, and SECAM formats) provides a general introduction to genetic programming and a visualization of actual computer runs for 22 of the problems discussed in the book GENETIC PROGRAMMING: ON THE PROGRAMMING OF COMPUTER BY MEANS OF NATURAL SELECTION. The problems include symbolic regression, the intertwined spirals, the artificial ant, the truck backer upper, broom balancing, wall following, box moving, the discrete pursuer-evader game, the differential pursuer- evader game, inverse kinematics for controlling a robot arm, emergent collecting behavior, emergent central place foraging, the integer randomizer, the one-dimensional cellular automaton randomizer, the two-dimensional cellular automaton randomizer, task prioritization (Pac Man), programmatic image compression, solving numeric equations for a numeric root, optimization of lizard foraging, Boolean function learning for the 11-multiplexer, co- evolution of game-playing strategies, and hierarchical automatic function definition as applied to learning the Boolean even-11- parity function. ---------------------------ORDER FORM---------------------- PHONE: 800-326-4471 TOLL-FREE or 617-625-8569 MAIL: The MIT Press, 55 Hayward Street, Cambridge, MA 02142 FAX: 617-625-9080 Please send ____ copies of the book GENETIC PROGRAMMING: ON THE PROGRAMMING OF COMPUTERS BY MEANS OF NATURAL SELECTION by John R. Koza (KOZGII) (ISBN 0-262-11170-5) @ $55.00. ____ copies of the one-hour videotape GENETIC PROGRAMMING: THE MOVIE by John R. Koza and James P. Rice in VHS NTSC format (KOZGVV) (ISBN 0-262-61084-1) @$34.95 ____ copies of the videotape in PAL format (KOZGPV) (ISBN 0-262- 61087-6) @$44.95 ____ copies of the videotape in SECAM format (KOZGSV) (ISBN 0- 262-61088-4) @44.95. Name __________________________________ Address_________________________________ City____________________________________ State_________________Zip________________ Country_________________________________ Phone Number ___________________________ $ _______ Total $ _______ Shipping and Handling ($3 per item. Outside U.S. and Canada, add $6 per item for surface rate or $22 per item for airmail) $ _______ Canada - Add 7% GST $ _______ Total due MIT Press __ Payment attached (check payable to The MIT Press in U.S. funds) __ Please charge to my VISA or MASTERCARD credit card Number ________________________________ Credit Card Expires _________________________________ Signature ________________________________ From eichler at pi18.arc.umn.edu Fri Nov 13 15:06:44 1992 From: eichler at pi18.arc.umn.edu (Rogene Eichler) Date: Fri, 13 Nov 92 14:06:44 CST Subject: Neural computing ideas and biological terms Message-ID: <9211132006.AA10502@pi18.arc.umn.edu> > Technically, I agree that a rather large schism exists between the > unknowns of neurophysiology and the creative endeavors of neural > computing. However, I disagree with the contention that "leaning > on the brain for moral support" is necessarily bad. I have been applying > neural computational models to analyze clinical data, and have found it > at times difficult to communicate the significance of these models to my > frequently non-mathematically oriented colleagues. Often, I have resorted > to explanations that are couched in biological terms rather than mathematical > ones, with the opinion that it is better to communicate something rather > than nothing at all. > > Craig Niederberger I would have to agree with Craig that it is necessary to frame one's model in biological terms when speaking to biologists. (Most cringe at the sight of a mathematical equation.) It is difficult to keep an audience's attention if you are speaking in a foreign tongue. It is your gain and their loss, to speak both languages. However, I grow tired of defending the validity of models to biologists who do not seem satisfied with any model that does not capture every last nuiance of complexity or that does not explain every last experimental finding. Modeling the brain will provide valuable insights into how we process information and how we can exploit those rules for artificial systems. But they do not need to duplicate every last brain dynamic to be useful or valid. And when modelers continue to make the claim 'just like the brain' for the sake of convincing you of the validity of the model, they are reinforcing the claim that the brain is the ONLY measure. It seems time for network modelers to develop a new set of measures, or perhaps some confidence in the significance of stand-alone models. That is my interpretation of what "leaning on the brain for moral support" is really about. - Rogene From N.E.Sharkey at dcs.ex.ac.uk Mon Nov 16 06:36:21 1992 From: N.E.Sharkey at dcs.ex.ac.uk (Noel Sharkey) Date: Mon, 16 Nov 92 11:36:21 GMT Subject: 4 research posts Message-ID: <1630.9211161136@propus.dcs.exeter.ac.uk> Research Posts at Centre for Connection Science Department of Computer Science University of Exeter UK Four new research posts will be available (expected start January 1st, 1993) at the Centre for Connection Science, Department of Computer Science, as part of a 3-year research project funded by the SERC/DTI and led by Noel Sharkey and Derek Partridge. The project will investigate the reliability of software systems implemented as neural nets using the multiversion programming strategy. Two of the posts will at the post-doctoral level (grade 1A). The ideal applicant will be proficient in both neural computing and software engineering (although training in one or the other may be given). In addition, there is a requirement for at least one of the successful applicants to work on the formal analysis of network implementation as a paradigm for reliable software. The other two posts will be for Research Assistants/Experimental Officers at grade 1b. One of these will be required to have a high level of proficiency in C programming and general computing skills. The other will be part-time, and preference will be given to an applicant with good mathematical and engineering skills (particulary control systems). For more information please contact Lyn Shackelton by email (lyn at dcs.ex.ac.uk or by telephone (0392-264066 mornings 10.00-2.00). From sofge at ai.mit.edu Mon Nov 16 19:25:25 1992 From: sofge at ai.mit.edu (Donald Sofge) Date: Mon, 16 Nov 92 19:25:25 EST Subject: Book Announcement Message-ID: <9211170025.AA02693@rice-chex> <<<<<---------------------- New Book Announcement ---------------------->>>> HANDBOOK OF INTELLIGENT CONTROL: Neural, Fuzzy, and Adaptive Approaches Edited by David A. White and Donald A. Sofge Handbook of Intelligent Control provides a hands-on approach to integrating intelligent control approaches with existing control architectures for the nonlinear control of complex multivariate systems. It is an attempt by leading industry and academic researchers to present the current state-of-the-art developments in intelligent control in a highly readable, often tutorial format such that many of the techniques described within may readily implemented by the reader. The goals of this approach are: % To provide firm mathematical and theoretical foundations for current intelligent control methods % To demonstrate how these methods may be effectively combined with existing control practices % To provide overviews and extensive bibliographic references to much current work in this field % To provide examples of real-world applications in robotics, aerospace, chemical engineering, and manufacturing which have served as a driving force behind new innovations in intelligent control. Discussions concerning these applications are provided for two main reasons: to demonstrate how existing intelligent control techniques may be applied to real-world problems, and to provide challenging real-world problems which serve as impetus for new innovations in intelligent control and for which intelligent control solutions may provide the only solutions. This book is an outgrowth of three major workshops held under the National Science Foundation Intelligent Control Initiative. The chapters included in this volume are implementations and extensions of creative new ideas discussed at these workshops. The application and integration of neural, fuzzy, and adaptive methods into "real-world" engineering problems makes this an ideal book for practicing engineers, as well as graduate and academic researchers. This volume contains the following chapters: Foreword - Paul Werbos, Elbert Marsh, Kishan Baheti, Maria Burka, and Howard Moraff Editors' Preface Donald Sofge and David White Introduction to Intelligent Control 1 Intelligent Control: An Overview and Evaluation Karl strom and Thomas McAvoy 2 An Introduction to Connectionist Learning Control Systems Walter Baker and Jay Farrell 3 Neurocontrol and Supervised Learning: An Overview and Evaluation Paul Werbos Conventional Control and Intelligent Approaches 4 Fuzzy Logic in Control Engineering Reza Langari, Hamid Berenji, and Lotfi Zadeh 5 Adaptive Control of Dynamical Systems Using Neural Networks K.S. Narendra 6 Optimal Control: A Foundation for Intelligent Control David White and Michael Jordan 7 Development and Application of CMAC Neural Network-based Control Gordan Kraft, Tom Miller, and D. Dietz Applications of Intelligent Control 8 Artificial Neural Networks in Manufacturing and Process Control Judy Franklin and David White 9 Applied Learning-Optimal Control for Manufacturing Donald Sofge and David White 10 Neural Networks, System Identification, and Control in the Chemical Process Industries Paul Werbos, Thomas McAvoy, and Ted Su 11 Flight, Propulsion, and Thermal Control of Advanced Aircraft and Hypersonic Vehicles David White, Albion Bowers, Ken Iliff, Greg Noffz, Mark Gonda, and John Menousek Advances in System Identification, Optimization and Learning Control Theory 12 Reinforcement Learning and Adaptive Critic Methods Andrew Barto 13 Approximate Dynamic Programming for Real-time Control and Neural Modeling Paul Werbos 14 The Role of Exploration in Learning Control Sebastian Thrun Publication Date: September 1992 568 pages Price: $59.95 Available from: Van Nostrand Reinhold VNR Order Processing P.O. Box 668 Florence, Kentucky 41022-0668 Call Toll-free: 1 (800) 926-2665 From N.E.Sharkey at dcs.ex.ac.uk Mon Nov 16 08:19:32 1992 From: N.E.Sharkey at dcs.ex.ac.uk (Noel Sharkey) Date: Mon, 16 Nov 92 13:19:32 GMT Subject: TR available Message-ID: <1692.9211161319@propus.dcs.exeter.ac.uk> TECH REPORT AVAILABLE: Computer Science TR R257 ADAPTIVE GENERALISATION AND THE TRANSFER OF KNOWLEDGE Noel E. Sharkey and Amanda J.C. Sharkey Center for Connection Science, University of Exeter Adaptive generalisation is the ability to use prior knowledge in the performance of novel tasks. Thus, if we are to model intelligent behaviour with neural nets, they must be able to generalise across task domains. Our objective is to elucidate the aetiology of transfer of information between connectionist nets. First, a method is described that provides a standardised score for the quantification of how much task structure a net has extracted, and to what degree knowledge has been transferred between tasks. This method is then applied in three simulation studies to examine Input-to-Hidden (IH) and Hidden-to-Output (HO) decision hyperplanes as determinants of transfer effects. In the first study, positive transfer is demonstrated between functions that require the vertices of their input spaces to be divided similarly, and negative transfer between functions that require decision regions of different shapes. In the other two studies, input and output similarity are varied independently in a series of paired associate learning tasks. Further explanation of transfer effects is provided through the use of a new technique that permits better visualisation of the entire computational space by showing both the relative position of inputs in Hidden Unit space, and the HO decision regions implemented by a set of weights. * This research was supported by an award from the Economic and Social Research Council, Grant No R000233441. An earlier version of this paper appears in the Proceedings of the Second Irish Neural Networks Conference, Belfast 1992. The current version will appear in an AI review special issue on Connectionism. For a postscript version, email: ptsec at uk.ac.exeter.dcs Some hardmail copies are available from the same source. Our hardmail address Mrs June Stevens, Dept. Computer Science University of Exeter Exeter EX4 4PT Devon U.K. From kolen-j at cis.ohio-state.edu Wed Nov 18 08:27:32 1992 From: kolen-j at cis.ohio-state.edu (john kolen) Date: Wed, 18 Nov 92 08:27:32 -0500 Subject: Neural computing ideas and biological terms In-Reply-To: Rogene Eichler's message of Fri, 13 Nov 92 14:06:44 CST <9211132006.AA10502@pi18.arc.umn.edu> Message-ID: <9211181327.AA07750@pons.cis.ohio-state.edu> Rogene Eichler writes However, I grow tired of defending the validity of models to biologists who do not seem satisfied with any model that does not capture every last nuiance of complexity or that does not explain every last experimental finding. Modeling the brain will provide valuable insights into how we process information and how we can exploit those rules for artificial systems. But they do not need to duplicate every last brain dynamic to be useful or valid. This is especially true if it is NOT the case that the details of brain function are the roots of brain behavior. These minute details may be washed out by dynamical principles which have their own behavioral "chemistry". For a mathematical example, look at the universality of symbol dynamics in unimodal iterated mapping (that's just a single bump, like the logistic function). As long as the mappings meet some fairly general qualifications, the iterated systems based on those mappings share the qualitative behavior, namely the bifurcation structure, regardless of the quantitative differences between the individual mappings. John Kolen From Robert.Kentridge at durham.ac.uk Wed Nov 18 12:15:13 1992 From: Robert.Kentridge at durham.ac.uk (Robert.Kentridge@durham.ac.uk) Date: Wed, 18 Nov 92 17:15:13 GMT Subject: Neural computing ideas and biological terms In-Reply-To: <9211132006.AA10502@pi18.arc.umn.edu>; from "Rogene Eichler" at Nov 13, 92 2:06 pm Message-ID: <27835.9211181715@deneb.dur.ac.uk> Rogene Eichler writes: > However, I grow tired of defending the validity of models to biologists > who do not seem satisfied with any model that does not capture every last > nuiance of complexity or that does not explain every last experimental > finding. Modeling the brain will provide valuable insights into how we > process information and how we can exploit those rules for artificial > systems. But they do not need to duplicate every last brain dynamic to > be useful or valid. And when modelers continue to make the claim 'just > like the brain' for the sake of convincing you of the validity of the > model, they are reinforcing the claim that the brain is the ONLY measure. I think a distinction can be drawn here between models which are simplifications of known biology but which don't include biological impossibilities and models which are simple and biologically impossible. Of course, this distinction might be a little in the eye of the beholder, for example I'd argue that, in a network, single compartment neurons are acceptable simplifications which still allow one to draw some conclusions about information processing in biological neural networks, but I know people who would disagree. On the other hand its pretty hard to argue that back-prop is any kind of simplification of biology. Of course, this assumes that your interest is in finding out about biology. If you just want to use networks in their own right then fine (but be wary of leaning on biology too much to justifiy their design!) cheers, bob -- Dr. R.W. Kentridge phone: +44 91 374 2621 Psychology Dept., email: robert.kentridge at durham.ac.uk University of Durham, Durham DH1 3LE, U.K. From sam at vaxserv.sarnoff.com Thu Nov 19 11:24:12 1992 From: sam at vaxserv.sarnoff.com (Scott A. Markel x2683) Date: Thu, 19 Nov 92 11:24:12 EST Subject: NIPS 92 Workshop on Training Issues Message-ID: <9211191624.AA16897@sarnoff.sarnoff.com> **************************** NIPS 92 Workshop **************************** "Computational Issues in Neural Network Training" or Why is Back-Propagation Still So Popular? ******************************************************************************* Roger Crane and I are are leading a NIPS '92 workshop on "Computational Issues in Neural Network Training". Our workshop will be on Saturday, 5 December, the second of two days of workshops in Vail. The discussion will focus on optimization techniques currently used by neural net researchers, and include some other techniques that are available. Back- propagation is still the optimization technique of choice even though there are obvious problems in training with BP: speed, convergence, ... . Several innovative algorithms have been proposed by the neural net community to improve upon BP, e.g., Scott Fahlman's QuickProp. We feel that there are classical optimization techniques that are superior to back-propagation. In fact, gradient descent (BP) fell out of favor with the mathematical optimization folks way backin the 60's! So why is BP still so popular? Topics along these lines include: * Why are classical methods generally ignored? * Computational speed * Convergence criteria (or lack thereof!) Broader issues to be discussed include: * Local minima * Selection of starting points * Conditioning (for higher order methods) * Characterization of the error surface If you would like to present something on any of these or similar topics, please contact me by e-mail and we can discuss details. Workshops are scheduled for a total of four hours. We're allowing for approxi- mately 8 presentations of 10-20 minutes each, since we want to make sure that ample time is reserved for discussion and informal presentations. We will encourage (incite) lively audience participation. By the way, none of the NIPS workshops are limited to presenters only. People who want to show up and just listen are more than welcome. Scott Markel Computational Science Research smarkel at sarnoff.com David Sarnoff Research Center Tel. 609-734-2683 CN 5300 FAX 609-734-2662 Princeton, NJ 08543-5300 From N.E.Sharkey at dcs.ex.ac.uk Fri Nov 20 08:56:14 1992 From: N.E.Sharkey at dcs.ex.ac.uk (Noel Sharkey) Date: Fri, 20 Nov 92 13:56:14 GMT Subject: workshop in UK. Message-ID: <2308.9211201356@propus.dcs.exeter.ac.uk> ******************* CALL FOR DISCUSSION ABSTRACTS ************************* WORKSHOP ON CONNECTIONISM, COGNITION AND A NEW AI A workshop at the 9th Biennial Conference on Artificial Intelligence (AISB-93) at the University of Birmingham, England, during 29th March - 2nd April 1993, organised by the Society for the Study of Artificial Intelligence and Simulation of Behaviour (SSAISB). A number of recent developments in Connectionist Research have strong implications for the future of AI and the study of Cognition. Among the most important are developments in Learning, Representation, and Productivity (or Generalisation). The aim of the workshop would be to focus on how these developments may change the way we look at AI and the study of Cognition. Our goal is to have a lively and invigorating debate on the state-of-the-art. SUGGESTED TOPICS INCLUDE (BUT ARE NOT RESTRICTED TO). * Connectionist representation * Generalisation and Transfer of Knowledge * Learning Machines and models of human deveopmental. * Symbolic Learning versus Connectionist learning * Advantages of Connectionist/Symbolic hybrids. * Modelling Cognitive Neuropsychology * Connectionist modelling of Creativity and music (or other arts). DEADLINE FOR SUBMISSION: 15th December, 1992 ORGANISER Noel Sharkey Centre for Connection Science, Computer Science, Exeter. COMMITTEE Andy Clark (Sussex). Glyn Humphries (Birmingham) Kim Plunkett (Oxford) Chris Thornton (Sussex) WORKSHOP ENTRANCE: Attendance at the workshop will be limited to 50 or 60 places, so please LET US KNOW AS SOON AS POSSIBLE IF YOU ARE PLANNING TO ATTEND, and to which of the following categories you belong. DISCUSSION PAPERS Acceptance of discussion papers will be decided on the basis of extended abstracts (try to keep them under 500 words please) clearly specifying a 15 to 20 minute discussion topic for oral presentation. There will also be a small number of invited contributors. ORDINARY PARTICIPANTS A limited number places will be available for participants who wish to sit in on the discussion but do not wish to present a paper. But please get in early with a short note saying what is your purpose in attending. Please send submissions to Noel E. Sharkey, Centre for Connection Science Dept. Computer Science University of Exeter Exeter EX4 4PT Devon U.K. or email noel at uk.ac.exeter.dcs From ingber at alumni.cco.caltech.edu Sun Nov 22 23:33:27 1992 From: ingber at alumni.cco.caltech.edu (Lester Ingber) Date: Sun, 22 Nov 1992 20:33:27 -0800 Subject: Very Fast Simulated Reannealing code available for beta testing Message-ID: <9211230433.AA16677@alumni.cco.caltech.edu> VERY FAST SIMULATED REANNEALING (VFSR) (C) Lester Ingber ingber at alumni.caltech.edu and Bruce Rosen rosen at ringer.cs.utsa.edu 1. License and Availability 1.1. GNU Copyleft License This Very Fast Simulated Reannealing (VFSR) code is being made available under a GNU COPYING-LIB "copyleft" license, and is owned jointly by Lester Ingber and Bruce Rosen[1]. Please read the copy of this license contained in this directory. 1.2. NETLIB Electronic Availability of VFSR You can obtain our code from NETLIB. This can be done interactively, or you can obtain it by electronic mail request. 1.2.1. Interactive From your local machine login to research.att.com: local% ftp research.att.com Name (research.att.com:your_login_name): netlib Password: [type in your_login_name or anything] ftp> cd opt ftp> binary ftp> get vfsr.Z ftp> quit After `uncompress vfsr.Z' read the header of vfsr for simple directions on obtaining your source files. For example, on most machines, after `sh vfsr' they will reside in a VFSR directory. 1.2.2. Electronic Mail Request Send the following one-line electronic mail request send vfsr from opt [For general NETLIB info, just use: send index] to one of the NETLIB sites: netlib at research.att.com (AT&T Bell Labs, NJ, USA) [most recent version] netlib at ornl.gov (Oak Ridge Natl Lab, TN, USA) netlib at ukc.ac.uk (U Kent, UK) netlib at nac.no (Oslo, Norway) netlib at cs.uow.edu.au (U Wollongong, NSW, Australia) 2. Background and Context VFSR was developed in 1987 to deal with the necessity of performing adaptive global optimization on multivariate nonlinear stochastic systems[2]. VFSR was recoded and applied to several complex systems, in combat analysis[3], finance[4], and neuro- science[5]. A comparison has shown VFSR to be superior to a standard genetic algorithm simulation on a suite of standard test problems[6], and VFSR has been examined in the context of a review of methods of simulated annealing[7]. A project comparing standard Boltzmann annealing with "fast" Cauchy annealing with VFSR has concluded that VFSR is a superior algorithm[8]. A paper has indicated how this technique can be enhanced by combining it with some other powerful algorithms[9]. 2.1. Efficiency Versus Necessity VFSR is not necessarily an "efficient" code. For example, if you know that your cost function to be optimized is something close to a parabola, then a simple gradient Newton search method most likely would be faster than VFSR. VFSR is believed to be faster and more robust than other simulated annealing techniques for most complex problems with multiple local optima; again, be careful to note that some problems are best treated by other algorithms. If you do not know much about the structure of your system, and especially if it has complex constraints, and you need to search for a global optimum, then we heartily recommend our VFSR code to you. 2.2. Outline of Use Set up the VFSR interface: Your program should be divided into two basic modules. (1) The user calling procedure, contain- ing the cost function to be minimized (or its negative if you require a global maximum), here is contained in user.c and user.h. (2) The VFSR optimization procedure, here is contained in vfsr.c and vfsr.h. Furthermore, there are some options to explore in the Makefile. We assume there will be no confusion over the standard uses of the term "parameter" in different con- texts, e.g., as an element passed by a subroutine or as a physi- cal coefficient in a cost function. In VFSR/TESTS we have included some user_out files from some sample runs, containing timed runs on a Sun4c/4.1.3 (SPARC-2) using compilers cc, acc and gcc-2.3.1, and on a Dec5100/Ultrix-4.2 using compilers cc and gcc-2.2.2. No attempt was made to optimize the use of any of these compilers, so that the runs do not really signify any testing of these compilers or architectures; rather they are meant to be used as a guide to determine what you might expect on your own machine. 3. Makefile This file was generated using `make doc'. The Makefile con- tains some options for formatting this file differently, includ- ing the PostScript version README.ps and the text version README. Since complex problems by their nature are often quite unique, it is unlikely that our default parameters are just right for your problem. However, our experience has shown that if you a priori do not have any reason to determine your own parameters, then you might do just fine using our defaults, and we recommend using them as a first-order guess. Most of our defaults can be changed simply by uncommenting lines in the Makefile. Remember to recompile the entire code every time you change any options. Depending on how you integrate VFSR into your own user modules, you may wish to modify this Makefile or at least use some of these options in your own compilation procedures. Read through all the options in the Makefile. As the com- ments therein suggest, it may be necessary to change some of them on some systems. Here are just a couple of examples you might consider: 3.1. SMALL_FLOAT For example, on one convex running our test problem in user.c the SMALL_FLOAT default was too small and the code crashed. A larger value was found to give reasonable results. The reason is that the fat tail of VFSR, associated with high parameter temperatures, is very important for searching the breadth of the ranges especially in the initial stages of search. However, the parameter temperatures require small values at the final stages of the search to converge to the best solution, albeit this is reached very quickly given the exponential sched- ule proven in the referenced publications to be permissible with VFSR. Note that our test problem in user.c is a particularly nasty one, with 1E20 local minima and requiring VFSR to search over many orders of magnitude of the cost function before cor- rectly finding the global minimum. In VFSR/TESTS We have included vfsr_out files comparing results using SMALL_FLOAT=1.0E-16, SMALL_FLOAT=1.0E-18 (the default), and SMALL_FLOAT=1.0E-20. Although the same final results were achieved, the intermediate calculations differ some- what. 3.2. HAVE_ANSI As another example, setting HAVE_ANSI=FALSE will permit you to use an older K&R C compiler. This option can be used if you do not have an ANSI compiler, overriding the default HAVE_ANSI=TRUE. 4. User Module We have set up this module as user.c and user.h. You may wish to combine them into one file, or you may wish to use our VFSR module as one component of a library required for a large project. 4.1. int main(int argc, char **argv) In main, set up your initializations and calling statements to vfsr. In the files user.c and user.h, we have provided a sam- ple program, as well as a sample cost function for your conve- nience. If you do not intend to pass parameters into main, then you can just declare it as main() without the argc and argv argu- ments. 4.2. void user_initialize_parameters() Before calling vfsr, the user must allocate storage and ini- tialize some of the passed parameters. A sample procedure is provided as a template. In this procedure the user should allo- cate storage for the passed arrays and define the minimum and maximum values. Below, we detail all the parameters which must be initialized. If your arrays are of size 1, still use them as arrays as described in user.c. 4.3. double user_cost_function(double *x, int *valid_flag) You can give any name to user_cost_function as long as you pass this name to vfsr. x (or whatever name you pass to vfsr) is an array of doubles representing a set of parameters to evaluate, and valid_flag (or whatever name you pass to vfsr) is the address of an integer. In user_cost_function, *valid_flag should be set to FALSE (0) if the parameters violate a set of user defined con- straints (e.g., as defined by a set of boundary conditions) or TRUE (1) if the parameters represent a valid state. If *valid_flag is FALSE, no acceptance test will be attempted, and a new set of trial parameters will be generated. The function returns a real value which VFSR will minimize. 4.4. double user_random_generator() A random number generator function must be passed next. It may be as simple as one of the UNIX random number generators (e.g. drand48), or may be user defined, but it should return a real value within [0,1) and not take any parameters. We have provided a good random number generator, randflt, and its auxil- iary routines with the code in the file user module. 4.5. void initialize_rng() Most random number generators should be "warmed-up" by call- ing a set of dummy random numbers. 4.6. void print_time(char *message) As a convenience, we have included this subroutine, and its auxiliary routine aux_print_time, to keep track of the time spent during optimization. It takes as its only parameter a string which will be printed. We have given an example in user_cost_function to illustrate how print_time may be called periodically every set number of calls by defining PRINT_FREQUENCY in user.h. 4.7. vfsr( user_cost_function, user_random_generator, number_parameters, parameter_type, parameter_initial_final, final_cost, parameter_minimum, parameter_maximum, tangents, curvature); This is the form of the call to vfsr from user.c. 4.8. void vfsr( double (*user_cost_function) (), double (*user_random_generator) (), int number_parameters, int *parameter_type, double *parameter_initial_final, double final_cost, double *parameter_minimum, double *parameter_maximum, double *tangents, double *curvature) This is how vfsr is defined in the VFSR module, contained in vfsr.c and vfsr.h. Each parameter is described below as it must be passed to this module from the user module. 4.8.1. double (*user_cost_function) () The parameter (*user_cost_function*) () is a pointer to the cost function that you defined in your user module. 4.8.2. double (*user_random_generator) () As discussed above, a pointer to the random number generator function, defined in the user module, must be passed next. 4.8.3. int number_parameters An integer containing the dimensionality of the state space is passed next. Each of the arrays that follow are to be of the size number_parameters. 4.8.4. int *parameter_type This integer array is passed next. Each element of this array (each flag) is either REAL_TYPE (0) (indicating the parame- ter is a real value) or INTEGER_TYPE (1) (indicating the parame- ter can take on only integer values). 4.8.5. double *parameter_initial_final Next, an array of doubles is passed. Initially, this array holds the set of starting parameters which should satisfy any constraints or boundary conditions. Upon return from the VFSR procedure, the array will contain the best set of parameters found by vfsr to minimize the user's cost function. Experience shows that any guesses within the acceptable ranges should suf- fice, since initially the system is at high annealing temperature and VFSR samples the breadth of the ranges. 4.8.6. double final_cost This double should be defined in the calling program. Upon return from the vfsr call, it will be the minimum cost value found by vfsr. 4.8.7. double *parameter_minimum 4.8.8. double *parameter_maximum These two arrays of doubles should also be passed. Since VFSR works only on bounded search spaces, these arrays should contain the minimum and maximum values each parameter can attain. If you aren't sure, try a factor of 10 or 100 times any reason- able values. The exponential temperature annealing schedule should quickly sharpen the search down to the most important region. 4.8.9. double *tangents 4.8.10. double *curvature These two arrays of doubles should be passed last. On return from vfsr, for real parameters, they contain the first and second derivatives of the cost function with respect to its parameters. These can be useful for determining the value of your fit. In this implementation of VFSR, the tangents are used to determine the relative reannealing among parameters. 5. Bug Reports While we do not have time to help you solve your own appli- cations, we do want VFSR to be helpful to a large community. Therefore, we welcome your bug reports and constructive critiques regarding our code. "Flames" will be rapidly quenched. References 1. L. Ingber and B. Rosen, "vfsr," Very Fast Simulated Rean- nealing (VFSR) Source Code, NETLIB Electronic Ftp Archive, netlib at research.att.com (1992). 2. L. Ingber, "Very fast simulated re-annealing," Mathl. Com- put. Modelling, 8, 12, pp. 967-973 (1989). 3. L. Ingber, H. Fujio, and M.F. Wehner, "Mathematical compari- son of combat computer models to exercise data," Mathl. Com- put. Modelling, 1, 15, pp. 65-90 (1991). 4. L. Ingber, "Statistical mechanical aids to calculating term structure models," Phys. Rev. A, 12, 42, pp. 7057-7064 (1990). 5. L. Ingber, "Statistical mechanics of neocortical interac- tions: A scaling paradigm applied to electroencephalogra- phy," Phys. Rev. A, 6, 44, pp. 4017-4060 (1991). 6. L. Ingber and B. Rosen, "Genetic algorithms and very fast simulated reannealing: A comparison," Mathl. Comput. Mod- elling, 11, 16, pp. 87-100 (1992). 7. L. Ingber, "Simulated annealing: Practice versus theory," Statistics Comput., p. (to be published) (1993). 8. B. Rosen, "Function optimization based on advanced simulated annealing," Report, University of Texas, San Antonio, TX (1992). 9. L. Ingber, "Generic mesoscopic neural networks based on sta- tistical mechanics of neocortical interactions," Phys. Rev. A, 4, 45, pp. R2183-R2186 (1992). [*] Some (p)reprints can be obtained via anonymous ftp from ftp.umiacs.umd.edu [128.8.120.23] in the pub/ingber direc- tory. | Prof. Lester Ingber ingber at alumni.caltech.edu # | P.O. Box 857 # | McLean, VA 22101 703-848-1859 = [10ATT]0-700-L-INGBER # From cateau at tkyux.phys.s.u-tokyo.ac.jp Mon Nov 23 21:26:06 1992 From: cateau at tkyux.phys.s.u-tokyo.ac.jp (Hideyuki Cateau) Date: Tue, 24 Nov 92 11:26:06 +0900 Subject: Neural computing ideas and biological terms Message-ID: <9211240226.AA04316@tkyux.phys.s.u-tokyo.ac.jp> In reply to the following discussion: >Rogene Eichler writes > However, I grow tired of defending the validity of models to biologists > who do not seem satisfied with any model that does not capture every last > nuiance of complexity or that does not explain every last experimental > finding. Modeling the brain will provide valuable insights into how we > process information and how we can exploit those rules for artificial > systems. But they do not need to duplicate every last brain dynamic to > be useful or valid. > >This is especially true if it is NOT the case that the details of brain >function are the roots of brain behavior. These minute details may be >washed out by dynamical principles which have their own behavioral >"chemistry". For a mathematical example, look at the universality of >symbol dynamics in unimodal iterated mapping (that's just a single bump, >like the logistic function). As long as the mappings meet some fairly >general qualifications, the iterated systems based on those mappings share >the qualitative behavior, namely the bifurcation structure, regardless of >the quantitative differences between the individual mappings. > >John Kolen I agree to Dr.Kolen. I would like the connectionists to pay attention to the possibility that neural network models can explain not only the qualitative aspects of our brain but also the "quantitative" one of it. In fact, I and my collaborators have found that the learning pace of the back propagation modeland the human brain are subject to a same power law with nearly equal values of the exponent. This is reported in "Power law in the human memory and in the neural network model, H.Cateau, T.Nakajima, H.Nunokawa and N.Fuchikami", which is placed in the neuroprose as a file cateau.power.tar.Z. Let us denote the time which is spent when one memorize M items by t(M). As M increases the learning pace slows down as you usually experience. A psychologist Foucault (M.Foucaut, Annee Psychol.19 (1913)218) found experimentally that this slowing down behavior is described by a following power law: t(M)= const*M^D where D is a constant. He expecially claimed that D=2. We have performed the same experiment by ourself and found that the power law is true with high statistical confidence and that D is between 1 and 2. Then we examined whether or not the back propagation(BP)network has the same property when it memorize some items. The answer was yes. The BP network is subject to the power law with a fairly nice precision. Furthermore the observed value of the exponent was two up to errors. All connectionists can easily check this interesting property by themselves and convince themselves that the fitting of the data to the above law is very good. When we make the BP memorize several items, the memories embedded in the connection weights interfere each other. Thus the slowing down of the learning is expected to occur also fo the BP. This is a qualitative expectation. But above result shows that the similarity is not only qualitative but also quantitative. I think this shows that the BP model, although it is too simple, surely simulate some essential feature of the real brain and that the studies of the neural network model cast a light on a secret of our brain. When we discuss whether or not the neural network model can explain some experimental results of the brain, we usually have, in our mind, physiological experiments. However, there are also many psychological experiments for the human brain. Many of such results are scientifically reliable because the statistical significance of the results were strictly checked. I belive that it is really meaningful as a study of the brain that we examine whether the existing neural network models can explain the many other psychological experiments. Hideyuki Cateau Particle theory group, Department of Physics,University of Tokyo,7-3-1, Hongo,Bunkyoku,113 Japan e-mail:cateau at star.phys.metro-u.ac.jp From lina at mimosa.physio.nwu.edu Mon Nov 23 17:55:19 1992 From: lina at mimosa.physio.nwu.edu (Lina Massone) Date: Mon, 23 Nov 92 16:55:19 CST Subject: paper available Message-ID: <9211232255.AA00498@mimosa.physio.nwu.edu> ************************************************* PLEASE DO NOT FORWARD TO OTHER BOARDS ************************************************ The following paper is available. A VELOCITY-BASED MODEL FOR CONTROL OF OCULAR SACCADES Lina L.E. Massone Department of Physiology Department of Electrical Eng. and Comp. Sci. Northwestern University This paper presents a computational closed-loop model of the saccadic system based on the experimental observation by Munoz, Pellisson and Guitton [1991] that the neural activity on the collicular motor map shifts, during eye movements, towards the rostral area of the superior colliculus. This assumption, together with other assumptions on how the colliculus projects to the burst cells in the brainstem and on the architecture of the network that translates the collicular signal into actual eye movements, results in a system that can: (1) perform a spatio-temporal transformation between a stimulation site on the collicular motor map and an eye movement, (2) spontaneously produce oblique saccades whose trajectories are, as dictated by experimental data, curved when the horizontal and vertical components of the motor error are unequal and straight when the horizontal and vertical components of the motor error are equal, (3) automatically maintain the eye position in the orbit at the end of a saccade by exploiting the internal dynamic of the network, (4) continuously produce efferent copies of the movements without the need for reset signals, (5) reproduce the outcome of the lidocaine experiment by Lee, Roher and Sparks [1988] without assuming a population averaging criterion to combine the activity of collicular cells. This model was developed as part of a theoretical study on the common properties of the eye and arm control systems and on the hypothetical role that the tecto- reticulo-spinal tract might play in the control of arm movements. Munoz, Pellison, Guitton [1991] Movement of neural activity on the superior colliculus motor map during gaze shifts, Science, 251, 1358-1360. Lee, Roher, Sparks [1988] Population coding of saccadic eye movements by neurons in the superior colliculus, Nature, 332, 357-360. A poster will be at NIPS. Email requests to: lina at mimosa.physio.nwu.edu From jlm at crab.psy.cmu.edu Tue Nov 24 09:38:22 1992 From: jlm at crab.psy.cmu.edu (James L. McClelland) Date: Tue, 24 Nov 92 09:38:22 EST Subject: cognition and biology Message-ID: <9211241438.AA18676@crab.psy.cmu.edu.noname> Hideyuki Cateau writes: > I believe that it is really meaningful as a study of the brain that we > examine whether the existing neural network models can explain the many > other psychological experiments. I hope by now nearly everyone agrees that it is very valuable to try to understand which robust phenomena of human cognition depend on which properties of the underlying mechanisms. Sometimes very abstract and general features that connectionist systems share with other systems are doing the work; other times it is going to turn out to be specific features not shared by a wide range of abstract models. The power law appears to be a case of the former, since it has probably been accounted for by more psychological models than any other phenomenon. There are clear cases in which listening to the brain has made a difference to our understanding at the abstract level. The phrase listening to the brain actually comes from a paper of Terry Sejnowski's in which he pointed out the stochastic character of neural activity. This observation contributed importantly to the development of the Boltzmann machine. More recently I have found that another robust regularity of psychological performance, called the independence law, arises from neural network models with bi-directional (symmetric) connections. This does not occur if the network uses the deterministic activation function of McClelland and Rumelhart's interactive activation model but it does occur if the network uses any of a wide range of stochastic activation functions, including the Boltzmann machine activation function and various continuous diffusion functions. These kinds of discoveries make it clear that abstraction is of the essence of understanding, but they also make it clear that it is important to abstract the right things. To me this argues forcefully for an interactive style of research, in which both details and abstractions matter. -- Jay McClelland From zipser at cogsci.UCSD.EDU Tue Nov 24 11:15:27 1992 From: zipser at cogsci.UCSD.EDU (David Zipser) Date: Tue, 24 Nov 1992 08:15:27 -0800 Subject: Neural computing ideas and biological terms Message-ID: <9211241618.AA29605@cogsci.UCSD.EDU> Some of you interested in realistic neural network models of the nervous system may want to look at a recent paper: Zipser, D. (1992). Identification models of the nervous system. Neuroscience, 47, 853-862. David Zipser From eichler at pi18.arc.umn.edu Tue Nov 24 12:54:34 1992 From: eichler at pi18.arc.umn.edu (Rogene Eichler) Date: Tue, 24 Nov 92 11:54:34 CST Subject: networks and biology Message-ID: <9211241754.AA06345@pi18.arc.umn.edu> > I agree to Dr.Kolen. I would like the connectionists to pay attention to > the possibility that neural network models can explain not only the > qualitative aspects of our brain but also the "quantitative" one of it. > > In fact, I and my collaborators have found that the learning pace of the > back propagation modeland the human brain are subject to a same power law > with nearly equal values of the exponent. This is reported in > "Power law in the human memory and in the neural network model, > H.Cateau, T.Nakajima, H.Nunokawa and N.Fuchikami", which is > placed in the neuroprose as a file cateau.power.tar.Z. The results of your work sound very exciting, indeed. But it is important not to get trapped in HUGE statements like ' neural network models can explain not only the qualitative aspects of our brain but also the "quantitative" one of it.' You are basing your statement on the ability of a subset of network models to explain a very small subset of the behaviors that are observable and testable by somewhat similar criteria. Furthermore, it could be argued that the criteria you are using for your comparison is qualitative in nature because of the testing methods employed to measure human performance in some cognitive tasks. Your work has shown that complex network systems can demonstrate similar emergent properties. That statement, supported by the performance measures you cited, is very powerful. But you have substituted one black box for another- nothing can be said quantitatively about how or where brain function occurs. - Rogene From rroberts at pstar.psy.du.edu Tue Nov 24 14:15:01 1992 From: rroberts at pstar.psy.du.edu (rroberts@pstar.psy.du.edu) Date: Tue Nov 24 12:15:01 MST 1992 Subject: No subject Message-ID: From floreen at cs.Helsinki.FI Wed Nov 25 08:09:29 1992 From: floreen at cs.Helsinki.FI (Patrik Floreen) Date: Wed, 25 Nov 92 15:09:29 +0200 Subject: 3 reports available Message-ID: <9211251309.AA20947@hydra.Helsinki.FI> The following 3 reports are now available: "Attraction Radii in Binary Hopfield Nets are Hard to Compute" by Patrik Floreen and Pekka Orponen, University of Helsinki. The name of the file is floreen.attrrad.ps.Z "Neural Networks and Complexity Theory" by Pekka Orponen, University of Helsinki. The name of the file is orponen.nncomp.ps.Z "On the Computational Power of Discrete Hopfield Nets" by Pekka Orponen, University of Helsinki. The name of the file is orponen.hoppow.ps.Z ------------ Abstracts of the papers: "Attraction Radii in Binary Hopfield Nets are Hard to Compute" ABSTRACT: We prove that it is an NP-hard problem to determine the attraction radius of a stable vector in a binary Hopfield memory network, and even that the attraction radius is hard to approximate. Under synchronous updating, the problems are already NP-hard for two-step attraction radii; direct (one-step) attraction radii can be computed in polynomial time. "Neural Networks and Complexity Theory" ABSTRACT: We survey some of the central results in the complexity theory of discrete neural networks, with pointers to the literature. "On the Computational Power of Discrete Hopfield Nets" ABSTRACT: We prove that polynomial size discrete synchronous Hopfield networks with hidden units compute exactly the class of Boolean functions PSPACE/poly, i.e., the same functions as are computed by polynomial space-bounded nonuniform Turing machines. As a corollary to the construction, we observe also that networks with polynomially bounded interconnection weights compute exactly the class of functions P/poly. --------------------------------------------------------------------------- To obtain copies of the postscript files, please use Jordan Pollack's service: Example: unix> ftp cheops.cis.ohio-state.edu # (or ftp 128.146.8.62) Name (cheops.cis.ohio-state.edu:): anonymous Password (cheops.cis.ohio-state.edu:anonymous): ftp> cd pub/neuroprose ftp> binary ftp> get (remote-file) floreen.attrrad.ps.Z (local-file) foo.ps.Z ftp> quit unix> uncompress foo.ps unix> lpr -P(your_local_postscript_printer) foo.ps Likewise for orponen.hoppow.ps.Z and orponen.nncomp.ps.Z ---------------------------------------------------------------------------- If you have any difficulties with the above, please send e-mail to floreen at cs.helsinki.fi. DO NOT "reply" to this message, please. From efiesler at idiap.ch Wed Nov 25 07:42:47 1992 From: efiesler at idiap.ch (E. Fiesler) Date: Wed, 25 Nov 92 13:42:47 +0100 Subject: Paper & software available on modified cascor. Message-ID: <9211251242.AA08738@idiap.ch> Paper available ---------------- The following paper has been published in Proc. Neuro-Nimes '92, Nimes, France, November 1992, pp. 455-466. This paper is available at the IDIAP ftp site. Instructions for obtaining a copy of this paper are given at the end of this message. ----------------------------------------------------------------------- Variations on the Cascade-Correlation Learning Architecture for Fast Convergence in Robot Control Natalio Simon Henk Corporaal Eugene Kerckhoffs Delft University of Technology, The Netherlands Abstract -------- Most applications of Neural Networks in Control Systems use a version of the Back-Propagation algorithm for training. Learning in these networks is generally a slow and very time consuming process. Cascade-Correlation is a supervised learning algorithm that automatically determines the size and topology of the network and is quicker than back-propagation in learning for several benchmarks. We present a modified version of the Cascade-Correlation learning algorithm, which is used to implement the inverse kinematic transformations of a robot arm controller with two and three degrees of freedom. This new version shows faster convergence than the original and scales better to bigger training sets and lower tolerances. ========================================================================= Public Domain Code available ---------------------------- The code of the modified cascade-correlation learning architecture, presented in the above report, is also available at the IDIAP ftp site. Instructions for obtaining a copy of this software are given at the end of this message. A description of the code follows: /********************************************************************************/ /* C implementation of the Modified Cascade-Correlation learning algorithm */ /* */ /* Modified by: N. Simon */ /* Department of Electrical Engineering */ /* Computer Architecture and Digital Systems */ /* Delft University of Technology */ /* 2600 GA Delft, The Netherlands */ /* */ /* E-mail: natalio at zen.et.tudelft.nl */ /* */ /* */ /* This code is a modification of the original code written by R. Scott */ /* Crowder of Carnegie Mellon University (version 1.32). */ /* That code is a port to C from the original Common Lisp implementation */ /* written by Scott E. Fahlman. (Version dated June 1 1990.) */ /* *//* */ /* For an explanation of the original algorithm, see "The */ /* Cascade-Correlation Learning Architecture" by Scott E. Fahlman and */ /* Christian Lebiere in D. S. Touretzky (ed.), "Advances in Neural */ /* Information Processing Systems 2", Morgan Kaufmann, 1990. A somewhat */ /* longer version is available as CMU Computer Science Tech Report */ /* CMU-CS-90-100. */ /* */ /* For an explanation of the Modified Cascade-Correlation learning */ /* see "Variations on the Cascade-Correlation Learning Architecture for */ /* Fast Convergence in Robot Control" by N. Simon, H. Corporaal and */ /* E. Kerckhoffs, in Proc. Neuro-Nimes '92, Nimes, France, 1992, */ /* pp. 455-466. */ /********************************************************************************/ Instructions for obtaining a copy of the paper: unix> ftp Maya.IDIAP.CH (or: ftp 192.33.221.1) login: anonymous password: ftp> cd pub/papers/neural ftp> binary ftp> get simon.variations.ps.Z ftp> bye unix> zcat simon.variations.ps.Z | lpr (or however you uncompress and print a postscript file) Instructions for obtaining a copy of the software: unix> ftp Maya.IDIAP.CH (or: ftp 192.33.221.1) login: anonymous password: ftp> cd pub/software/neural ftp> binary ftp> get mcascor.c.Z ftp> bye unix> uncompress mcascor.c.Z E. Fiesler IDIAP From tomy at maxwell.ee.washington.edu Wed Nov 25 19:03:28 1992 From: tomy at maxwell.ee.washington.edu (Thomas A. Tucker) Date: Wed, 25 Nov 92 16:03:28 PST Subject: NIPS 92 Young Scientists' Network Announcement Message-ID: <9211260003.AA23313@shockley.ee.washington.edu> ANNOUNCING AN INFORMAL NIPS92 PRESENCE -- The Young Scientists' Network -- The YSN is a mailing list dedicated to the "discussion of issues involving the employment of scientists, especially those just beginning their careers." YSN started about two years ago when a group of physics PhD's noted a common thread among their difficulties in securing academic and industrial employment -- but since then scientists in other fields have reported similar situations. Our conclusion has been that there's a glut of scientists, not the shortage that had been predicted for years. Consequently, many well-trained persons are finding themselves under- or un-employed, and, contrary to traditional attitudes, this poor employment record is NOT a reflection of individual abilities so much as it is an indictment of the priorities of American science. The discussions on YSN have identified many contributors to this phenomenon: universities producing too many graduates, insubstantial R&D investments by US companies, research funds drying up, the growing tendency of universities to create non-tenure-track positions when facing budget constraints, and a host of less tangible but undeniably real interpersonal and political pressures. We'd like to offer the opportunity to learn about and get involved with the Young Scientists' Network next week at NIPS. With luck, we'll be able to set a meeting time in Denver by next Monday, and this information should be available when you check in. Our concrete goals: -- an opportunity to learn more about the aims and means of the Young Scientists' Network, with email contact information and sample YSN Digests. -- solicitation of ideas that might be presented in a national legislative or administrative forum (Congress or the NSF, for example), to help foster economic opportunities for research. -- compiling a directory of academic and industrial organizations which might offer potential employment (the responses we receive, the more complete the directory). -- compiling a possible roster of sources for funding. -- compiling a list of graduate programs offering study in computational neuroscience or other connectionist fields with an emphasis on interdisciplinary work. -- we hope to make available survey information which will help to document the nature and magnitude of YSN members' concerns. Of course, we're open to any suggestions you may wish to put forward, and we'll be available for casual discussion throughout the conference. We look forward to seeing you there, Thomas A. Tucker (tomy at ee.washington.edu) Pamela A. Abshire (pa11722 at medtronic.com) From jbower at cns.caltech.edu Wed Nov 25 17:58:25 1992 From: jbower at cns.caltech.edu (Jim Bower) Date: Wed, 25 Nov 92 14:58:25 PST Subject: Brains and Braun Message-ID: <9211252258.AA21496@smaug.cns.caltech.edu> With respect to the recent Biology / abstract modeling discussion. I would like to suggest that the distance between Cognitive Psychology and the details of brain circuitry might very well be as substantial as the distance between connectionist models and brain structure. Accordingly, relations between abstract descriptions of brain function and the performance of abstract networks does not necessarily make me, as a computational neurobiologist, more comfortable with either. Jim Bower From jbower at cns.caltech.edu Wed Nov 25 21:59:54 1992 From: jbower at cns.caltech.edu (Jim Bower) Date: Wed, 25 Nov 92 18:59:54 PST Subject: Details, who needs them?? Message-ID: <9211260259.AA21750@smaug.cns.caltech.edu> >I grow tired of defending the validity of models to biologists > who do not seem satisfied with any model that does not capture >every last nuance of complexity or that does not explain every last >experimental finding. In response to this and several similar statements, I have to say that in now many years of building biologically motivated neural models, and attending neural modeling meetings of all sorts, I have never yet met such a neurobiologist. I have of course met many who object to the kind of "brain-hype" that originally prompted my remarks. However, there can be no question that the issue of the level of detail necessary to account for brain function, or to do something really interesting with neural networks is a subject of active debate. With respect to neural networks, I would point out that this question has been around from the begining of neural network research. Further, not that long ago, many believed, and argued loudly that simple networks could do everything. Several of us said that if that were true, the brain would be simple and because it is not, it is likely that artificial networks would have to get more complex to do anything real or even very interesting. As we head to the NIPS meeting, it is fairly clear that the simple neural networks have not done very well evolutionarily. Further, the derivatives are clear. With respect to the detail necessary to understand the brain, this is also an area of active debate in the young field of computational neuroscience. However, from our own work and that of others, maybe it is time to make the statement that it appears as though the details might matter a great deal. For example, through the interaction of realistic models and experimental work, we have recently stumbled across a regulatory mechanism in the olfactory cerebral cortex that may be involved in switching the network from a learning to a recall state. If correct, this switching mechanism serves to reduce the possibility of the corruption of new memories with old memories. While it would be inappropriate to describe the results in this forum in detail, it turns out that the mechanism bears a resemblance to an approach used by Kohonen to avoid the same problem. Further, when the more elaborate details of the biologically derived mechanism are placed in a Kohonen associative memory, the performance of the original Kohonen net is improved. In this case, however, the connection to Kohonen's work was made only after we performed the biological experiments. This is not because we did not know Kohonen's work, but because the basic mechanism was so unbiological that it would have made little sense to specifically look for it in the network. The biological modeling now done, we can see that Kohonen's approach appears as a minimal implementation of a much more sophisticated, complicated, and apparently more effective memory regulation mechanism. While it is not the common practice on this network or in this field to point out ones own shortcomings, it turns out that we did not know, prior to doing the realistic modeling, which biological details might matter the most. Once they were discovered, it was fairly trivial to modify an abstract model to include them. The point here is that only through paying close attention to the biological details was this mechanism discovered. From this and a few other examples in the new and growing field of computational neuroscience, it may very well be that we will actually have to pay very close attention to the structure of the nervous system if we are going to learn anything new about how machines like the brain work. I acknowledge that this may or may not be eventually relevant to neural networks and connectionism as I am yet to be convinced that these are particularly good models for whatever type of computational object the brain is. However, if there is some connection, it might be necessary to have those interested in advancing the state of artificial networks seek more information about neurobiology than they can obtain at their favorite annual neural network meeting, from a basic neurobiology textbook, or from some "overview" published by some certified leader of the field. Who knows, it might even be necessary to learn how to use an electrode. Jim Bower For those interested in a very general overview of the work described above, I have placed the following review article in neuroprose: "The Modulation of Learning State in a Biological Associative Memory: An in vitro, in vivo, and in computo Study of Object Recognition in Mammalian Olfactory Cortex." James M. Bower To retrieve from neuroprose: unix> ftp cheops.cis.ohio-state.edu Name (cheops.cis.ohio-state.edu:becker): anonymous Password: (use your email address) ftp> cd pub/neuroprose ftp> get bower.ACH.asci.Z 200 PORT command successful. 150 Opening BINARY mode data connection for bower.ACH.asci.Z 226 Transfer complete. ###### bytes received in ## seconds (## Kbytes/s) ftp> quit From cateau at tkyux.phys.s.u-tokyo.ac.jp Thu Nov 26 00:16:29 1992 From: cateau at tkyux.phys.s.u-tokyo.ac.jp (Hideyuki Cateau) Date: Thu, 26 Nov 92 14:16:29 +0900 Subject: cognition and biology Message-ID: <9211260516.AA05354@tkyux.phys.s.u-tokyo.ac.jp> Jay McClelland writes: >Sometimes very >abstract and general features that connectionist systems share with >other systems are doing the work; other times it is going to turn out >to be specific features not shared by a wide range of abstract models. >The power law appears to be a case of the former, since it has >probably been accounted for by more psychological models than any >other phenomenon. I have two things which I would like to say in relation to his comments. First, since I am not a psychologist, I have asked many psychologists about the power law which I am interested in. Then I knew that the power laws are frequently found in psychological experiments such as Stevens' law. I also knew there are various psychological models which derive some of the power laws. But up to now, I have never found in literatures or heard from psychologist, that there is a non-neural-network-based psychological model which explains exactly the same experiment in question. If anyone know such work, please tell it to me. It is very intriguing to me to examine which model is better. Second, I am actually a physisist majoring in an elementary particle physics. Particle physisists generally believe that all the phenomena occuring in this world must be explained, after all, from fundamental laws of the elemenary particles, because this world consists of the elementary particles. In just the same way, I believe that every intellectual phenomenum of our brain is derived from th activities of the neurons of which our brain consists. So I have a tendency to prefer the neural-network-based model to other psychological models. This is the reason why I think my work is meaningful although there might be other psychological models which also explain the the experiment in question. Of course it is wrong to say that one way of thinking is correct and another way is incorrect. Both non-neural-network-based way and neural- network-based way will be useful for our understanding of nature. In a community of particle physics, the two different stand points are clearly separated. Those who on the former stand point are called phenomenologists, while those who on the latter stand point are called theoretical theorists. The former people are trying to find a simple law which reproduces experimental facts based on some assumptipons, but not so serious about why such law holds. The latter people are trying to derive the fundamental laws of physics from the first principle, but they frequently fall into the study of the toy models which are only of academic interest. Anyway, both ways of thinking is necessary for understanding of nature. Hideyuki Cateau From cateau at tkyux.phys.s.u-tokyo.ac.jp Thu Nov 26 07:14:43 1992 From: cateau at tkyux.phys.s.u-tokyo.ac.jp (Hideyuki Cateau) Date: Thu, 26 Nov 92 21:14:43 +0900 Subject: Neural computing ideas ... Message-ID: <9211261214.AA01048@tkyux.phys.s.u-tokyo.ac.jp> Bard Ermentrout writes: >Bacterial cultures in a nutrient depleted medium with inhibited >motility produce fractal colonies that have the same fractal >dimension as diffusion-limited aggregation processes. However the >mechanism for one is totally different from the other. All Hopf >bifurcations from rest scale in the smae fashion. All of these >examples are consequences of the fact that many mathematical >phenomena obey fixed scaling laws. But to say that because backprop >and humnan memory learn with the same scaling laws implies that backprop >has something to do with real learning is at best specious. Just becausee >two phenomena scale the same way doenst mean that the mechanisms are >identical. It is easy to find good examples just like it is easy to find bad ones. Ising model of the magnetism is really simpified model. No one might be able to predict this model is a very good abstraction of the real complex material. However, the scaling behavior in the vicinity of the critical point of metal is correctly understood by Ising model. The model of the superconductivity proposed by B.C.S. was also a good model. It deepen the understanding of the superconductivity of the real material. There are countless number of good examples as well as bad examples. So it does not seem to make sense to judge that our proposal is a bad one being based on the fact that there are many bad examples in the world. As he said it is true that there are various kinds of scaling behavior in nature. There is a potential danger that we are incorrectly convinced that two of them are derived from the same mechanism just because the values of the exponent coincide each other. But we are not only based on the accordance of the exponents but also based on other circumstances in which we are working. As I have noted in the original mail, the reason for the slowing down of the learning pace is considered to be an interfarence between different items which are memorized in the brain. Back prop slows down by the same reason. The configuration of the connection weights Wij which is good for one vector to memorize is not good for another vector to memorize in general. Back prop must find out common grounds to memorize serveral items. It costs time and slows down the memory. At least for us, under these circumstances, it seems natural to expect an analogy between the two slowing downs. Then we performed a simulation to obtain a plausible result. Watching one aspect of the matter is always dangerous. Taking a very simple example, we use a trigonometric function when we measure the hieght of a tall tree. On the other hand, we also have a trigonometric function when we solve the wave eqution. Of course it is unnatural to expect some deep connection between the two situations. But for our case it is natural at least to us. We agree that the evidences might still be not enough. Our work is on the process of exploring more detailed ananlogy between the two systems, the brain and the back prop. If the result of our simulation were negative, the exploring program had quickly reached the end. Our conclusion in that case would have been that there were at least one evidence which indicated back prop was not a good model for our brain. But, at this level of exploration, we think we have got a plausible evidence and it is worth reporting. In the future, we might find an aspect which is not common between the back prop and brain. To reach such conclusion is also meaningful because our purpose is not to prove that the back prop is the best model of the brain but to know to what extent the back prop is a good model of our brain. We think that it is the starting point of the understanding of the brain based on the neural network models. H.Cateau From jlm at crab.psy.cmu.edu Thu Nov 26 08:45:44 1992 From: jlm at crab.psy.cmu.edu (James L. McClelland) Date: Thu, 26 Nov 92 08:45:44 EST Subject: independence law Message-ID: <9211261345.AA27057@crab.psy.cmu.edu.noname> Several readers of connectionists have asked for clarification of the independence law and/or pointers to publications. The independence law (I call it that) is a finding in the literature on the effect of context on perception of near-threshold stimuli (Morton, Psychological Review, 1968). It arises in studies in which one manipulates contextual support for a particular response (e.g. by providing a context such as 'I like coffee with cream and _____' vs 'The next word will be ______'). The '_____' represents the near-threshold stimulus. The context manipulation increases p = the probability of choosing some response of interest (e.g. 'sugar') relative to the neutral condition. Now, the independence law is the finding that context exterts the same effect on the variable log(p/(1-p)), independent of the exact nature of the stimulus. Of course, one needs a marginal stimulus to avoid p = 1 or 0. It had been claimed (Massaro, Cognitive Psychology, 1989) that the independence law is inconsistent with symmetrically connected networks, but this claim was based on simulations of the (non-stochastic) interactive activation model of McClelland and Rumelhart (Psych Review, 1981). I found, though, (McClelland, Cognitive Psychology, 1991) that in fact stochastic, symmetrically connected networks can -- indeed, must -- adhere to the independence law if they adhere to what I called the structural independence constraint on architecture. The constraint is roughly the following: a) the network be structured so that there are three separate pools of units: one for receiving and representing the stimulus input, one for recieving and representing the context, and one for combining these influences to represent the set of possible response alternatives (e.g., words). b) there can be no connections between any of the units in the stimulus input part of the network and any of the units in the context part of the network. In my 91 paper the result was developed for Boltzmann machines using localist representations of letters or words. In recent work with Javier Movellan (not yet ready for circulation) we are establishing considerable generality to these results and relating the findings to other literatures. It turns out that what I called structural independence in a stochastic symmetric network is tantamount to adherence to an assumption called 'conditional independence' that is used in pattern recognition in order to select the response with the maximum a posteriori probability (MAP) given context and stimulus information. We will announce this paper when it is ready on connectionists. From cateau at tkyux.phys.s.u-tokyo.ac.jp Thu Nov 26 08:44:11 1992 From: cateau at tkyux.phys.s.u-tokyo.ac.jp (Hideyuki Cateau) Date: Thu, 26 Nov 92 22:44:11 +0900 Subject: networks and biology Message-ID: <9211261344.AA11468@tkyux.phys.s.u-tokyo.ac.jp> Rogene Eichler writes: >................. >................. >You are basing your statement on the ability of a subset of network models >to explain a very small subset of the behaviors that are observable and >testable by somewhat similar criteria. Furthermore, it could be argued that >the criteria you are using for your comparison is qualitative in nature >because of the testing methods employed to measure human performance in >some cognitive tasks. > >Your work has shown that complex network systems can demonstrate similar >emergent properties. That statement, supported by the performance measures >you cited, is very powerful. But you have substituted one black box for >another- nothing can be said quantitatively about how or where brain function >occurs. I agree with the first statement of the upper paragraph above. We should examine various kinds of neural network model to see the universality of the power law, although we have checked it for various parameters of the back prop model. He further says that "... explain a very small subset of the behaviors ...". He is completely correct. But I am not so serious about the point. At the first sight, the back prop model is too simple a model to be regarded as model of the brain. I never think that the back prop can model the whole behavior of our complex brain. Only one point of similarity between the back prop and the brain is surprising to me, and I thought it is worth reporting. I know that it is not the first dicovery of the similiarity between neural network models and the real brain. I argued that the novelty of our result is that our result is quantitative instead of qualitative. At this point our opinions disagree with each other. He writes: >... >the criteria you are using for your comparison is qualitative in nature >because of the testing methods employed to measure human performance in >some cognitive tasks. >... First of all, I do not think that measuring the human cognitive tasks are always qualitative. I am a physisist. So, when I was not familiar with works of psychologists, I certainly thought that most psychological experiments would be only qualitative. I believed that they had in general low reproducibility, compared with physical experiments. But I changed my idea after I read many psychological papers and I myself performed a psycho- logical experiment of the power law in question. The power law was very stable. The value of the exponent varies depending on persons or other factors, but the value always fell within the range between 1 and 2. This is a definite quantitative fact. The exponent for back prop was two up to errors, as I wrote in the original mail. This is nothing but a quatitative accordance. If the exponent for back prop had been observed to be 10, for example, we would have concluded that the behaviors of the brain and back prop qualitatively coincided, but the accordance were not quantitative. I agree that our result have not opened a black box, which he mentioned in his last paragraph. It would be a very long way to the point when we open the black box when we finally see the secret of the brain. What we could do now is to hit the black box and carefully hear the sound it emits, to tumble it down and observe the reaction etc. We, in some sense, hit the black box by the psychological experiment and got the power law as reaction of it. On the other hand we verified that it was the same reaction when we hit the back prop. It is encouraging to the neural networkers who believe that the study of the neural network model is useful for the study of the brain. H.Cateau From schwaber at eplrx7.es.duPont.com Wed Nov 25 10:11:01 1992 From: schwaber at eplrx7.es.duPont.com (Jim Schwaber) Date: Wed, 25 Nov 92 10:11:01 EST Subject: NIPS workshop - REAL biological computation Message-ID: <9211251511.AA12325@eplrx7.es.duPont.com> -----------NIPS 92 WORKSHOP---------------------- Real Applications of Real Biological Circuits or "If back-prop is not enough how will we get more?" or "Is anybody really getting anywhere with biology?" --------------------------------------------------- When: Friday, Dec. 4th ==== Intended Audience: Those interested in detailed biological modeling. ================== Those interested in nonlinear control. Those interested in neuronal signal processing. Those interested in connecting the above. Organizers: =========== Richard Granger Jim Schwaber granger at ics.uci.edu schwaber at eplrx7.es.dupont.com Agenda: ======= Morning Session, 7:30 - 9:30, Brain Control Systems and Chemical --------------- Process Control Jim Schwaber Brainstem reflexes as adaptive controllers Dupont Babatunde Ogunnaike Reverse engineering brain control systems DuPont Frank Doyle Neurons as nonlinear systems for control Purdue John Hopfield Discussant Caltech Afternoon Session, 4:30 - 6:30, Real biological modeling, nonlinear ----------------- systems and signal processing Richard Granger Signal processing in real neural systems: is UC Irvine it applicable? Gary Green The single neuron as a nonlinear system - its Newcastle Volterra kernels as described by neural networks. Program: ======== We anticipate that the topic will generate several points of view. Thus, presenters will restrict themselves to a very, very few slides intended to make a point for discussion. Given that there now are concrete examples of taking biological principles to application, we expect the discussion will center more on how, and at what level, rather than whether "reverse engineering the brain" is useful. Granger (UC Irvine): ------- The architectures, performance rules and learning rules of most artificial neural networks are at odds with the anatomy and physiology of real biological neural circuitry. For example, mammalian telencephelon (forebrain) is characterized by extremely sparse connectivity (~1-5%), almost entirely lacks dense recurrent connections, and has extensive lateral local circuit connections; inhibition is delayed-onset and relatively long-lasting (100s of milliseconds) compared to rapid-onset brief excitation (10s of milliseconds), and they are not interchangeable. Excitatory connections learn, but there is very little evidence for plasticity in inhibitory connections. Real synaptic plasticity rules are sensitive to temporal information, are not Hebbian, and do not contain "supervision" signals in any form related to those common in ANNs. These discrepancies between natural and artificial NNs raise the question of whether such biological details are largely extraneous to the behavioral and computational utility of neural circuitry, or whether such properties may yield novel rules that confer useful computational abilities to networks that use them. In this workshop we will explicitly analyze the power and utility of a range of novel algorithms derived from detailed biology, and illustrate specific industrial applicatons of these algorithms in the fields of process control and signal processing. Ogunnaike (DuPont): ----------- REVERSE ENGINEERING BRAIN CONTROL SYSTEMS: EXPLORING THE POTENTIAL FOR APPLICATIONS IN CHEMICAL PROCESS CONTROL. ===================================================================== The main motivation for our efforts lies in the simple fact that there are remarkable analogies between the human body and the chemical process plant. Furthermore, it is known that while the brain has been quite successful in performing its task as the central supervisor of intricate control systems operating under conditions which leave very little margin for error, the control computer in the chemical process plant has not been so successful. We have been concerned with seeking answers to the following question: ``Is it possible to ``reverse engineer'' a biological control system and use the understanding to develop novel approaches to chemical process control systems design and analysis?'' Our discussion will provide an overview of the tentative answers we have to date. We will first provide a brief summary of the salient features and main problems of chemical process control; we will then introduce the biological control system under study (the baroreceptor vagal reflex); finally we will present an actual industrial process whose main features indicate that it may benefit from the knowledge garnered from the neurobiological studies. Doyle (Purdue): ------ We are focusing our research on two levels: 1) Neuron level: investigating novel building blocks for process modeling applications which are motivated by realistic biological neurons. 2) Network Level: looking for novel approaches to nonlinear dynamic scheduling algorithms for process control and modeling (again, motivated by biological signal processing in the baroreceptor reflex). Green (Newcastle): ------- I would love to tell the NIPS people about Volterra series, especially as we have now made a connection between neural networks, Volterra series and the differential geometric representation of networks. This allows us to say why one, two or more layers are necessary for a particular analytic problem. We can also say how to invert nets which are homeomorphic in their mappings. More importantly for us biologists we can turn the state equations of membrane currents, using neural networks into approximate Volterra kernels which I think (!) helps understand the dynamics. This gives a solution to the differential equations, albeit an approximate one in practical terms. The equations are invertible and therefore allow a formal link between current clamp and voltage clamp at the equation level. The method we have used to do this is of interest to chem. eng. people because we can use the same concepts in non-linear control. It appears at first glance that we can link the everyday use of neural networks to well established theory through a study of tangent spaces of networks. We construct a state space model of a plant, calculate the differential of the rate of change of output with respect to the input. Calculate the same for a neural network. Compare coefficients. The solution to the set of simultaneous equations for the coefficents produces a network which is formally equivalent to the solution of the original differential equation which defined the state equations. We will be making the claim that analytic solutions of non-linear differential equations is possible using neural networks for some problems. For all other problems an approximate solution is possible but the architecture that must be used can be defined. Last I'll show how this is related to the old techniques using Volterra series and why the kernels and inverse transforms can be directly extracted from networks. I think it is a new method of solving what is a very old problem. All in 20 minutes ! From zhang at psych.lsa.umich.edu Fri Nov 27 10:12:04 1992 From: zhang at psych.lsa.umich.edu (zhang@psych.lsa.umich.edu) Date: Fri, 27 Nov 92 10:12:04 EST Subject: No subject Message-ID: <9211271512.AA00495@parvo.psych.lsa.umich.edu> Position in Cognitive Psychology University of Michigan The University of Michigan Department of Psychology invites applications for a tenure-track position in the area of Cognition, beginning September 1, 1993. The appointment will most likely be made at the Assistant Professor level, but it is possible at any rank. We seek candidates with primary interests and technical skills in cognitive psychology. Our primary goal is to hire an outstanding cognitive psychologist, and thus we will look at candidates with any specific research interest. We have a preference for candidates interested in higher mental processes or for candidates with computational modeling skills (including connectionism). Responsibilities include graduate and undergraduate teaching, as well as research and research supervision. Send curriculum vitae, letters of reference, copies of recent publications, and a statement of research and teaching interests no later than January 8, 1993 to: Gary Olson, Chair, Cognitive Processes Search Committee, Department of Psychology, University of Michigan, 330 Packard Road, Ann Arbor, Michigan 48104. The University of Michigan is an Equal Opportunity/Affirmative Action employer. From zhang at psych.lsa.umich.edu Sun Nov 29 17:47:23 1992 From: zhang at psych.lsa.umich.edu (zhang@psych.lsa.umich.edu) Date: Sun, 29 Nov 92 17:47:23 EST Subject: No subject Message-ID: <9211292247.AA00606@parvo.psych.lsa.umich.edu> Position in Cognitive Psychology University of Michigan The University of Michigan Department of Psychology invites applications for a tenure-track position in the area of Cognition, beginning September 1, 1993. The appointment will most likely be made at the Assistant Professor level, but it is possible at any rank. We seek candidates with primary interests and technical skills in cognitive psychology. Our primary goal is to hire an outstanding cognitive psychologist, and thus we will look at candidates with any specific research interest. We have a preference for candidates interested in higher mental processes or for candidates with computational modeling skills (including connectionism). Responsibilities include graduate and undergraduate teaching, as well as research and research supervision. Send curriculum vitae, letters of reference, copies of recent publications, and a statement of research and teaching interests no later than January 8, 1993 to: Gary Olson, Chair, Cognitive Processes Search Committee, Department of Psychology, University of Michigan, 330 Packard Road, Ann Arbor, Michigan 48104. The University of Michigan is an Equal Opportunity/Affirmative Action employer. From ken at cns.caltech.edu Sun Nov 29 09:29:29 1992 From: ken at cns.caltech.edu (Ken Miller) Date: Sun, 29 Nov 92 06:29:29 PST Subject: No subject Message-ID: <9211291429.AA06827@zenon.cns.caltech.edu> POSTDOCTORAL POSITIONS COMPUTATIONAL NEUROSCIENCE UNIVERSITY OF CALIFORNIA, SAN FRANCISCO I will soon be beginning a new lab at UCSF, and anticipate several positions for postdocs beginning in 1993 and 1994 (prospective graduate students are also encouraged to apply to the UCSF Neuroscience Program). The lab will focus on understanding both development and mature processing in the cerebral cortex. Theoretical, computational, and experimental approaches will be taken. Candidates should have skills relevant to one or more of those approaches. The most important criteria are demonstrated scientific ability and creativity, and a deep interest in grappling with the details of neurobiology and the brain. Past work has focused on modeling of development in visual cortex under Hebbian and similar ``correlation-based" rules of synaptic plasticity. The goal has been to understand these rules in a general way that allows experimental predictions to be made. Models have been formulated for the development of ocular dominance and orientation columns. A few references are listed below. Future work of the lab will extend the developmental modeling, and will also take various approaches to understanding mature cortical function. These will include detailed biophysical modeling of visual cortical networks, many-cell recording from visual cortex, and use of a number of theoretical methods to guide and interpret this recording. There will also be opportunities for theoretical forays in new directions, in particular in collaborations with the other Neuroscientists at UCSF. Facilities to develop new experimental directions that are relevant to the lab's program, for example slice studies and use of optical methods, will also exist. I will be part of the Keck Center for Systems Neuroscience at UCSF, which will be a very interactive environment for Systems Neurobiology. Other members will include: * Alan Basbaum (pain systems); * Allison Doupe (song learning in songbirds); * Steve Lisberger (oculomotor system); * Michael Merzenich (adult cortical plasticity); * Christof Schreiner (auditory system); * Michael Stryker (visual system, development and plasticity); Closely related faculty members include Roger Nicoll (hippocampus, LTP); Rob Malenka (hippocampus, LTP); Howard Fields (pain systems); and Henry Ralston (spinal cord and thalamus). Please send a letter describing your interests and a C.V., and arrange to have three letters of recommendation sent to Ken Miller Division of Biology 216-76 Caltech Pasadena, CA 91125 ken at cns.caltech.edu Some References: Miller, K.D. (1992). ``Models of Activity-Dependent Neural Development." Seminars in the Neurosciences, 4:61-73. Miller, K.D. (1992). ``Development of Orientation Columns Via Competition Between ON- and OFF-Center Inputs." NeuroReport 3:73-76. MacKay, D.J.C. and K.D. Miller (1990). ``Analysis of Linsker's simulations of Hebbian rules," Neural Computation 2:169-182. Miller, K.D. (1990). ``Correlation-based mechanisms of neural development," in Neuroscience and Connectionist Theory, M.A. Gluck and D.E. Rumelhart, Eds. (Lawrence Erlbaum Associates, Hillsdale NJ), pp. 267-353. Miller, K.D., J.B. Keller and M.P. Stryker (1989). ``Ocular dominance column development: analysis and simulation," Science 245:605-615. Miller, K.D., B. Chapman and M.P. Stryker (1989). ``Responses of cells in cat visual cortex depend on NMDA receptors," Proc. Nat. Acad. Sci. USA 86:5183-5187. From moody at chianti.cse.ogi.edu Sat Nov 28 17:35:24 1992 From: moody at chianti.cse.ogi.edu (John Moody) Date: Sat, 28 Nov 92 14:35:24 -0800 Subject: NATO ASI on Statistics and Neural Networks Message-ID: <9211282235.AA08058@chianti.cse.ogi.edu> NATO Advanced Studies Institute (ASI) on Statistics and Neural Networks June 21 - July 2, 1993, Les Arcs, France Directors: Professor Vladimir Cherkassky Department of Electrical Eng. University of Minnesota, Minneapolis, MN 55455 tel.(612) 625-9597 fax (612) 625- 4583 email: cherkass at ee.umn.edu Professor Jerome H. Friedman Statistics Department Stanford University Stanford, CA 94309 tel (415 )723-9329 fax(415) 926-3329 email: jhf at playfair.stanford.edu Professor Harry Wechsler Computer Science Department George Mason University Fairfax VA 22030 tel (703) 993-1533 fax (703) 993-1521 email: wechsler at gmuvax2.gmu.edu List of invited lecturers: I. Alexander, L. Almeida, A. Barron, A. Buja, E. Bienenstock, G. Carpenter, V. Cherkassky, T. Hastie, F. Fogelman, J. Friedman, H. Freeman, F. Girosi, S. Grossberg, J. Kittler, R. Lippmann, J. Moody, G. Palm, R. Tibshirani, H. Wechsler, C. Wellekens. Objective, Agenda and Participants: Nonparametric estimation is a problem of fundamental importance for many applications involving pattern classification and discrimination. This problem has been addressed in Statistics, Pattern Recognition, Chaotic Systems Theory, and more recently in Artificial Neural Network (ANN) research. This ASI will bring together leading researchers from these fields to present an up-to-date review of the current state-of-the art, to identify fundamental concepts and trends for future development, to assess the relative advantages and limitations of statistical vs neural network techniques for various pattern recognition applications, and to develop a coherent framework for the joint study of Statistics and ANNs. Topics range from theoretical modeling and adaptive computational methods to empirical comparisons between statistical and neural network techniques. Lectures will be presented in a tutorial manner to benefit the participants of ASI. A two-week programme is planned, complete with lectures, industrial/government sessions, poster sessions and social events. It is expected that over seventy students (which can be researchers or practitioners at the post-graduate or graduate level) will attend, drawn from each NATO country and from Central and Eastern Europe. The proceedings of ASI will be published by Springer-Verlag. Applications: Applications for participation at the ASI are sought. Prospective students, industrial or government participants should send a brief statement of what they intend to accomplish and what form their participation would take. Each application should include a curriculum vitae, with a brief summary of relevant scientific or professional accomplishments, and a documented statement of financial need (if funds are applied for). Optionally, applications may include a one page summary for making a short presentation at the poster session. Poster presentations focusing on comparative evaluation of statistical and neural network methods and application studies are especially sought. For junior applicants, support letters from senior members of the professional community familiar with the applicant's work would strengthen the application. Prospective participants from Greece, Portugal and Turkey are especially encouraged to apply. Costs and Funding: The estimated cost of hotel accommodations and meals for the two-week duration of the ASI is US$1,600. In addition, participants from industry will be charged an industrial registration fee, not to exceed US$1,000. Participants representing industrial sponsors will be exempt from the fee. We intend to subsidize costs of participants to the maximum extent possible by available funding. Prospective participants should also seek support from their national scientific funding agencies. The agencies, such as the American NSF or the German DFG, may provide some ASI travel funds upon the recommendation of an ASI director. Additional funds exist for students from Greece, Portugal and Turkey. We are also seeking additional sponsorship of ASI. Every sponsor will be fully acknowledged at the ASI site as well as in the printed proceedings. Correspondence and Registration: Applications should be forwarded to Dr. Cherkassky at the above address. Applications arriving after March 1, 1993 may not be considered. All approved applicants will be informed of the exact registration arrangements. Informal email inquiries can be addressed to Dr. Cherkassky at nato_asi at ee.umn.edu. From jbower at cns.caltech.edu Mon Nov 30 20:00:30 1992 From: jbower at cns.caltech.edu (Jim Bower) Date: Mon, 30 Nov 92 17:00:30 PST Subject: paper Message-ID: <9212010100.AA25727@smaug.cns.caltech.edu> Sorry, the paper I refered to in my previous posting is now in neuroprose and renamed: bower.ach.asc.Z Jim Bower From ingber at alumni.cco.caltech.edu Mon Nov 30 10:17:21 1992 From: ingber at alumni.cco.caltech.edu (Lester Ingber) Date: Mon, 30 Nov 1992 07:17:21 -0800 Subject: Very Fast Simulated Reannealing version 6.20 Message-ID: <9211301517.AA01764@alumni.cco.caltech.edu> VERY FAST SIMULATED REANNEALING (VFSR) (C) Lester Ingber ingber at alumni.caltech.edu and Bruce Rosen rosen at ringer.cs.utsa.edu The good news is that the people who have gotten our beta version of VFSR to work on their applications are very pleased. The bad news is that because of some blunders made in the process of making the code user-friendly, the code has to be modified to use as a standalone function call. This bug is corrected and some other fixes/changes are made in version v6.20. This version is now updated in netlib at research.att.com. It will eventually find its way into the other NETLIB archives. To access the new version: Interactive local% ftp research.att.com Name (research.att.com:your_login_name): netlib Password: [type in your_login_name or anything] ftp> cd opt ftp> binary ftp> get vfsr.Z ftp> quit local% uncompress vfsr.Z local% sh vfsr Electronic Mail Request local% mail netlib at research.att.com [mail netlib at ornl.gov] [mail netlib at ukc.ac.uk] [mail netlib at nac.no] [mail netlib at cs.uow.edu.au] send vfsr from opt ^D [or however you send mail] Lester || Prof. Lester Ingber ingber at alumni.caltech.edu || || P.O. Box 857 || || McLean, VA 22101 703-848-1859 = [10ATT]0-700-L-INGBER || From BARNEV%BGEARN.BITNET at BITNET.CC.CMU.EDU Mon Nov 2 13:03:24 1992 From: BARNEV%BGEARN.BITNET at BITNET.CC.CMU.EDU (IT&P) Date: Mon, 02 Nov 92 13:03:24 BG Subject: Conference Announcement Message-ID: ----------------------------Original message---------------------------- DEAR COLLEAGUE, I would like to invite you to the 18. issue of the International INFORMATION TECHNOLOGIES AND PROGRAMMING conference to be held in Sofia, the capital of Bulgaria from 27 June to 4 July 1993. MAIN TOPICS: - Information Technologies and Telecommunications in Business and Public Administration - Hypertext and Multimedia Systems - Graphical Methods for Scientific and Technical Computing PROGRAMME COMMITTEE: L. AIELLO (Universita di Roma 'La Sapienza', Italy) M. Mac an AIRCHINNIGH (Trinity College, Dublin, Ireland) P. BARNEV (Institute of Mathematics, Sofia, Bulgaria) - chairman J. HOFFER (Indiana University, Bloomington, USA) S. KERPEDJIEV (Institute of Mathematics, Sofia, Bulgaria) B. KOKINOV (Institute of Mathematics, Sofia, Bulgaria) V. KOTOV (Institute of Informatics Systems, Novosibirsk, Russia) N. SPYRATOS (Universite de Paris-Sud, Paris, France) N. STREITZ (Gesellschaft fuer Mathematik und Daterverarbeitung, Darmstadt, Germany) C. THANOS (Instituto de Elaborazione della Informazione, Pisa, Italy) T. VAMOS (Hungarian Academy of Sciences, Budapest, Hungary) The deadline for submitting papers is February15, 1993. Electronic submission is welcome but the final camera-ready form should be in hard-copy form. All papers will be reviewed by members of the International Programme Committee. As a member of the Program Committee I would like to encourage the submission of connectionist papers. To receive more information, please contact me or Prof. Barnev (barnev at bgearn). Sincerely yours, Boicho Kokinov