From rich at gte.com Fri Feb 1 14:56:12 1991 From: rich at gte.com (Rich Sutton) Date: Fri, 1 Feb 91 14:56:12 -0500 Subject: New Book: Neural Networks for Control Message-ID: <9102011956.AA19613@bunny.gte.com> In the final hours of 1990, MIT Press printed a new book that I hope will be of interest to readers of this mailing list: NEURAL NETWORKS FOR CONTROL edited by W. T. Miller, R. S. Sutton and P. J. Werbos This is an edited collection with articles by Barto, Narendra, Widrow, Albus, Anderson, Selfridge, Sanderson, Kawato, Atkeson, Mel, Williams, and many others. - Rich Sutton From mehra%aquinas at uxc.cso.uiuc.edu Fri Feb 1 18:04:52 1991 From: mehra%aquinas at uxc.cso.uiuc.edu (Pankaj Mehra) Date: Fri, 1 Feb 91 17:04:52 CST Subject: Usefulness of chaotic dynamics Message-ID: <9102012304.AA00422@hobbes> Fellow Connectionists: It was with fascination that I read Gleick's book on chaos recently. My initial reaction was that the thoery of chaotic dynamics of nonlinear systems represents a significant advance in science by allowing us to model and understand complex phenomena, such as turbulence. Yesterday, John Hopfield was in Champaign, talking about ``Neural Computation'' at the centennial celebration of Physics department. During his talk, he made a distinction between useful and ``unlikely to be useful'' dynamics of neural networks with feedback. Essentially, dynamics which show convergent behavior are considered useful, and those that show chaotic or divergent behavior, not useful. The crux of such an argument lies in the fact that ``similar but unresolvable initial conditions lead to large divergences in trajectories'' [1] for a chaotic system. This is an antithesis of generalization, which is considered a defining trait of connectionist models. However, a recent paper by Frison [1] shows how the behavior of chaotic systems may be predicted using neural networks. The applications suggested in this paper include weather forecasting and securities analysis. In my opinion, there is an emerging trend to move away from convergent dynamics. Section 4 of Hirsch's paper [2] presents arguments for/against using chaotic dynamics. In a limited sense, the work of Michail Zak [3] shows how breaking some traditional assumptions of convergent behavior can help solve the problems of spurious and local minima. I would like to hear more about the pros and cons of this issue. - Pankaj Mehra {p-mehra at uiuc.edu, mehra at cs.uiuc.edu, mehra at aquinas.csl.uiuc.edu} [1] Frison, T. W., "Predicting Nonlinear and Chaotic Systems Behavior Using Neural Networks," Jrnl. Neural Network Computing, pp. 31-39, Fall 1990. [2] Hirsch, M. W., "Convergent Activation Dynamics is Continuous Time Networks," Neural Networks, vol. 2, pp. 331-349, Pergamon Press, 1989. [3] Zak, M., "Creative Dynamics Approach to Neural Intelligence," Biological Cybernetics, vol. 64, pp. 15-23, Springer-Verlag, 1990. From koch%CITIAGO.BITNET at vma.CC.CMU.EDU Sat Feb 2 16:05:15 1991 From: koch%CITIAGO.BITNET at vma.CC.CMU.EDU (Christof Koch) Date: Sat, 2 Feb 91 13:05:15 PST Subject: Tech report on 'Awareness" Message-ID: <910202130456.204011e6@Iago.Caltech.Edu> The following paper is available by anyonymous FTP from Ohio State University from pub/neuroprose. The manuscript, a verbatim copy of the same named manuscript which appeared in "Seminars in the Neurosciences", Volume 2, pages 263-273, 1990, is stored in three files called koch.awareness1.ps.Z, koch.awareness2.ps.Z and koch.awareness3.ps.Z For instructions on how to get this file, see below. TOWARDS A NEUROBIOLOGICAL THEORY OF CONSCIOUSNESS Francis Crick and Christof Koch Visual awareness is a favorable form of consciousness to study neurobiologically. We propose that it takes two forms: a very fast form, linked to iconic memory, that may be difficult to study; and a somewhat slower one involving visual attention and short-term memory. In the slower form an attentional mechanism transiently binds together all those neurons whose activity relates to the relevant features of a single visual object. We suggest that this is done by generating coherent semi-synchronous oscillations, probably in the 40 Hz range. These oscillations then activate a transient short-term (working) memory. We outline several lines of experimental work that might advance the understanding of the neural mechanisms involved. The neural basis of very short-term memory especially needs more experimental study. Key words: consciousness / awareness/ visual attention / 40 Hz oscillations / short-term memory. For comments, send e-mail to koch at iago.caltech.edu. Christof P.S. And this is how you can FTP and print the file: unix> ftp cheops.cis.ohio-state.edu (or 128.146.8.62) Name: anonymous Password: neuron ftp> cd pub/neuroprose (actually, cd neuroprose) ftp> binary ftp> get koch.awareness1.ps.Z ftp> get koch.awareness2.ps.Z ftp> get koch.awareness3.ps.Z ftp> quit unix> uncompress koch.awareness1.ps.Z unix> lpr koch.awareness1.ps.Z unix> uncompress koch.awareness2.ps.Z unix> lpr koch.awareness2.ps.Z unix> uncompress koch.awareness3.ps.Z unix> lpr koch.awareness3.ps.Z Done! From HORN at vm.tau.ac.il Mon Feb 4 15:51:36 1991 From: HORN at vm.tau.ac.il (David Horn) Date: Mon, 04 Feb 91 15:51:36 IST Subject: Usefulness of chaotic dynamics Message-ID: Pankaj Mehra has raised the issue of usefulness of chaotic dynamics in neural network models following a statement by Hopfield who made a distinction between useful dynamics, which have convergent behavior, and "unlikely to be useful" dynamics which are divergent. I would like to point out the importance of dynamics which are convergent on a short time scale and divergent on a long time scale. We have worked on neural networks which display such behavior. In particular we can model a system which converges to a set of fixed points on a short time scale (thus performing some "useful" computation), and is "free" to move between them on a longer time scale. This kind of freedom stems from the unpredictability of chaotic systems. Such networks will undoubtedly be very useful for modelling unguided thinking processes and decision making. We may still be far from meaningful models of higher cognitive phenomena, but many will agree that these are valuable long-term goals. In the meantime, these networks are of interest because they form good examples of oscillating systems. Our networks are feedback systems of formal neurons to which we attribute dynamical thresholds. Letting a threshold rise when the neuron with which it is associated keeps firing, we simulate fatigue. Some short time after the network moves into an attractor, the fatigue effect destabilizes the attractor and throws the system into a different basin of attraction. This can go on indefinitely. Naturally it leads to an oscillatory behavior. The recent observations of cortical oscillations led to increased interest in oscillating neural networks, of which ours are particular examples. We have developped several models, all of which display spontaneous and induced transitions between memory patterns. Such models are simple testing grounds for hypotheses about the relevance of oscillations to questions of segmentation and binding. All this is possible because we work with systems which allow for periodic and chaotic motion between centers of attraction. References: D. Horn and M. Usher: --------------------- Neural Networks with Dynamical Thresholds, Phys. Rev. A40 (1989) 1036-1044. Motion in the Space of Memory Patterns, Proceedings of the 1989 Int. Joint Conf. on Neural Networks, I-61-69. Excitatory-Inhibitory Networks with Dynamical Thresholds, Int. Journal of Neural Systems 1 (1990) 249-257. Parallel Activation of Memories in an Oscillatory Neural Network Neural Computation 3 (1991) 31 - 43. Segmentation and Binding in an Oscillatory Neural Network submitted to IJCNN-91-Seattle. O. Hendin, D. Horn and M. Usher: -------------------------------- Chaotic behavior of a neural network with dynamical thresholds, Int. J. of Neural Systems, to be published in the next issue. -- David Horn From p-mehra at uiuc.edu Mon Feb 4 13:58:19 1991 From: p-mehra at uiuc.edu (Pankaj Mehra) Date: Mon, 04 Feb 91 12:58:19 CST Subject: Summary of responses: usefulness of chaotic dynamics Message-ID: <9102041858.AA00974@hobbes> **** PLEASE DO NOT FORWARD TO OTHER LISTS/BULLETIN BOARDS **** A few notes: 1. If replying to my message did not work, try one of the e-mail addresses at the end of my original message. 2. The comment: ``Chaos is an antithesis of generalization, a defining trait of connectionist models,'' was mine, not Hopfield's. 3. All the responses so far seem to suggest that chaos is useful in an evolving system. We can have a more focussed discussion if we can answer: 3a. Is chaos a precise quantitative way of stating one's ignorance of the dynamics of the process being modeled/controlled? 3b. What methods are available for implementing controlled chaos? 3c. How can chaos and learning be integrated in neural networks? Of course, discussion on cognitive/engineering significance of chaos is still welcome. --------- RESPONSES RECEIVED Paraphrased portions are enclosed in {}. ********************************************************************** From: Richard Rohwer I think that it is still an open question what sort of dynamics is cognitively useful. I can see the sense in Hopfield's intuition that convergent dynamics is good for generalization, but this doesn't really rule out chaotic dynamics, because although trajectories don't converge to fixed points, they do converge to attractors. Even if the attractors are chaotic, they lie "arbitrarily close" (in an epsilon- delta sense) to specific manifolds in state space which stay put. So there is a contracting mapping between the basin of attraction and the attractor. Anyway, I don't accept that generalization is the only phenomenon in cognition. I find it at least plausible that thought processes do come to critical junctures at which small perturbations can make large differences in how the thoughts evolve. { Pointers to relevant papers: Steve Renals and Richard Rohwer, "A study of network dynamics", J. Statistical Physics, vol. 58, pp.825-847, 1990. Stephen J. Renals, "Speech and Neural Network Dynamics", Ph.D. thesis, Edinburgh University, 1990. Steve Renals, "Chaos in neural networks" in Neural Networks (Almeida and Wellekens, eds.), Lecture Notes in Computer Science 412, Springer -Verlag, Berlin, pp. 90-99, 1990. } ********************************************************************** {some portions deleted} From: Jordan B Pollack There is a lot more to chaos than Sensitivity to initial conditions; there are self-organizing dynamics and computational properties beyond simple limit-point systems. Chaos is almost unavoidable in NN, and has to be suppressed with artifacts like symettric weights and synchronous recurrence. If it is so prevalent in nature, it must be adaptive! (If your heart converges, you die; if your brain converges, you die.) Hopfield is accurate in that neural networks which converge might be as commercially useful as logic gates, but they won't address the question of how high-dimensional dynamical systems self-organize into complex algorithmic structures. My research slogan might be "chaos in brain -> fractals in mind", and believe that more complex dynamics than convergence are quite necessary to getting non-trivial neural cognitive models. Had one paper in NIPS 1 outlining some research proposals, and 2 forthcoming in NIPS3. There are groups in Tel-Aviv, Warsaw, and Edinburgh, at least, working on complex neuro-dynamics. ********************************************************************** From: andreas%psych at Forsythe.Stanford.EDU (Andreas Weigend) {Pointers to related publications: Andreas S. Weigend, Bernardo A. Huberman, and David E. Rumelhart, "Predicting the future: a connectionist approach", International Journal of Neural Systems, vol. 1, pp. 193-209, 1990. Andreas S. Weigend, Bernardo A. Huberman, and David E. Rumelhart, "Back-propagation, weight-elimination and time series prediction" in Proc. 1990 Connectionist Models Summer School, Morgan Kaufmann, pp. 105-116, 1990. Andreas S. Weigend, Bernardo A. Huberman, and David E. Rumelhart, 1990 Lectures in Complex Systems, ?? (eds. Nadel and Stein), Addison-Wesley, 1991. Andreas S. Weigend, Bernardo A. Huberman, and David E. Rumelhart, "Predicting Sunspots and Currency Rates with Connectionist Networks", in Proc. NATO Workshop on Nonlinear Modeling and Forecasting (Santa Fe, Sept. 1990). *********************************************************************** From: David Horn I would like to point out the importance of dynamics which are convergent on a short time scale and divergent on a long time scale. We have worked on neural networks which display such behavior. In particular we can model a system which converges to a set of fixed points on a short time scale (thus performing some "useful" computation), and is "free" to move between them on a longer time scale. This kind of freedom stems from the unpredictability of chaotic systems. {deleted; message directly mailed to Connectionists} *********************************************************************** END OF RESPONSES From aboulanger at BBN.COM Mon Feb 4 20:38:55 1991 From: aboulanger at BBN.COM (Albert G. Boulanger) Date: Mon, 4 Feb 91 20:38:55 EST Subject: Summary of responses: usefulness of chaotic dynamics In-Reply-To: Pankaj Mehra's message of Mon, 04 Feb 91 12:58:19 CST <9102041858.AA00974@hobbes> Message-ID: While the following reference is not directly tied to NNets, it is tied to the the broader program of making computational use of chaos: "Chaotic Optimization and the Construction of Fractals: Solution of an Inverse Problem", Giorgio Mantica & Alan Sloan, Complex Systems 3(1989) 37-62. The agenda here is to make use of the ergodic properties of a dynamical system driven to be chaotic. (Briefly, an ergodic system is one that will pass through every possible dynamical state compatible with its energy.) This corresponds to a high temperature setting in simulated annealing. (A key to the way annealing works is that it, too, is ergodic.) Then they drive the parameter down, and it becomes controlled by repulsive (objective function: worse) and attractive (objective function: better) Coulomb charges. These charges are placed at the successive sites visited. (They also window the number of charges. The repulsive charge makes this system have tabu-search like properties.) Thus, they endow the system with memory at the lower setting of the dynamical parameter. They claim the memory allows more efficient convergence of the algorithm than annealing. Here are some short references on chaos and ergodic theory: "Modern Ergodic Theory" Joel Lebowitz, & Oliver Penrose Physics Today, Feb, 1973, 23-29 "Chaos, Entropy, and, the Arrow of Time" Peter Coveney New Scientist, 29 Sept, 1990, 49-52 (nontechnical) "The Second Law of Thermodynamics: Entropy, Irreversibility, and Dynamics", Peter Coveney, Nature, Vol 333, 2 June 1988, 409-415. (technical) "Strange Attractors, Chaotic Behavior, and Information Flow" Robert Shaw, Z. Naturforsch, 86a(1981), 80-112 "Ergodic Theory, Randomness, and 'Chaos'" D.S. Ornstein Science, Vol 243, 13 Jan 1989, 182- The papers by Coveney and the one by Shaw get into another possible use of chaos involving many-body "nonlinear systems, far from equilibrium". Because of the sensitivity of chaotic systems to external couplings, one can get such systems to act as information (or noise) amplifiers. Shaw puts it as getting information to flow from the microscale to the macroscale. Such self-organizing many-body systems can be used in a generate-and-test architecture as the pattern generators. In neural networks, competitive dynamics can give rise to such behavior. Eric Mjolsness worked on a fingerprint "hallucinator" that worked like this (as I remember). Optical NNets using 4-wave mixing with photorefractive crystals have this kind of dynamics too. Seeking structure in chaos, Albert Boulanger aboulanger at bbn.com From B344DSL at UTARLG.UTA.EDU Tue Feb 5 00:25:00 1991 From: B344DSL at UTARLG.UTA.EDU (B344DSL@UTARLG.UTA.EDU) Date: Mon, 4 Feb 91 23:25 CST Subject: Chaos Message-ID: From: IN%"mehra%aquinas at uxc.cso.uiuc.edu" "Pankaj Mehra" 2-FEB-1991 17:29:15.49 To: Connectionists at CS.CMU.EDU CC: p-mehra at uiuc.edu Subj: Usefulness of chaotic dynamics From mehra%aquinas at uxc.cso.uiuc.edu Fri Feb 1 18:04:52 1991 From: mehra%aquinas at uxc.cso.uiuc.edu (Pankaj Mehra) Date: Fri, 1 Feb 91 17:04:52 CST Subject: Usefulness of chaotic dynamics Message-ID: <9102012304.AA00422@hobbes> Fellow Connectionists: It was with fascination that I read Gleick's book on chaos recently. My initial reaction was that the thoery of chaotic dynamics of nonlinear systems represents a significant advance in science by allowing us to model and understand complex phenomena, such as turbulence. Yesterday, John Hopfield was in Champaign, talking about ``Neural Computation'' at the centennial celebration of Physics department. During his talk, he made a distinction between useful and ``unlikely to be useful'' dynamics of neural networks with feedback. Essentially, dynamics which show convergent behavior are considered useful, and those that show chaotic or divergent behavior, not useful. The crux of such an argument lies in the fact that ``similar but unresolvable initial conditions lead to large divergences in trajectories'' [1] for a chaotic system. This is an antithesis of generalization, which is considered a defining trait of connectionist models. However, a recent paper by Frison [1] shows how the behavior of chaotic systems may be predicted using neural networks. The applications suggested in this paper include weather forecasting and securities analysis. In my opinion, there is an emerging trend to move away from convergent dynamics. Section 4 of Hirsch's paper [2] presents arguments for/against using chaotic dynamics. In a limited sense, the work of Michail Zak [3] shows how breaking some traditional assumptions of convergent behavior can help solve the problems of spurious and local minima. I would like to hear more about the pros and cons of this issue. - Pankaj Mehra {p-mehra at uiuc.edu, mehra at cs.uiuc.edu, mehra at aquinas.csl.uiuc.edu} [1] Frison, T. W., "Predicting Nonlinear and Chaotic Systems Behavior Using Neural Networks," Jrnl. Neural Network Computing, pp. 31-39, Fall 1990. [2] Hirsch, M. W., "Convergent Activation Dynamics is Continuous Time Networks," Neural Networks, vol. 2, pp. 331-349, Pergamon Press, 1989. [3] Zak, M., "Creative Dynamics Approach to Neural Intelligence," Biological Cybernetics, vol. 64, pp. 15-23, Springer-Verlag, 1990. From koch%CITIAGO.BITNET at VMA.CC.CMU.EDU Tue Feb 5 02:00:45 1991 From: koch%CITIAGO.BITNET at VMA.CC.CMU.EDU (Christof Koch) Date: Mon, 4 Feb 91 23:00:45 PST Subject: Memo on "Awareness" Message-ID: <910204230039.2040195e@Iago.Caltech.Edu> For those of you trying to get the Crick and Koch memo on "Towards a neurobiological theory of consciousness". Once you grabed and uncompressed the file, you have to print the uncompressed version of course, i.e. lpr koch.awareness1.ps (and not, as I had indicated, lpr koch.awareness1.ps.Z) Sorry about that, Christof From h1705erd at ella.hu Tue Feb 5 12:00:00 1991 From: h1705erd at ella.hu (Erdi Peter) Date: Tue, 05 Feb 91 12:00:00 Subject: usefulness of chaotic dynamics Message-ID: <9102051104.AA10986@sztaki.hu> Technical report available: SELF-ORGANIZATION IN THE NERVOUS SYSTEM: NETWORK STRUCTURE AND STABILITY (P. Erdi, Central Reseacrh Inst. Physics, Hung. Acad. Sci., H-1525 Budapest, P.O. Box 49, Hungary) to appear in: Martyhematical Approaches to Brain Functioning Diagnostics; Manchaster Univ. Press. (23 pages) 1. Self-organization and neurodynamivcs 1.1. General remarks 1.2. Real neural networks: some remarks 2. Neural network models 2.1. Conceptual and mathematical skeleton 2.2. Neurodynamic problems 3. Neural architectures and dynamic behaviour 3.1. On the "structure - function" problem 3.2. The connectivity - stability dilemma 3.3. Qualitative stability and instability 4. Regular and periodic behaviour in neural networks 4.1. Some remarks on the experimental background of the oscillatory behaviour 4.2. Network architecturw of possyble rhythm generators 5. Network structure and chaos: some designing principles (59 references) From p-mehra at uiuc.edu Tue Feb 5 22:03:52 1991 From: p-mehra at uiuc.edu (Pankaj Mehra) Date: Tue, 05 Feb 91 21:03:52 CST Subject: More responses: Chaos Message-ID: <9102060303.AA01523@hobbes> First, a small note: In his response (sent directly to the list), Albert Boulanger makes a connection between annealing and ergodicity. He then describes an optimization procedure based on slowly constraining the dynamics of a search process. He claims that the algorithm has better convergence than annealing. RESPONSES RECEIVED ********** From: B344DSL at UTARLG.UTA.EDU (???) On the usefulness of chaos in neural models (particularly biologically- related ones), I recommend the following two articles: 1. Skarda, C. & Freeman, W. J. (1987). How brains make chaos to make sense of the world. Behavioral and Brain Sciences 10: 161-195. (This is about the olfactory system, with some speculative generalizations to other mamm- alian sensory systems.) 2. Mpitsos, G. J., Burton, R. M., Creech, H. C., & Soinila, S. O. (1988). Evidence for chaos in spike trains of neurons that generate rhythmic motor patterns. Brain Research Bulletin 21: 529-538. (This is about motor systems in mollusks.) Both of these authors see chaos as a useful biological device for promoting behavioral variability. In the Skarda-Freeman article, there are several commentaries by other people that deal with this issue, including commentaries by Grossberg and by myself that suggest that chaos is not the only possible method for achieving this goal of variability. (There are also responses to the commentaries by the author.) Both Freeman and Mpitsos have pursued this theme in several other articles. ********** From: pluto at cs.UCSD.EDU (Mark Plutowski) With regards to: 3a. Is chaos a precise quantitative way of stating one's ignorance of the dynamics of the process being modeled/controlled? Formally, probability theory (in particular, the theory of stochastic processes) is a precise quantitative way of stating one's ignorance of the dynamics of a process. (Say, by approximation of the empirical data by a sequence of processes which converge to the data probabilistically.) According to my knowledge, a chaotic attractor gives a concise representation which can describe processes which are complicated enough to appear (behaviorally) indistinguishable from random processes. In other words, the benefit of a chaotic model is in its richness of descriptive capability, and its concise formulation. Perhaps chaotic attractors could be employed to provide deterministic models which give a plausible account of the uncertainties inherent in the probabilistic models of our empirical data. Whereas the probabilistic models can predict population behavior, or, the asymptotic behavior of a single element of the population over time, an equally accurate chaotic model may as well be able to predict the behavior of a single element of the population over a finite range of time. I anticipate that the real challenge would be in inferring a chaotic attractor model which can give a testable hypothesis to tell us something that we cannot ascertain by population statistics or asymptotic arguments. Without this latter condition, the chaotic model may be just an unnecessary generalization - why use a chaotic model if more well-understood formalisms are sufficient? However, if it is used to provided a model which is formally convenient, then, the benefit is not necessarily due to descriptive capability, and so, the resulting models need not necessarily give a mechanistic model of the empirical data (and hence, potentially reducing the predictive capability of the models.) Of course, this is all IMHO, due to my expectation that noise inherent in our best experimental data will tend to cause researchers to first fit the data with stochastic processes and classical dynamical systems, and then try to refine these models by using deterministic chaotic models. Perhaps someone better informed on the use of these models could comment on whether this perspective holds water? It may be the case that I am missing out on capabilities of chaotic models which are not apparent in the popular literature. ********* END OF RESPONSES From pollack at cis.ohio-state.edu Tue Feb 5 23:25:42 1991 From: pollack at cis.ohio-state.edu (Jordan B Pollack) Date: Tue, 5 Feb 91 23:25:42 -0500 Subject: Neuroprose Message-ID: <9102060425.AA00927@dendrite.cis.ohio-state.edu> **Do not forward to other lists** Neuroprose seems to be working alright in general, although all recent announcements have just appeared in NEURON-DIGEST, and might have a nasty effect on cheop's load. When you announce a paper, you should consider whether (A) you want it automatically forwarded to other groups, like NEURON-DIGEST, (which gets posted to comp.ai.neural-networks) and if you want to provide (B) FREE or PREPAID alternative hard copies. One author reported that when they allowed combination AB, the rattling around of their "free paper offer" on the worldwide data net generated over 400 hardcopy requests! Jordan Pollack Assistant Professor CIS Dept/OSU Laboratory for AI Research 2036 Neil Ave Email: pollack at cis.ohio-state.edu Columbus, OH 43210 Fax/Phone: (614) 292-4890 From IP%IRMKANT.BITNET at VMA.CC.CMU.EDU Wed Feb 6 12:24:05 1991 From: IP%IRMKANT.BITNET at VMA.CC.CMU.EDU (stefano nolfi) Date: Wed, 06 Feb 91 13:24:05 EDT Subject: Technical report available Message-ID: The following technical report is now available. The paper has been submitted to ICGA-91. Send request to stiva at irmkant.Bitnet e-mail comments and related references are appreciated AUTO-TEACHING: NETWORKS THAT DEVELOP THEIR OWN TEACHING INPUT Stefano Nolfi Domenico Parisi Institute of Psychology CNR - Rome E-mail: stiva at irmkant.Bitnet ABSTRACT Back-propagation learning (Rumelhart, Hinton and Williams, 1986) is a useful research tool but it has a number of undesiderable features such as having the experimenter decide from outside what should be learned. We describe a number of simulations of neural networks that internally generate their own teaching input. The networks generate the teaching input by trasforming the network input through connection weights that are evolved using a form of genetic algorithm. What results is an innate (evolved) capacity not to behave efficiently in an environment but to learn to behave efficiently. The analysis of what these networks evolve to learn shows some interesting results. references Rumelhart, D.E., Hinton G.E., and Williams, R.J. (1986). Learning internal representations by error propagation. In D.E. Rumelhart, and J.L. McClelland, (eds.), Parallel Distributed Processing. Vol.1: Foundations. Cambridge, Mass.: MIT Press. From noordewi at cs.rutgers.edu Wed Feb 6 18:45:45 1991 From: noordewi at cs.rutgers.edu (noordewi@cs.rutgers.edu) Date: Wed, 6 Feb 91 18:45:45 EST Subject: Rutgers Neural Network Seminar Message-ID: <9102062345.AA13716@porthos.rutgers.edu> RUTGERS UNIVERSITY Dept. of Computer Science/Dept. of Mathematics Neural Networks Colloquium Series --- Spring 1991 Stephen Judd Siemens The Complexity of Learning in Families of Neural Networks Abstract What exactly does it mean for Neural Networks to `learn'? We formalize a notion of learning that characterizes the simple training of feed-forward Neural Networks. The formulation is intended to model the objectives of the current mode of connectionist research in which one searches for powerful and efficient `learning rules' to stick in the `neurons'. By showing the learning problem to be NP-complete, we demonstrate that in general the set of things that a network can learn to do is smaller than the set of things it can do. No reasonable learning rule exists to train all families of networks. Naturally this provokes questions about easier cases, and we explore how the problem does or does not get easier as the neurons are made more powerful, or as various constraints are placed on the architecture of the network. We study one particular family of networks called `shallow architectures' which are defined in such a way as to bound their depth but let them grow very wide -- a description inspired by certain neuro-anatomical structures. The results seem to be robust in the face of all choices for what the neurons are able to compute individually. February 13, 1991 Busch Campus --- 4:30 p.m., room 217 SEC host: Mick Noordewier (201/932-3698) finger noordewi at cs.rutgers.edu for further schedule information From worth at park.bu.edu Thu Feb 7 15:03:28 1991 From: worth at park.bu.edu (Andrew J. Worth) Date: Thu, 7 Feb 91 15:03:28 -0500 Subject: Survey Results: Connectionism vs AI Message-ID: <9102072003.AA07670@park.bu.edu> On January 6th, I suggested a survey as part of the "Connectionism vs AI" discussion. Though the replies were not overwhelming, some of the main participants in the debate did respond. The text of the responses has (or will soon) be placed in the connectionist archive. Instructions to obtain this via anonymous ftp are given below. If you do not have access to the archive, I can send you the file by email directly. The file is plain text and has 455 lines, 3273 words, and 22732 characters. % ftp B.GP.CS.CMU.EDU (or ftp 128.2.242.8) Name: anonymous Password: (your username) ftp> cd /usr/connect/connectionists/archives ftp> get connect-vs-ai ftp> quit Thanks to all those that responded! Andy. ----------------------------------------------------------------------- Andrew J. Worth worth at park.bu.edu (617) 353-6741 Cognitive & Neural Systems Boston University Center for Adaptive Systems 111 Cummington St. Room 244 (617) 353-7857 Boston, MA 02215 USA From leon%FRLRI61.BITNET at CUNYVM.CUNY.EDU Thu Feb 7 06:02:44 1991 From: leon%FRLRI61.BITNET at CUNYVM.CUNY.EDU (leon%FRLRI61.BITNET@CUNYVM.CUNY.EDU) Date: Thu, 07 Feb 91 12:02:44 +0100 Subject: preprint available Message-ID: <9102071102.AA02071@sun3d.lri.fr> The following paper has been placed in the neuroprose archives at Ohio State University: A Framework for the Cooperation of Learning Algorithms Leon Bottou & Patrick Gallinari Laboratoire de Recherche en Informatique Universite de Paris XI 91405 Orsay Cedex - France Abstract We introduce a framework for training architectures composed of several modules. This framework, which uses a statistical formulation of learning systems, provides a single formalism for describing many classical connectionist algorithms as well as complex systems where several algorithms interact. It allows to design hybrid systems which combine the advantages of connectionist algorithms as well as other learning algorithms. This paper will appear in the NIPS-90 proceedings. To retrieve it by anonymous ftp, do the following: unix> ftp cheops.cis.ohio-state.edu # (or ftp 128.146.8.62) Name (cheops.cis.ohio-state.edu:): anonymous Password (cheops.cis.ohio-state.edu:anonymous): ftp> cd pub/neuroprose ftp> binary ftp> get bottou.cooperation.ps.Z ftp> quit unix> unix> zcat bottou.cooperation.ps.Z | lpr -P From shavlik at cs.wisc.edu Fri Feb 8 14:56:12 1991 From: shavlik at cs.wisc.edu (Jude Shavlik) Date: Fri, 8 Feb 91 13:56:12 -0600 Subject: Faculty Position Message-ID: <9102081956.AA00311@steves.cs.wisc.edu> Due to a state-wide hiring freeze, our ad to CACM announcing an open faculty position was delayed. One of our target areas is artificial intelligence, so if you are looking for a faculty position this year, you may wish to send your application to me. Please include your vita, the names of at least three references, and a couple of sample publications. Sincerely, Jude Shavlik shavlik at cs.wisc.edu Assistant Professor Computer Sciences Dept University of Wisconsin 1210 W. Dayton Street Madison, WI 53706 (608) 262-1204 (608) 262-9777 (fax) From INAM%MUSICB.MCGILL.CA at BITNET.CC.CMU.EDU Sun Feb 10 16:27:24 1991 From: INAM%MUSICB.MCGILL.CA at BITNET.CC.CMU.EDU (Tony Marley) Date: Sun, 10 Feb 91 16:27:24 EST Subject: Symposium on Models of Human Identification and Categorization Message-ID: <10FEB91.17773400.0250.MUSIC@MUSICB.MCGILL.CA> Department of Psychology McGill University 1205 Avenue Dr Penfield Montreal PQ H3A 1B1 Canada February 10, 1991 MODELS OF HUMAN IDENTIFICATION AND CATEGORIZATION Symposium at the Twenty-Fourth Annual Mathematical Psychology Meeting A. A. J. Marley, Symposium Organizer The Society for Mathematical Psychology Sponsored by Indiana University, Bloomington, Indiana August 10-13, 1991 At each of its Annual Meetings, the Society for Mathematical Psychology has one or more symposia on topics of current interest. I believe that this is an opportune time to have the proposed session since much exciting work is being done, plus Robert Nosofsky is an organizer of the conference at Indiana, and he has recently developed many empirical and theoretical ideas that have encouraged others to (re)enter this area. Each presenter in the symposium will have 20 to 30 minutes available to them, plus there will be time scheduled for general discussion. This meeting is a good place to present your theoretical ideas in detail, although simulation and empirical results are naturally also welcome. Remember, the Cognitive Science Society is meeting at the University of Chicago August 7- 10, i.e. just prior to this meeting; thus, by splitting your material in an appropriate manner between the two meetings, you will have an excellent forum within a period of a week to present your work in detail. If you are interested in participating in this symposium, please contact me with a TITLE and ABSTRACT. I would also be interested in suggestions of other participants with (if possible) an email address for them. To give you a clearer idea of the kind of work that I consider of direct relevance, I mention a few researchers and some recent papers. This list is meant to be illustrative, so please don't be annoyed if I have omitted your favourite work (including your own). REFERENCES AHA, D. W., & MCNULTY, D. (1990). Learning attribute relevance in context in instance-based learning algorithms. In Proceedings of the Twelfth Annual Conference of the Cognitive Science Society. Hillsdale, NJ: Erlbaum. ASHBY, F. G. (Ed.). (in press). Probabilistic Multidimensional Models of Perception and Cognition. Hillsdale, NJ: Erlbaum. ESTES, W. K., CAMPBELL, J. A., HATSPOULOS, N., & HURWITZ, J. B. (1989). Base-rate effects in category learning: A comparison of parallel network and memory storage-retrieval models. Journal of Experimental Psychology: Learning, Memory, and Cognition, 15, 556-571. GLUCK, M. A., & BOWER, G. H. (1989). Evaluating an adaptive network model of human learning. Journal of Memory and Language, 27, 166-195. HURWITZ, J. B. (1990). A hidden-pattern network model of category learning. Ph. D. Thesis, Department of Psychology, Harvard. KRUSCKE, J. K. (1990). ALCOVE: A connectionist model of category learning. Research Report 19, Cognitive Science, Indiana University. LACOUTURE, Y., & MARLEY, A. A. J. (1990). A connectionist model of choice and reaction time in absolute identification. Manuscript, Universite Laval & McGill University. NOSOFSKY, R. M., & GLUCK, M. A. (1989). Adaptive networks, exemplars, and classification rule learning. Thirtieth Annual Meeting of the Psychonomic Society, Atlanta, Georgia. RATCLIFF, R. (1990). Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions. Psychological Review, 97, 285-308. SHEPARD, R. N. (1989). A law of generalization and connectionist learning. Plenary Session, Eleventh Annual Conference of the Cognitive Science Society, University of Michigan, Ann Arbor. Regards Tony A. A. J. Marley Professor Director of the McGill Cognitive Science Centre From MURRE%rulfsw.LeidenUniv.nl at BITNET.CC.CMU.EDU Tue Feb 12 16:38:00 1991 From: MURRE%rulfsw.LeidenUniv.nl at BITNET.CC.CMU.EDU (MURRE%rulfsw.LeidenUniv.nl@BITNET.CC.CMU.EDU) Date: Tue, 12 Feb 91 16:38 MET Subject: Neurosimulators Message-ID: Dear connectionist researchers, We are compiling a list of neurosimulators for inclusion in a review paper. The table below presents the 45 simulators that we have been able to track down so far. We have not been able to find out all the details. We would, therefore, appreciate it when users or developers could fill us in on the white spots in the list (or point out any mistakes). Also, if anyone knows of other simulators that should be included, please, drop us a note. We would especially welcome any (pointers to) papers describing neurosimulators. This would enable us to refine and extend the list of features. Thanks! Jaap Murre Steven Kleynenberg E-mail: MURRE at HLERUL55.Bitnet Surface mail: Jacob M.J. Murre Unit of Experimental and Theoretical Psychology Leiden University P.O. Box 9555 2300 RB Leiden The Netherlands To save precious bytes, we have configured the table below in a 132 column format. It may be easier to send the file to a line printer, then to read it behind the terminal. (On a VAX use: set term /width=132.) TABLE: NEUROSIMULATORS Name Manufacturer Language Models Hardware Referenc e Price ($) -------------------------------------------------------------------------------- ----------------- ADAPTICS Adaptic [AZEMA] ANNE Oregon Grad. Cent. HLL Intel hypercube [AZEMA] ANSE TRW TRW [AZEMA] ANSIM SAIC several IBM [COHEN] 495.00 ANSKIT SAIC several many [BARGA][ BYTE] ANSPEC SIAC HLL many IBM,MAC,SUN,VAX 995.00 AWARENESS Neural Systems IBM [BYTE] 275.00 AXON HNC HLL many HNC neurocomp. [AZEMA][ BYTE] 1950.00 BOSS [REGGIA] BRAIN SIMULATOR Abbot,Foster, & Hauser IBM 99.00 BRAINMAKER Cal.Scient.Software bp IBM [BYTE] 195.00 CABLE VAX [MILLER] CASENET Prolog [DOBBINS ] COGNITRON Cognitive Software Lisp many MAC,IBM [ZEITV][ BYTE] 600.00 CONE IBM Palo Alto HLL IBM [AZEMA] CONNECTIONS hopf IBM [BYTE] 87.00 CORTEX [REGGIA] DESIRE/NEUNET IBM [KORN] EXPLORENET 3000 HNC HLL many IBM,VAX [BYTE][C OHEN] GENESIS Neural Systems IBM [MILLER] 1095.00 GRADSIM Univ. of Penns. C several GRIFFIN Texas Instruments [AZEMA] HYPERBRAIN Neurix MAC [BYTE] 995.00 MACBRAIN Neurix many MAC [BYTE] 995.00 MACTIVATION Univ. of Colorado? METANET Leiden University HLL many IBM,VAX [MURRE] MIRROR 2 HLL several [REGGIA] N-NET AIWare C bp IBM,VAX [BYTE] 695.00 N1000 Nestor IBM,SUN [BYTE] 19000.00 N500 Nestor IBM [BYTE] NEMOSYS IBM RS/6000 [MILLER] NESTOR DEV. SYSTEM Nestor IBM,MAC 9950.00 NET [REGGIA] NETSET 2 HNC many IBM,SUN,VAX 19500.00 NETWURKZ Dair Computer IBM [BYTE] 79.95 NEURALWORKS NeuralWare HLL many IBM,MAC,SUN [BYTE][C OHEN] 1495.00 NEUROCLUSTERS VAX [AZMY] NEURON [MILLER] NEUROSHELL Ward Systems Group bp IBM [BYTE] 195.00 NEUROSOFT HNC NEUROSYM NeuroSym many IBM 179.00 NEURUN Dare Research bp IBM NN3 GMD Bonn HLL many [LINDEN] NNSIM [NIJHUIS ] OWL Olmsted & Watkins many IBM,MAC,SUN,VAX [BYTE] 1495.00 P3 UCSD HLL many Symbolics [ZIPSER] PABLO [REGGIA] PLATO/ARISTOTLE NeuralTech [AZEMA] PLEXI Symbolics Lisp bp,hopf Symbolics PREENS Nijmegen University HLL many SUN PYGMALION Esprit C many SUN,VAX [AZEMA] RCS Rochester University C many SUN [AZEMA] SFINX UCLA HLL [AZEMA] Explanation of abbreviations and terms: Languages: HLL = High Level Language (i.e., network definition language; if spec ific programming languages are mentioned networks can be defined using high level functions in thes e languages) Models: several = a fixed number of models is (and will be) supported many = the systems can be (or will be) extended with new models bp = backpropagation hopf = hopfield (if specific models are mentioned these are th e only ones su pported) References: see list below (We welcome any additional references.) [AZEMA] Azema-Barac, M., M. Heweston, M. Recce, J. Taylor, P. Treleaven, M. Vellasco (1990). Pygmalion, neural network progamming environment. [BARGA] Barga, R.S., R.B. Melton (1990). Framework for distributed artificial neural system simulation. Proceedings of the IJCNN-90-Washington DC, 2, 94-97. [BYTE] Byte (product listing) (1989). BYTE, 14(8), 244-245. [COHEN] Cohen, H. (1989). How useful are current neural network software tools? Neural Network Review, 3, 102-113. [DOBBINS] Dobbins, R.W., R.C. Eberhart (1990). Casenet, computer aided neural network generation tool. Proceedings of the IJCNN-90-Washington DC, 2, 122-125. [KORN] Korn, G.A. (1989). A new environment for interactive neural network experiments. Neural Networks, 2, 229-237. [LINDEN] Linden, A., Ch. Tietz (in prep.). Research and development software environment for modular adaptive systems. Technical Report NN3-1, GMD Birlinghoven, Sankt Augustin, Germany. [MILLER] Miller, J.P. (1990). Computer modelling at the single-neuron level. Nature, 347, 783-784. [MURRE] Murre, J.M.J., S.E. Kleynenberg (submitted). Extending the MetaNet Network Environment: process control and machine independence. [NIJHUIS] Nijhuis, J., L. Spaanenburg, F. Warkowski (1989). Structure and application of NNSIM: a general purpose neural network simulator. Microprocessing and Microprogramming, 27, 189-194. [REGGIA] Reggia, J.A., C.L. D'Autrechy, G.C. Sutton III, S.M. Goodall (1988). A general-purpose simulation environment of developing connectionist models. Simulation, 51, 5-19. [VIBERT] Vibert, J.F., N. Azmy (1990). Neuro_Clusters: A biological plausible neural networks simulator tool. [ZEITV] Zeitvogel, R.K. (1989). Cognitive Software's Cognitron 1.2 (review). Neural Network Review, 3, 11-16. [ZIPSER] Zipser, D., D.E. Rabin (1986). P3: a parallel network simulation system. In: D.E. Rumelhart, J.L. McClelland (1986). Parallel distributed processing. Volume 1. Cambridge MA: MIT Press. From Connectionists-Request at CS.CMU.EDU Wed Feb 13 13:35:19 1991 From: Connectionists-Request at CS.CMU.EDU (Connectionists-Request@CS.CMU.EDU) Date: Wed, 13 Feb 91 13:35:19 EST Subject: Bi-monthly Reminder Message-ID: <20706.666470119@B.GP.CS.CMU.EDU> This is an automatically posted bi-monthly reminder about how the CONNECTIONISTS list works and how to access various online resources. CONNECTIONISTS is not an edited forum like the Neuron Digest, or a free-for-all newsgroup like comp.ai.neural-nets. It's somewhere in between, relying on the self-restraint of its subscribers. Membership in CONNECTIONISTS is restricted to persons actively involved in neural net research. The following posting guidelines are designed to reduce the amount of irrelevant messages sent to the list. Before you post, please remember that this list is distributed to over a thousand busy people who don't want their time wasted on trivia. Also, many subscribers pay cash for each kbyte; they shouldn't be forced to pay for junk mail. Happy hacking. -- Dave Touretzky & Scott Crowder --------------------------------------------------------------------- What to post to CONNECTIONISTS ------------------------------ - The list is primarily intended to support the discussion of technical issues relating to neural computation. - We encourage people to post the abstracts of their latest papers and tech reports. - Conferences and workshops may be announced on this list AT MOST twice: once to send out a call for papers, and once to remind non-authors about the registration deadline. A flood of repetitive announcements about the same conference is not welcome here. - Requests for ADDITIONAL references. This has been a particularly sensitive subject lately. Please try to (a) demonstrate that you have already pursued the quick, obvious routes to finding the information you desire, and (b) give people something back in return for bothering them. The easiest way to do both these things is to FIRST do the library work to find the basic references, then POST these as part of your query. Here's an example: WRONG WAY: "Can someone please mail me all references to cascade correlation?" RIGHT WAY: "I'm looking for references to work on cascade correlation. I've already read Fahlman's paper in NIPS 2, his NIPS 3 abstract, and found the code in the nn-bench archive. Is anyone aware of additional work with this algorithm? I'll summarize and post results to the list." - Announcements of job openings related to neural computation. - Short reviews of new text books related to neural computation. To send mail to everyone on the list, address it to Connectionists at CS.CMU.EDU ------------------------------------------------------------------- What NOT to post to CONNECTIONISTS: ----------------------------------- - Requests for addition to the list, change of address and other administrative matters should be sent to: "Connectionists-Request at cs.cmu.edu" (note the exact spelling: many "connectionists", one "request"). If you mention our mailing list to someone who may apply to be added to it, please make sure they use the above and NOT "Connectionists at cs.cmu.edu". - Requests for e-mail addresses of people who are believed to subscribe to CONNECTIONISTS should be sent to postmaster at appropriate-site. If the site address is unknown, send your request to Connectionists-Request at cs.cmu.edu and we'll do our best to help. A phone call to the appropriate institution may sometimes be simpler and faster. - Note that in many mail programs a reply to a message is automatically "CC"-ed to all the addresses on the "To" and "CC" lines of the original message. If the mailer you use has this property, please make sure your personal response (request for a Tech Report etc.) is NOT broadcast over the net. - Do NOT tell a friend about Connectionists at cs.cmu.edu. Tell him or her only about Connectionists-Request at cs.cmu.edu. This will save your friend from public embarrassment if she/he tries to subscribe. - Limericks should not be posted here. ------------------------------------------------------------------------------- The CONNECTIONISTS Archive: --------------------------- All e-mail messages sent to "Connectionists at cs.cmu.edu" starting 27-Feb-88 are now available for public perusal. A separate file exists for each month. The files' names are: arch.yymm where yymm stand for the obvious thing. Thus the earliest available data are in the file: arch.8802 Files ending with .Z are compressed using the standard unix compress program. To browse through these files (as well as through other files, see below) you must FTP them to your local machine. ------------------------------------------------------------------------------- How to FTP Files from the CONNECTIONISTS Archive ------------------------------------------------ 1. Open an FTP connection to host B.GP.CS.CMU.EDU (Internet address 128.2.242.8). 2. Login as user anonymous with password your username. 3. 'cd' directly to one of the following directories: /usr/connect/connectionists/archives /usr/connect/connectionists/bibliographies 4. The archives and bibliographies directories are the ONLY ones you can access. You can't even find out whether any other directories exist. If you are using the 'cd' command you must cd DIRECTLY into one of these two directories. Access will be denied to any others, including their parent directory. 5. The archives subdirectory contains back issues of the mailing list. Some bibliographies are in the bibliographies subdirectory. Problems? - contact us at "Connectionists-Request at cs.cmu.edu". ------------------------------------------------------------------------------- How to FTP Files from the Neuroprose Archive -------------------------------------------- Anonymous FTP on cheops.cis.ohio-state.edu (128.146.8.62) pub/neuroprose directory This directory contains technical reports as a public service to the connectionist and neural network scientific community. Researchers may place electronic versions of their preprints or articles in this directory, announce availability, and other interested researchers can rapidly retrieve and print the postscripts. This saves copying, postage and handling, by having the interested reader supply the paper. To place a file, put it in the Inbox subdirectory, and send mail to pollack at cis.ohio-state.edu. Within a couple of days, I will move and protect it, and suggest a different name if necessary. Current naming convention is author.title.filetype[.Z] where title is enough to discriminate among the files of the same author. The filetype is usually "ps" for postscript, our desired universal printing format, but may be tex, which requires more local software than a spooler. Very large files (e.g. over 200k) must be squashed (with either a sigmoid function :) or the standard unix "compress" utility, which results in the .Z affix. To place or retrieve .Z files, make sure to issue the FTP command "BINARY" before transfering files. After retrieval, call the standard unix "uncompress" utility, which removes the .Z affix. An example of placing a file is attached as an appendix, and a shell script called Getps in the directory can perform the necessary retrival operations. For further questions contact: Jordan Pollack Email: pollack at cis.ohio-state.edu Here is an example of naming and placing a file: gvax> cp i-was-right.txt.ps rosenblatt.reborn.ps gvax> compress rosenblatt.reborn.ps gvax> ftp cheops.cis.ohio-state.edu Connected to cheops.cis.ohio-state.edu. 220 cheops.cis.ohio-state.edu FTP server ready. Name: anonymous 331 Guest login ok, send ident as password. Password:neuron 230 Guest login ok, access restrictions apply. ftp> binary 200 Type set to I. ftp> cd pub/neuroprose/Inbox 250 CWD command successful. ftp> put rosenblatt.reborn.ps.Z 200 PORT command successful. 150 Opening BINARY mode data connection for rosenblatt.reborn.ps.Z 226 Transfer complete. 100000 bytes sent in 3.14159 seconds ftp> quit 221 Goodbye. gvax> mail pollack at cis.ohio-state.edu Subject: file in Inbox. Jordan, I just placed the file rosenblatt.reborn.ps.Z in the Inbox. The INDEX sentence is "Boastful statements by the deceased leader of the neurocomputing field." Please let me know when it is ready to announce to Connectionists at cmu. BTW, I enjoyed reading your review of the new edition of Perceptrons! Frank ------------------------------------------------------------------------ How to FTP Files from the NN-Bench Collection --------------------------------------------- 1. Create an FTP connection from wherever you are to machine "pt.cs.cmu.edu" (128.2.254.155). 2. Log in as user "anonymous" with password your username. 3. Change remote directory to "/afs/cs/project/connect/bench". Any subdirectories of this one should also be accessible. Parent directories should not be. 4. At this point FTP should be able to get a listing of files in this directory and fetch the ones you want. Problems? - contact us at "nn-bench-request at cs.cmu.edu". From vincew at cse.ogi.edu Wed Feb 13 18:30:36 1991 From: vincew at cse.ogi.edu (Vince Weatherill) Date: Wed, 13 Feb 91 15:30:36 -0800 Subject: Speech Recognition & NNs preprints/reprints available Message-ID: <9102132330.AA01394@cse.ogi.edu> Reprints and preprints are now available for the following publications of the OGI Speech Group. Please respond directly to me by e-mail or surface mail. Don't forget to include your address with your request. Unless you indicate otherwise, I will send all 6 reports. Vince Weatherill Dept. of Computer Science and Engineering Oregon Graduate Institute 19600 NW von Neumann Drive Beaverton, OR 97006-1999 Barnard, E., Cole, R.A., Vea, M.P., and Alleva, F. "Pitch detection with a neural-net classifier," IEEE Transactions on Acoustics, Speech & Signal Processing, (February, 1991). Cole, R.A., M. Fanty, M. Gopalakrishnan, and R.D.T. Janssen, "Speaker-independent name retrieval from spellings using a database of 50,000 names," Proceedings of the IEEE Interna- tional Conference on Acoustics, Speech and Signal Process- ing, Toronto, Canada, May 14-17, (1991). Muthusamy, Y. K., R.A. Cole, and M. Gopalakrishnan, "A segment- based approach to automatic language identification," Proceedings of the 1991 IEEE International Conference on Acoustics, Speech and Signal Processing, Toronto, Canada, May 14-17, (1991). Fanty, M., R. A. Cole, and , "Spoken Letter Recognition," Proceedings of the Neural Information Processing Systems Conference, Denver, CO, (Nov. 1990). Janssen, R.D.T, M. Fanty, and R.A. Cole, "Speaker-independent phonetic classification in continuous English letters," Proceedings of the International Joint Conference on Neural Networks, Seattle, WA, Jul 8-12, (1991), submitted for publication. Fanty, M., R. A. Cole, and , "Speaker-independent English alpha- bet recognition: Experiments with the E-Set," Proceedings of the 1990 International Conference on Spoken Language Pro- cessing, Kobe, Japan, (Nov. 1990). **************************************************************** PITCH DETECTION WITH A NEURAL-NET CLASSIFIER Etienne Barnard, Ronald Cole, M. P. Vea and Fil Alleva ABSTRACT Pitch detection based on neural-net classifiers is investi- gated. To this end, the extent of generalization attainable with neural nets is first examined, and it is shown that a suitable choice of features is required to utilize this pro- perty. Specifically, invariant features should be used whenever possible. For pitch detection, two feature sets, one based on waveform samples and the other based on proper- ties of waveform peaks, are introduced. Experiments with neural classifiers demonstrate that the latter feature set --which has better invariance properties--performs more suc- cessfully. It is found that the best neural-net pitch tracker approaches the level of agreement of human labelers on the same data set, and performs competitively in comparison to a sophisticated feature-based tracker.An analysis of the errors committed by the neural net (relative to the hand labels used for training) reveals that they are mostly due to inconsistent hand labeling of ambigu- ous waveform peaks. ************************************************************* SPEAKER-INDEPENDENT NAME RETRIEVAL FROM SPELLINGS USING A DATABASE OF 50,000 NAMES Ronald Cole, Mark Fanty, Murali Gopalakrishnan, Rik Janssen ABSTRACT We describe a system that recognizes names spelled with pauses between letters using high quality speech. The sys- tem uses neural network classifiers to locate and classify letters, then searches a database of names to find the best match to the letter scores. The directory name retrieval system was evaluated on 1020 names provided by 34 speakers who were not used to train the system. Using a database of 50,000 names, 972, or 95.3%, were correctly identified as the first choice. Of the remaining 48 names, all but 10 were in the top 3 choices. Ninty nine percent of letters were correctly located, although speakers failed to pause completely about 10% of the time. Classification of indivi- dual spoken letters that were correctly located was 93%. ************************************************************* A SEGMENT-BASED APPROACH TO AUTOMATIC LANGUAGE IDENTIFICATION Yeshwant K. Muthusamy, Ronald A. Cole and Murali Gopalakrishnan ABSTRACT A segment-based approach to automatic language identifica- tion is based on the idea that the acoustic structure of languages can be estimated by segmenting speech into broad phonetic categories. Automatic language identification can then be achieved by computing features that describe the phonetic and prosodic characteristics of the language, and using these feature measurements to train a classifier to distinguish between languages. As a first step in this approach, we have built a multi-language, neural network- based segmentation and broad classification algorithm using seven broad phonetic categories. The algorithm was trained and tested on separate sets of speakers of American English, Japanese, Mandarin Chinese and Tamil. It currently performs with an accuracy of 82.3% on the utterances of the test set. ************************************************************* SPOKEN LETTER RECOGNITION Mark Fanty and Ron Cole ABSTRACT Through the use of neural network classifiers and careful feature selection, we have achieved high-accuracy speaker- independent spoken letter recognition. For isolated letters, a broad-category segmentation is performed Location of segment boundaries allows us to measure features at From rosen at CS.UCLA.EDU Thu Feb 14 14:11:46 1991 From: rosen at CS.UCLA.EDU (Bruce E Rosen) Date: Thu, 14 Feb 91 11:11:46 -0800 Subject: Adaptive Range Coding - Tech Report Available Message-ID: <9102141911.AA28582@lanai.cs.ucla.edu> REPORT AVAILABLE ON ADAPTIVE RANGE CODING At the request of a few people at NIPS, I placed in the connectionists archive the postscript version of my report describing adaptive range coding. Below are the abstract and instructions on ftp retrieval. I would very much welcome any discussion of this subject. If you want, send email to me and I can summarize later for the net. Thanks Bruce ----------------------------------------------------------------------------- Report DMI-90-4, UCLA Distributed Machine Intelligence Laboratory, January 1991 Adaptive Range Coding Abstract This paper examines a class of neuron based learning systems for dynamic control that rely on adaptive range coding of sensor inputs. Sensors are assumed to provide binary coded range vectors that coarsely describe the system state. These vectors are input to neuron-like processing elements. Output decisions generated by these "neurons" in turn affect the system state, subsequently producing new inputs. Reinforcement signals from the environment are received at various intervals and evaluated. The neural weights as well as the range boundaries determining the output decisions are then altered with the goal of maximizing future reinforcement from the environment. Preliminary experiments show the promise of adapting "neural receptive fields" when learning dynamical control. The observed performance with this method exceeds that of earlier approaches. ----------------------------------------------------------------------- To obtain copies of the postscript file, please use Jordan Pollack's service: Example: unix> ftp cheops.cis.ohio-state.edu # (or ftp 128.146.8.62) Name (cheops.cis.ohio-state.edu:): anonymous Password (cheops.cis.ohio-state.edu:anonymous): ftp> cd pub/neuroprose ftp> binary ftp> get (remote-file) rosen.adaptrange.ps.Z (local-file) rosen.adaptrange.ps.Z ftp> quit unix> uncompress rosen.adaptrange.ps unix> lpr -P(your_local_postscript_printer) rosen.adaptrange.ps ---------------------------------------------------------------------------- If you have any difficulties with the above, please send e-mail to rosen at cs.ucla.edu. DO NOT "reply" to this message, please. From yoshua at homer.cs.mcgill.ca Sun Feb 17 21:20:42 1991 From: yoshua at homer.cs.mcgill.ca (Yoshua BENGIO) Date: Sun, 17 Feb 91 21:20:42 EST Subject: TR available: Yet another ANN/HMM hybrid. Message-ID: <9102180220.AA08811@homer.cs.mcgill.ca> The following technical report is now available by ftp from neuroprose: Yoshua Bengio, Renato De Mori, Giovanni Flammia, and Ralf Kompe (1990), "Global Optimization of a Neural Network - Hidden Markov Model Hybrid", Technical Report TR-SOCS-90.22, December 1990, School of Computer Science, McGill University. Abstract: Global Optimization of a Neural Network - Hidden Markov Model Hybrid Yoshua Bengio, Renato De Mori, Giovanni Flammia, Ralf Kompe TR-SOCS-90.22, December 1990 In this paper a method for integrating Artificial Neural Networks (ANN) with Hidden Markov Models (HMM) is proposed and evaluated. ANNs are suitable to perform phonetic classification, whereas HMMs have been proven successful at modeling the temporal structure of the speech signal. In the approach described here, the ANN outputs constitute the sequence of observation vectors for the HMM. An algorithm is proposed for global optimization of all the parameters. An incremental design method is described in which specialized networks are integrated to the recognition system in order to improve its performance. Results on speaker-independent recognition experiments using this integrated ANN-HMM system on the TIMIT continuous speech database are reported. --------------------------------------------------------------------------- Copies of the postscript file bengio.hybrid.ps.Z may be obtained from the pub/neuroprose directory in cheops.cis.ohio-state.edu. Either use the Getps script or do this: unix-1> ftp cheops.cis.ohio-state.edu # (or ftp 128.146.8.62) Connected to cheops.cis.ohio-state.edu. Name (cheops.cis.ohio-state.edu:): anonymous 331 Guest login ok, sent ident as password. Password: neuron 230 Guest login ok, access restrictions apply. ftp> cd pub/neuroprose ftp> binary ftp> get bengio.hybrid.ps.Z ftp> quit unix-2> uncompress bengio.hybrid.ps.Z unix-3> lpr -P(your_local_postscript_printer) bengio.hybrid.ps Or, order a hardcopy by sending your physical mail address to yoshua at cs.mcgill.ca, mentioning Technical Report TR-SOCS-90.22. PLEASE do this only if you cannot use the ftp method described above. ---------------------------------------------------------------------------- From nzt at research.att.com Mon Feb 18 15:55:29 1991 From: nzt at research.att.com (nzt@research.att.com) Date: Mon, 18 Feb 91 15:55:29 EST Subject: mailing list Message-ID: <9102182055.AA30788@minos.att.com> Hi: Please add my name to the connectionists mailing list. Naftali Tishby, AT&T Bell Laboratory. nzt at research.att.com Thanks. From pluto at cs.UCSD.EDU Tue Feb 19 15:21:17 1991 From: pluto at cs.UCSD.EDU (Mark Plutowski) Date: Tue, 19 Feb 91 12:21:17 PST Subject: Tech Report Available in Neuroprose Message-ID: <9102192021.AA00628@cornelius> The following report has been placed in the neuroprose archives at Ohio State University: UCSD CSE Technical Report No. CS91-180 Active selection of training examples for network learning in noiseless environments. Mark Plutowski Department of Computer Science and Engineering, UCSD, and Halbert White Institute for Neural Computation and Department of Economics, UCSD. Abstract: We derive a method for {\sl actively selecting} examples to be used in estimating an unknown mapping with a multilayer feedforward network architecture. Active selection chooses from among a set of available examples an example which, when added to the previous set of training examples and learned, maximizes the decrement of network error over the input space. New examples are chosen according to network performance on previous training examples. In practice, this amounts to incrementally growing the training set as necessary to achieve the desired level of accuracy. The objective is to minimize the data requirement of learning. Towards this end, we choose a general criterion for selecting training examples that works well in conjunction with the criterion used for learning, here, least squares. Examples are chosen to minimize Integrated Mean Square Error (IMSE). IMSE embodies the effects of bias (misspecification of the network model) and variance (sampling variation due to noise). We consider a special case of IMSE, Integrated Squared Bias, (ISB) to derive a selection criterion ($\Delta ISB$) which we maximize to select new training examples. $\Delta ISB$ is applicable whenever sampling variation due to noise can be ignored. We conclude with graphical illustrations of the method, and demonstrate its use during network training. -=-=-=-=-=-=-=-=-=-=-=-=-=-= How to obtain a copy -=-=-=-=-=-=-=-=-=-=-=-=-=-= Copies may be obtained by a) FTP directly from the Neuroprose directory, or b) by land mail from the CSE dept. at UCSE. a) via FTP: To obtain a copy from Neuroprose, either use the "getps" program, or ftp the file as follows: % ftp cheops.cis.ohio-state.edu Connected to cheops.cis.ohio-state.edu. 220 cheops.cis.ohio-state.edu FTP server (Version 5.49 Tue May 9 14:01:04 EDT 1989) ready. Name (cheops.cis.ohio-state.edu:your-ident): anonymous [2331 Guest login ok, send ident as password. Password: your-ident 230 Guest login ok, access restrictions apply. ftp> cd pub/neuroprose 250 CWD command successful. ftp> binary 200 Type set to I. ftp> get plutowski.active.ps.Z 200 PORT command successful. 150 Opening BINARY mode data connection for plutowski.active.ps.Z (348443 bytes). 226 Transfer complete. local: plutowski.active.ps.Z remote: plutowski.active.ps.Z 348443 bytes received in 44 seconds (7.2 Kbytes/s) ftp> quit % uncompress plutowski.active.ps.Z % lpr -P plutowski.active.ps b) via postal mail: Requests for hardcopies may be sent to: Kay Hutcheson CSE Department, 0114 UCSD La Jolla, CA 92093-0114 and enclose a check for $5.00 payable to "UC Regents." The report number is: Technical Report No. CS91-180 From yoshua at homer.cs.mcgill.ca Tue Feb 19 14:33:08 1991 From: yoshua at homer.cs.mcgill.ca (Yoshua BENGIO) Date: Tue, 19 Feb 91 14:33:08 EST Subject: header for TR on ANN/HMM hybrid in neuroprose Message-ID: <9102191933.AA08798@homer.cs.mcgill.ca> ---------------------------------------------------------------------------- The following technical report available by ftp from neuroprose was recently advertised: Yoshua Bengio, Renato De Mori, Giovanni Flammia, and Ralf Kompe (1990), "Global Optimization of a Neural Network - Hidden Markov Model Hybrid", Technical Report TR-SOCS-90.22, December 1990, School of Computer Science, McGill University. However, it was not mentionned that the front pages of the TR are in bengio.hybrid_header.ps.Z whereas the paper itself is in: bengio.hybrid.ps.Z Sorry for the inconvenience, Yoshua Bengio School of Computer Science, McGill University ---------------------------------------------------------------------------- From whart at cs.UCSD.EDU Wed Feb 20 01:26:06 1991 From: whart at cs.UCSD.EDU (Bill Hart) Date: Tue, 19 Feb 91 22:26:06 PST Subject: New TR Available Message-ID: <9102200626.AA01316@beowulf.ucsd.edu> The following TR has been placed in the neuroprose archives at Ohio State University. --Bill -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= UCSD CSE Technical Report No. CS91-180 Active selection of training examples for network learning in noiseless environments. Mark Plutowski Department of Computer Science and Engineering, UCSD, and Halbert White Institute for Neural Computation and Department of Economics, UCSD. Abstract: We derive a method for {\sl actively selecting} examples to be used in estimating an unknown mapping with a multilayer feedforward network architecture. Active selection chooses from among a set of available examples an example which, when added to the previous set of training examples and learned, maximizes the decrement of network error over the input space. %New examples are chosen according to %network performance on previous training examples. In practice, this amounts to incrementally growing the training set as necessary to achieve the desired level of accuracy. The objective is to minimize the data requirement of learning. Towards this end, we choose a general criterion for selecting training examples that works well in conjunction with the criterion used for learning, here, least squares. Examples are chosen to minimize Integrated Mean Square Error (IMSE). IMSE embodies the effects of bias (misspecification of the network model) and variance (sampling variation due to noise). We consider a special case of IMSE, Integrated Squared Bias, (ISB) to derive a selection criterion ($\Delta ISB$) which we maximize to select new training examples. $\Delta ISB$ is applicable whenever sampling variation due to noise can be ignored. We conclude with graphical illustrations of the method, and demonstrate its use during network training. -=-=-=-=-=-=-=-=-=-=-=-=-=-= How to obtain a copy -=-=-=-=-=-=-=-=-=-=-=-=-=-= a) via FTP: To obtain a copy from Neuroprose, either use the "getps" program, or ftp the file as follows: % ftp cheops.cis.ohio-state.edu Connected to cheops.cis.ohio-state.edu. 220 cheops.cis.ohio-state.edu FTP server (Version 5.49 Tue May 9 14:01:04 EDT 1989) ready. Name (cheops.cis.ohio-state.edu:your-ident): anonymous [2331 Guest login ok, send ident as password. Password: your-ident 230 Guest login ok, access restrictions apply. ftp> cd pub/neuroprose 250 CWD command successful. ftp> binary 200 Type set to I. ftp> get plutowski.active.ps.Z 200 PORT command successful. 150 Opening BINARY mode data connection for plutowski.active.ps.Z (325222 bytes). 226 Transfer complete. local: plutowski.active.ps.Z remote: plutowski.active.ps.Z 325222 bytes received in 44 seconds (7.2 Kbytes/s) ftp> quit % uncompress plutowski.active.ps.Z % lpr -P plutowski.active.ps b) via postal mail: Requests for hardcopies may be sent to: Kay Hutcheson CSE Department, 0114 UCSD La Jolla, CA 92093-0114 and enclose a check for $5.00 payable to "UC Regents." The report number is: Technical Report No. CS91-180 From dwunsch at atc.boeing.com Wed Feb 20 13:40:15 1991 From: dwunsch at atc.boeing.com (Don Wunsch) Date: Wed, 20 Feb 91 10:40:15 PST Subject: An IJCNN question Message-ID: <9102201840.AA21635@atc.boeing.com> In December, I posted a call for papers for IJCNN. The paper deadline has passed, but this question came in the mail recently. It was a good question that others among you might have, so I'm posting my reply. Barring some major issue, this is the last you'll hear from me re IJCNN till the late preregistration deadline in June. From zoran at theory.cs.psu.edu Sun Feb 17 14:05:10 1991 From: zoran at theory.cs.psu.edu (Zoran Obradovic) Date: Sun, 17 Feb 91 14:05:10 EST Subject: IJCNN-91-Seattle Message-ID: >I just realized that I do not have any conference registration form for >IJCNN-91-Seattle. I would appreciate very much if you email me the form. >If you do not have one on-line, please tell me if you know exactly whom >to make cheque payable and what information I have to provide (in addition >to my mailing address). >Regards, >Zoran Obradovic Thanks for a good question. You don't say whether you are a student or not, or an IEEE or INNS member, so here's the generic scoop: First of all, your question is quite timely. Registrations postmarked by March 1, 1991 will enjoy significant savings over later ones. The complete cost breakdown: Preregistration: Members $195 Nonmembers $295 Students $50 by March 1! Late preregistration: Members $295 Nonmembers $395 Students $75 by June 1. On site: Members $395 Nonmembers $495 Students $95 Tutorials Members $195 Nonmembers $195 Students $85 This gets you into ALL the tutorials! You must register for the conference, however. They want the tutorial registration by April 30. One day fee is $125 for members and $175 for nonmembers. If you want to register as a student, get a letter from your department verifying full-time status. Be sure to bring your student ID to the conference. Sorry, no proceedings at that rate. If you are at all thinking about coming, you really should register now. If you cancel anytime up to June 21, you'll get everything back except for $30. Note also that you might as well get an IEEE or INNS membership if you're not using the student rate--it's cheaper that way. I believe IEEE and INNS both can sign you up by phone. The info they ask for is: Title (Dr. Mr. Ms.), Name, Affiliation (for your badge), Full mailing address, phone #, FAX #, e-mail address. You should include the information about what you are signing up for, and what rate you are paying. Enclose the letter if you are claiming student status. For one-day registration, say which day. You may send a check, made out to University of Washington. You can also use MasterCard or VISA. If you do that, you can register by phone or FAX. Phone: (206) 543-2310 FAX: (206) 685-9359 The address to write to is: IJCNN-91-Seattle Conference Management, Attn: Sarah Eck, MS GH-22, Suite 108 5001 25th Ave. NE Seattle, WA 98195 Sarah's number is (206) 543-0888, and she is the best person to ask most questions, although I'll do my best to help also. Now I have a favor to ask you. May I copy your mail and post this to the net? There are probably many people out there with the same question. Thanks! Don From khaines at galileo.ece.cmu.edu Thu Feb 21 09:53:44 1991 From: khaines at galileo.ece.cmu.edu (Karen Haines) Date: Thu, 21 Feb 91 09:53:44 EST Subject: An IJCNN question Message-ID: <9102211453.AA16257@galileo.ECE.CMU.EDU> In regards to the recent post, student registration includes proceedings and admittance into all the social events. Karen Haines From CADEPS%BBRNSF11.BITNET at BITNET.CC.CMU.EDU Thu Feb 21 12:02:09 1991 From: CADEPS%BBRNSF11.BITNET at BITNET.CC.CMU.EDU (JANSSEN Jacques) Date: Thu, 21 Feb 91 18:02:09 +0100 Subject: No subject Message-ID: IS BEHAVIORAL LIMIT CYCLING SURPRISING? I have a question I would like to pose to theorists of neural net dynamics. A behavioral phenomenon has cropped up in my work which surprised me and I would like to know if this phenomenon is to be expected by NN dynamics theorists or not. Before describing the phenomenon, I need to give some technical background. I work in the field of "Genetic Programming", i.e. using the Genetic Algorithm (GA) to build/evolve systems which function, but are (probably) too complex to analyse mathematically. I have applied these GP techniques to building artificia nervous systems and artificial embryos. To build an artificial nervous system, I evolve the weights of fully connected neural nets (GenNets), such that their time dependent outputs control some process e.g. the angles of legs on a simulated artificial creature. This way one can evolve behaviours of the creature, e.g. to get it to walk straight ahead, choose the GA fitness to be the distance covered over a given number of cycles. To get it to turn, choose the fitness to be the angle rotated etc. The surprise comes when one tries to combine these motions. To do so, take the output of one GenNet (with its set of weights) and input it into the second GenNet. For example, if one wants the creature to walk straight ahead and later to turn right, then use the GenNet evolved for straight ahead walking for the time you want. Then take the output of this first GenNet and input it into the second GenNet which was evolved to get the creature to turn. What I found was that INDEPENDENTLY of the state of the creatures legs i.e. their angles (which are proportional to the output values), which were input into the second GenNet, one got the desired qualitative behaviour, i.e. it turned. I find this extremely useful phenomenon puzzling. It can be used for smooth transitions between behaviours, but why does it work? It looks as though the GenNet has evolved a sort of limit cycle in its behaviour, so that no matter what the starting state (i.e. the initial input values to the GenNet) the desired limit cycle behaviour will occur (e.g. straight ahead walking or turning etc). I call this phenomenon "Behavioral Limit Cycling" (BLC). Is this phenomenon to be expected? Is it old-hat to the theorists or have I stumbled onto something new and exciting? (I will certainly be able to use it when switching between behavioral GenNets amongst a whole GenNet library. This library constitutes an artificial nervous system and will be useful in forging a link between the two fields of neural networks (i.e. multi networks, not just one, which is what most NN papers are about) and the hot new field of Artificial Life). I pose this question for open discussion. Cheers, Hugo de Garis, University of Brussels & George Mason University VA. email CADEPS at BBRNSF11.BITNET From andycl at syma.sussex.ac.uk Mon Feb 18 15:25:47 1991 From: andycl at syma.sussex.ac.uk (Andy Clark) Date: Mon, 18 Feb 91 20:25:47 GMT Subject: No subject Message-ID: <6587.9102182025@syma.sussex.ac.uk> Dear People, Here is a short ad concerning a new course which may be of interest to you or your students. NOTICE OF NEW M.A.COURSE BEGINNING OCT. 1991 UNIVERSITY OF SUSSEX, BRIGHTON, ENGLAND SCHOOL OF COGNITIVE AND COMPUTING SCIENCES M.A. in the PHILOSOPHY OF COGNITIVE SCIENCE This is a one year taught course which examines issues relating to computational models of mind. A specific focus concerns the significance of connectionist models and the role of rules and symbolic representation in cognitive science. Students would combine work towards a 20,000 word philosophy dissertation with subsidiary courses introducing aspects of A.I. and the other Cognitive Sciences. For information about this new course contact Dr Andy Clark, School of Cognitive and Computing Sciences, University of Sussex,Brighton, BN1 9QH, U.K. E-mail: andycl at uk.ac.sussex.syma From lazzaro at sake.Colorado.EDU Fri Feb 22 02:10:21 1991 From: lazzaro at sake.Colorado.EDU (John Lazzaro) Date: Fri, 22 Feb 91 00:10:21 MST Subject: No subject Message-ID: <9102220710.AA03865@sake.colorado.edu> An announcement of a preprint on the neuroprose server ... A Delay-Line Based Motion Detection Chip Tim Horiuchi, John Lazzaro*, Andrew Moore, Christof Koch CNS Program, Caltech and *Optoelectronics Center, CU Boulder Abstract -------- Inspired by a visual motion detection model for the rabbit retina and by a computational architecture used for early audition in the barn owl, we have designed a chip that employs a correlation model to report the one-dimensional field motion of a scene in real time. Using subthreshold analog VLSI techniques, we have fabricated and successfully tested a 8000 transistor chip using a standard MOSIS process. ----- To retrieve ... >cheops.cis.ohio-state.edu >Name (cheops.cis.ohio-state.edu:lazzaro): anonymous >331 Guest login ok, send ident as password. >Password: your_username >230 Guest login ok, access restrictions apply. >cd pub/neuroprose >binary >get horiuchi.motion.ps.Z >quit %uncompress horiuchi.motion.ps.Z %lpr horiuchi.motion.ps ---- --jl From st at gmdzi.uucp Fri Feb 22 07:29:04 1991 From: st at gmdzi.uucp (Sebastian Thrun) Date: Fri, 22 Feb 91 11:29:04 -0100 Subject: No subject Message-ID: <9102221029.AA25000@gmdzi.gmd.de> Technical Reports available: Planning with an Adaptive World Model S. Thrun, K. Moeller, A. Linden We present a new connectionist planning method. By interaction with an unknown environment, a world model is progressively constructed using gradient descent. For deriving optimal actions with respect to future reinforcement, planning is applied in two steps: an experience network proposes a plan which is subsequently optimized by gradient descent with a chain of world models, so that an optimal reinforcement may be obtained when it is actually run. The appropriateness of this method is demonstrated by a robotics application and a pole balancing task. (to appear in proceedings NIPS*90) ------------------------------------------------------------------------- A General Feed-Forward Algorithm for Gradient Descent in Connectionist Networks S. Thrun, F. Smieja An extended feed-forward algorithm for recurrent connectionist networks is presented. This algorithm, which works locally in time, is derived both for discrete-in-time networks and for continuous networks. Several standard gradient descent algorithms for connectionist networks (e.g. Williams/Zipser 88, Pineda 87, Pearlmutter 88, Gherrity 89, Rohwer 87, Waibel 88, especially the backpropagation algorithm Rumelhart/Hinton/Williams 86, are mathematically derived from this algorithm. The learning rule presented in this paper is a superset of gradient descent learning algorithms for multilayer networks, recurrent networks and time-delay networks that allows any combinations of their components. In addition, the paper presents feed-forward approximation procedures for initial activations and external input values. The former one is used for optimizing starting values of the so-called context nodes, the latter one turned out to be very useful for finding spurious input attractors of a trained connectionist network. Finally, we compare time, processor and space complexities of this algorithm with backpropagation for an unfolded-in-time network and present some simulation results. (in: "GMD Arbeitspapiere Nr. 483") ------------------------------------------------------------------------- Both reports can be received by ftp: unix> ftp cis.ohio-state.edu Name: anonymous Guest Login ok, send ident as password Password: neuron ftp> binary ftp> cd pub ftp> cd neuroprose ftp> get thrun.nips90.ps.Z ftp> get thrun.grad-desc.ps.Z ftp> bye unix> uncompress thrun.nips90.ps unix> uncompress thrun.grad-desc.ps unix> lpr thrun.nips90.ps unix> lpr thrun.grad-desc.ps ------------------------------------------------------------------------- To all European guys: The same files can be retrieved from gmdzi.gmd.de (129.26.1.90), directory pub/gmd, which is probably a bit cheaper. ------------------------------------------------------------------------- If you have trouble in ftping the files, do not hesitate to contact me. --- Sebastian Thrun (st at gmdzi.uucp, st at gmdzi.gmd.de) From psykimp at aau.dk Fri Feb 22 05:47:37 1991 From: psykimp at aau.dk (Kim Plunkett) Date: Fri, 22 Feb 91 11:47:37 +0100 Subject: No subject Message-ID: <9102221047.AA13756@aau.dk> The following technical report is now available. For a copy, email "psyklone at aau.dk" and include your ordinary mail address. Kim Plunkett +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Connectionism and Developmental Theory Kim Plunkett and Chris Sinha University of Aarhus, Denmark Abstract The main goal of this paper is to argue for an ``epigenetic developmental interpretation'' of connectionist modelling of human cognitive processes, and to propose that parallel dis- tributed processing (PDP) models provide a superior account of developmental phenomena than that offered by cognitivist (symbolic) computational theories. After comparing some of the general characteristics of epigeneticist and cognitivist theories, we provide a brief overview of the operating prin- ciples underlying artificial neural networks (ANNs) and their associated learning procedures. Four applications of different PDP architectures to developmental phenomena are described. First, we assess the current status of the debate between symbolic and connectionist accounts of the process of English past tense formation. Second, we introduce a connectionist model of concept formation and vocabulary growth and show how it provides an account of aspects of semantic development in early childhood. Next, we take up the problem of compositionality and structure dependency in connectionist nets, and demonstrate that PDP models can be architecturally designed to capture the structural princi- ples characteristic of human cognition. Finally, we review a connectionist model of cognitive development which yields stage-like behavioural properties even though structural and input assumptions remain constant throughout training. It is shown how the organisational characteristics of the model provide a simple but precise account of the equilibration of the processes of accommodation and assimilation. The authors conclude that a coherent epigenetic-developmental interpretation of PDP modelling requires the rejection of so-called hybrid-architecture theories of human cognition. From marshall at cs.unc.edu Fri Feb 22 11:03:09 1991 From: marshall at cs.unc.edu (Jonathan Marshall) Date: Fri, 22 Feb 91 11:03:09 -0500 Subject: Paper available -- visual orientation multiplexing Message-ID: <9102221603.AA23754@marshall.cs.unc.edu> **** Please do not re-post to other bboards. **** Papers available, hardcopy only. ---------------------------------------------------------------------- ADAPTIVE NEURAL METHODS FOR MULTIPLEXING ORIENTED EDGES Jonathan A. Marshall Department of Computer Science University of North Carolina at Chapel Hill Edge linearization operators are often used in computer vision and in neural network models of vision to reconstruct noisy or incomplete edges. Such operators gather evidence for the presence of an edge at various orientations across all image locations and then choose the orientation that best fits the data at each point. One disadvantage of such methods is that they often function in a winner-take-all fashion: the presence of only a single orientation can be represented at any point; multiple edges cannot be represented where they intersect. For example, the neural Boundary Contour System of Grossberg and Mingolla implements a form of winner-take-all competition between orthogonal orientations at each spatial location, to promote sharpening of noisy, uncertain image data. But that competition may produce rivalry, oscillation, instability, or mutual suppression when intersecting edges (e.g., a cross) are present. This "cross problem" exists for all techniques, including Markov Random Fields, where a representation of a chosen favored orientation suppresses representations of alternate orientations. A new adaptive technique, using both an inhibitory learning rule and an excitatory learning rule, weakens inhibition between neurons representing poorly correlated orientations. It may reasonably be assumed that neurons coding dissimilar orientations are less likely to be coactivated than neurons coding similar orientations. Multiplexing by superposition is ordinarily generated: combinations of intersecting edges become represented by simultaneous activation of multiple neurons, each of which represents a single supported oriented edge. Unsupported or weakly supported orientations are suppressed. The cross problem is thereby solved. [to appear in Proceedings of the SPIE Conference on Advances in Intelligent Systems, Boston, November 1990.] ---------------------------------------------------------------------- Also available: J.A. Marshall, "A Self-Organizing Scale-Sensitive Neural Network." In Proceedings of the International Joint Conference on Neural Networks, San Diego, June 1990, Vol.III., pp.649-654. J.A. Marshall, "Self-Organizing Neural Networks for Perception of Visual Motion." Neural Networks, 3, pp.45-74 (1990). = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = Jonathan A. Marshall marshall at cs.unc.edu = = Department of Computer Science = = CB 3175, Sitterson Hall = = University of North Carolina Office 919-962-1887 = = Chapel Hill, NC 27599-3175, U.S.A. Fax 919-962-1799 = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = **** Please do not re-post to other bboards. **** From issnnet at park.bu.edu Fri Feb 22 12:54:24 1991 From: issnnet at park.bu.edu (Student Society Account) Date: Fri, 22 Feb 91 12:54:24 -0500 Subject: Student Conference Sponsorships Message-ID: <9102221754.AA22635@park.bu.edu> ---- THIS NOTE CONTAINS MATERIAL OF INTEREST TO ALL STUDENTS ---- (and some non-students) This message is a brief update of the International Student Society for Neural Networks (ISSNNet), and also an ANNOUNCEMENT OF AVAILABLE STUDENT SPONSORSHIP AT UPCOMING NNets CONFERENCES. ---------------------------------------------------------------------- NOTE TO ISSNNet MEMBERS: If you have joined the society but did not receive our second newsletter, or have not heard from us in the recent past, please send e-mail to . ---------------------------------------------------------------------- 1) We had a problem with our Post Office Box (in the USA), which was inadvertently shut down for about one month between Christmas and the end of January. If you sent us surface mail which was returned to you, please send it again. Our apologies for the inconvenience. 2) We are about to distribute the third newsletter. This is a special issue that includes our bylaws, a description of the various officer positions, and a call for nominations for upcoming elections. In addition, a complete membership list will be included. ISSNNet MEMBERS: If you have not sent us a note about your interests in NNets, you must do so by about the end of next week to insure it appears in the membership list. Also, all governors should send us a list of their members if they have not done so already. 3) STUDENT SPONSORSHIPS AVAILABLE: We have been in contact with the IEEE Neural Networks Council. We donated $500 (about half of our savings to this point) to pay the registration for students who are presenting articles (posters or oral presentations) at the Helsinki (ICANN), Seattle (IJCNN), or Singapore (IJCNN) conferences. The IEEE Neural Networks Council has augmented our donation with an additional $5,000 to be distributed between the two IJCNN conferences, and we are currently in contact with the International Neural Networks Society (INNS) regarding a similar donation. We are hoping to pay for registration and proceedings for an approximately equal number of students at each of the three conferences. Depending on other donations and on how many people are eligible, only a limited number of sponsorships may be available. The details of eligibility will be officially published in our next newsletter and on other mailing lists, but generally you will need to be the person presenting the paper (if co-authored), and you must receive only partial or no support from your department. Forms will be included with the IJCNN paper acceptance notifications. It is not necessary to be a member of any of these societies, although we hope this will encourage future student support and increased society membership. IF YOU HAVE SUBMITTED A PAPER TO ONE OF THE IJCNN CONFERENCES, YOU WILL RECEIVE DETAILS WITH YOUR NOTIFICATION FROM IJCNN. Details on the ICANN conference will be made available in the near future. For other questions send us some e-mail . ISSNNet, Inc. PO Box 557, New Town Branch Boston, MA 02258 Sponsors will be officially recognized in our future newsletters, and will be mentioned by the sponsored student during the presentations and posters. 6) We are considering the possibility of including abstracts from papers written by ISSNNet members in future newsletters. Depending on how many ISSNNet papers are accepted at the three conferences, we may be able to publish the abstracts in the fourth newsletter, which should come out before the ICANN conference. This would give the presenters some additional publicity, and would give ISSNNet members a sneak preview of what other people are doing. MORE DETAIL ON THESE TOPICS WILL BE INCLUDED IN OUR NEXT NEWSLETTER, WHICH WE EXPECT TO PUBLISH AROUND THE END OF THIS MONTH. For more details on ISSNNet, or to receive a sample newsletter, send e-mail to . You need not be a student to become a member! From bms at dcs.leeds.ac.uk Fri Feb 22 09:58:44 1991 From: bms at dcs.leeds.ac.uk (B M Smith) Date: Fri, 22 Feb 91 14:58:44 GMT Subject: Item for Distribution Message-ID: <7743.9102221458@csunb0.dcs.leeds.ac.uk> ************************************************************************** ***************** * * * A I S B 9 1 * * * ***************** UNIVERSITY OF LEEDS, UK 16 - 19 APRIL 1991 TUTORIAL PROGRAMME 16 APRIL TECHNICAL PROGRAMME 17-19 APRIL with sessions on: * Distributed Intelligent Agents * Situatedness and Emergence in Autonomous Agents * New Modes of Reasoning * The Knowledge Level Perspective * Theorem Proving * Machine Learning Programmes and registration forms are now available from: Barbara Smith AISB91 Local Organizer School of Computer Sudies University of Leeds Leeds LS2 9JT, UK email: aisb91 at ai.leeds.ac.uk *************************************************************************** From jcp at vaxserv.sarnoff.com Fri Feb 22 14:58:54 1991 From: jcp at vaxserv.sarnoff.com (John Pearson W343 x2385) Date: Fri, 22 Feb 91 14:58:54 EST Subject: NIPS Call for Papers Message-ID: <9102221958.AA20764@sarnoff.sarnoff.com> CALL FOR PAPERS Neural Information Processing Systems -Natural and Synthetic- Monday, December 2 - Thursday, December 5, 1991 Denver, Colorado This is the fifth meeting of an inter-disciplinary conference which brings together neuroscientists, engineers, computer scientists, cognitive scientists, physicists, and mathematicians interested in all aspects of neural processing and computation. There will be an afternoon of tutorial presentations (Dec 2) preceding the regular session and two days of focused workshops will follow at a nearby ski area (Dec 6-7). Major categories and examples of subcategories for paper submissions are the following; Neuroscience: Studies and Analyses of Neurobiological Systems, Inhibition in cortical circuits, Signals and noise in neural computation, Theoretical Biology and Biophysics. Theory: Computational Learning Theory, Complexity Theory, Dynamical Systems, Statistical Mechanics, Probability and Statistics, Approximation Theory. Implementation and Simulation: VLSI, Optical, Software Simulators, Implementation Languages, Parallel Processor Design and Benchmarks. Algorithms and Architectures: Learning Algorithms, Constructive and Pruning Algorithms, Localized Basis Functions, Tree Structured Networks, Performance Comparisons, Recurrent Networks, Combinatorial Optimization, Genetic Algorithms. Cognitive Science & AI: Natural Language, Human Learning and Memory, Perception and Psychophysics, Symbolic Reasoning. Visual Processing: Stereopsis, Visual Motion Processing, Image Coding and Classification. Speech and Signal Processing: Speech Recognition, Coding, and Synthesis, Text-to-Speech, Adaptive Equalization, Nonlinear Noise Removal. Control, Navigation, and Planning Navigation and Planning, Learning Internal Models of the World, Trajectory Planning, Robotic Motor Control, Process Control. Applications Medical Diagnosis or Data Analysis, Financial and Economic Analysis, Timeseries Prediction, Protein Structure Prediction, Music Processing, Expert Systems. Technical Program: Plenary, contributed and poster sessions will be held. There will be no parallel sessions. The full text of presented papers will be published. Submission Procedures: Original research contributions are solicited, and will be carefully refereed. Authors must submit six copies of both a 1000-word (or less) summary and six copies of a separate single- page 50-100 word abstract clearly stating their results postmarked by May 17, 1991. Accepted abstracts will be published in the conference program. Summaries are for program committee use only. At the bottom of each abstract page and on the first summary page indicate preference for oral or poster presentation and specify one of the above nine broad categories and, if appropriate, sub-categories (For example: Poster, Applications- Expert Systems; Oral, Implementation-Analog VLSI). Include addresses of all authors at the front of the summary and the abstract and indicate to which author correspondence should be addressed. Submissions will not be considered that lack category information, separate abstract sheets, the required six copies, author addresses, or are late. Mail Submissions To: Stephen J. Hanson NIPS*91 Submissions Siemens Research Center 755 College Road East Princeton NJ, 08540 Mail For Registration Material To: NIPS*91 Registration Siemens Research Center 755 College Road East Princeton, NJ, 08540 All submitting authors will be sent registration material automatically. Program committee decisions will be sent to the correspondence author only. NIPS*91 Organizing Committee: General Chair, John Moody, Yale U.; Program Chair, Stephen J. Hanson, Siemens Research & Princeton U.; Publications Chair, Richard Lippmann, MIT Lincoln Laboratory; Publicity Chair, John Pearson, SRI, David Sarnoff Research Center; Treasurer, Bob Allen, Bellcore; Local Arrangements, Mike Mozer, University of Colorado; Program Co-Chairs:, David Ackley, Bellcore; Pierre Baldi, JPL & Caltech; William Bialek, NEC; Lee Giles, NEC; Mike Jordan, MIT; Steve Omohundro, ICSI; John Platt, Synaptics; Terry Sejnowski, Salk Institute; David Stork, Ricoh & Stanford; Alex Waibel, CMU; Tutorial Chair: John Moody, Workshop CoChairs: Gerry Tesauro, IBM & Scott Kirkpatrick, IBM; Domestic Liasons: IEEE Liaison, Rodney Goodman, Caltech; APS Liaison, Eric Baum, NEC; Neurobiology Liaison, Tom Brown, Yale U.; Government & Corporate Liaison, Lee Giles, NEC; Overseas Liasons: Mitsuo Kawato, ATR; Marwan Jabri, University of Sydney; Benny Lautrup, Niels Bohr Institute; John Bridle, RSRE; Andreas Meier, Simon Bolivar U. DEADLINE FOR SUMMARIES & ABSTRACTS IS MAY 17, 1991 please post From jcp at vaxserv.sarnoff.com Fri Feb 22 15:12:52 1991 From: jcp at vaxserv.sarnoff.com (John Pearson W343 x2385) Date: Fri, 22 Feb 91 15:12:52 EST Subject: NIPS-91 Workshop Message-ID: <9102222012.AA20814@sarnoff.sarnoff.com> CALL FOR WORKSHOPS NIPS*91 Post-Conference Workshops December 6 and 7, 1991 Vail, Colorado Request for Proposals Following the regular NIPS program, workshops on current topics on Neural Information Processing will be held on December 6 and 7, 1991, in Vail, Colorado. Proposals by qualified individuals interested in chairing one of these workshops are solicited. Past topics have included: Rules and Connectionist Models; Speech; Vision; Sensory Biophysics; Neural Network Dynamics; Neurobiology; Computational Complexity Issues; Fault Tolerance in Neural Networks; Benchmarking and Comparing Neural Network Applications; Architectural Issues; Fast Training Techniques; Control; Optimization, Statistical Inference, Genetic Algorithms; VLSI and Optical Implementations; Integration of Neural Networks with Conventional Software. The format of the workshop is informal. Beyond reporting on past research, the goal is to provide a forum for scientists actively working in the field to freely discuss current issues of concern and interest. Sessions will meet in the morning and in the afternoon of both days, with free time in between for the ongoing individual exchange or outdoor activities. Specific open and/or controversial issues are encouraged and preferred as workshop topics. Individuals interested in chairing a workshop must propose a topic of current interest and must be willing to accept responsibility for their group's discussion. Discussion leaders' responsibilities include: arrange brief informal presentations by experts working on this topic, moderate or lead the discussion, and report its high points, findings and conclusions to the group during evening plenary sessions, and in a short (2 page) summary. Submission Procedure: Interested parties should submit a short proposal for a workshop of interest by May 17, 1991. Proposals should include a title and a short description of what the workshop is to address and accomplish. It should state why the topic is of interest or controversial, why it should be discussed and what the targeted group of participants is. In addition, please send a brief resume of the prospective workshop chair, list of publications and evidence of scholarship in the field of interest. Mail submissions to: Dr. Gerald Tesauro, Co-Chair, Attn: NIPS91 Workshops, IBM Research P.O. Box 704 Yorktown Heights, NY 10598 USA Name, mailing address, phone number, and e-mail net address (if applicable) must be on all submissions. Workshop CoChairs: G. Tesauro & S. Kirkpatrick, IBM PROPOSALS MUST BE RECEIVED BY MAY 17,1991 Please Post From erol at ehei.ehei.fr Fri Feb 22 13:11:25 1991 From: erol at ehei.ehei.fr (Erol Gelenbe) Date: Fri, 22 Feb 91 18:13:25 +2 Subject: Preprint on Texture Generation with the Random Neural Network Message-ID: <9102230855.AA15700@inria.inria.fr> The following paper, accepted for oral presentation at ICANN-91, is available as a preprint : Texture Generation with the Random Neural Network Model by Volkan Atalay, Erol Gelenbe, Nese Yalabik A copy may be obtained by e-mailing your request to : erol at ehei.ehei.fr Erol Gelenbe EHEI Universite Rene Descartes (Paris V) 45 rue des Saints-Peres 75006 Paris From thrun at gmdzi.uucp Sun Feb 24 17:56:27 1991 From: thrun at gmdzi.uucp (Sebastian Thrun) Date: Sun, 24 Feb 91 21:56:27 -0100 Subject: No subject Message-ID: <9102242056.AA06846@gmdzi.gmd.de> Subject: Wrong host name Sorry, in my last posting I made the mistake to write "cis.ohio-state.edu" instead of the correct "cheops.cis-ohio-state.edu" as the address of the neuroprose archieve. Of course, the TRs I announced can only be retrieved from the latter machine. To all those who failed: Please try again with this new address. ---- thanks, sebastian From MURRE%rulfsw.LeidenUniv.nl at BITNET.CC.CMU.EDU Mon Feb 25 16:57:00 1991 From: MURRE%rulfsw.LeidenUniv.nl at BITNET.CC.CMU.EDU (MURRE%rulfsw.LeidenUniv.nl@BITNET.CC.CMU.EDU) Date: Mon, 25 Feb 91 16:57 MET Subject: Limited precision implementations (updated posting) Message-ID: Connectionist researchers, Here is an updated posting on limited precision implementations of neural networks. It is my impression that research in this area is still fragmentary. This is surprising, because the literature on analog and digital implementations is growing very fast. There is a wide range of possibly applicable rules of thumb. Claims about sufficient precision differ from single bits to 20 bits or more for certain models. Hard problems may need higher precision. There may be a trade-off between few weights (nodes) with high precision weights (activations) versus many weights (nodes) with low precision weights (act.). The precise relation between precision in weights and activations remains unclear, as does the relation between the effect of precision on learning and recall. Thanks for all comments so far. Jaap Jacob M.J. Murre Unit of Experimental and Theoretical Psychology Leiden University P.O. Box 9555 2300 RB Leiden The Netherlands General comments by researchers By Soheil Shams: As far as the required precision for neural computation is concerned, the precision is directly proportional to the difficulty of the problem you are trying to solve. For example in training a back-propagation network to discriminate between two very similar classes of inputs, you will need to have high precision values and arithmetic to effectively find the narrow region in the space that the separating hyperplane has to be drawn at. I believe that the lack of analytical information in this area is due to this relationship between the specific application and the required precision . At the NIPS90 workshop on massively parallel implementations, some people indicated they have determined, EMPERICALLY, that for most problems, 16-bit precision is required for learning and 8-bit for recall of back-propagation. By Roni Rosenfeld: Santosh Venkatesh (of Penn State, I believe, or is it U. Penn?) did some work a few years ago on how many bits are needed per weight. The surprising result was that 1 bit/weight did most of the work, with additional bits contributing surprisingly little. By Thomas Baker: ... We have found that for backprop learning, between twelve and sixteen bits are needed. I have seen several other papers with these same results. After learning, we have been able to reduce the weights to four to eight bits with no loss in network performance. I have also seen others with similar results. One method that optical and analog engineers use is to calculate the error by running the feed forward calculations with limited precision, and learning the weights with a higher precision. The weights are quantized and updated during training. I am currently collecting a bibliography on limited precision papers. ... I will try to keep in touch with others that are doing research in this area. References Brause, R. (1988). Pattern recognition and fault tolerance in non-linear neural networks. Biological Cybernetics, 58, 129-139. Hollis, P.W., J.S. Harper, J.J. Paulos (1990). The effects of precision constraints in a backpropagation learning network. Neural Computation, 2, 363-373. Holt, J.L., & J-N. Hwang (in prep.). Finite precision error analysis of neural network hardware implementations. Univ. of Washington, FT-10, WA 98195. (Comments by the authors: We are in the process of finishing up a paper which gives a theoretical (systematic) derivation of the finite precision neural network computation. The idea is a nonlinear extension of "general compound operators" widely used for error analysis of linear computation. We derive several mathematical formula for both retrieving and learning of neural networks. The finite precision error in the retrieving phase can be written as a function of several parameters, e.g., number of bits of weights, number of bits for multiplication and accumlation, size of nonlinear table-look-up, truncation/rounding or jamming approaches, and etc. Then we are able to extend this retrieving phase error analysis to iterative learning to predict the necessary number of bits. This can be shown using a ratio between the finite precision error and the (floating point) back-propagated error. Simulations have been conducted and matched the theoretical prediction quite well.) Hong, J. (1987). On connectionist models. Tech. Rep., Dept. Comp. Sci., Univ. of Chicago, May 1987. (Demonstrates that a network of perceptrons needs only finite-precision weights.) Jou, J., & J.A. Abraham (1986). Fault-tolerant matrix arithmetic and signal processing on highly concurrent computing structures. Proceedings of the IEEE, 74, 732-741. Kampf, F., P. Koch, K. Roy, M. Sullivan, Z. Delalic, & S. DasGupta (1989). Digital implementation of a neural network. Tech. Rep. Temple Univ., Philadelphia PA, Elec. Eng. Div. Moore, W.R. (1988). Conventional fault-tolerance and neural computers. In: R. Eckmiller, & C. Von der Malsburg (Eds.). Neural Computers. NATO ASI Series, F41, (Berling: Springer-Verlag), 29-37. Nadal, J.P. (1990). On the storage capacity with sign-constrained synaptic couplings. Network, 1, 463-466. Nijhuis, J., & L. Spaanenburg (1989). Fault tolerance of neural associative memories. IEE Proceedings, 136, 389-394. Rao, A., M.R. Walker, L.T. Clark, & L.A. Akers (1989). Integrated circuit emulation of ART networks. Proc. First IEEE Conf. Artificial Neural Networks, 37-41, Institution of Electrical Engineers, London. Rao, A., M.R. Walker, L.T. Clark, L.A. Akers, & R.O. Grondin (1990). VLSI implementation of neural classifiers. Neural Computation, 2, 35-43. (The paper by Rao et al. give an equation for the number of bits of resolution required for the bottom-up weights in ART 1: t = (3 log N) / log(2), where N is the size of the F1 layer in nodes.) From giles at fuzzy.nec.com Mon Feb 25 13:48:06 1991 From: giles at fuzzy.nec.com (Lee Giles) Date: Mon, 25 Feb 91 13:48:06 EST Subject: possible academic positions Message-ID: <9102251848.AA15611@fuzzy.nec.com> The February issue of IEEE Spectrum posted 7 academic positions that mentioned "neural networks" or "neural computing" in the job description. -- C. Lee Giles NEC Research Institute 4 Independence Way Princeton, NJ 08540 USA Internet: giles at research.nj.nec.com UUCP: princeton!nec!giles PHONE: (609) 951-2642 FAX: (609) 951-2482 From giles at fuzzy.nec.com Mon Feb 25 13:52:01 1991 From: giles at fuzzy.nec.com (Lee Giles) Date: Mon, 25 Feb 91 13:52:01 EST Subject: Summer Position Message-ID: <9102251852.AA15615@fuzzy.nec.com> NEC Research Institute in Princeton, N.J. has available a 3 month summer research and programming position. The research emphasis will be on exploring the computational capabilities of recurrent neural networks. The successful candidate will have a background in neural networks and strong programming skills in the C/Unix environment. Computer science background preferred. Interested applicants should send their resumes by mail, fax, or email to the address below. The application deadline is March 25, 1991. Applicants must show documentation of eligibility for employment. Because this is a summer position, the only expenses to be paid will be salary. NEC is an equal opportunity employer. -- C. Lee Giles NEC Research Institute 4 Independence Way Princeton, NJ 08540 USA Internet: giles at research.nj.nec.com UUCP: princeton!nec!giles PHONE: (609) 951-2642 FAX: (609) 951-2482 From gluck%psych at Forsythe.Stanford.EDU Mon Feb 25 14:03:38 1991 From: gluck%psych at Forsythe.Stanford.EDU (Mark Gluck) Date: Mon, 25 Feb 91 11:03:38 PST Subject: Postdoc & Research Assistant openings in the COGNITIVE & NEURAL BASES OF LEARNING (Rutgers, NJ) Message-ID: <9102251903.AA25039@psych> Postdoctoral & Research/Programming Positions in: THE COGNITIVE & NEURAL BASES OF LEARNING ---------------------------------------------------------------------------- Rutgers University Center for Molecular & Behavioral Neuroscience 195 University Avenue Newark, NJ 07102 Postdoctoral Positions in: -------------------------- 1. EMPIRICAL STUDIES OF HUMAN LEARNING: Including: designing and conducting studies of human learning and decision making, especially categorization learning. These are primarily motivated by a desire to evaluate and refine adaptive network models of learning and memory (see, e.g., the experimental studies described in Gluck & Bower, 1988a; Pavel, Gluck, & Henkle, 1988). This work requires a familiarity with psychological methods of experimental design and data analysis. 2. COMPUTATIONAL MODELS OF ANIMAL & HUMAN LEARNING: Including: developing and extending current network models of learning to more accurately reflect a wider range of animal and human learning behaviors. This work requires strong programming skills, familiarity with adaptive network theories and methods, and some degree of mathematical and analytic training. 3. COMPUTATIONAL MODELS OF THE NEUROBIOLOGY OF LEARNING & MEMORY: Including: (1) Models and theories of the neural bases of classical and operant conditioning; (2) Neural mechansims for human associative learning; (3) Theoretical studies which seek to form links, behavioral or biological, between animal and human learning (see, e.g., Gluck, Reifsnider, & Thompson (1989), in Gluck & Rumelhart (Eds.) Neuroscience & Connectionist Theory). and Connectionist Theory). Full or Part-Time Research & Programming Positions: --------------------------------------------------- These positions are ideal for someone who has just graduated with an undergraduate degree and would like a year or two of "hands on" experience in research before applying to graduate school in one of the cognitive sciences (e.g., neuroscience, psychology, computer science). We are looking for two types of people: 1) a RESEARCH PROGRAMMER with strong computational skills (especially with C/Unix and SUN systems) and experience with PDP models and theory, and (2) an EXPERIMENTAL RESEARCH ASSISTANT to assist in running and designing human learning experiments. Some research experience required (familiarity with Apple MACs a plus). Application Procedure: ---------------------- For more information on learning research at the CMBN/Rutgers or to apply for these positions, please send a cover letter with a statement of your research interests, a CV, copies of relevant preprints, and the the names & phone numbers of references to: Dr. Mark A. Gluck Phone: (415) 725-2434 Dept. of Psychology <-[Current address to 4/91] FAX: (415) 725-5699 Jordan Hall; Bldg. 420 Stanford University email: gluck at psych.stanford.edu Stanford, CA 94305-2130 --------------------------end of notice-------------------------- From holt at pierce.ee.washington.edu Mon Feb 25 14:33:44 1991 From: holt at pierce.ee.washington.edu (Jordan Holt) Date: Mon, 25 Feb 91 11:33:44 PST Subject: Technical Report Available Message-ID: <9102251933.AA01454@pierce.ee.washington.edu.Jaimie> Technical Report Available: Finite Precision Error Analysis of Neural Network Hardware Implementations Jordan Holt, Jenq-Neng Hwang The high speed desired in the implementation of many neural network algorithms, such as back-propagation learning in a multilayer perceptron (MLP), may be attained through the use of finite precision hardware. This finite precision hardware; however, is prone to errors. A method of theoretically deriving and statistically evaluating this error is presented and could be used as a guide to the details of hardware design and algorithm implementation. The paper is devoted to the derivation of the techniques involved as well as the details of the back-propagation example. The intent is to provide a general framework by which most neural network algorithms under any set of hardware constraints may be evaluated. Section 2 demonstrates the sources of error due to finite precision computation and their statistical properties. A general error model is also derived by which an equation for the error at the output of a general compound operator may be written. As an example, error equations are derived in Section 3 for each of the operations required in the forward retrieving and error back- propagation steps of an MLP. Statistical analysis and simulation results of the resulting distribution of errors for each individual step of an MLP are also included in this section. These error equations are then integrated, in Section 4, to predict the influence of finite precision computation on several stages (early, middle, final stages) of back-propagation learning. Finally, concluding remarks are given in Section 5. ---------------------------------------------------------------------------- The report can be received by ftp: unix> ftp cheops.cis.ohio-state.edu Name: anonymous Guest Login ok, send ident as password Password: neuron ftp> binary ftp> cd pub ftp> cd neuroprose ftp> get holt.finite_error.ps.Z ftp> bye unix> uncompress holt.finite_error.ps unix> lpr holt.finite_error.ps From ecdbcf at ukc.ac.uk Mon Feb 25 12:07:30 1991 From: ecdbcf at ukc.ac.uk (ecdbcf@ukc.ac.uk) Date: Mon, 25 Feb 91 17:07:30 +0000 Subject: Boolean Models(GSN) Message-ID: Dear Connectionists, Most people who read this mail will probably be working with continuous/analogue models. There is, however, a growing interest in Boolean neuron models, and some readers might be interested to know that I have recently successfully completed a Ph.D thesis which deals with a particular kind of Boolean neuron. Some brief details are given below, together with some references to more detailed material. ----------------------------------------------------------------------- Abstract This thesis is concerned with the investigation of Boolean neural networks based on a novel RAM-based Goal-Seeking Neuron(GSN). Boolean neurons are particularly suited to the solution of Boolean or logic problems such as the recognition and associative recall of binarised patterns. One main advantage of Boolean neural networks is the ease with which they can be implemented in hardware. This can result in very fast operation. The GSN has been formulated to ensure this implementation advantage is not lost. The GSN model operates through the interaction of a number of local low level goals and is applicable to practical problems in pattern recognition with only a single pass of the training data(one-shot learning). The thesis explores different architectures for GSNs (feed-forward, feedback and self-organising) together with different learning rules, and investigates a wide range of alternative configurations within these three architectures. Practical results are demonstrated in the context of a character recognition problem. ----------------------------------------------------------------------- Highlights of GSNs, Learning Algorithms, Architectures and Main Contributions The main advantage of RAM-based neural networks in comparison with networks based on sum-of-products functions is the ease with which they can be implemented in hardware. This derives from their essentially logical rather than continuous nature. The GSN model has a natural propensity to solve the main problems associated with other RAM-based neurons. Specific classes of computational activity can be more appropriately realised by using a particular goal seeking function, and different kinds of goal seeking functions can be sought in order to provide a range of suitable behaviour, creating effectively a family of GSNs. The main experimental results have demonstrated the viability of the one-shot learning algorithms: partial pattern association, quasi-self-organisation, and self-organisation. The one-shot learning is only possible because of the the GSN's ability to validate the possibility of learning a given input pattern using a single presentation. The partial pattern association and the quasi-self-organising learning have been applied in feed-forward architectures. These two kinds of learning have given similar performance, though the quasi-self-organising learning gives slightly better results when a small training size is considered. The work reported has established the viability and basic effectiveness of the GSN concept. The GSN proposal provides a new range of computational units, learning algorithms, architectures, and new concepts related to the fundamental processes of computation using Boolean networks. In all of these ideas further modifications, extensions, and applications can be considered in order fully to establish Boolean neural networks as a strong candidate for solving Boolean-type problems. A great deal of additional research can be identified for immediate investigation as follows. One of the most important contributions of this work is the idea of flexible local goals in RAM-based neurons which allows the application of RAM-based neurons and architectures to a wider range of problems. The definition of the goal seeking functions for all the GSN models used in the feed-forward, feedback and self-organising architectures are important because they provide local goals which try to maximise the memory capacity and to improve the recall of correct output patterns. Although the supervised pattern association learning is not the kind of learning most suitable for use with GSN networks, because it demands multi-presentations of the training set and causes a fast saturation of the neurons' contents, the variety of solutions presented to the problem of conflict of learning can help to achieve correct learning with a relatively small number of activations compared to the traditional way of erasing a path without taking care to keep the maximum number of stored patterns. The partial pattern association, quasi-self-organising, and the self-organising learning have managed to break away from the traditional necessity for many thousands of presentations of the training set, and instead have concentrated on providing one-shot learning. This is made possible by the propagation of the undefined value between the neurons in conjunction with the local goal used in the validating state. Due to the partial coverage area and the limited functionality of the pyramids, which can cause an inability to learn particular patterns, it is important to change the desired output patterns in order to be able to learn these classes. The network produces essentially self-desired output patterns which are similar to the desired output patterns, but not necessarily the same. The differences between the desired output patterns and the self-desired output patterns can be observed in the learning phase by looking at the output values of each pyramid and the desired output values. The definition of the self-desired and the learning probability recall rules have provided a way of sensing the changes in the desired output patterns, and of achieving the required pattern classification. The principle of low connectivity and partial coverage area make possible more realistic VLSI implementations in terms of memory requirements and overall connection complexity associated with the traditional problem of fan-in and fan-out for high connectivity neurons. The feedback architecture is able to achieve associative recall and pattern completion, demonstrating that it is possible to have a cascade of feedback networks that incrementally increases the similarity between a prototype and the output patterns. The utilisation of the freeze feedback operation has given a high percentage of correct convergences and fast stabilisation of the output patterns. The analysis of the saturation problem has demonstrated that the traditional way of using uniform connectivity for all the layers impedes the advance of the learning process and many memory addresses remain unused. This is because saturation is not at the same level for each of the layers. Thus, a new approach has been developed to assign a varied connectivity to the architecture which can achieve a better capacity of learning, a lower level of saturation and a smaller residue of unused memory. In terms of architectures and learning, an important result is the design of the GSN self-organising network which incorporates some principles related to the Adaptive Resonance Theory(ART). The self-organising network contains intrinsic mechanisms to prevent the explosion of the number of clusters necessary for self-stabilising a given training pattern set. Several interesting properties are found in the GSN self-organising network such as: attention, discrimination, generalisation, self-stabilisation, and so on. References @conference{key210, author = "D L Bisset And E C D B C Filho And M C Fairhurst", title = "A Comparative study of neural network structures for practical application in a pattern recognition enviroment", publisher= "IEE", booktitle= "Proc. First IEE International Conference on Artificial Neural Networks", address = "London, UK", month = "October", pages = "378-382", year = "1989" } @conference{key214, author = "E C D B C Filho And D L Bisset And M C Fairhurst", title = "A Goal Seeking Neuron For {B}oolean Neural Networks", publisher= "IEEE", booktitle= "Proc. International Neural Networks Conference", address = "Paris, France", month = "July", volume = "2", pages = "894-897", year = "1990" } @article{key279, author = "E C D B C Filho And D L Bisset And M C Fairhurst", title = "Architectures for Goal-Seeking Neurons", journal= "International Journal of Intelligent Systems", publisher= "John Wiley & Sons, Inc", note = "To Appear", year = "1991" } @article{key280, author = "E C D B C Filho And M C Fairhurst And D L Bisset", title = "Adaptive Pattern Recognition Using Goal-Seeking Neurons", journal= "Pattern Recognition Letters", publisher= "North Holland", month = "March" year = "1991" } All the best, Edson ... Filho -- Before 10-Mar-91 ---------------------------------------------------------- ! Edson Costa de Barros Carvalho Filho ! ARPA: ecdbcf%ukc.ac.uk at cs.ucl.ac.uk ! ! Electronic Engineering Laboratories ! UUCP: ecdbcf at ukc.ac.uk ! ! University of Kent at Canterbury ! Phone: (227) 764000x3718 ! ! Canterbury Kent CT2 7NT England ! ! -- After 10-Mar-91 ----------------------------------------------------------- ! Universidade Federal de Pernambuco ! e-mail: edson at di0001.ufpe.anpe.br ! ! Departamento de Informatica ! Phone: (81) 2713052 ! ! Av. Prof. Luis Freire, S/N ! ! ! Recife --- PE --- Brazil --- 50739 ! ! ------------------------------------------------------------------------------ From fellous%pipiens.usc.edu at usc.edu Mon Feb 25 16:39:48 1991 From: fellous%pipiens.usc.edu at usc.edu (Jean-Marc Fellous) Date: Mon, 25 Feb 91 13:39:48 PST Subject: No subject Message-ID: <9102252139.AA18397@pipiens.usc.edu> Subject: CNE Workshop on Emotions ***************************************************************************** ** C.N.E W O R K S H O P O N E M O T I O N S ** ***************************************************************************** The Center for Neural Engineering of the University of Southern California is happy to announce that its student Workshop on Emotions will be held Monday March 18th from 8.30am to 4.00pm in the Hedco Neuro-Science Building Auditorium (on U.S.C campus). The papers presented will be the following: Affect versus Cognitive-repair Behaviors. Sharon Ruth Gross - U.S.C (Social Psychology) A Mathematical representation of Emotions. Charles Rapp - Illinois Institute of Technology (Computer Science). Cognitive and Emotional disorders in Parkinson's Disease. Peter Dominey - U.S.C (C.N.E, Gerontology). Cognitive-Emotional interaction using subsymbolic paradigm. Aluizio Fausto Ribeiro Araujo - University of Sussex (U.K) (School of cognitive and Computing Sciences) Emotional expressions conceptualized as uniquely effective communication devices Heidi M. Lincer - U.S.C (Psychology). Taxi world: Computing Emotions. Clark Eliott - Northwestern University. (Artificial Intelligence and Cognitive Sciences). Zeal: A Sociological perspective on Emotion, cognition and organizational structure. Gerardo Marti - U.S.C (Gerontology, Sociology) In addition, the following papers have been accepted but will not be presented orally during the Workshop. They will be put on loan during the Workshop. Emotions and autonomous machinery. Douglas A. Kerns - California Institute of Technology (Electrical Engineering). Representation, Action, and Emotion. Michael Travers - M.I.T (Media-Lab). Toward an Emotional Computer: Models of Emotions. Jean-Marc Fellous - U.S.C (C.N.E Computer Science) There will not be any registration fees but, as to get an estimation of the number of persons attending the Workshop, interested people are invited to announce their attendance by email (or surface mail). We remind the participants that this event being a Workshop not a Conference they are strongly encouraged to participate to the debates by their comments and questions to the speakers. Thank you for forwarding this announcement to potentialy interested persons/instituions/mailing_lists. Further informations requests (and email registration) can be addressed to Jean-Marc FELLOUS Center For Neural Engineering University of Southern California U.S.C - University Park Los Angeles CA 90089-2520 U.S.A Tel: (213) 740-3506 Fax: (213) 746-2863 email: fellous at rana.usc.edu .. From R14502%BBRBFU01.BITNET at BITNET.CC.CMU.EDU Wed Feb 27 10:16:14 1991 From: R14502%BBRBFU01.BITNET at BITNET.CC.CMU.EDU (R14502%BBRBFU01.BITNET@BITNET.CC.CMU.EDU) Date: Wed, 27 Feb 91 16:16:14 +0100 Subject: A book on Self-Organization Message-ID: To all interested in the concept of Selforganization ------------------------------------------------------ The subject matter of selforganization has drawn a lot of attention from physicists, chemists, and theoretical biologists before becoming so popular with the Neural Network researches. Anybody interested in the field with few hours to spare may find the basics and typical examples of self-organization of complex systems in an elementary book which I wrote few years ago. The title is: "Molecules, Dynamics and Life: An introduction to self-organization of matter", Wiley, New York, 1986. A. Babloyantz University of Brussels " Dr. Babloyantz has produced an engaging and earnest introduction to the field of self-organization in chemical and biological systems. Dr. Babloyantz proves herself to be a pleasant, practical and reliable guide to new territory which is still largely uncharted and inhospitable to tourists. Her style falls halfway between that found in a popular account and that of a txtbook. She tells her story in a chatty, down-to-earth way, while also giving serious scientific consideration to fundamental issues of the self-organization of matter." (Nature) " The issue of self-organization has at the center of a larger theoretical revolution in physics - the belief that the fundamental laws of nature are irreversible and random, rather than determinstic and reversible. The concepts and processes underlying this new way of thinking are formidable. Molecules, Dynamics and Life makes these concepts and processes accessible, for the first time, to students and researchers in physics, chemistry, biology, and the social sciences." (Physics Briefs) " In Molecules, Dynamics and Life, Dr. Agnes Babloyantz develops a clear and easy to read presentation of this developing field of knowledge. Because only a few advanced research treatises are available so far, this book is especially welcomed. It offers an excellent introduction to an interdisciplinary domain, involving physics and biology, chemistry and mathematics. Obviously, putting together all these topics and making them readable to a large audience was really a challenge." (Biosystem's) " With this fine book Agnessa Babloyantz has provided a successful and welcome summary of what has been accomplished so far in the study of self-organization of matter according to the Prigogine school in Brussels. Dr. Babloyantz's book can be highly recommended to all those interested in self-organization in the fields of chemistry and biochemistry." (Bull. Math. Biology) From gac at cs.brown.edu Wed Feb 27 14:19:59 1991 From: gac at cs.brown.edu (Glenn Carroll) Date: Wed, 27 Feb 91 14:19:59 -0500 Subject: position available Message-ID: <9102271919.AA05483@tolstoy.cs.brown.edu> I'm forwarding this for a friend--please note the BITNET address below for replies. From: CSA::GYULASSY 21-FEB-1991 08:32:48.00 To: CARROLL CC: GYULASSY Subj: net jon Research Position Available Effective March 1,1990 Place: Nuclear Science Division Lawrence Berkeley Laboratory Area: Neural Network Computing Research with Application to Complex Pattern Recognition Problems in High Energy and Nuclear Physics Description: Experiments in high energy and nuclear physics are confronted with increasingly difficult pattern recognition problems, for example in tracking charged particles and identifying jets in very high multiplicity and noisy environments. In 1990, a generic R&D program was initiated at LBL to develop new computational strategies to solve such problems. The emphasis is on developing and testing artificial neural network algorithms. Last year we developed a new Elastic Network type tracking algorithm that is able to track at densities an order of magnitude higher than conventional Road Finding algorithms and even Hopfield Net type algorithms. This year we plan on a number of followup studies and extensions of that work as well as begin research on jet finding algorithm. Jets are formed through the fragmentation of high energy quarks and gluons, via a rare process in high energy collisions of hadrons or nuclei. The problem of identifying such jets via calorimetric or tracking detectors is greatly complicated by the very high multiplicity of fragments produced via other processes. The research will involve developing new preprocessing strategies and network architectures to be trained by simulated Monte Carlo data. Required Qualifications: General understanding of basic neural computing algorithms such as multilayer feed forward and recurrent nets and a variety of training algorithms. Proficiency in programing in Fortran and C on a variety of systems VAX/VMS and/or Sparc/UNIX. Interested applicants should contact Miklos Gyulassy Mailstop 70A-3307 LBL Berkeley, CA 94720 E-mail: GYULASSY at LBL.Bitnet Telephone: (415) 486-5239 From BHAVSAR%UNB.CA at UNBMVS1.csd.unb.ca Thu Feb 28 14:58:50 1991 From: BHAVSAR%UNB.CA at UNBMVS1.csd.unb.ca (V. C. Bhavsar) Date: Thu, 28 Feb 91 15:58:50 AST Subject: Supercomputing Symposium,June3-5,Canada Message-ID: SUPERCOMPUTING SYMPOSIUM'91 June 3-5,1991,Fredericton,N.B.,Canada The fifth annual Canadian Supercomputing Symposium, sponsored by the Canadian Special Interest Group on Supercomputing (SUPERCAN) and the Faculty of Computer Science at U.N.B, will be held in Fredericton, N. B. Canada, From June 3 to 5 1991. Previous symposiums were held in Calgary, Edmonton, Toronto, and Montreal. SS91 promises to be a very exiting event. Over 30 papers have been received as of Feburary28,1991. The invited speakers are: Dr. Andrew Bjerring, Director, Computing & Communication Services University of Western Ontario, London ON, Canada. 'Scientific Computing in Canada'(Tentative title) Dr. John Caulfield, Director, Center for Applied Optics, University of Alabama, Hunstville, AL, USA 'Progress in Quantum Optical Computing' Dr. Narenda Karmarkar, AT&T Bell Labs., Murray Hill, New Jersey, USA 'A New Supercomputer Architecture for Scientific Computation based on Finite Geometries' Dr.K.J.M.Moriarty,Institute for Computational Studies, Dalhousie University,Halifax,N.S.,Canada Topic: Computational Physics Dr. Louise Nielson, Director, High Performance Computing, IBM, Kingston, NY, USA 'Future Directions in High Performance Computing'(Tentative title) (Inaugural Talk) Dr. Richard Peltier, Department of Physics, Universtiy of Toronto, Toronto, ON, Canada. 'Imaging Hydrodynamic Complexity with Supercomputers' ************************ CALL FOR PAPERS Papers are solicited on significant research results in the development and use of supercomputing systems in (but not restrcted to ) the follwing areas : Applications of Supercomputing Neural Computing Supercomputing Algorithms Systems Software Performance Analysis Applications Development Tools Design of Supercomputing Systems Scientific Visualization Optical Computing Supercomputing Education Biomolecular Computing Networks and Communications DEADLINES Mar 11,1991 -Two copies of extended summary (~2 pages) Mar 18,1991 -Notification to authors Apr 18,1991 -Camera-ready copy May 1,1991 -Advance Registration ******************** EXHIBITORS Manufacturers and suppliers of the supercomputers and desktop supercomputer products are encouraged to contact the organizers. ******************* Organizers: Virendra Bhavsar -General Chairman bhavsar at unb.ca Uday G. Gujar -Program Chairman uday at unb.ca Kirby Keyser -Exhibits Chairman kmk at unb.ca John M.DeDourek -Local Arrangements Chairman dedourek at unb.ca Snail Mail: Supercomputing Group Faculty of Computer Science University of New Brunswick Fredericton, N.B., E3B 5A3, Canada Voice: (506) 453-4566 Fax: (506) 453-3566 --------------------- End of forwarded message --------------------- From rich at gte.com Fri Feb 1 14:56:12 1991 From: rich at gte.com (Rich Sutton) Date: Fri, 1 Feb 91 14:56:12 -0500 Subject: New Book: Neural Networks for Control Message-ID: <9102011956.AA19613@bunny.gte.com> In the final hours of 1990, MIT Press printed a new book that I hope will be of interest to readers of this mailing list: NEURAL NETWORKS FOR CONTROL edited by W. T. Miller, R. S. Sutton and P. J. Werbos This is an edited collection with articles by Barto, Narendra, Widrow, Albus, Anderson, Selfridge, Sanderson, Kawato, Atkeson, Mel, Williams, and many others. - Rich Sutton From mehra%aquinas at uxc.cso.uiuc.edu Fri Feb 1 18:04:52 1991 From: mehra%aquinas at uxc.cso.uiuc.edu (Pankaj Mehra) Date: Fri, 1 Feb 91 17:04:52 CST Subject: Usefulness of chaotic dynamics Message-ID: <9102012304.AA00422@hobbes> Fellow Connectionists: It was with fascination that I read Gleick's book on chaos recently. My initial reaction was that the thoery of chaotic dynamics of nonlinear systems represents a significant advance in science by allowing us to model and understand complex phenomena, such as turbulence. Yesterday, John Hopfield was in Champaign, talking about ``Neural Computation'' at the centennial celebration of Physics department. During his talk, he made a distinction between useful and ``unlikely to be useful'' dynamics of neural networks with feedback. Essentially, dynamics which show convergent behavior are considered useful, and those that show chaotic or divergent behavior, not useful. The crux of such an argument lies in the fact that ``similar but unresolvable initial conditions lead to large divergences in trajectories'' [1] for a chaotic system. This is an antithesis of generalization, which is considered a defining trait of connectionist models. However, a recent paper by Frison [1] shows how the behavior of chaotic systems may be predicted using neural networks. The applications suggested in this paper include weather forecasting and securities analysis. In my opinion, there is an emerging trend to move away from convergent dynamics. Section 4 of Hirsch's paper [2] presents arguments for/against using chaotic dynamics. In a limited sense, the work of Michail Zak [3] shows how breaking some traditional assumptions of convergent behavior can help solve the problems of spurious and local minima. I would like to hear more about the pros and cons of this issue. - Pankaj Mehra {p-mehra at uiuc.edu, mehra at cs.uiuc.edu, mehra at aquinas.csl.uiuc.edu} [1] Frison, T. W., "Predicting Nonlinear and Chaotic Systems Behavior Using Neural Networks," Jrnl. Neural Network Computing, pp. 31-39, Fall 1990. [2] Hirsch, M. W., "Convergent Activation Dynamics is Continuous Time Networks," Neural Networks, vol. 2, pp. 331-349, Pergamon Press, 1989. [3] Zak, M., "Creative Dynamics Approach to Neural Intelligence," Biological Cybernetics, vol. 64, pp. 15-23, Springer-Verlag, 1990. From koch%CITIAGO.BITNET at vma.CC.CMU.EDU Sat Feb 2 16:05:15 1991 From: koch%CITIAGO.BITNET at vma.CC.CMU.EDU (Christof Koch) Date: Sat, 2 Feb 91 13:05:15 PST Subject: Tech report on 'Awareness" Message-ID: <910202130456.204011e6@Iago.Caltech.Edu> The following paper is available by anyonymous FTP from Ohio State University from pub/neuroprose. The manuscript, a verbatim copy of the same named manuscript which appeared in "Seminars in the Neurosciences", Volume 2, pages 263-273, 1990, is stored in three files called koch.awareness1.ps.Z, koch.awareness2.ps.Z and koch.awareness3.ps.Z For instructions on how to get this file, see below. TOWARDS A NEUROBIOLOGICAL THEORY OF CONSCIOUSNESS Francis Crick and Christof Koch Visual awareness is a favorable form of consciousness to study neurobiologically. We propose that it takes two forms: a very fast form, linked to iconic memory, that may be difficult to study; and a somewhat slower one involving visual attention and short-term memory. In the slower form an attentional mechanism transiently binds together all those neurons whose activity relates to the relevant features of a single visual object. We suggest that this is done by generating coherent semi-synchronous oscillations, probably in the 40 Hz range. These oscillations then activate a transient short-term (working) memory. We outline several lines of experimental work that might advance the understanding of the neural mechanisms involved. The neural basis of very short-term memory especially needs more experimental study. Key words: consciousness / awareness/ visual attention / 40 Hz oscillations / short-term memory. For comments, send e-mail to koch at iago.caltech.edu. Christof P.S. And this is how you can FTP and print the file: unix> ftp cheops.cis.ohio-state.edu (or 128.146.8.62) Name: anonymous Password: neuron ftp> cd pub/neuroprose (actually, cd neuroprose) ftp> binary ftp> get koch.awareness1.ps.Z ftp> get koch.awareness2.ps.Z ftp> get koch.awareness3.ps.Z ftp> quit unix> uncompress koch.awareness1.ps.Z unix> lpr koch.awareness1.ps.Z unix> uncompress koch.awareness2.ps.Z unix> lpr koch.awareness2.ps.Z unix> uncompress koch.awareness3.ps.Z unix> lpr koch.awareness3.ps.Z Done! From HORN at vm.tau.ac.il Mon Feb 4 15:51:36 1991 From: HORN at vm.tau.ac.il (David Horn) Date: Mon, 04 Feb 91 15:51:36 IST Subject: Usefulness of chaotic dynamics Message-ID: Pankaj Mehra has raised the issue of usefulness of chaotic dynamics in neural network models following a statement by Hopfield who made a distinction between useful dynamics, which have convergent behavior, and "unlikely to be useful" dynamics which are divergent. I would like to point out the importance of dynamics which are convergent on a short time scale and divergent on a long time scale. We have worked on neural networks which display such behavior. In particular we can model a system which converges to a set of fixed points on a short time scale (thus performing some "useful" computation), and is "free" to move between them on a longer time scale. This kind of freedom stems from the unpredictability of chaotic systems. Such networks will undoubtedly be very useful for modelling unguided thinking processes and decision making. We may still be far from meaningful models of higher cognitive phenomena, but many will agree that these are valuable long-term goals. In the meantime, these networks are of interest because they form good examples of oscillating systems. Our networks are feedback systems of formal neurons to which we attribute dynamical thresholds. Letting a threshold rise when the neuron with which it is associated keeps firing, we simulate fatigue. Some short time after the network moves into an attractor, the fatigue effect destabilizes the attractor and throws the system into a different basin of attraction. This can go on indefinitely. Naturally it leads to an oscillatory behavior. The recent observations of cortical oscillations led to increased interest in oscillating neural networks, of which ours are particular examples. We have developped several models, all of which display spontaneous and induced transitions between memory patterns. Such models are simple testing grounds for hypotheses about the relevance of oscillations to questions of segmentation and binding. All this is possible because we work with systems which allow for periodic and chaotic motion between centers of attraction. References: D. Horn and M. Usher: --------------------- Neural Networks with Dynamical Thresholds, Phys. Rev. A40 (1989) 1036-1044. Motion in the Space of Memory Patterns, Proceedings of the 1989 Int. Joint Conf. on Neural Networks, I-61-69. Excitatory-Inhibitory Networks with Dynamical Thresholds, Int. Journal of Neural Systems 1 (1990) 249-257. Parallel Activation of Memories in an Oscillatory Neural Network Neural Computation 3 (1991) 31 - 43. Segmentation and Binding in an Oscillatory Neural Network submitted to IJCNN-91-Seattle. O. Hendin, D. Horn and M. Usher: -------------------------------- Chaotic behavior of a neural network with dynamical thresholds, Int. J. of Neural Systems, to be published in the next issue. -- David Horn From p-mehra at uiuc.edu Mon Feb 4 13:58:19 1991 From: p-mehra at uiuc.edu (Pankaj Mehra) Date: Mon, 04 Feb 91 12:58:19 CST Subject: Summary of responses: usefulness of chaotic dynamics Message-ID: <9102041858.AA00974@hobbes> **** PLEASE DO NOT FORWARD TO OTHER LISTS/BULLETIN BOARDS **** A few notes: 1. If replying to my message did not work, try one of the e-mail addresses at the end of my original message. 2. The comment: ``Chaos is an antithesis of generalization, a defining trait of connectionist models,'' was mine, not Hopfield's. 3. All the responses so far seem to suggest that chaos is useful in an evolving system. We can have a more focussed discussion if we can answer: 3a. Is chaos a precise quantitative way of stating one's ignorance of the dynamics of the process being modeled/controlled? 3b. What methods are available for implementing controlled chaos? 3c. How can chaos and learning be integrated in neural networks? Of course, discussion on cognitive/engineering significance of chaos is still welcome. --------- RESPONSES RECEIVED Paraphrased portions are enclosed in {}. ********************************************************************** From: Richard Rohwer I think that it is still an open question what sort of dynamics is cognitively useful. I can see the sense in Hopfield's intuition that convergent dynamics is good for generalization, but this doesn't really rule out chaotic dynamics, because although trajectories don't converge to fixed points, they do converge to attractors. Even if the attractors are chaotic, they lie "arbitrarily close" (in an epsilon- delta sense) to specific manifolds in state space which stay put. So there is a contracting mapping between the basin of attraction and the attractor. Anyway, I don't accept that generalization is the only phenomenon in cognition. I find it at least plausible that thought processes do come to critical junctures at which small perturbations can make large differences in how the thoughts evolve. { Pointers to relevant papers: Steve Renals and Richard Rohwer, "A study of network dynamics", J. Statistical Physics, vol. 58, pp.825-847, 1990. Stephen J. Renals, "Speech and Neural Network Dynamics", Ph.D. thesis, Edinburgh University, 1990. Steve Renals, "Chaos in neural networks" in Neural Networks (Almeida and Wellekens, eds.), Lecture Notes in Computer Science 412, Springer -Verlag, Berlin, pp. 90-99, 1990. } ********************************************************************** {some portions deleted} From: Jordan B Pollack There is a lot more to chaos than Sensitivity to initial conditions; there are self-organizing dynamics and computational properties beyond simple limit-point systems. Chaos is almost unavoidable in NN, and has to be suppressed with artifacts like symettric weights and synchronous recurrence. If it is so prevalent in nature, it must be adaptive! (If your heart converges, you die; if your brain converges, you die.) Hopfield is accurate in that neural networks which converge might be as commercially useful as logic gates, but they won't address the question of how high-dimensional dynamical systems self-organize into complex algorithmic structures. My research slogan might be "chaos in brain -> fractals in mind", and believe that more complex dynamics than convergence are quite necessary to getting non-trivial neural cognitive models. Had one paper in NIPS 1 outlining some research proposals, and 2 forthcoming in NIPS3. There are groups in Tel-Aviv, Warsaw, and Edinburgh, at least, working on complex neuro-dynamics. ********************************************************************** From: andreas%psych at Forsythe.Stanford.EDU (Andreas Weigend) {Pointers to related publications: Andreas S. Weigend, Bernardo A. Huberman, and David E. Rumelhart, "Predicting the future: a connectionist approach", International Journal of Neural Systems, vol. 1, pp. 193-209, 1990. Andreas S. Weigend, Bernardo A. Huberman, and David E. Rumelhart, "Back-propagation, weight-elimination and time series prediction" in Proc. 1990 Connectionist Models Summer School, Morgan Kaufmann, pp. 105-116, 1990. Andreas S. Weigend, Bernardo A. Huberman, and David E. Rumelhart, 1990 Lectures in Complex Systems, ?? (eds. Nadel and Stein), Addison-Wesley, 1991. Andreas S. Weigend, Bernardo A. Huberman, and David E. Rumelhart, "Predicting Sunspots and Currency Rates with Connectionist Networks", in Proc. NATO Workshop on Nonlinear Modeling and Forecasting (Santa Fe, Sept. 1990). *********************************************************************** From: David Horn I would like to point out the importance of dynamics which are convergent on a short time scale and divergent on a long time scale. We have worked on neural networks which display such behavior. In particular we can model a system which converges to a set of fixed points on a short time scale (thus performing some "useful" computation), and is "free" to move between them on a longer time scale. This kind of freedom stems from the unpredictability of chaotic systems. {deleted; message directly mailed to Connectionists} *********************************************************************** END OF RESPONSES From aboulanger at BBN.COM Mon Feb 4 20:38:55 1991 From: aboulanger at BBN.COM (Albert G. Boulanger) Date: Mon, 4 Feb 91 20:38:55 EST Subject: Summary of responses: usefulness of chaotic dynamics In-Reply-To: Pankaj Mehra's message of Mon, 04 Feb 91 12:58:19 CST <9102041858.AA00974@hobbes> Message-ID: While the following reference is not directly tied to NNets, it is tied to the the broader program of making computational use of chaos: "Chaotic Optimization and the Construction of Fractals: Solution of an Inverse Problem", Giorgio Mantica & Alan Sloan, Complex Systems 3(1989) 37-62. The agenda here is to make use of the ergodic properties of a dynamical system driven to be chaotic. (Briefly, an ergodic system is one that will pass through every possible dynamical state compatible with its energy.) This corresponds to a high temperature setting in simulated annealing. (A key to the way annealing works is that it, too, is ergodic.) Then they drive the parameter down, and it becomes controlled by repulsive (objective function: worse) and attractive (objective function: better) Coulomb charges. These charges are placed at the successive sites visited. (They also window the number of charges. The repulsive charge makes this system have tabu-search like properties.) Thus, they endow the system with memory at the lower setting of the dynamical parameter. They claim the memory allows more efficient convergence of the algorithm than annealing. Here are some short references on chaos and ergodic theory: "Modern Ergodic Theory" Joel Lebowitz, & Oliver Penrose Physics Today, Feb, 1973, 23-29 "Chaos, Entropy, and, the Arrow of Time" Peter Coveney New Scientist, 29 Sept, 1990, 49-52 (nontechnical) "The Second Law of Thermodynamics: Entropy, Irreversibility, and Dynamics", Peter Coveney, Nature, Vol 333, 2 June 1988, 409-415. (technical) "Strange Attractors, Chaotic Behavior, and Information Flow" Robert Shaw, Z. Naturforsch, 86a(1981), 80-112 "Ergodic Theory, Randomness, and 'Chaos'" D.S. Ornstein Science, Vol 243, 13 Jan 1989, 182- The papers by Coveney and the one by Shaw get into another possible use of chaos involving many-body "nonlinear systems, far from equilibrium". Because of the sensitivity of chaotic systems to external couplings, one can get such systems to act as information (or noise) amplifiers. Shaw puts it as getting information to flow from the microscale to the macroscale. Such self-organizing many-body systems can be used in a generate-and-test architecture as the pattern generators. In neural networks, competitive dynamics can give rise to such behavior. Eric Mjolsness worked on a fingerprint "hallucinator" that worked like this (as I remember). Optical NNets using 4-wave mixing with photorefractive crystals have this kind of dynamics too. Seeking structure in chaos, Albert Boulanger aboulanger at bbn.com From B344DSL at UTARLG.UTA.EDU Tue Feb 5 00:25:00 1991 From: B344DSL at UTARLG.UTA.EDU (B344DSL@UTARLG.UTA.EDU) Date: Mon, 4 Feb 91 23:25 CST Subject: Chaos Message-ID: From: IN%"mehra%aquinas at uxc.cso.uiuc.edu" "Pankaj Mehra" 2-FEB-1991 17:29:15.49 To: Connectionists at CS.CMU.EDU CC: p-mehra at uiuc.edu Subj: Usefulness of chaotic dynamics From mehra%aquinas at uxc.cso.uiuc.edu Fri Feb 1 18:04:52 1991 From: mehra%aquinas at uxc.cso.uiuc.edu (Pankaj Mehra) Date: Fri, 1 Feb 91 17:04:52 CST Subject: Usefulness of chaotic dynamics Message-ID: <9102012304.AA00422@hobbes> Fellow Connectionists: It was with fascination that I read Gleick's book on chaos recently. My initial reaction was that the thoery of chaotic dynamics of nonlinear systems represents a significant advance in science by allowing us to model and understand complex phenomena, such as turbulence. Yesterday, John Hopfield was in Champaign, talking about ``Neural Computation'' at the centennial celebration of Physics department. During his talk, he made a distinction between useful and ``unlikely to be useful'' dynamics of neural networks with feedback. Essentially, dynamics which show convergent behavior are considered useful, and those that show chaotic or divergent behavior, not useful. The crux of such an argument lies in the fact that ``similar but unresolvable initial conditions lead to large divergences in trajectories'' [1] for a chaotic system. This is an antithesis of generalization, which is considered a defining trait of connectionist models. However, a recent paper by Frison [1] shows how the behavior of chaotic systems may be predicted using neural networks. The applications suggested in this paper include weather forecasting and securities analysis. In my opinion, there is an emerging trend to move away from convergent dynamics. Section 4 of Hirsch's paper [2] presents arguments for/against using chaotic dynamics. In a limited sense, the work of Michail Zak [3] shows how breaking some traditional assumptions of convergent behavior can help solve the problems of spurious and local minima. I would like to hear more about the pros and cons of this issue. - Pankaj Mehra {p-mehra at uiuc.edu, mehra at cs.uiuc.edu, mehra at aquinas.csl.uiuc.edu} [1] Frison, T. W., "Predicting Nonlinear and Chaotic Systems Behavior Using Neural Networks," Jrnl. Neural Network Computing, pp. 31-39, Fall 1990. [2] Hirsch, M. W., "Convergent Activation Dynamics is Continuous Time Networks," Neural Networks, vol. 2, pp. 331-349, Pergamon Press, 1989. [3] Zak, M., "Creative Dynamics Approach to Neural Intelligence," Biological Cybernetics, vol. 64, pp. 15-23, Springer-Verlag, 1990. From koch%CITIAGO.BITNET at VMA.CC.CMU.EDU Tue Feb 5 02:00:45 1991 From: koch%CITIAGO.BITNET at VMA.CC.CMU.EDU (Christof Koch) Date: Mon, 4 Feb 91 23:00:45 PST Subject: Memo on "Awareness" Message-ID: <910204230039.2040195e@Iago.Caltech.Edu> For those of you trying to get the Crick and Koch memo on "Towards a neurobiological theory of consciousness". Once you grabed and uncompressed the file, you have to print the uncompressed version of course, i.e. lpr koch.awareness1.ps (and not, as I had indicated, lpr koch.awareness1.ps.Z) Sorry about that, Christof From h1705erd at ella.hu Tue Feb 5 12:00:00 1991 From: h1705erd at ella.hu (Erdi Peter) Date: Tue, 05 Feb 91 12:00:00 Subject: usefulness of chaotic dynamics Message-ID: <9102051104.AA10986@sztaki.hu> Technical report available: SELF-ORGANIZATION IN THE NERVOUS SYSTEM: NETWORK STRUCTURE AND STABILITY (P. Erdi, Central Reseacrh Inst. Physics, Hung. Acad. Sci., H-1525 Budapest, P.O. Box 49, Hungary) to appear in: Martyhematical Approaches to Brain Functioning Diagnostics; Manchaster Univ. Press. (23 pages) 1. Self-organization and neurodynamivcs 1.1. General remarks 1.2. Real neural networks: some remarks 2. Neural network models 2.1. Conceptual and mathematical skeleton 2.2. Neurodynamic problems 3. Neural architectures and dynamic behaviour 3.1. On the "structure - function" problem 3.2. The connectivity - stability dilemma 3.3. Qualitative stability and instability 4. Regular and periodic behaviour in neural networks 4.1. Some remarks on the experimental background of the oscillatory behaviour 4.2. Network architecturw of possyble rhythm generators 5. Network structure and chaos: some designing principles (59 references) From p-mehra at uiuc.edu Tue Feb 5 22:03:52 1991 From: p-mehra at uiuc.edu (Pankaj Mehra) Date: Tue, 05 Feb 91 21:03:52 CST Subject: More responses: Chaos Message-ID: <9102060303.AA01523@hobbes> First, a small note: In his response (sent directly to the list), Albert Boulanger makes a connection between annealing and ergodicity. He then describes an optimization procedure based on slowly constraining the dynamics of a search process. He claims that the algorithm has better convergence than annealing. RESPONSES RECEIVED ********** From: B344DSL at UTARLG.UTA.EDU (???) On the usefulness of chaos in neural models (particularly biologically- related ones), I recommend the following two articles: 1. Skarda, C. & Freeman, W. J. (1987). How brains make chaos to make sense of the world. Behavioral and Brain Sciences 10: 161-195. (This is about the olfactory system, with some speculative generalizations to other mamm- alian sensory systems.) 2. Mpitsos, G. J., Burton, R. M., Creech, H. C., & Soinila, S. O. (1988). Evidence for chaos in spike trains of neurons that generate rhythmic motor patterns. Brain Research Bulletin 21: 529-538. (This is about motor systems in mollusks.) Both of these authors see chaos as a useful biological device for promoting behavioral variability. In the Skarda-Freeman article, there are several commentaries by other people that deal with this issue, including commentaries by Grossberg and by myself that suggest that chaos is not the only possible method for achieving this goal of variability. (There are also responses to the commentaries by the author.) Both Freeman and Mpitsos have pursued this theme in several other articles. ********** From: pluto at cs.UCSD.EDU (Mark Plutowski) With regards to: 3a. Is chaos a precise quantitative way of stating one's ignorance of the dynamics of the process being modeled/controlled? Formally, probability theory (in particular, the theory of stochastic processes) is a precise quantitative way of stating one's ignorance of the dynamics of a process. (Say, by approximation of the empirical data by a sequence of processes which converge to the data probabilistically.) According to my knowledge, a chaotic attractor gives a concise representation which can describe processes which are complicated enough to appear (behaviorally) indistinguishable from random processes. In other words, the benefit of a chaotic model is in its richness of descriptive capability, and its concise formulation. Perhaps chaotic attractors could be employed to provide deterministic models which give a plausible account of the uncertainties inherent in the probabilistic models of our empirical data. Whereas the probabilistic models can predict population behavior, or, the asymptotic behavior of a single element of the population over time, an equally accurate chaotic model may as well be able to predict the behavior of a single element of the population over a finite range of time. I anticipate that the real challenge would be in inferring a chaotic attractor model which can give a testable hypothesis to tell us something that we cannot ascertain by population statistics or asymptotic arguments. Without this latter condition, the chaotic model may be just an unnecessary generalization - why use a chaotic model if more well-understood formalisms are sufficient? However, if it is used to provided a model which is formally convenient, then, the benefit is not necessarily due to descriptive capability, and so, the resulting models need not necessarily give a mechanistic model of the empirical data (and hence, potentially reducing the predictive capability of the models.) Of course, this is all IMHO, due to my expectation that noise inherent in our best experimental data will tend to cause researchers to first fit the data with stochastic processes and classical dynamical systems, and then try to refine these models by using deterministic chaotic models. Perhaps someone better informed on the use of these models could comment on whether this perspective holds water? It may be the case that I am missing out on capabilities of chaotic models which are not apparent in the popular literature. ********* END OF RESPONSES From pollack at cis.ohio-state.edu Tue Feb 5 23:25:42 1991 From: pollack at cis.ohio-state.edu (Jordan B Pollack) Date: Tue, 5 Feb 91 23:25:42 -0500 Subject: Neuroprose Message-ID: <9102060425.AA00927@dendrite.cis.ohio-state.edu> **Do not forward to other lists** Neuroprose seems to be working alright in general, although all recent announcements have just appeared in NEURON-DIGEST, and might have a nasty effect on cheop's load. When you announce a paper, you should consider whether (A) you want it automatically forwarded to other groups, like NEURON-DIGEST, (which gets posted to comp.ai.neural-networks) and if you want to provide (B) FREE or PREPAID alternative hard copies. One author reported that when they allowed combination AB, the rattling around of their "free paper offer" on the worldwide data net generated over 400 hardcopy requests! Jordan Pollack Assistant Professor CIS Dept/OSU Laboratory for AI Research 2036 Neil Ave Email: pollack at cis.ohio-state.edu Columbus, OH 43210 Fax/Phone: (614) 292-4890 From IP%IRMKANT.BITNET at VMA.CC.CMU.EDU Wed Feb 6 12:24:05 1991 From: IP%IRMKANT.BITNET at VMA.CC.CMU.EDU (stefano nolfi) Date: Wed, 06 Feb 91 13:24:05 EDT Subject: Technical report available Message-ID: The following technical report is now available. The paper has been submitted to ICGA-91. Send request to stiva at irmkant.Bitnet e-mail comments and related references are appreciated AUTO-TEACHING: NETWORKS THAT DEVELOP THEIR OWN TEACHING INPUT Stefano Nolfi Domenico Parisi Institute of Psychology CNR - Rome E-mail: stiva at irmkant.Bitnet ABSTRACT Back-propagation learning (Rumelhart, Hinton and Williams, 1986) is a useful research tool but it has a number of undesiderable features such as having the experimenter decide from outside what should be learned. We describe a number of simulations of neural networks that internally generate their own teaching input. The networks generate the teaching input by trasforming the network input through connection weights that are evolved using a form of genetic algorithm. What results is an innate (evolved) capacity not to behave efficiently in an environment but to learn to behave efficiently. The analysis of what these networks evolve to learn shows some interesting results. references Rumelhart, D.E., Hinton G.E., and Williams, R.J. (1986). Learning internal representations by error propagation. In D.E. Rumelhart, and J.L. McClelland, (eds.), Parallel Distributed Processing. Vol.1: Foundations. Cambridge, Mass.: MIT Press. From noordewi at cs.rutgers.edu Wed Feb 6 18:45:45 1991 From: noordewi at cs.rutgers.edu (noordewi@cs.rutgers.edu) Date: Wed, 6 Feb 91 18:45:45 EST Subject: Rutgers Neural Network Seminar Message-ID: <9102062345.AA13716@porthos.rutgers.edu> RUTGERS UNIVERSITY Dept. of Computer Science/Dept. of Mathematics Neural Networks Colloquium Series --- Spring 1991 Stephen Judd Siemens The Complexity of Learning in Families of Neural Networks Abstract What exactly does it mean for Neural Networks to `learn'? We formalize a notion of learning that characterizes the simple training of feed-forward Neural Networks. The formulation is intended to model the objectives of the current mode of connectionist research in which one searches for powerful and efficient `learning rules' to stick in the `neurons'. By showing the learning problem to be NP-complete, we demonstrate that in general the set of things that a network can learn to do is smaller than the set of things it can do. No reasonable learning rule exists to train all families of networks. Naturally this provokes questions about easier cases, and we explore how the problem does or does not get easier as the neurons are made more powerful, or as various constraints are placed on the architecture of the network. We study one particular family of networks called `shallow architectures' which are defined in such a way as to bound their depth but let them grow very wide -- a description inspired by certain neuro-anatomical structures. The results seem to be robust in the face of all choices for what the neurons are able to compute individually. February 13, 1991 Busch Campus --- 4:30 p.m., room 217 SEC host: Mick Noordewier (201/932-3698) finger noordewi at cs.rutgers.edu for further schedule information From worth at park.bu.edu Thu Feb 7 15:03:28 1991 From: worth at park.bu.edu (Andrew J. Worth) Date: Thu, 7 Feb 91 15:03:28 -0500 Subject: Survey Results: Connectionism vs AI Message-ID: <9102072003.AA07670@park.bu.edu> On January 6th, I suggested a survey as part of the "Connectionism vs AI" discussion. Though the replies were not overwhelming, some of the main participants in the debate did respond. The text of the responses has (or will soon) be placed in the connectionist archive. Instructions to obtain this via anonymous ftp are given below. If you do not have access to the archive, I can send you the file by email directly. The file is plain text and has 455 lines, 3273 words, and 22732 characters. % ftp B.GP.CS.CMU.EDU (or ftp 128.2.242.8) Name: anonymous Password: (your username) ftp> cd /usr/connect/connectionists/archives ftp> get connect-vs-ai ftp> quit Thanks to all those that responded! Andy. ----------------------------------------------------------------------- Andrew J. Worth worth at park.bu.edu (617) 353-6741 Cognitive & Neural Systems Boston University Center for Adaptive Systems 111 Cummington St. Room 244 (617) 353-7857 Boston, MA 02215 USA From leon%FRLRI61.BITNET at CUNYVM.CUNY.EDU Thu Feb 7 06:02:44 1991 From: leon%FRLRI61.BITNET at CUNYVM.CUNY.EDU (leon%FRLRI61.BITNET@CUNYVM.CUNY.EDU) Date: Thu, 07 Feb 91 12:02:44 +0100 Subject: preprint available Message-ID: <9102071102.AA02071@sun3d.lri.fr> The following paper has been placed in the neuroprose archives at Ohio State University: A Framework for the Cooperation of Learning Algorithms Leon Bottou & Patrick Gallinari Laboratoire de Recherche en Informatique Universite de Paris XI 91405 Orsay Cedex - France Abstract We introduce a framework for training architectures composed of several modules. This framework, which uses a statistical formulation of learning systems, provides a single formalism for describing many classical connectionist algorithms as well as complex systems where several algorithms interact. It allows to design hybrid systems which combine the advantages of connectionist algorithms as well as other learning algorithms. This paper will appear in the NIPS-90 proceedings. To retrieve it by anonymous ftp, do the following: unix> ftp cheops.cis.ohio-state.edu # (or ftp 128.146.8.62) Name (cheops.cis.ohio-state.edu:): anonymous Password (cheops.cis.ohio-state.edu:anonymous): ftp> cd pub/neuroprose ftp> binary ftp> get bottou.cooperation.ps.Z ftp> quit unix> unix> zcat bottou.cooperation.ps.Z | lpr -P From shavlik at cs.wisc.edu Fri Feb 8 14:56:12 1991 From: shavlik at cs.wisc.edu (Jude Shavlik) Date: Fri, 8 Feb 91 13:56:12 -0600 Subject: Faculty Position Message-ID: <9102081956.AA00311@steves.cs.wisc.edu> Due to a state-wide hiring freeze, our ad to CACM announcing an open faculty position was delayed. One of our target areas is artificial intelligence, so if you are looking for a faculty position this year, you may wish to send your application to me. Please include your vita, the names of at least three references, and a couple of sample publications. Sincerely, Jude Shavlik shavlik at cs.wisc.edu Assistant Professor Computer Sciences Dept University of Wisconsin 1210 W. Dayton Street Madison, WI 53706 (608) 262-1204 (608) 262-9777 (fax) From INAM%MUSICB.MCGILL.CA at BITNET.CC.CMU.EDU Sun Feb 10 16:27:24 1991 From: INAM%MUSICB.MCGILL.CA at BITNET.CC.CMU.EDU (Tony Marley) Date: Sun, 10 Feb 91 16:27:24 EST Subject: Symposium on Models of Human Identification and Categorization Message-ID: <10FEB91.17773400.0250.MUSIC@MUSICB.MCGILL.CA> Department of Psychology McGill University 1205 Avenue Dr Penfield Montreal PQ H3A 1B1 Canada February 10, 1991 MODELS OF HUMAN IDENTIFICATION AND CATEGORIZATION Symposium at the Twenty-Fourth Annual Mathematical Psychology Meeting A. A. J. Marley, Symposium Organizer The Society for Mathematical Psychology Sponsored by Indiana University, Bloomington, Indiana August 10-13, 1991 At each of its Annual Meetings, the Society for Mathematical Psychology has one or more symposia on topics of current interest. I believe that this is an opportune time to have the proposed session since much exciting work is being done, plus Robert Nosofsky is an organizer of the conference at Indiana, and he has recently developed many empirical and theoretical ideas that have encouraged others to (re)enter this area. Each presenter in the symposium will have 20 to 30 minutes available to them, plus there will be time scheduled for general discussion. This meeting is a good place to present your theoretical ideas in detail, although simulation and empirical results are naturally also welcome. Remember, the Cognitive Science Society is meeting at the University of Chicago August 7- 10, i.e. just prior to this meeting; thus, by splitting your material in an appropriate manner between the two meetings, you will have an excellent forum within a period of a week to present your work in detail. If you are interested in participating in this symposium, please contact me with a TITLE and ABSTRACT. I would also be interested in suggestions of other participants with (if possible) an email address for them. To give you a clearer idea of the kind of work that I consider of direct relevance, I mention a few researchers and some recent papers. This list is meant to be illustrative, so please don't be annoyed if I have omitted your favourite work (including your own). REFERENCES AHA, D. W., & MCNULTY, D. (1990). Learning attribute relevance in context in instance-based learning algorithms. In Proceedings of the Twelfth Annual Conference of the Cognitive Science Society. Hillsdale, NJ: Erlbaum. ASHBY, F. G. (Ed.). (in press). Probabilistic Multidimensional Models of Perception and Cognition. Hillsdale, NJ: Erlbaum. ESTES, W. K., CAMPBELL, J. A., HATSPOULOS, N., & HURWITZ, J. B. (1989). Base-rate effects in category learning: A comparison of parallel network and memory storage-retrieval models. Journal of Experimental Psychology: Learning, Memory, and Cognition, 15, 556-571. GLUCK, M. A., & BOWER, G. H. (1989). Evaluating an adaptive network model of human learning. Journal of Memory and Language, 27, 166-195. HURWITZ, J. B. (1990). A hidden-pattern network model of category learning. Ph. D. Thesis, Department of Psychology, Harvard. KRUSCKE, J. K. (1990). ALCOVE: A connectionist model of category learning. Research Report 19, Cognitive Science, Indiana University. LACOUTURE, Y., & MARLEY, A. A. J. (1990). A connectionist model of choice and reaction time in absolute identification. Manuscript, Universite Laval & McGill University. NOSOFSKY, R. M., & GLUCK, M. A. (1989). Adaptive networks, exemplars, and classification rule learning. Thirtieth Annual Meeting of the Psychonomic Society, Atlanta, Georgia. RATCLIFF, R. (1990). Connectionist models of recognition memory: Constraints imposed by learning and forgetting functions. Psychological Review, 97, 285-308. SHEPARD, R. N. (1989). A law of generalization and connectionist learning. Plenary Session, Eleventh Annual Conference of the Cognitive Science Society, University of Michigan, Ann Arbor. Regards Tony A. A. J. Marley Professor Director of the McGill Cognitive Science Centre From MURRE%rulfsw.LeidenUniv.nl at BITNET.CC.CMU.EDU Tue Feb 12 16:38:00 1991 From: MURRE%rulfsw.LeidenUniv.nl at BITNET.CC.CMU.EDU (MURRE%rulfsw.LeidenUniv.nl@BITNET.CC.CMU.EDU) Date: Tue, 12 Feb 91 16:38 MET Subject: Neurosimulators Message-ID: Dear connectionist researchers, We are compiling a list of neurosimulators for inclusion in a review paper. The table below presents the 45 simulators that we have been able to track down so far. We have not been able to find out all the details. We would, therefore, appreciate it when users or developers could fill us in on the white spots in the list (or point out any mistakes). Also, if anyone knows of other simulators that should be included, please, drop us a note. We would especially welcome any (pointers to) papers describing neurosimulators. This would enable us to refine and extend the list of features. Thanks! Jaap Murre Steven Kleynenberg E-mail: MURRE at HLERUL55.Bitnet Surface mail: Jacob M.J. Murre Unit of Experimental and Theoretical Psychology Leiden University P.O. Box 9555 2300 RB Leiden The Netherlands To save precious bytes, we have configured the table below in a 132 column format. It may be easier to send the file to a line printer, then to read it behind the terminal. (On a VAX use: set term /width=132.) TABLE: NEUROSIMULATORS Name Manufacturer Language Models Hardware Referenc e Price ($) -------------------------------------------------------------------------------- ----------------- ADAPTICS Adaptic [AZEMA] ANNE Oregon Grad. Cent. HLL Intel hypercube [AZEMA] ANSE TRW TRW [AZEMA] ANSIM SAIC several IBM [COHEN] 495.00 ANSKIT SAIC several many [BARGA][ BYTE] ANSPEC SIAC HLL many IBM,MAC,SUN,VAX 995.00 AWARENESS Neural Systems IBM [BYTE] 275.00 AXON HNC HLL many HNC neurocomp. [AZEMA][ BYTE] 1950.00 BOSS [REGGIA] BRAIN SIMULATOR Abbot,Foster, & Hauser IBM 99.00 BRAINMAKER Cal.Scient.Software bp IBM [BYTE] 195.00 CABLE VAX [MILLER] CASENET Prolog [DOBBINS ] COGNITRON Cognitive Software Lisp many MAC,IBM [ZEITV][ BYTE] 600.00 CONE IBM Palo Alto HLL IBM [AZEMA] CONNECTIONS hopf IBM [BYTE] 87.00 CORTEX [REGGIA] DESIRE/NEUNET IBM [KORN] EXPLORENET 3000 HNC HLL many IBM,VAX [BYTE][C OHEN] GENESIS Neural Systems IBM [MILLER] 1095.00 GRADSIM Univ. of Penns. C several GRIFFIN Texas Instruments [AZEMA] HYPERBRAIN Neurix MAC [BYTE] 995.00 MACBRAIN Neurix many MAC [BYTE] 995.00 MACTIVATION Univ. of Colorado? METANET Leiden University HLL many IBM,VAX [MURRE] MIRROR 2 HLL several [REGGIA] N-NET AIWare C bp IBM,VAX [BYTE] 695.00 N1000 Nestor IBM,SUN [BYTE] 19000.00 N500 Nestor IBM [BYTE] NEMOSYS IBM RS/6000 [MILLER] NESTOR DEV. SYSTEM Nestor IBM,MAC 9950.00 NET [REGGIA] NETSET 2 HNC many IBM,SUN,VAX 19500.00 NETWURKZ Dair Computer IBM [BYTE] 79.95 NEURALWORKS NeuralWare HLL many IBM,MAC,SUN [BYTE][C OHEN] 1495.00 NEUROCLUSTERS VAX [AZMY] NEURON [MILLER] NEUROSHELL Ward Systems Group bp IBM [BYTE] 195.00 NEUROSOFT HNC NEUROSYM NeuroSym many IBM 179.00 NEURUN Dare Research bp IBM NN3 GMD Bonn HLL many [LINDEN] NNSIM [NIJHUIS ] OWL Olmsted & Watkins many IBM,MAC,SUN,VAX [BYTE] 1495.00 P3 UCSD HLL many Symbolics [ZIPSER] PABLO [REGGIA] PLATO/ARISTOTLE NeuralTech [AZEMA] PLEXI Symbolics Lisp bp,hopf Symbolics PREENS Nijmegen University HLL many SUN PYGMALION Esprit C many SUN,VAX [AZEMA] RCS Rochester University C many SUN [AZEMA] SFINX UCLA HLL [AZEMA] Explanation of abbreviations and terms: Languages: HLL = High Level Language (i.e., network definition language; if spec ific programming languages are mentioned networks can be defined using high level functions in thes e languages) Models: several = a fixed number of models is (and will be) supported many = the systems can be (or will be) extended with new models bp = backpropagation hopf = hopfield (if specific models are mentioned these are th e only ones su pported) References: see list below (We welcome any additional references.) [AZEMA] Azema-Barac, M., M. Heweston, M. Recce, J. Taylor, P. Treleaven, M. Vellasco (1990). Pygmalion, neural network progamming environment. [BARGA] Barga, R.S., R.B. Melton (1990). Framework for distributed artificial neural system simulation. Proceedings of the IJCNN-90-Washington DC, 2, 94-97. [BYTE] Byte (product listing) (1989). BYTE, 14(8), 244-245. [COHEN] Cohen, H. (1989). How useful are current neural network software tools? Neural Network Review, 3, 102-113. [DOBBINS] Dobbins, R.W., R.C. Eberhart (1990). Casenet, computer aided neural network generation tool. Proceedings of the IJCNN-90-Washington DC, 2, 122-125. [KORN] Korn, G.A. (1989). A new environment for interactive neural network experiments. Neural Networks, 2, 229-237. [LINDEN] Linden, A., Ch. Tietz (in prep.). Research and development software environment for modular adaptive systems. Technical Report NN3-1, GMD Birlinghoven, Sankt Augustin, Germany. [MILLER] Miller, J.P. (1990). Computer modelling at the single-neuron level. Nature, 347, 783-784. [MURRE] Murre, J.M.J., S.E. Kleynenberg (submitted). Extending the MetaNet Network Environment: process control and machine independence. [NIJHUIS] Nijhuis, J., L. Spaanenburg, F. Warkowski (1989). Structure and application of NNSIM: a general purpose neural network simulator. Microprocessing and Microprogramming, 27, 189-194. [REGGIA] Reggia, J.A., C.L. D'Autrechy, G.C. Sutton III, S.M. Goodall (1988). A general-purpose simulation environment of developing connectionist models. Simulation, 51, 5-19. [VIBERT] Vibert, J.F., N. Azmy (1990). Neuro_Clusters: A biological plausible neural networks simulator tool. [ZEITV] Zeitvogel, R.K. (1989). Cognitive Software's Cognitron 1.2 (review). Neural Network Review, 3, 11-16. [ZIPSER] Zipser, D., D.E. Rabin (1986). P3: a parallel network simulation system. In: D.E. Rumelhart, J.L. McClelland (1986). Parallel distributed processing. Volume 1. Cambridge MA: MIT Press. From Connectionists-Request at CS.CMU.EDU Wed Feb 13 13:35:19 1991 From: Connectionists-Request at CS.CMU.EDU (Connectionists-Request@CS.CMU.EDU) Date: Wed, 13 Feb 91 13:35:19 EST Subject: Bi-monthly Reminder Message-ID: <20706.666470119@B.GP.CS.CMU.EDU> This is an automatically posted bi-monthly reminder about how the CONNECTIONISTS list works and how to access various online resources. CONNECTIONISTS is not an edited forum like the Neuron Digest, or a free-for-all newsgroup like comp.ai.neural-nets. It's somewhere in between, relying on the self-restraint of its subscribers. Membership in CONNECTIONISTS is restricted to persons actively involved in neural net research. The following posting guidelines are designed to reduce the amount of irrelevant messages sent to the list. Before you post, please remember that this list is distributed to over a thousand busy people who don't want their time wasted on trivia. Also, many subscribers pay cash for each kbyte; they shouldn't be forced to pay for junk mail. Happy hacking. -- Dave Touretzky & Scott Crowder --------------------------------------------------------------------- What to post to CONNECTIONISTS ------------------------------ - The list is primarily intended to support the discussion of technical issues relating to neural computation. - We encourage people to post the abstracts of their latest papers and tech reports. - Conferences and workshops may be announced on this list AT MOST twice: once to send out a call for papers, and once to remind non-authors about the registration deadline. A flood of repetitive announcements about the same conference is not welcome here. - Requests for ADDITIONAL references. This has been a particularly sensitive subject lately. Please try to (a) demonstrate that you have already pursued the quick, obvious routes to finding the information you desire, and (b) give people something back in return for bothering them. The easiest way to do both these things is to FIRST do the library work to find the basic references, then POST these as part of your query. Here's an example: WRONG WAY: "Can someone please mail me all references to cascade correlation?" RIGHT WAY: "I'm looking for references to work on cascade correlation. I've already read Fahlman's paper in NIPS 2, his NIPS 3 abstract, and found the code in the nn-bench archive. Is anyone aware of additional work with this algorithm? I'll summarize and post results to the list." - Announcements of job openings related to neural computation. - Short reviews of new text books related to neural computation. To send mail to everyone on the list, address it to Connectionists at CS.CMU.EDU ------------------------------------------------------------------- What NOT to post to CONNECTIONISTS: ----------------------------------- - Requests for addition to the list, change of address and other administrative matters should be sent to: "Connectionists-Request at cs.cmu.edu" (note the exact spelling: many "connectionists", one "request"). If you mention our mailing list to someone who may apply to be added to it, please make sure they use the above and NOT "Connectionists at cs.cmu.edu". - Requests for e-mail addresses of people who are believed to subscribe to CONNECTIONISTS should be sent to postmaster at appropriate-site. If the site address is unknown, send your request to Connectionists-Request at cs.cmu.edu and we'll do our best to help. A phone call to the appropriate institution may sometimes be simpler and faster. - Note that in many mail programs a reply to a message is automatically "CC"-ed to all the addresses on the "To" and "CC" lines of the original message. If the mailer you use has this property, please make sure your personal response (request for a Tech Report etc.) is NOT broadcast over the net. - Do NOT tell a friend about Connectionists at cs.cmu.edu. Tell him or her only about Connectionists-Request at cs.cmu.edu. This will save your friend from public embarrassment if she/he tries to subscribe. - Limericks should not be posted here. ------------------------------------------------------------------------------- The CONNECTIONISTS Archive: --------------------------- All e-mail messages sent to "Connectionists at cs.cmu.edu" starting 27-Feb-88 are now available for public perusal. A separate file exists for each month. The files' names are: arch.yymm where yymm stand for the obvious thing. Thus the earliest available data are in the file: arch.8802 Files ending with .Z are compressed using the standard unix compress program. To browse through these files (as well as through other files, see below) you must FTP them to your local machine. ------------------------------------------------------------------------------- How to FTP Files from the CONNECTIONISTS Archive ------------------------------------------------ 1. Open an FTP connection to host B.GP.CS.CMU.EDU (Internet address 128.2.242.8). 2. Login as user anonymous with password your username. 3. 'cd' directly to one of the following directories: /usr/connect/connectionists/archives /usr/connect/connectionists/bibliographies 4. The archives and bibliographies directories are the ONLY ones you can access. You can't even find out whether any other directories exist. If you are using the 'cd' command you must cd DIRECTLY into one of these two directories. Access will be denied to any others, including their parent directory. 5. The archives subdirectory contains back issues of the mailing list. Some bibliographies are in the bibliographies subdirectory. Problems? - contact us at "Connectionists-Request at cs.cmu.edu". ------------------------------------------------------------------------------- How to FTP Files from the Neuroprose Archive -------------------------------------------- Anonymous FTP on cheops.cis.ohio-state.edu (128.146.8.62) pub/neuroprose directory This directory contains technical reports as a public service to the connectionist and neural network scientific community. Researchers may place electronic versions of their preprints or articles in this directory, announce availability, and other interested researchers can rapidly retrieve and print the postscripts. This saves copying, postage and handling, by having the interested reader supply the paper. To place a file, put it in the Inbox subdirectory, and send mail to pollack at cis.ohio-state.edu. Within a couple of days, I will move and protect it, and suggest a different name if necessary. Current naming convention is author.title.filetype[.Z] where title is enough to discriminate among the files of the same author. The filetype is usually "ps" for postscript, our desired universal printing format, but may be tex, which requires more local software than a spooler. Very large files (e.g. over 200k) must be squashed (with either a sigmoid function :) or the standard unix "compress" utility, which results in the .Z affix. To place or retrieve .Z files, make sure to issue the FTP command "BINARY" before transfering files. After retrieval, call the standard unix "uncompress" utility, which removes the .Z affix. An example of placing a file is attached as an appendix, and a shell script called Getps in the directory can perform the necessary retrival operations. For further questions contact: Jordan Pollack Email: pollack at cis.ohio-state.edu Here is an example of naming and placing a file: gvax> cp i-was-right.txt.ps rosenblatt.reborn.ps gvax> compress rosenblatt.reborn.ps gvax> ftp cheops.cis.ohio-state.edu Connected to cheops.cis.ohio-state.edu. 220 cheops.cis.ohio-state.edu FTP server ready. Name: anonymous 331 Guest login ok, send ident as password. Password:neuron 230 Guest login ok, access restrictions apply. ftp> binary 200 Type set to I. ftp> cd pub/neuroprose/Inbox 250 CWD command successful. ftp> put rosenblatt.reborn.ps.Z 200 PORT command successful. 150 Opening BINARY mode data connection for rosenblatt.reborn.ps.Z 226 Transfer complete. 100000 bytes sent in 3.14159 seconds ftp> quit 221 Goodbye. gvax> mail pollack at cis.ohio-state.edu Subject: file in Inbox. Jordan, I just placed the file rosenblatt.reborn.ps.Z in the Inbox. The INDEX sentence is "Boastful statements by the deceased leader of the neurocomputing field." Please let me know when it is ready to announce to Connectionists at cmu. BTW, I enjoyed reading your review of the new edition of Perceptrons! Frank ------------------------------------------------------------------------ How to FTP Files from the NN-Bench Collection --------------------------------------------- 1. Create an FTP connection from wherever you are to machine "pt.cs.cmu.edu" (128.2.254.155). 2. Log in as user "anonymous" with password your username. 3. Change remote directory to "/afs/cs/project/connect/bench". Any subdirectories of this one should also be accessible. Parent directories should not be. 4. At this point FTP should be able to get a listing of files in this directory and fetch the ones you want. Problems? - contact us at "nn-bench-request at cs.cmu.edu". From vincew at cse.ogi.edu Wed Feb 13 18:30:36 1991 From: vincew at cse.ogi.edu (Vince Weatherill) Date: Wed, 13 Feb 91 15:30:36 -0800 Subject: Speech Recognition & NNs preprints/reprints available Message-ID: <9102132330.AA01394@cse.ogi.edu> Reprints and preprints are now available for the following publications of the OGI Speech Group. Please respond directly to me by e-mail or surface mail. Don't forget to include your address with your request. Unless you indicate otherwise, I will send all 6 reports. Vince Weatherill Dept. of Computer Science and Engineering Oregon Graduate Institute 19600 NW von Neumann Drive Beaverton, OR 97006-1999 Barnard, E., Cole, R.A., Vea, M.P., and Alleva, F. "Pitch detection with a neural-net classifier," IEEE Transactions on Acoustics, Speech & Signal Processing, (February, 1991). Cole, R.A., M. Fanty, M. Gopalakrishnan, and R.D.T. Janssen, "Speaker-independent name retrieval from spellings using a database of 50,000 names," Proceedings of the IEEE Interna- tional Conference on Acoustics, Speech and Signal Process- ing, Toronto, Canada, May 14-17, (1991). Muthusamy, Y. K., R.A. Cole, and M. Gopalakrishnan, "A segment- based approach to automatic language identification," Proceedings of the 1991 IEEE International Conference on Acoustics, Speech and Signal Processing, Toronto, Canada, May 14-17, (1991). Fanty, M., R. A. Cole, and , "Spoken Letter Recognition," Proceedings of the Neural Information Processing Systems Conference, Denver, CO, (Nov. 1990). Janssen, R.D.T, M. Fanty, and R.A. Cole, "Speaker-independent phonetic classification in continuous English letters," Proceedings of the International Joint Conference on Neural Networks, Seattle, WA, Jul 8-12, (1991), submitted for publication. Fanty, M., R. A. Cole, and , "Speaker-independent English alpha- bet recognition: Experiments with the E-Set," Proceedings of the 1990 International Conference on Spoken Language Pro- cessing, Kobe, Japan, (Nov. 1990). **************************************************************** PITCH DETECTION WITH A NEURAL-NET CLASSIFIER Etienne Barnard, Ronald Cole, M. P. Vea and Fil Alleva ABSTRACT Pitch detection based on neural-net classifiers is investi- gated. To this end, the extent of generalization attainable with neural nets is first examined, and it is shown that a suitable choice of features is required to utilize this pro- perty. Specifically, invariant features should be used whenever possible. For pitch detection, two feature sets, one based on waveform samples and the other based on proper- ties of waveform peaks, are introduced. Experiments with neural classifiers demonstrate that the latter feature set --which has better invariance properties--performs more suc- cessfully. It is found that the best neural-net pitch tracker approaches the level of agreement of human labelers on the same data set, and performs competitively in comparison to a sophisticated feature-based tracker.An analysis of the errors committed by the neural net (relative to the hand labels used for training) reveals that they are mostly due to inconsistent hand labeling of ambigu- ous waveform peaks. ************************************************************* SPEAKER-INDEPENDENT NAME RETRIEVAL FROM SPELLINGS USING A DATABASE OF 50,000 NAMES Ronald Cole, Mark Fanty, Murali Gopalakrishnan, Rik Janssen ABSTRACT We describe a system that recognizes names spelled with pauses between letters using high quality speech. The sys- tem uses neural network classifiers to locate and classify letters, then searches a database of names to find the best match to the letter scores. The directory name retrieval system was evaluated on 1020 names provided by 34 speakers who were not used to train the system. Using a database of 50,000 names, 972, or 95.3%, were correctly identified as the first choice. Of the remaining 48 names, all but 10 were in the top 3 choices. Ninty nine percent of letters were correctly located, although speakers failed to pause completely about 10% of the time. Classification of indivi- dual spoken letters that were correctly located was 93%. ************************************************************* A SEGMENT-BASED APPROACH TO AUTOMATIC LANGUAGE IDENTIFICATION Yeshwant K. Muthusamy, Ronald A. Cole and Murali Gopalakrishnan ABSTRACT A segment-based approach to automatic language identifica- tion is based on the idea that the acoustic structure of languages can be estimated by segmenting speech into broad phonetic categories. Automatic language identification can then be achieved by computing features that describe the phonetic and prosodic characteristics of the language, and using these feature measurements to train a classifier to distinguish between languages. As a first step in this approach, we have built a multi-language, neural network- based segmentation and broad classification algorithm using seven broad phonetic categories. The algorithm was trained and tested on separate sets of speakers of American English, Japanese, Mandarin Chinese and Tamil. It currently performs with an accuracy of 82.3% on the utterances of the test set. ************************************************************* SPOKEN LETTER RECOGNITION Mark Fanty and Ron Cole ABSTRACT Through the use of neural network classifiers and careful feature selection, we have achieved high-accuracy speaker- independent spoken letter recognition. For isolated letters, a broad-category segmentation is performed Location of segment boundaries allows us to measure features at From rosen at CS.UCLA.EDU Thu Feb 14 14:11:46 1991 From: rosen at CS.UCLA.EDU (Bruce E Rosen) Date: Thu, 14 Feb 91 11:11:46 -0800 Subject: Adaptive Range Coding - Tech Report Available Message-ID: <9102141911.AA28582@lanai.cs.ucla.edu> REPORT AVAILABLE ON ADAPTIVE RANGE CODING At the request of a few people at NIPS, I placed in the connectionists archive the postscript version of my report describing adaptive range coding. Below are the abstract and instructions on ftp retrieval. I would very much welcome any discussion of this subject. If you want, send email to me and I can summarize later for the net. Thanks Bruce ----------------------------------------------------------------------------- Report DMI-90-4, UCLA Distributed Machine Intelligence Laboratory, January 1991 Adaptive Range Coding Abstract This paper examines a class of neuron based learning systems for dynamic control that rely on adaptive range coding of sensor inputs. Sensors are assumed to provide binary coded range vectors that coarsely describe the system state. These vectors are input to neuron-like processing elements. Output decisions generated by these "neurons" in turn affect the system state, subsequently producing new inputs. Reinforcement signals from the environment are received at various intervals and evaluated. The neural weights as well as the range boundaries determining the output decisions are then altered with the goal of maximizing future reinforcement from the environment. Preliminary experiments show the promise of adapting "neural receptive fields" when learning dynamical control. The observed performance with this method exceeds that of earlier approaches. ----------------------------------------------------------------------- To obtain copies of the postscript file, please use Jordan Pollack's service: Example: unix> ftp cheops.cis.ohio-state.edu # (or ftp 128.146.8.62) Name (cheops.cis.ohio-state.edu:): anonymous Password (cheops.cis.ohio-state.edu:anonymous): ftp> cd pub/neuroprose ftp> binary ftp> get (remote-file) rosen.adaptrange.ps.Z (local-file) rosen.adaptrange.ps.Z ftp> quit unix> uncompress rosen.adaptrange.ps unix> lpr -P(your_local_postscript_printer) rosen.adaptrange.ps ---------------------------------------------------------------------------- If you have any difficulties with the above, please send e-mail to rosen at cs.ucla.edu. DO NOT "reply" to this message, please. From yoshua at homer.cs.mcgill.ca Sun Feb 17 21:20:42 1991 From: yoshua at homer.cs.mcgill.ca (Yoshua BENGIO) Date: Sun, 17 Feb 91 21:20:42 EST Subject: TR available: Yet another ANN/HMM hybrid. Message-ID: <9102180220.AA08811@homer.cs.mcgill.ca> The following technical report is now available by ftp from neuroprose: Yoshua Bengio, Renato De Mori, Giovanni Flammia, and Ralf Kompe (1990), "Global Optimization of a Neural Network - Hidden Markov Model Hybrid", Technical Report TR-SOCS-90.22, December 1990, School of Computer Science, McGill University. Abstract: Global Optimization of a Neural Network - Hidden Markov Model Hybrid Yoshua Bengio, Renato De Mori, Giovanni Flammia, Ralf Kompe TR-SOCS-90.22, December 1990 In this paper a method for integrating Artificial Neural Networks (ANN) with Hidden Markov Models (HMM) is proposed and evaluated. ANNs are suitable to perform phonetic classification, whereas HMMs have been proven successful at modeling the temporal structure of the speech signal. In the approach described here, the ANN outputs constitute the sequence of observation vectors for the HMM. An algorithm is proposed for global optimization of all the parameters. An incremental design method is described in which specialized networks are integrated to the recognition system in order to improve its performance. Results on speaker-independent recognition experiments using this integrated ANN-HMM system on the TIMIT continuous speech database are reported. --------------------------------------------------------------------------- Copies of the postscript file bengio.hybrid.ps.Z may be obtained from the pub/neuroprose directory in cheops.cis.ohio-state.edu. Either use the Getps script or do this: unix-1> ftp cheops.cis.ohio-state.edu # (or ftp 128.146.8.62) Connected to cheops.cis.ohio-state.edu. Name (cheops.cis.ohio-state.edu:): anonymous 331 Guest login ok, sent ident as password. Password: neuron 230 Guest login ok, access restrictions apply. ftp> cd pub/neuroprose ftp> binary ftp> get bengio.hybrid.ps.Z ftp> quit unix-2> uncompress bengio.hybrid.ps.Z unix-3> lpr -P(your_local_postscript_printer) bengio.hybrid.ps Or, order a hardcopy by sending your physical mail address to yoshua at cs.mcgill.ca, mentioning Technical Report TR-SOCS-90.22. PLEASE do this only if you cannot use the ftp method described above. ---------------------------------------------------------------------------- From nzt at research.att.com Mon Feb 18 15:55:29 1991 From: nzt at research.att.com (nzt@research.att.com) Date: Mon, 18 Feb 91 15:55:29 EST Subject: mailing list Message-ID: <9102182055.AA30788@minos.att.com> Hi: Please add my name to the connectionists mailing list. Naftali Tishby, AT&T Bell Laboratory. nzt at research.att.com Thanks. From pluto at cs.UCSD.EDU Tue Feb 19 15:21:17 1991 From: pluto at cs.UCSD.EDU (Mark Plutowski) Date: Tue, 19 Feb 91 12:21:17 PST Subject: Tech Report Available in Neuroprose Message-ID: <9102192021.AA00628@cornelius> The following report has been placed in the neuroprose archives at Ohio State University: UCSD CSE Technical Report No. CS91-180 Active selection of training examples for network learning in noiseless environments. Mark Plutowski Department of Computer Science and Engineering, UCSD, and Halbert White Institute for Neural Computation and Department of Economics, UCSD. Abstract: We derive a method for {\sl actively selecting} examples to be used in estimating an unknown mapping with a multilayer feedforward network architecture. Active selection chooses from among a set of available examples an example which, when added to the previous set of training examples and learned, maximizes the decrement of network error over the input space. New examples are chosen according to network performance on previous training examples. In practice, this amounts to incrementally growing the training set as necessary to achieve the desired level of accuracy. The objective is to minimize the data requirement of learning. Towards this end, we choose a general criterion for selecting training examples that works well in conjunction with the criterion used for learning, here, least squares. Examples are chosen to minimize Integrated Mean Square Error (IMSE). IMSE embodies the effects of bias (misspecification of the network model) and variance (sampling variation due to noise). We consider a special case of IMSE, Integrated Squared Bias, (ISB) to derive a selection criterion ($\Delta ISB$) which we maximize to select new training examples. $\Delta ISB$ is applicable whenever sampling variation due to noise can be ignored. We conclude with graphical illustrations of the method, and demonstrate its use during network training. -=-=-=-=-=-=-=-=-=-=-=-=-=-= How to obtain a copy -=-=-=-=-=-=-=-=-=-=-=-=-=-= Copies may be obtained by a) FTP directly from the Neuroprose directory, or b) by land mail from the CSE dept. at UCSE. a) via FTP: To obtain a copy from Neuroprose, either use the "getps" program, or ftp the file as follows: % ftp cheops.cis.ohio-state.edu Connected to cheops.cis.ohio-state.edu. 220 cheops.cis.ohio-state.edu FTP server (Version 5.49 Tue May 9 14:01:04 EDT 1989) ready. Name (cheops.cis.ohio-state.edu:your-ident): anonymous [2331 Guest login ok, send ident as password. Password: your-ident 230 Guest login ok, access restrictions apply. ftp> cd pub/neuroprose 250 CWD command successful. ftp> binary 200 Type set to I. ftp> get plutowski.active.ps.Z 200 PORT command successful. 150 Opening BINARY mode data connection for plutowski.active.ps.Z (348443 bytes). 226 Transfer complete. local: plutowski.active.ps.Z remote: plutowski.active.ps.Z 348443 bytes received in 44 seconds (7.2 Kbytes/s) ftp> quit % uncompress plutowski.active.ps.Z % lpr -P plutowski.active.ps b) via postal mail: Requests for hardcopies may be sent to: Kay Hutcheson CSE Department, 0114 UCSD La Jolla, CA 92093-0114 and enclose a check for $5.00 payable to "UC Regents." The report number is: Technical Report No. CS91-180 From yoshua at homer.cs.mcgill.ca Tue Feb 19 14:33:08 1991 From: yoshua at homer.cs.mcgill.ca (Yoshua BENGIO) Date: Tue, 19 Feb 91 14:33:08 EST Subject: header for TR on ANN/HMM hybrid in neuroprose Message-ID: <9102191933.AA08798@homer.cs.mcgill.ca> ---------------------------------------------------------------------------- The following technical report available by ftp from neuroprose was recently advertised: Yoshua Bengio, Renato De Mori, Giovanni Flammia, and Ralf Kompe (1990), "Global Optimization of a Neural Network - Hidden Markov Model Hybrid", Technical Report TR-SOCS-90.22, December 1990, School of Computer Science, McGill University. However, it was not mentionned that the front pages of the TR are in bengio.hybrid_header.ps.Z whereas the paper itself is in: bengio.hybrid.ps.Z Sorry for the inconvenience, Yoshua Bengio School of Computer Science, McGill University ---------------------------------------------------------------------------- From whart at cs.UCSD.EDU Wed Feb 20 01:26:06 1991 From: whart at cs.UCSD.EDU (Bill Hart) Date: Tue, 19 Feb 91 22:26:06 PST Subject: New TR Available Message-ID: <9102200626.AA01316@beowulf.ucsd.edu> The following TR has been placed in the neuroprose archives at Ohio State University. --Bill -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= UCSD CSE Technical Report No. CS91-180 Active selection of training examples for network learning in noiseless environments. Mark Plutowski Department of Computer Science and Engineering, UCSD, and Halbert White Institute for Neural Computation and Department of Economics, UCSD. Abstract: We derive a method for {\sl actively selecting} examples to be used in estimating an unknown mapping with a multilayer feedforward network architecture. Active selection chooses from among a set of available examples an example which, when added to the previous set of training examples and learned, maximizes the decrement of network error over the input space. %New examples are chosen according to %network performance on previous training examples. In practice, this amounts to incrementally growing the training set as necessary to achieve the desired level of accuracy. The objective is to minimize the data requirement of learning. Towards this end, we choose a general criterion for selecting training examples that works well in conjunction with the criterion used for learning, here, least squares. Examples are chosen to minimize Integrated Mean Square Error (IMSE). IMSE embodies the effects of bias (misspecification of the network model) and variance (sampling variation due to noise). We consider a special case of IMSE, Integrated Squared Bias, (ISB) to derive a selection criterion ($\Delta ISB$) which we maximize to select new training examples. $\Delta ISB$ is applicable whenever sampling variation due to noise can be ignored. We conclude with graphical illustrations of the method, and demonstrate its use during network training. -=-=-=-=-=-=-=-=-=-=-=-=-=-= How to obtain a copy -=-=-=-=-=-=-=-=-=-=-=-=-=-= a) via FTP: To obtain a copy from Neuroprose, either use the "getps" program, or ftp the file as follows: % ftp cheops.cis.ohio-state.edu Connected to cheops.cis.ohio-state.edu. 220 cheops.cis.ohio-state.edu FTP server (Version 5.49 Tue May 9 14:01:04 EDT 1989) ready. Name (cheops.cis.ohio-state.edu:your-ident): anonymous [2331 Guest login ok, send ident as password. Password: your-ident 230 Guest login ok, access restrictions apply. ftp> cd pub/neuroprose 250 CWD command successful. ftp> binary 200 Type set to I. ftp> get plutowski.active.ps.Z 200 PORT command successful. 150 Opening BINARY mode data connection for plutowski.active.ps.Z (325222 bytes). 226 Transfer complete. local: plutowski.active.ps.Z remote: plutowski.active.ps.Z 325222 bytes received in 44 seconds (7.2 Kbytes/s) ftp> quit % uncompress plutowski.active.ps.Z % lpr -P plutowski.active.ps b) via postal mail: Requests for hardcopies may be sent to: Kay Hutcheson CSE Department, 0114 UCSD La Jolla, CA 92093-0114 and enclose a check for $5.00 payable to "UC Regents." The report number is: Technical Report No. CS91-180 From dwunsch at atc.boeing.com Wed Feb 20 13:40:15 1991 From: dwunsch at atc.boeing.com (Don Wunsch) Date: Wed, 20 Feb 91 10:40:15 PST Subject: An IJCNN question Message-ID: <9102201840.AA21635@atc.boeing.com> In December, I posted a call for papers for IJCNN. The paper deadline has passed, but this question came in the mail recently. It was a good question that others among you might have, so I'm posting my reply. Barring some major issue, this is the last you'll hear from me re IJCNN till the late preregistration deadline in June. From zoran at theory.cs.psu.edu Sun Feb 17 14:05:10 1991 From: zoran at theory.cs.psu.edu (Zoran Obradovic) Date: Sun, 17 Feb 91 14:05:10 EST Subject: IJCNN-91-Seattle Message-ID: >I just realized that I do not have any conference registration form for >IJCNN-91-Seattle. I would appreciate very much if you email me the form. >If you do not have one on-line, please tell me if you know exactly whom >to make cheque payable and what information I have to provide (in addition >to my mailing address). >Regards, >Zoran Obradovic Thanks for a good question. You don't say whether you are a student or not, or an IEEE or INNS member, so here's the generic scoop: First of all, your question is quite timely. Registrations postmarked by March 1, 1991 will enjoy significant savings over later ones. The complete cost breakdown: Preregistration: Members $195 Nonmembers $295 Students $50 by March 1! Late preregistration: Members $295 Nonmembers $395 Students $75 by June 1. On site: Members $395 Nonmembers $495 Students $95 Tutorials Members $195 Nonmembers $195 Students $85 This gets you into ALL the tutorials! You must register for the conference, however. They want the tutorial registration by April 30. One day fee is $125 for members and $175 for nonmembers. If you want to register as a student, get a letter from your department verifying full-time status. Be sure to bring your student ID to the conference. Sorry, no proceedings at that rate. If you are at all thinking about coming, you really should register now. If you cancel anytime up to June 21, you'll get everything back except for $30. Note also that you might as well get an IEEE or INNS membership if you're not using the student rate--it's cheaper that way. I believe IEEE and INNS both can sign you up by phone. The info they ask for is: Title (Dr. Mr. Ms.), Name, Affiliation (for your badge), Full mailing address, phone #, FAX #, e-mail address. You should include the information about what you are signing up for, and what rate you are paying. Enclose the letter if you are claiming student status. For one-day registration, say which day. You may send a check, made out to University of Washington. You can also use MasterCard or VISA. If you do that, you can register by phone or FAX. Phone: (206) 543-2310 FAX: (206) 685-9359 The address to write to is: IJCNN-91-Seattle Conference Management, Attn: Sarah Eck, MS GH-22, Suite 108 5001 25th Ave. NE Seattle, WA 98195 Sarah's number is (206) 543-0888, and she is the best person to ask most questions, although I'll do my best to help also. Now I have a favor to ask you. May I copy your mail and post this to the net? There are probably many people out there with the same question. Thanks! Don From khaines at galileo.ece.cmu.edu Thu Feb 21 09:53:44 1991 From: khaines at galileo.ece.cmu.edu (Karen Haines) Date: Thu, 21 Feb 91 09:53:44 EST Subject: An IJCNN question Message-ID: <9102211453.AA16257@galileo.ECE.CMU.EDU> In regards to the recent post, student registration includes proceedings and admittance into all the social events. Karen Haines From CADEPS%BBRNSF11.BITNET at BITNET.CC.CMU.EDU Thu Feb 21 12:02:09 1991 From: CADEPS%BBRNSF11.BITNET at BITNET.CC.CMU.EDU (JANSSEN Jacques) Date: Thu, 21 Feb 91 18:02:09 +0100 Subject: No subject Message-ID: IS BEHAVIORAL LIMIT CYCLING SURPRISING? I have a question I would like to pose to theorists of neural net dynamics. A behavioral phenomenon has cropped up in my work which surprised me and I would like to know if this phenomenon is to be expected by NN dynamics theorists or not. Before describing the phenomenon, I need to give some technical background. I work in the field of "Genetic Programming", i.e. using the Genetic Algorithm (GA) to build/evolve systems which function, but are (probably) too complex to analyse mathematically. I have applied these GP techniques to building artificia nervous systems and artificial embryos. To build an artificial nervous system, I evolve the weights of fully connected neural nets (GenNets), such that their time dependent outputs control some process e.g. the angles of legs on a simulated artificial creature. This way one can evolve behaviours of the creature, e.g. to get it to walk straight ahead, choose the GA fitness to be the distance covered over a given number of cycles. To get it to turn, choose the fitness to be the angle rotated etc. The surprise comes when one tries to combine these motions. To do so, take the output of one GenNet (with its set of weights) and input it into the second GenNet. For example, if one wants the creature to walk straight ahead and later to turn right, then use the GenNet evolved for straight ahead walking for the time you want. Then take the output of this first GenNet and input it into the second GenNet which was evolved to get the creature to turn. What I found was that INDEPENDENTLY of the state of the creatures legs i.e. their angles (which are proportional to the output values), which were input into the second GenNet, one got the desired qualitative behaviour, i.e. it turned. I find this extremely useful phenomenon puzzling. It can be used for smooth transitions between behaviours, but why does it work? It looks as though the GenNet has evolved a sort of limit cycle in its behaviour, so that no matter what the starting state (i.e. the initial input values to the GenNet) the desired limit cycle behaviour will occur (e.g. straight ahead walking or turning etc). I call this phenomenon "Behavioral Limit Cycling" (BLC). Is this phenomenon to be expected? Is it old-hat to the theorists or have I stumbled onto something new and exciting? (I will certainly be able to use it when switching between behavioral GenNets amongst a whole GenNet library. This library constitutes an artificial nervous system and will be useful in forging a link between the two fields of neural networks (i.e. multi networks, not just one, which is what most NN papers are about) and the hot new field of Artificial Life). I pose this question for open discussion. Cheers, Hugo de Garis, University of Brussels & George Mason University VA. email CADEPS at BBRNSF11.BITNET From andycl at syma.sussex.ac.uk Mon Feb 18 15:25:47 1991 From: andycl at syma.sussex.ac.uk (Andy Clark) Date: Mon, 18 Feb 91 20:25:47 GMT Subject: No subject Message-ID: <6587.9102182025@syma.sussex.ac.uk> Dear People, Here is a short ad concerning a new course which may be of interest to you or your students. NOTICE OF NEW M.A.COURSE BEGINNING OCT. 1991 UNIVERSITY OF SUSSEX, BRIGHTON, ENGLAND SCHOOL OF COGNITIVE AND COMPUTING SCIENCES M.A. in the PHILOSOPHY OF COGNITIVE SCIENCE This is a one year taught course which examines issues relating to computational models of mind. A specific focus concerns the significance of connectionist models and the role of rules and symbolic representation in cognitive science. Students would combine work towards a 20,000 word philosophy dissertation with subsidiary courses introducing aspects of A.I. and the other Cognitive Sciences. For information about this new course contact Dr Andy Clark, School of Cognitive and Computing Sciences, University of Sussex,Brighton, BN1 9QH, U.K. E-mail: andycl at uk.ac.sussex.syma From lazzaro at sake.Colorado.EDU Fri Feb 22 02:10:21 1991 From: lazzaro at sake.Colorado.EDU (John Lazzaro) Date: Fri, 22 Feb 91 00:10:21 MST Subject: No subject Message-ID: <9102220710.AA03865@sake.colorado.edu> An announcement of a preprint on the neuroprose server ... A Delay-Line Based Motion Detection Chip Tim Horiuchi, John Lazzaro*, Andrew Moore, Christof Koch CNS Program, Caltech and *Optoelectronics Center, CU Boulder Abstract -------- Inspired by a visual motion detection model for the rabbit retina and by a computational architecture used for early audition in the barn owl, we have designed a chip that employs a correlation model to report the one-dimensional field motion of a scene in real time. Using subthreshold analog VLSI techniques, we have fabricated and successfully tested a 8000 transistor chip using a standard MOSIS process. ----- To retrieve ... >cheops.cis.ohio-state.edu >Name (cheops.cis.ohio-state.edu:lazzaro): anonymous >331 Guest login ok, send ident as password. >Password: your_username >230 Guest login ok, access restrictions apply. >cd pub/neuroprose >binary >get horiuchi.motion.ps.Z >quit %uncompress horiuchi.motion.ps.Z %lpr horiuchi.motion.ps ---- --jl From st at gmdzi.uucp Fri Feb 22 07:29:04 1991 From: st at gmdzi.uucp (Sebastian Thrun) Date: Fri, 22 Feb 91 11:29:04 -0100 Subject: No subject Message-ID: <9102221029.AA25000@gmdzi.gmd.de> Technical Reports available: Planning with an Adaptive World Model S. Thrun, K. Moeller, A. Linden We present a new connectionist planning method. By interaction with an unknown environment, a world model is progressively constructed using gradient descent. For deriving optimal actions with respect to future reinforcement, planning is applied in two steps: an experience network proposes a plan which is subsequently optimized by gradient descent with a chain of world models, so that an optimal reinforcement may be obtained when it is actually run. The appropriateness of this method is demonstrated by a robotics application and a pole balancing task. (to appear in proceedings NIPS*90) ------------------------------------------------------------------------- A General Feed-Forward Algorithm for Gradient Descent in Connectionist Networks S. Thrun, F. Smieja An extended feed-forward algorithm for recurrent connectionist networks is presented. This algorithm, which works locally in time, is derived both for discrete-in-time networks and for continuous networks. Several standard gradient descent algorithms for connectionist networks (e.g. Williams/Zipser 88, Pineda 87, Pearlmutter 88, Gherrity 89, Rohwer 87, Waibel 88, especially the backpropagation algorithm Rumelhart/Hinton/Williams 86, are mathematically derived from this algorithm. The learning rule presented in this paper is a superset of gradient descent learning algorithms for multilayer networks, recurrent networks and time-delay networks that allows any combinations of their components. In addition, the paper presents feed-forward approximation procedures for initial activations and external input values. The former one is used for optimizing starting values of the so-called context nodes, the latter one turned out to be very useful for finding spurious input attractors of a trained connectionist network. Finally, we compare time, processor and space complexities of this algorithm with backpropagation for an unfolded-in-time network and present some simulation results. (in: "GMD Arbeitspapiere Nr. 483") ------------------------------------------------------------------------- Both reports can be received by ftp: unix> ftp cis.ohio-state.edu Name: anonymous Guest Login ok, send ident as password Password: neuron ftp> binary ftp> cd pub ftp> cd neuroprose ftp> get thrun.nips90.ps.Z ftp> get thrun.grad-desc.ps.Z ftp> bye unix> uncompress thrun.nips90.ps unix> uncompress thrun.grad-desc.ps unix> lpr thrun.nips90.ps unix> lpr thrun.grad-desc.ps ------------------------------------------------------------------------- To all European guys: The same files can be retrieved from gmdzi.gmd.de (129.26.1.90), directory pub/gmd, which is probably a bit cheaper. ------------------------------------------------------------------------- If you have trouble in ftping the files, do not hesitate to contact me. --- Sebastian Thrun (st at gmdzi.uucp, st at gmdzi.gmd.de) From psykimp at aau.dk Fri Feb 22 05:47:37 1991 From: psykimp at aau.dk (Kim Plunkett) Date: Fri, 22 Feb 91 11:47:37 +0100 Subject: No subject Message-ID: <9102221047.AA13756@aau.dk> The following technical report is now available. For a copy, email "psyklone at aau.dk" and include your ordinary mail address. Kim Plunkett +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Connectionism and Developmental Theory Kim Plunkett and Chris Sinha University of Aarhus, Denmark Abstract The main goal of this paper is to argue for an ``epigenetic developmental interpretation'' of connectionist modelling of human cognitive processes, and to propose that parallel dis- tributed processing (PDP) models provide a superior account of developmental phenomena than that offered by cognitivist (symbolic) computational theories. After comparing some of the general characteristics of epigeneticist and cognitivist theories, we provide a brief overview of the operating prin- ciples underlying artificial neural networks (ANNs) and their associated learning procedures. Four applications of different PDP architectures to developmental phenomena are described. First, we assess the current status of the debate between symbolic and connectionist accounts of the process of English past tense formation. Second, we introduce a connectionist model of concept formation and vocabulary growth and show how it provides an account of aspects of semantic development in early childhood. Next, we take up the problem of compositionality and structure dependency in connectionist nets, and demonstrate that PDP models can be architecturally designed to capture the structural princi- ples characteristic of human cognition. Finally, we review a connectionist model of cognitive development which yields stage-like behavioural properties even though structural and input assumptions remain constant throughout training. It is shown how the organisational characteristics of the model provide a simple but precise account of the equilibration of the processes of accommodation and assimilation. The authors conclude that a coherent epigenetic-developmental interpretation of PDP modelling requires the rejection of so-called hybrid-architecture theories of human cognition. From marshall at cs.unc.edu Fri Feb 22 11:03:09 1991 From: marshall at cs.unc.edu (Jonathan Marshall) Date: Fri, 22 Feb 91 11:03:09 -0500 Subject: Paper available -- visual orientation multiplexing Message-ID: <9102221603.AA23754@marshall.cs.unc.edu> **** Please do not re-post to other bboards. **** Papers available, hardcopy only. ---------------------------------------------------------------------- ADAPTIVE NEURAL METHODS FOR MULTIPLEXING ORIENTED EDGES Jonathan A. Marshall Department of Computer Science University of North Carolina at Chapel Hill Edge linearization operators are often used in computer vision and in neural network models of vision to reconstruct noisy or incomplete edges. Such operators gather evidence for the presence of an edge at various orientations across all image locations and then choose the orientation that best fits the data at each point. One disadvantage of such methods is that they often function in a winner-take-all fashion: the presence of only a single orientation can be represented at any point; multiple edges cannot be represented where they intersect. For example, the neural Boundary Contour System of Grossberg and Mingolla implements a form of winner-take-all competition between orthogonal orientations at each spatial location, to promote sharpening of noisy, uncertain image data. But that competition may produce rivalry, oscillation, instability, or mutual suppression when intersecting edges (e.g., a cross) are present. This "cross problem" exists for all techniques, including Markov Random Fields, where a representation of a chosen favored orientation suppresses representations of alternate orientations. A new adaptive technique, using both an inhibitory learning rule and an excitatory learning rule, weakens inhibition between neurons representing poorly correlated orientations. It may reasonably be assumed that neurons coding dissimilar orientations are less likely to be coactivated than neurons coding similar orientations. Multiplexing by superposition is ordinarily generated: combinations of intersecting edges become represented by simultaneous activation of multiple neurons, each of which represents a single supported oriented edge. Unsupported or weakly supported orientations are suppressed. The cross problem is thereby solved. [to appear in Proceedings of the SPIE Conference on Advances in Intelligent Systems, Boston, November 1990.] ---------------------------------------------------------------------- Also available: J.A. Marshall, "A Self-Organizing Scale-Sensitive Neural Network." In Proceedings of the International Joint Conference on Neural Networks, San Diego, June 1990, Vol.III., pp.649-654. J.A. Marshall, "Self-Organizing Neural Networks for Perception of Visual Motion." Neural Networks, 3, pp.45-74 (1990). = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = Jonathan A. Marshall marshall at cs.unc.edu = = Department of Computer Science = = CB 3175, Sitterson Hall = = University of North Carolina Office 919-962-1887 = = Chapel Hill, NC 27599-3175, U.S.A. Fax 919-962-1799 = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = **** Please do not re-post to other bboards. **** From issnnet at park.bu.edu Fri Feb 22 12:54:24 1991 From: issnnet at park.bu.edu (Student Society Account) Date: Fri, 22 Feb 91 12:54:24 -0500 Subject: Student Conference Sponsorships Message-ID: <9102221754.AA22635@park.bu.edu> ---- THIS NOTE CONTAINS MATERIAL OF INTEREST TO ALL STUDENTS ---- (and some non-students) This message is a brief update of the International Student Society for Neural Networks (ISSNNet), and also an ANNOUNCEMENT OF AVAILABLE STUDENT SPONSORSHIP AT UPCOMING NNets CONFERENCES. ---------------------------------------------------------------------- NOTE TO ISSNNet MEMBERS: If you have joined the society but did not receive our second newsletter, or have not heard from us in the recent past, please send e-mail to . ---------------------------------------------------------------------- 1) We had a problem with our Post Office Box (in the USA), which was inadvertently shut down for about one month between Christmas and the end of January. If you sent us surface mail which was returned to you, please send it again. Our apologies for the inconvenience. 2) We are about to distribute the third newsletter. This is a special issue that includes our bylaws, a description of the various officer positions, and a call for nominations for upcoming elections. In addition, a complete membership list will be included. ISSNNet MEMBERS: If you have not sent us a note about your interests in NNets, you must do so by about the end of next week to insure it appears in the membership list. Also, all governors should send us a list of their members if they have not done so already. 3) STUDENT SPONSORSHIPS AVAILABLE: We have been in contact with the IEEE Neural Networks Council. We donated $500 (about half of our savings to this point) to pay the registration for students who are presenting articles (posters or oral presentations) at the Helsinki (ICANN), Seattle (IJCNN), or Singapore (IJCNN) conferences. The IEEE Neural Networks Council has augmented our donation with an additional $5,000 to be distributed between the two IJCNN conferences, and we are currently in contact with the International Neural Networks Society (INNS) regarding a similar donation. We are hoping to pay for registration and proceedings for an approximately equal number of students at each of the three conferences. Depending on other donations and on how many people are eligible, only a limited number of sponsorships may be available. The details of eligibility will be officially published in our next newsletter and on other mailing lists, but generally you will need to be the person presenting the paper (if co-authored), and you must receive only partial or no support from your department. Forms will be included with the IJCNN paper acceptance notifications. It is not necessary to be a member of any of these societies, although we hope this will encourage future student support and increased society membership. IF YOU HAVE SUBMITTED A PAPER TO ONE OF THE IJCNN CONFERENCES, YOU WILL RECEIVE DETAILS WITH YOUR NOTIFICATION FROM IJCNN. Details on the ICANN conference will be made available in the near future. For other questions send us some e-mail . ISSNNet, Inc. PO Box 557, New Town Branch Boston, MA 02258 Sponsors will be officially recognized in our future newsletters, and will be mentioned by the sponsored student during the presentations and posters. 6) We are considering the possibility of including abstracts from papers written by ISSNNet members in future newsletters. Depending on how many ISSNNet papers are accepted at the three conferences, we may be able to publish the abstracts in the fourth newsletter, which should come out before the ICANN conference. This would give the presenters some additional publicity, and would give ISSNNet members a sneak preview of what other people are doing. MORE DETAIL ON THESE TOPICS WILL BE INCLUDED IN OUR NEXT NEWSLETTER, WHICH WE EXPECT TO PUBLISH AROUND THE END OF THIS MONTH. For more details on ISSNNet, or to receive a sample newsletter, send e-mail to . You need not be a student to become a member! From bms at dcs.leeds.ac.uk Fri Feb 22 09:58:44 1991 From: bms at dcs.leeds.ac.uk (B M Smith) Date: Fri, 22 Feb 91 14:58:44 GMT Subject: Item for Distribution Message-ID: <7743.9102221458@csunb0.dcs.leeds.ac.uk> ************************************************************************** ***************** * * * A I S B 9 1 * * * ***************** UNIVERSITY OF LEEDS, UK 16 - 19 APRIL 1991 TUTORIAL PROGRAMME 16 APRIL TECHNICAL PROGRAMME 17-19 APRIL with sessions on: * Distributed Intelligent Agents * Situatedness and Emergence in Autonomous Agents * New Modes of Reasoning * The Knowledge Level Perspective * Theorem Proving * Machine Learning Programmes and registration forms are now available from: Barbara Smith AISB91 Local Organizer School of Computer Sudies University of Leeds Leeds LS2 9JT, UK email: aisb91 at ai.leeds.ac.uk *************************************************************************** From jcp at vaxserv.sarnoff.com Fri Feb 22 14:58:54 1991 From: jcp at vaxserv.sarnoff.com (John Pearson W343 x2385) Date: Fri, 22 Feb 91 14:58:54 EST Subject: NIPS Call for Papers Message-ID: <9102221958.AA20764@sarnoff.sarnoff.com> CALL FOR PAPERS Neural Information Processing Systems -Natural and Synthetic- Monday, December 2 - Thursday, December 5, 1991 Denver, Colorado This is the fifth meeting of an inter-disciplinary conference which brings together neuroscientists, engineers, computer scientists, cognitive scientists, physicists, and mathematicians interested in all aspects of neural processing and computation. There will be an afternoon of tutorial presentations (Dec 2) preceding the regular session and two days of focused workshops will follow at a nearby ski area (Dec 6-7). Major categories and examples of subcategories for paper submissions are the following; Neuroscience: Studies and Analyses of Neurobiological Systems, Inhibition in cortical circuits, Signals and noise in neural computation, Theoretical Biology and Biophysics. Theory: Computational Learning Theory, Complexity Theory, Dynamical Systems, Statistical Mechanics, Probability and Statistics, Approximation Theory. Implementation and Simulation: VLSI, Optical, Software Simulators, Implementation Languages, Parallel Processor Design and Benchmarks. Algorithms and Architectures: Learning Algorithms, Constructive and Pruning Algorithms, Localized Basis Functions, Tree Structured Networks, Performance Comparisons, Recurrent Networks, Combinatorial Optimization, Genetic Algorithms. Cognitive Science & AI: Natural Language, Human Learning and Memory, Perception and Psychophysics, Symbolic Reasoning. Visual Processing: Stereopsis, Visual Motion Processing, Image Coding and Classification. Speech and Signal Processing: Speech Recognition, Coding, and Synthesis, Text-to-Speech, Adaptive Equalization, Nonlinear Noise Removal. Control, Navigation, and Planning Navigation and Planning, Learning Internal Models of the World, Trajectory Planning, Robotic Motor Control, Process Control. Applications Medical Diagnosis or Data Analysis, Financial and Economic Analysis, Timeseries Prediction, Protein Structure Prediction, Music Processing, Expert Systems. Technical Program: Plenary, contributed and poster sessions will be held. There will be no parallel sessions. The full text of presented papers will be published. Submission Procedures: Original research contributions are solicited, and will be carefully refereed. Authors must submit six copies of both a 1000-word (or less) summary and six copies of a separate single- page 50-100 word abstract clearly stating their results postmarked by May 17, 1991. Accepted abstracts will be published in the conference program. Summaries are for program committee use only. At the bottom of each abstract page and on the first summary page indicate preference for oral or poster presentation and specify one of the above nine broad categories and, if appropriate, sub-categories (For example: Poster, Applications- Expert Systems; Oral, Implementation-Analog VLSI). Include addresses of all authors at the front of the summary and the abstract and indicate to which author correspondence should be addressed. Submissions will not be considered that lack category information, separate abstract sheets, the required six copies, author addresses, or are late. Mail Submissions To: Stephen J. Hanson NIPS*91 Submissions Siemens Research Center 755 College Road East Princeton NJ, 08540 Mail For Registration Material To: NIPS*91 Registration Siemens Research Center 755 College Road East Princeton, NJ, 08540 All submitting authors will be sent registration material automatically. Program committee decisions will be sent to the correspondence author only. NIPS*91 Organizing Committee: General Chair, John Moody, Yale U.; Program Chair, Stephen J. Hanson, Siemens Research & Princeton U.; Publications Chair, Richard Lippmann, MIT Lincoln Laboratory; Publicity Chair, John Pearson, SRI, David Sarnoff Research Center; Treasurer, Bob Allen, Bellcore; Local Arrangements, Mike Mozer, University of Colorado; Program Co-Chairs:, David Ackley, Bellcore; Pierre Baldi, JPL & Caltech; William Bialek, NEC; Lee Giles, NEC; Mike Jordan, MIT; Steve Omohundro, ICSI; John Platt, Synaptics; Terry Sejnowski, Salk Institute; David Stork, Ricoh & Stanford; Alex Waibel, CMU; Tutorial Chair: John Moody, Workshop CoChairs: Gerry Tesauro, IBM & Scott Kirkpatrick, IBM; Domestic Liasons: IEEE Liaison, Rodney Goodman, Caltech; APS Liaison, Eric Baum, NEC; Neurobiology Liaison, Tom Brown, Yale U.; Government & Corporate Liaison, Lee Giles, NEC; Overseas Liasons: Mitsuo Kawato, ATR; Marwan Jabri, University of Sydney; Benny Lautrup, Niels Bohr Institute; John Bridle, RSRE; Andreas Meier, Simon Bolivar U. DEADLINE FOR SUMMARIES & ABSTRACTS IS MAY 17, 1991 please post From jcp at vaxserv.sarnoff.com Fri Feb 22 15:12:52 1991 From: jcp at vaxserv.sarnoff.com (John Pearson W343 x2385) Date: Fri, 22 Feb 91 15:12:52 EST Subject: NIPS-91 Workshop Message-ID: <9102222012.AA20814@sarnoff.sarnoff.com> CALL FOR WORKSHOPS NIPS*91 Post-Conference Workshops December 6 and 7, 1991 Vail, Colorado Request for Proposals Following the regular NIPS program, workshops on current topics on Neural Information Processing will be held on December 6 and 7, 1991, in Vail, Colorado. Proposals by qualified individuals interested in chairing one of these workshops are solicited. Past topics have included: Rules and Connectionist Models; Speech; Vision; Sensory Biophysics; Neural Network Dynamics; Neurobiology; Computational Complexity Issues; Fault Tolerance in Neural Networks; Benchmarking and Comparing Neural Network Applications; Architectural Issues; Fast Training Techniques; Control; Optimization, Statistical Inference, Genetic Algorithms; VLSI and Optical Implementations; Integration of Neural Networks with Conventional Software. The format of the workshop is informal. Beyond reporting on past research, the goal is to provide a forum for scientists actively working in the field to freely discuss current issues of concern and interest. Sessions will meet in the morning and in the afternoon of both days, with free time in between for the ongoing individual exchange or outdoor activities. Specific open and/or controversial issues are encouraged and preferred as workshop topics. Individuals interested in chairing a workshop must propose a topic of current interest and must be willing to accept responsibility for their group's discussion. Discussion leaders' responsibilities include: arrange brief informal presentations by experts working on this topic, moderate or lead the discussion, and report its high points, findings and conclusions to the group during evening plenary sessions, and in a short (2 page) summary. Submission Procedure: Interested parties should submit a short proposal for a workshop of interest by May 17, 1991. Proposals should include a title and a short description of what the workshop is to address and accomplish. It should state why the topic is of interest or controversial, why it should be discussed and what the targeted group of participants is. In addition, please send a brief resume of the prospective workshop chair, list of publications and evidence of scholarship in the field of interest. Mail submissions to: Dr. Gerald Tesauro, Co-Chair, Attn: NIPS91 Workshops, IBM Research P.O. Box 704 Yorktown Heights, NY 10598 USA Name, mailing address, phone number, and e-mail net address (if applicable) must be on all submissions. Workshop CoChairs: G. Tesauro & S. Kirkpatrick, IBM PROPOSALS MUST BE RECEIVED BY MAY 17,1991 Please Post From erol at ehei.ehei.fr Fri Feb 22 13:11:25 1991 From: erol at ehei.ehei.fr (Erol Gelenbe) Date: Fri, 22 Feb 91 18:13:25 +2 Subject: Preprint on Texture Generation with the Random Neural Network Message-ID: <9102230855.AA15700@inria.inria.fr> The following paper, accepted for oral presentation at ICANN-91, is available as a preprint : Texture Generation with the Random Neural Network Model by Volkan Atalay, Erol Gelenbe, Nese Yalabik A copy may be obtained by e-mailing your request to : erol at ehei.ehei.fr Erol Gelenbe EHEI Universite Rene Descartes (Paris V) 45 rue des Saints-Peres 75006 Paris From thrun at gmdzi.uucp Sun Feb 24 17:56:27 1991 From: thrun at gmdzi.uucp (Sebastian Thrun) Date: Sun, 24 Feb 91 21:56:27 -0100 Subject: No subject Message-ID: <9102242056.AA06846@gmdzi.gmd.de> Subject: Wrong host name Sorry, in my last posting I made the mistake to write "cis.ohio-state.edu" instead of the correct "cheops.cis-ohio-state.edu" as the address of the neuroprose archieve. Of course, the TRs I announced can only be retrieved from the latter machine. To all those who failed: Please try again with this new address. ---- thanks, sebastian From MURRE%rulfsw.LeidenUniv.nl at BITNET.CC.CMU.EDU Mon Feb 25 16:57:00 1991 From: MURRE%rulfsw.LeidenUniv.nl at BITNET.CC.CMU.EDU (MURRE%rulfsw.LeidenUniv.nl@BITNET.CC.CMU.EDU) Date: Mon, 25 Feb 91 16:57 MET Subject: Limited precision implementations (updated posting) Message-ID: Connectionist researchers, Here is an updated posting on limited precision implementations of neural networks. It is my impression that research in this area is still fragmentary. This is surprising, because the literature on analog and digital implementations is growing very fast. There is a wide range of possibly applicable rules of thumb. Claims about sufficient precision differ from single bits to 20 bits or more for certain models. Hard problems may need higher precision. There may be a trade-off between few weights (nodes) with high precision weights (activations) versus many weights (nodes) with low precision weights (act.). The precise relation between precision in weights and activations remains unclear, as does the relation between the effect of precision on learning and recall. Thanks for all comments so far. Jaap Jacob M.J. Murre Unit of Experimental and Theoretical Psychology Leiden University P.O. Box 9555 2300 RB Leiden The Netherlands General comments by researchers By Soheil Shams: As far as the required precision for neural computation is concerned, the precision is directly proportional to the difficulty of the problem you are trying to solve. For example in training a back-propagation network to discriminate between two very similar classes of inputs, you will need to have high precision values and arithmetic to effectively find the narrow region in the space that the separating hyperplane has to be drawn at. I believe that the lack of analytical information in this area is due to this relationship between the specific application and the required precision . At the NIPS90 workshop on massively parallel implementations, some people indicated they have determined, EMPERICALLY, that for most problems, 16-bit precision is required for learning and 8-bit for recall of back-propagation. By Roni Rosenfeld: Santosh Venkatesh (of Penn State, I believe, or is it U. Penn?) did some work a few years ago on how many bits are needed per weight. The surprising result was that 1 bit/weight did most of the work, with additional bits contributing surprisingly little. By Thomas Baker: ... We have found that for backprop learning, between twelve and sixteen bits are needed. I have seen several other papers with these same results. After learning, we have been able to reduce the weights to four to eight bits with no loss in network performance. I have also seen others with similar results. One method that optical and analog engineers use is to calculate the error by running the feed forward calculations with limited precision, and learning the weights with a higher precision. The weights are quantized and updated during training. I am currently collecting a bibliography on limited precision papers. ... I will try to keep in touch with others that are doing research in this area. References Brause, R. (1988). Pattern recognition and fault tolerance in non-linear neural networks. Biological Cybernetics, 58, 129-139. Hollis, P.W., J.S. Harper, J.J. Paulos (1990). The effects of precision constraints in a backpropagation learning network. Neural Computation, 2, 363-373. Holt, J.L., & J-N. Hwang (in prep.). Finite precision error analysis of neural network hardware implementations. Univ. of Washington, FT-10, WA 98195. (Comments by the authors: We are in the process of finishing up a paper which gives a theoretical (systematic) derivation of the finite precision neural network computation. The idea is a nonlinear extension of "general compound operators" widely used for error analysis of linear computation. We derive several mathematical formula for both retrieving and learning of neural networks. The finite precision error in the retrieving phase can be written as a function of several parameters, e.g., number of bits of weights, number of bits for multiplication and accumlation, size of nonlinear table-look-up, truncation/rounding or jamming approaches, and etc. Then we are able to extend this retrieving phase error analysis to iterative learning to predict the necessary number of bits. This can be shown using a ratio between the finite precision error and the (floating point) back-propagated error. Simulations have been conducted and matched the theoretical prediction quite well.) Hong, J. (1987). On connectionist models. Tech. Rep., Dept. Comp. Sci., Univ. of Chicago, May 1987. (Demonstrates that a network of perceptrons needs only finite-precision weights.) Jou, J., & J.A. Abraham (1986). Fault-tolerant matrix arithmetic and signal processing on highly concurrent computing structures. Proceedings of the IEEE, 74, 732-741. Kampf, F., P. Koch, K. Roy, M. Sullivan, Z. Delalic, & S. DasGupta (1989). Digital implementation of a neural network. Tech. Rep. Temple Univ., Philadelphia PA, Elec. Eng. Div. Moore, W.R. (1988). Conventional fault-tolerance and neural computers. In: R. Eckmiller, & C. Von der Malsburg (Eds.). Neural Computers. NATO ASI Series, F41, (Berling: Springer-Verlag), 29-37. Nadal, J.P. (1990). On the storage capacity with sign-constrained synaptic couplings. Network, 1, 463-466. Nijhuis, J., & L. Spaanenburg (1989). Fault tolerance of neural associative memories. IEE Proceedings, 136, 389-394. Rao, A., M.R. Walker, L.T. Clark, & L.A. Akers (1989). Integrated circuit emulation of ART networks. Proc. First IEEE Conf. Artificial Neural Networks, 37-41, Institution of Electrical Engineers, London. Rao, A., M.R. Walker, L.T. Clark, L.A. Akers, & R.O. Grondin (1990). VLSI implementation of neural classifiers. Neural Computation, 2, 35-43. (The paper by Rao et al. give an equation for the number of bits of resolution required for the bottom-up weights in ART 1: t = (3 log N) / log(2), where N is the size of the F1 layer in nodes.) From giles at fuzzy.nec.com Mon Feb 25 13:48:06 1991 From: giles at fuzzy.nec.com (Lee Giles) Date: Mon, 25 Feb 91 13:48:06 EST Subject: possible academic positions Message-ID: <9102251848.AA15611@fuzzy.nec.com> The February issue of IEEE Spectrum posted 7 academic positions that mentioned "neural networks" or "neural computing" in the job description. -- C. Lee Giles NEC Research Institute 4 Independence Way Princeton, NJ 08540 USA Internet: giles at research.nj.nec.com UUCP: princeton!nec!giles PHONE: (609) 951-2642 FAX: (609) 951-2482 From giles at fuzzy.nec.com Mon Feb 25 13:52:01 1991 From: giles at fuzzy.nec.com (Lee Giles) Date: Mon, 25 Feb 91 13:52:01 EST Subject: Summer Position Message-ID: <9102251852.AA15615@fuzzy.nec.com> NEC Research Institute in Princeton, N.J. has available a 3 month summer research and programming position. The research emphasis will be on exploring the computational capabilities of recurrent neural networks. The successful candidate will have a background in neural networks and strong programming skills in the C/Unix environment. Computer science background preferred. Interested applicants should send their resumes by mail, fax, or email to the address below. The application deadline is March 25, 1991. Applicants must show documentation of eligibility for employment. Because this is a summer position, the only expenses to be paid will be salary. NEC is an equal opportunity employer. -- C. Lee Giles NEC Research Institute 4 Independence Way Princeton, NJ 08540 USA Internet: giles at research.nj.nec.com UUCP: princeton!nec!giles PHONE: (609) 951-2642 FAX: (609) 951-2482 From gluck%psych at Forsythe.Stanford.EDU Mon Feb 25 14:03:38 1991 From: gluck%psych at Forsythe.Stanford.EDU (Mark Gluck) Date: Mon, 25 Feb 91 11:03:38 PST Subject: Postdoc & Research Assistant openings in the COGNITIVE & NEURAL BASES OF LEARNING (Rutgers, NJ) Message-ID: <9102251903.AA25039@psych> Postdoctoral & Research/Programming Positions in: THE COGNITIVE & NEURAL BASES OF LEARNING ---------------------------------------------------------------------------- Rutgers University Center for Molecular & Behavioral Neuroscience 195 University Avenue Newark, NJ 07102 Postdoctoral Positions in: -------------------------- 1. EMPIRICAL STUDIES OF HUMAN LEARNING: Including: designing and conducting studies of human learning and decision making, especially categorization learning. These are primarily motivated by a desire to evaluate and refine adaptive network models of learning and memory (see, e.g., the experimental studies described in Gluck & Bower, 1988a; Pavel, Gluck, & Henkle, 1988). This work requires a familiarity with psychological methods of experimental design and data analysis. 2. COMPUTATIONAL MODELS OF ANIMAL & HUMAN LEARNING: Including: developing and extending current network models of learning to more accurately reflect a wider range of animal and human learning behaviors. This work requires strong programming skills, familiarity with adaptive network theories and methods, and some degree of mathematical and analytic training. 3. COMPUTATIONAL MODELS OF THE NEUROBIOLOGY OF LEARNING & MEMORY: Including: (1) Models and theories of the neural bases of classical and operant conditioning; (2) Neural mechansims for human associative learning; (3) Theoretical studies which seek to form links, behavioral or biological, between animal and human learning (see, e.g., Gluck, Reifsnider, & Thompson (1989), in Gluck & Rumelhart (Eds.) Neuroscience & Connectionist Theory). and Connectionist Theory). Full or Part-Time Research & Programming Positions: --------------------------------------------------- These positions are ideal for someone who has just graduated with an undergraduate degree and would like a year or two of "hands on" experience in research before applying to graduate school in one of the cognitive sciences (e.g., neuroscience, psychology, computer science). We are looking for two types of people: 1) a RESEARCH PROGRAMMER with strong computational skills (especially with C/Unix and SUN systems) and experience with PDP models and theory, and (2) an EXPERIMENTAL RESEARCH ASSISTANT to assist in running and designing human learning experiments. Some research experience required (familiarity with Apple MACs a plus). Application Procedure: ---------------------- For more information on learning research at the CMBN/Rutgers or to apply for these positions, please send a cover letter with a statement of your research interests, a CV, copies of relevant preprints, and the the names & phone numbers of references to: Dr. Mark A. Gluck Phone: (415) 725-2434 Dept. of Psychology <-[Current address to 4/91] FAX: (415) 725-5699 Jordan Hall; Bldg. 420 Stanford University email: gluck at psych.stanford.edu Stanford, CA 94305-2130 --------------------------end of notice-------------------------- From holt at pierce.ee.washington.edu Mon Feb 25 14:33:44 1991 From: holt at pierce.ee.washington.edu (Jordan Holt) Date: Mon, 25 Feb 91 11:33:44 PST Subject: Technical Report Available Message-ID: <9102251933.AA01454@pierce.ee.washington.edu.Jaimie> Technical Report Available: Finite Precision Error Analysis of Neural Network Hardware Implementations Jordan Holt, Jenq-Neng Hwang The high speed desired in the implementation of many neural network algorithms, such as back-propagation learning in a multilayer perceptron (MLP), may be attained through the use of finite precision hardware. This finite precision hardware; however, is prone to errors. A method of theoretically deriving and statistically evaluating this error is presented and could be used as a guide to the details of hardware design and algorithm implementation. The paper is devoted to the derivation of the techniques involved as well as the details of the back-propagation example. The intent is to provide a general framework by which most neural network algorithms under any set of hardware constraints may be evaluated. Section 2 demonstrates the sources of error due to finite precision computation and their statistical properties. A general error model is also derived by which an equation for the error at the output of a general compound operator may be written. As an example, error equations are derived in Section 3 for each of the operations required in the forward retrieving and error back- propagation steps of an MLP. Statistical analysis and simulation results of the resulting distribution of errors for each individual step of an MLP are also included in this section. These error equations are then integrated, in Section 4, to predict the influence of finite precision computation on several stages (early, middle, final stages) of back-propagation learning. Finally, concluding remarks are given in Section 5. ---------------------------------------------------------------------------- The report can be received by ftp: unix> ftp cheops.cis.ohio-state.edu Name: anonymous Guest Login ok, send ident as password Password: neuron ftp> binary ftp> cd pub ftp> cd neuroprose ftp> get holt.finite_error.ps.Z ftp> bye unix> uncompress holt.finite_error.ps unix> lpr holt.finite_error.ps From ecdbcf at ukc.ac.uk Mon Feb 25 12:07:30 1991 From: ecdbcf at ukc.ac.uk (ecdbcf@ukc.ac.uk) Date: Mon, 25 Feb 91 17:07:30 +0000 Subject: Boolean Models(GSN) Message-ID: Dear Connectionists, Most people who read this mail will probably be working with continuous/analogue models. There is, however, a growing interest in Boolean neuron models, and some readers might be interested to know that I have recently successfully completed a Ph.D thesis which deals with a particular kind of Boolean neuron. Some brief details are given below, together with some references to more detailed material. ----------------------------------------------------------------------- Abstract This thesis is concerned with the investigation of Boolean neural networks based on a novel RAM-based Goal-Seeking Neuron(GSN). Boolean neurons are particularly suited to the solution of Boolean or logic problems such as the recognition and associative recall of binarised patterns. One main advantage of Boolean neural networks is the ease with which they can be implemented in hardware. This can result in very fast operation. The GSN has been formulated to ensure this implementation advantage is not lost. The GSN model operates through the interaction of a number of local low level goals and is applicable to practical problems in pattern recognition with only a single pass of the training data(one-shot learning). The thesis explores different architectures for GSNs (feed-forward, feedback and self-organising) together with different learning rules, and investigates a wide range of alternative configurations within these three architectures. Practical results are demonstrated in the context of a character recognition problem. ----------------------------------------------------------------------- Highlights of GSNs, Learning Algorithms, Architectures and Main Contributions The main advantage of RAM-based neural networks in comparison with networks based on sum-of-products functions is the ease with which they can be implemented in hardware. This derives from their essentially logical rather than continuous nature. The GSN model has a natural propensity to solve the main problems associated with other RAM-based neurons. Specific classes of computational activity can be more appropriately realised by using a particular goal seeking function, and different kinds of goal seeking functions can be sought in order to provide a range of suitable behaviour, creating effectively a family of GSNs. The main experimental results have demonstrated the viability of the one-shot learning algorithms: partial pattern association, quasi-self-organisation, and self-organisation. The one-shot learning is only possible because of the the GSN's ability to validate the possibility of learning a given input pattern using a single presentation. The partial pattern association and the quasi-self-organising learning have been applied in feed-forward architectures. These two kinds of learning have given similar performance, though the quasi-self-organising learning gives slightly better results when a small training size is considered. The work reported has established the viability and basic effectiveness of the GSN concept. The GSN proposal provides a new range of computational units, learning algorithms, architectures, and new concepts related to the fundamental processes of computation using Boolean networks. In all of these ideas further modifications, extensions, and applications can be considered in order fully to establish Boolean neural networks as a strong candidate for solving Boolean-type problems. A great deal of additional research can be identified for immediate investigation as follows. One of the most important contributions of this work is the idea of flexible local goals in RAM-based neurons which allows the application of RAM-based neurons and architectures to a wider range of problems. The definition of the goal seeking functions for all the GSN models used in the feed-forward, feedback and self-organising architectures are important because they provide local goals which try to maximise the memory capacity and to improve the recall of correct output patterns. Although the supervised pattern association learning is not the kind of learning most suitable for use with GSN networks, because it demands multi-presentations of the training set and causes a fast saturation of the neurons' contents, the variety of solutions presented to the problem of conflict of learning can help to achieve correct learning with a relatively small number of activations compared to the traditional way of erasing a path without taking care to keep the maximum number of stored patterns. The partial pattern association, quasi-self-organising, and the self-organising learning have managed to break away from the traditional necessity for many thousands of presentations of the training set, and instead have concentrated on providing one-shot learning. This is made possible by the propagation of the undefined value between the neurons in conjunction with the local goal used in the validating state. Due to the partial coverage area and the limited functionality of the pyramids, which can cause an inability to learn particular patterns, it is important to change the desired output patterns in order to be able to learn these classes. The network produces essentially self-desired output patterns which are similar to the desired output patterns, but not necessarily the same. The differences between the desired output patterns and the self-desired output patterns can be observed in the learning phase by looking at the output values of each pyramid and the desired output values. The definition of the self-desired and the learning probability recall rules have provided a way of sensing the changes in the desired output patterns, and of achieving the required pattern classification. The principle of low connectivity and partial coverage area make possible more realistic VLSI implementations in terms of memory requirements and overall connection complexity associated with the traditional problem of fan-in and fan-out for high connectivity neurons. The feedback architecture is able to achieve associative recall and pattern completion, demonstrating that it is possible to have a cascade of feedback networks that incrementally increases the similarity between a prototype and the output patterns. The utilisation of the freeze feedback operation has given a high percentage of correct convergences and fast stabilisation of the output patterns. The analysis of the saturation problem has demonstrated that the traditional way of using uniform connectivity for all the layers impedes the advance of the learning process and many memory addresses remain unused. This is because saturation is not at the same level for each of the layers. Thus, a new approach has been developed to assign a varied connectivity to the architecture which can achieve a better capacity of learning, a lower level of saturation and a smaller residue of unused memory. In terms of architectures and learning, an important result is the design of the GSN self-organising network which incorporates some principles related to the Adaptive Resonance Theory(ART). The self-organising network contains intrinsic mechanisms to prevent the explosion of the number of clusters necessary for self-stabilising a given training pattern set. Several interesting properties are found in the GSN self-organising network such as: attention, discrimination, generalisation, self-stabilisation, and so on. References @conference{key210, author = "D L Bisset And E C D B C Filho And M C Fairhurst", title = "A Comparative study of neural network structures for practical application in a pattern recognition enviroment", publisher= "IEE", booktitle= "Proc. First IEE International Conference on Artificial Neural Networks", address = "London, UK", month = "October", pages = "378-382", year = "1989" } @conference{key214, author = "E C D B C Filho And D L Bisset And M C Fairhurst", title = "A Goal Seeking Neuron For {B}oolean Neural Networks", publisher= "IEEE", booktitle= "Proc. International Neural Networks Conference", address = "Paris, France", month = "July", volume = "2", pages = "894-897", year = "1990" } @article{key279, author = "E C D B C Filho And D L Bisset And M C Fairhurst", title = "Architectures for Goal-Seeking Neurons", journal= "International Journal of Intelligent Systems", publisher= "John Wiley & Sons, Inc", note = "To Appear", year = "1991" } @article{key280, author = "E C D B C Filho And M C Fairhurst And D L Bisset", title = "Adaptive Pattern Recognition Using Goal-Seeking Neurons", journal= "Pattern Recognition Letters", publisher= "North Holland", month = "March" year = "1991" } All the best, Edson ... Filho -- Before 10-Mar-91 ---------------------------------------------------------- ! Edson Costa de Barros Carvalho Filho ! ARPA: ecdbcf%ukc.ac.uk at cs.ucl.ac.uk ! ! Electronic Engineering Laboratories ! UUCP: ecdbcf at ukc.ac.uk ! ! University of Kent at Canterbury ! Phone: (227) 764000x3718 ! ! Canterbury Kent CT2 7NT England ! ! -- After 10-Mar-91 ----------------------------------------------------------- ! Universidade Federal de Pernambuco ! e-mail: edson at di0001.ufpe.anpe.br ! ! Departamento de Informatica ! Phone: (81) 2713052 ! ! Av. Prof. Luis Freire, S/N ! ! ! Recife --- PE --- Brazil --- 50739 ! ! ------------------------------------------------------------------------------ From fellous%pipiens.usc.edu at usc.edu Mon Feb 25 16:39:48 1991 From: fellous%pipiens.usc.edu at usc.edu (Jean-Marc Fellous) Date: Mon, 25 Feb 91 13:39:48 PST Subject: No subject Message-ID: <9102252139.AA18397@pipiens.usc.edu> Subject: CNE Workshop on Emotions ***************************************************************************** ** C.N.E W O R K S H O P O N E M O T I O N S ** ***************************************************************************** The Center for Neural Engineering of the University of Southern California is happy to announce that its student Workshop on Emotions will be held Monday March 18th from 8.30am to 4.00pm in the Hedco Neuro-Science Building Auditorium (on U.S.C campus). The papers presented will be the following: Affect versus Cognitive-repair Behaviors. Sharon Ruth Gross - U.S.C (Social Psychology) A Mathematical representation of Emotions. Charles Rapp - Illinois Institute of Technology (Computer Science). Cognitive and Emotional disorders in Parkinson's Disease. Peter Dominey - U.S.C (C.N.E, Gerontology). Cognitive-Emotional interaction using subsymbolic paradigm. Aluizio Fausto Ribeiro Araujo - University of Sussex (U.K) (School of cognitive and Computing Sciences) Emotional expressions conceptualized as uniquely effective communication devices Heidi M. Lincer - U.S.C (Psychology). Taxi world: Computing Emotions. Clark Eliott - Northwestern University. (Artificial Intelligence and Cognitive Sciences). Zeal: A Sociological perspective on Emotion, cognition and organizational structure. Gerardo Marti - U.S.C (Gerontology, Sociology) In addition, the following papers have been accepted but will not be presented orally during the Workshop. They will be put on loan during the Workshop. Emotions and autonomous machinery. Douglas A. Kerns - California Institute of Technology (Electrical Engineering). Representation, Action, and Emotion. Michael Travers - M.I.T (Media-Lab). Toward an Emotional Computer: Models of Emotions. Jean-Marc Fellous - U.S.C (C.N.E Computer Science) There will not be any registration fees but, as to get an estimation of the number of persons attending the Workshop, interested people are invited to announce their attendance by email (or surface mail). We remind the participants that this event being a Workshop not a Conference they are strongly encouraged to participate to the debates by their comments and questions to the speakers. Thank you for forwarding this announcement to potentialy interested persons/instituions/mailing_lists. Further informations requests (and email registration) can be addressed to Jean-Marc FELLOUS Center For Neural Engineering University of Southern California U.S.C - University Park Los Angeles CA 90089-2520 U.S.A Tel: (213) 740-3506 Fax: (213) 746-2863 email: fellous at rana.usc.edu .. From R14502%BBRBFU01.BITNET at BITNET.CC.CMU.EDU Wed Feb 27 10:16:14 1991 From: R14502%BBRBFU01.BITNET at BITNET.CC.CMU.EDU (R14502%BBRBFU01.BITNET@BITNET.CC.CMU.EDU) Date: Wed, 27 Feb 91 16:16:14 +0100 Subject: A book on Self-Organization Message-ID: To all interested in the concept of Selforganization ------------------------------------------------------ The subject matter of selforganization has drawn a lot of attention from physicists, chemists, and theoretical biologists before becoming so popular with the Neural Network researches. Anybody interested in the field with few hours to spare may find the basics and typical examples of self-organization of complex systems in an elementary book which I wrote few years ago. The title is: "Molecules, Dynamics and Life: An introduction to self-organization of matter", Wiley, New York, 1986. A. Babloyantz University of Brussels " Dr. Babloyantz has produced an engaging and earnest introduction to the field of self-organization in chemical and biological systems. Dr. Babloyantz proves herself to be a pleasant, practical and reliable guide to new territory which is still largely uncharted and inhospitable to tourists. Her style falls halfway between that found in a popular account and that of a txtbook. She tells her story in a chatty, down-to-earth way, while also giving serious scientific consideration to fundamental issues of the self-organization of matter." (Nature) " The issue of self-organization has at the center of a larger theoretical revolution in physics - the belief that the fundamental laws of nature are irreversible and random, rather than determinstic and reversible. The concepts and processes underlying this new way of thinking are formidable. Molecules, Dynamics and Life makes these concepts and processes accessible, for the first time, to students and researchers in physics, chemistry, biology, and the social sciences." (Physics Briefs) " In Molecules, Dynamics and Life, Dr. Agnes Babloyantz develops a clear and easy to read presentation of this developing field of knowledge. Because only a few advanced research treatises are available so far, this book is especially welcomed. It offers an excellent introduction to an interdisciplinary domain, involving physics and biology, chemistry and mathematics. Obviously, putting together all these topics and making them readable to a large audience was really a challenge." (Biosystem's) " With this fine book Agnessa Babloyantz has provided a successful and welcome summary of what has been accomplished so far in the study of self-organization of matter according to the Prigogine school in Brussels. Dr. Babloyantz's book can be highly recommended to all those interested in self-organization in the fields of chemistry and biochemistry." (Bull. Math. Biology) From gac at cs.brown.edu Wed Feb 27 14:19:59 1991 From: gac at cs.brown.edu (Glenn Carroll) Date: Wed, 27 Feb 91 14:19:59 -0500 Subject: position available Message-ID: <9102271919.AA05483@tolstoy.cs.brown.edu> I'm forwarding this for a friend--please note the BITNET address below for replies. From: CSA::GYULASSY 21-FEB-1991 08:32:48.00 To: CARROLL CC: GYULASSY Subj: net jon Research Position Available Effective March 1,1990 Place: Nuclear Science Division Lawrence Berkeley Laboratory Area: Neural Network Computing Research with Application to Complex Pattern Recognition Problems in High Energy and Nuclear Physics Description: Experiments in high energy and nuclear physics are confronted with increasingly difficult pattern recognition problems, for example in tracking charged particles and identifying jets in very high multiplicity and noisy environments. In 1990, a generic R&D program was initiated at LBL to develop new computational strategies to solve such problems. The emphasis is on developing and testing artificial neural network algorithms. Last year we developed a new Elastic Network type tracking algorithm that is able to track at densities an order of magnitude higher than conventional Road Finding algorithms and even Hopfield Net type algorithms. This year we plan on a number of followup studies and extensions of that work as well as begin research on jet finding algorithm. Jets are formed through the fragmentation of high energy quarks and gluons, via a rare process in high energy collisions of hadrons or nuclei. The problem of identifying such jets via calorimetric or tracking detectors is greatly complicated by the very high multiplicity of fragments produced via other processes. The research will involve developing new preprocessing strategies and network architectures to be trained by simulated Monte Carlo data. Required Qualifications: General understanding of basic neural computing algorithms such as multilayer feed forward and recurrent nets and a variety of training algorithms. Proficiency in programing in Fortran and C on a variety of systems VAX/VMS and/or Sparc/UNIX. Interested applicants should contact Miklos Gyulassy Mailstop 70A-3307 LBL Berkeley, CA 94720 E-mail: GYULASSY at LBL.Bitnet Telephone: (415) 486-5239 From BHAVSAR%UNB.CA at UNBMVS1.csd.unb.ca Thu Feb 28 14:58:50 1991 From: BHAVSAR%UNB.CA at UNBMVS1.csd.unb.ca (V. C. Bhavsar) Date: Thu, 28 Feb 91 15:58:50 AST Subject: Supercomputing Symposium,June3-5,Canada Message-ID: SUPERCOMPUTING SYMPOSIUM'91 June 3-5,1991,Fredericton,N.B.,Canada The fifth annual Canadian Supercomputing Symposium, sponsored by the Canadian Special Interest Group on Supercomputing (SUPERCAN) and the Faculty of Computer Science at U.N.B, will be held in Fredericton, N. B. Canada, From June 3 to 5 1991. Previous symposiums were held in Calgary, Edmonton, Toronto, and Montreal. SS91 promises to be a very exiting event. Over 30 papers have been received as of Feburary28,1991. The invited speakers are: Dr. Andrew Bjerring, Director, Computing & Communication Services University of Western Ontario, London ON, Canada. 'Scientific Computing in Canada'(Tentative title) Dr. John Caulfield, Director, Center for Applied Optics, University of Alabama, Hunstville, AL, USA 'Progress in Quantum Optical Computing' Dr. Narenda Karmarkar, AT&T Bell Labs., Murray Hill, New Jersey, USA 'A New Supercomputer Architecture for Scientific Computation based on Finite Geometries' Dr.K.J.M.Moriarty,Institute for Computational Studies, Dalhousie University,Halifax,N.S.,Canada Topic: Computational Physics Dr. Louise Nielson, Director, High Performance Computing, IBM, Kingston, NY, USA 'Future Directions in High Performance Computing'(Tentative title) (Inaugural Talk) Dr. Richard Peltier, Department of Physics, Universtiy of Toronto, Toronto, ON, Canada. 'Imaging Hydrodynamic Complexity with Supercomputers' ************************ CALL FOR PAPERS Papers are solicited on significant research results in the development and use of supercomputing systems in (but not restrcted to ) the follwing areas : Applications of Supercomputing Neural Computing Supercomputing Algorithms Systems Software Performance Analysis Applications Development Tools Design of Supercomputing Systems Scientific Visualization Optical Computing Supercomputing Education Biomolecular Computing Networks and Communications DEADLINES Mar 11,1991 -Two copies of extended summary (~2 pages) Mar 18,1991 -Notification to authors Apr 18,1991 -Camera-ready copy May 1,1991 -Advance Registration ******************** EXHIBITORS Manufacturers and suppliers of the supercomputers and desktop supercomputer products are encouraged to contact the organizers. ******************* Organizers: Virendra Bhavsar -General Chairman bhavsar at unb.ca Uday G. Gujar -Program Chairman uday at unb.ca Kirby Keyser -Exhibits Chairman kmk at unb.ca John M.DeDourek -Local Arrangements Chairman dedourek at unb.ca Snail Mail: Supercomputing Group Faculty of Computer Science University of New Brunswick Fredericton, N.B., E3B 5A3, Canada Voice: (506) 453-4566 Fax: (506) 453-3566 --------------------- End of forwarded message ---------------------