From tsejnowski at UCSD.EDU Tue Jan 1 16:38:04 1991 From: tsejnowski at UCSD.EDU (Terry Sejnowski) Date: Tue, 1 Jan 91 13:38:04 PST Subject: NEURAL COMPUTATION 2:4 Message-ID: <9101012138.AA20315@sdbio2.UCSD.EDU> NEURAL COMPUTATION Volume 2 Issue 4 NOTES Leonid Kruglyak How to Solve the N Bit Encoder Problem with Just Two Hidden Units Zoran Obradovic and Peiyuan Yan Small Depth Polynomial Size Neural Networks LETTERS Steven D. Whitehead and Dana H. Ballard Active Perception and Reinforcement Learning Lucia M. Vaina, Norberto M. Grzywacz, and Marjorie LeMay Structure from Motion with Impaired Local-Speed and Global Motion-Field Computations Pierre Baldi and Ronny Meir Computing with Arrays of Coupled Oscillators An Application to Preattentive Texture Discrimination Reza Shadmehr Learning Virtual Equilibrium Trajectories for Control of a Robot Arm Michael C. Mozer and Jonathan Bachrach Discovering the Structure of a Reactive Environment by Exploration Alan J. Katz, Michael T. Gately, and Dean R. Collin Robust Classifiers without Robust Features William G. Baxt Use of an Artificial Neural Network for Data Analysis in Clinical Decision-Making: the Diagnosis of Acute Coronary Occlusion Ronald J. Williams and Jing Peng An Efficient Gradient-Based Algorithm for On-Line Training of Recurrent Network Trajectories Michael Georgiopoulos, Gregory L. Heileman, and Juxin Huang Convergence Properties of Learning in ART1 Eric B. Baum A Polynomial Time Algorithm That Learns Two Hidden Unit Nets James A. Reggia and Mark Edwards Phase Transitions in Connectionist Models Having Rapidly Varying Connection Strengths R. Bijjani and P. Das An M-ary Neural Network Model Mark J. Brady Guaranteed Learning Algorithm for Network with Units Having Periodic Threshold Output Function SUBSCRIPTIONS: ***** Last opportunity to obtain all issues in Volume 2 ***** ______ $35 Student ______ $50 Individual ______ $100 Institution Add $12. for postage outside USA and Canada surface mail. Add $18. for air mail. (Back issues of volume 1 are available for $27 each.) MIT Press Journals, 55 Hayward Street, Cambridge, MA 02142. (617) 253-2889. ----- From GOLDFARB%UNB.CA at UNBMVS1.csd.unb.ca Tue Jan 1 22:35:28 1991 From: GOLDFARB%UNB.CA at UNBMVS1.csd.unb.ca (Lev Goldfarb) Date: Tue, 01 Jan 91 23:35:28 AST Subject: On the shape of things to come Message-ID: In view of the multidisciplinary background of members of this mailing list and, more particularly, in view of some resent remarks (which I don't want to quote since they may appear to have been taken out of the context) about the specific ways of proceeding to our common goal -- an analytical model of an "intelligent" system -- I feel compelled to draw your attention to the following apparent paradox, if we are to accept a more "direct" or "hands on" approach to the modeling of the brain. Most of us would probably agree that to model "intelligent" (biological, irreversible) processes a fundamentally new mathematical models are necessary. Is it possible then to proceed to construct these models in a piecemeal fashion using "pieces" of the old mathematical models (that were constructed for entirely different purposes) as one is compelled to do under the "direct approach" philosophy? If it has been impossible to do this within the confines of the same science, physics, it will prove to be even less possible to do so for the biological processes. I for one would be much less sure of the feasibility of the new model proposed by me if one could show that it is reducible to one of the existing mathematical models. Incidentally, the NN models are not new mathematical models. --Lev Goldfarb From VAINA at buenga.bu.edu Wed Jan 2 11:13:00 1991 From: VAINA at buenga.bu.edu (VAINA@buenga.bu.edu) Date: Wed, 2 Jan 91 11:13 EST Subject: The computing Brain lecture series-at BU: Engineering Message-ID: From: BUENGA::CORTEX 20-DEC-1990 16:02 To: @CORTEX-NEW,IN%CORTEX-IN-DISTRIBUTION CC: CORTEX Subj: COMPUTING BRAIN LECTURE SERIES - Marvin Minsky *************************************************************************** THE COMPUTING BRAIN LECTURE SERIES *************************************************************************** " SOCIETY OF MIND II " MARVIN MINSKY Toshiba Professor of Media Arts and Sciences Massachusetts Institute of Technology Wednesday, January 9, 1991 at 4 pm Old Engineering Building - Room 150 110 Cummington Street, Boston, Ma Tea at 3 pm in Room 129 (same address as above) Lecture open to all For further information contact: Professor Lucia M. Vaina 353-2455 or vaina at buenga.bu.edu *************************************************************************** From jbower at smaug.cns.caltech.edu Wed Jan 2 13:49:15 1991 From: jbower at smaug.cns.caltech.edu (Jim Bower) Date: Wed, 2 Jan 91 10:49:15 PST Subject: flag waving Message-ID: <9101021849.AA09749@smaug.cns.caltech.edu> I am extremely loath to continue to participate in this debate in that it HAS become too redundant, may very well have been a waste of time to begin with, and is fast becoming an exercise in patriotic flag waving (Paul Munro's ridiculous equivalence for example). If anyone is still unclear about what I am, or am not trying to say, or wants to take personal exception to some comment I have made, please communicate with me directly. But I would like to make two brief comments with respect to Lev Goldfarb's posting. First, it is not clear to me that an "analytical model of an intelligent system" is the common goal of the people on this mailing list. Second, the assumption that a "direct approach" to modeling the brain is limited to old tools is simply not correct. Both Terry Sejnowski's example from history, or the current work of Nancy Kopel (BU) and Bard Ermentrout (UPitt) are examples to the contrary. It may even be that taking a direct approach applies more pressure to come up with new mathematical tools, especially if one has learned enough about the biology to know that the old models are off the mark. Jim Bower From simic at kastor.ccsf.caltech.edu Wed Jan 2 14:02:55 1991 From: simic at kastor.ccsf.caltech.edu (Petar Simic) Date: Wed, 2 Jan 91 19:02:55 GMT Subject: There is no use for bad data Message-ID: <1991Jan2.190255.6920@nntp-server.caltech.edu> Connectionism and NN made a bold guess of what should be the zeroth order approximation in modeling neural information processing, and I think that the phrase 'biological plausibility' (or 'neural style') is meant to indicate, (1) fine-grained parallel processing, (2) reasonable although no-doubt crude model-neuron . While I heard that some not-only-IN PRINCIPLE intelligent people think that parallelism is 'JUST a question of speed', I think that so defined biological plausibility is not to be discounted in discussing the differences between the two approaches to modeling intelligence, AI and Connectionism/NN&ALL THAT. Perhaps the phrase 'biological plausibility' should be changed to 'physical plausibility' indicating that Connectionist/NN models have natural implementation in physical hardware. This is not to say that Connectionism/NN&ALL THAT should pose as biology. It doesn't need to ---it has its own subject of study (which in combination with traditional AI makes what one may call 'modern AI') and it need not be theology (J.B) or pure math, providing it is willing to impose on itself some of the hard constraints of either engineering, or natural science (or both). The engineering constraint is that understanding of intelligence should not be too far from building it (in software or physical hardware). This is not the unfamiliar constraint for connectionists, and is also healthy part of the traditional AI, since they both make computational models, and as they move from toy problems to larger scales, concepts and representations they develop should be softly constrained by the plausibility of implementation today, or perhaps tomorrow. The natural science constraint is in traying to reverse-engineer the real brain (J.B), but I would suggest that this constraint is 'softer' then what seem to be suggested by Jim Bower, and that the reason for this is not in our underestimate of the complexity of the brain, but in the theoretical depth of the reverse-engineering problem. I think that at the level of the detailed modeling of specific neural circuits, Connectionism/NN provide a set of tools which may or may not be useful, depending on the problem at hand, and how these tools are applied. The interesting question, therefore, is not how useful is Connectionism/NN to neurobiology ---how useful is sharp pencil or PC for neurobiology? --- but how useful is neurobiology, as practiced today, to the study of the information processing phenomena across the levels, in natural (and, why not, artificial) systems. I would think that the present situation in which Connectionist/NN models are ignoring many of the 'details' should be a source of challenge to theoretically minded neurobiologists ---especially to the ones who think that the theoretical tools needed to describe the transition between between the levels are just the question of reading a chapter from some math textbook ---and that they should come up with computational models and convince everybody that particular detail does matter, in the sense that it is 'visible' at the higher information processing level, and can account for some useful computational phenomena which simplified model can not, or can but in a more complicated way. Modeling firmly rooted in biological fact if to detailed, might not be directly useful as modeling-component at higher level, except as a good starting point for simplification. That simplifications are essential, not DESPITE but BECAUSE the complexity of the brain, should be self-evident to all who believe that an understanding of the brain is possible. What is not self-evident is which details should not be thrown out, and how are they related to the variables useful at the higher information processing level. Even in simple systems such as the ones studied in physics, where one knows exactly the microscopic equations, the question of continuity between the correct theoretical descriptions at two different levels is very deep one. Just "..a cursory look at the brain ..." (J.B.) should not be enough to disqualify simplified phenomenological models which are meant to describe phenomenology at higher (coarser) level, as wrong. For example, if you want to model the heat distribution in some material, you use macroscopic heat-equation, and the basic variable there (the temperature) is related, but in rather unobvious way, to moving particles and their collisions, which are microscopic details of which heat is made of. Yet, the heat equation is the correct phenomenological description of the 'heat processing phenomenology'. This is not to say that the very simplified connectionist/NN models are good, but just warning that the theoretical modeling across the different levels of description is as hard as it is fascinating, it needs to be attacked simultaneously from all ends, and more often then not, one finds that the relationship between the microscopic variables and the variables useful at the next level, is neither obvious nor simple. I would suggest that if we are to understand the brain and the connection between its levels, one should not underestimate the theoretical depth of the problem. While it is definitely a good idea for a theorist to know well the phenomenology he is theorizing about, and it is an imperative for him, in this field, to build computational models, I think that it is somewhat silly to condition the relevance of his work on his ability to do neurobiological experiments. There is no use for bad data. From clee at ICSI.Berkeley.EDU Wed Jan 2 14:32:50 1991 From: clee at ICSI.Berkeley.EDU (Connie Lee) Date: Wed, 2 Jan 91 11:32:50 PST Subject: TR request Message-ID: <9101021932.AA04164@icsid.Berkeley.EDU> From Mailer-Daemon at icsib.Berkeley.EDU Wed Jan 2 14:30:33 1991 From: Mailer-Daemon at icsib.Berkeley.EDU (Mail Delivery Subsystem) Date: Wed, 2 Jan 91 11:30:33 PST Subject: Returned mail: User unknown Message-ID: <9101021930.AB20207@icsib> ----- Transcript of session follows ----- Connected to cs.cmu.edu: >>> RCPT To: <<< 550-(USER) Unknown user name in "connectionist at cs.cmu.edu" <<< 550-Some of the nearest matches are: <<< 550- <<< 550- connectionists-archive (connectionists-archive) <<< 550- Connectionists (connectionists) <<< 550- connectionists-request-list (connectionists-request-list) <<< 550- connectionist-request (connectionist-request) <<< 550- connectionist-requests (connectionist-requests) <<< 550- Connectionists-Requests (connectionists-requests) <<< 550- Connectionists-Request (Connectionists-Request) <<< 550 550 ... User unknown ----- Unsent message follows ----- From clee at ICSI.Berkeley.EDU Wed Jan 2 14:31:06 1991 From: clee at ICSI.Berkeley.EDU (Connie Lee) Date: Wed, 2 Jan 91 11:31:06 PST Subject: TR request Message-ID: <9101021931.AA04142@icsid.Berkeley.EDU> Please send Report FKI-140-90 "Beyond Hebb Synapses: Biological Building Blocks for Unsupervised Learning in Artifician Neural Networks by Patrick V. Thomas to: Dr. Jerome A. Feldman Director International Computer Science Institute 1947 Center Street Berkeley, CA 94704 Thank you, Connie Lee Admin. Assist. From jbower at smaug.cns.caltech.edu Wed Jan 2 15:27:39 1991 From: jbower at smaug.cns.caltech.edu (Jim Bower) Date: Wed, 2 Jan 91 12:27:39 PST Subject: Summer course Message-ID: <9101022027.AA09884@smaug.cns.caltech.edu> Summer Course Announcement Methods in Computational Neurobiology August 4th - August 31st Marine Biological Laboratory Woods Hole, MA This course is for advanced graduate students and postdoctoral fellows in neurobiology, physics, electrical engineering, computer science and psychology with an interest in "Computational Neuroscience." 20 such students will be enrolled. In addition, this coming summer the course has allocated 5 additional positions for participants who are currently faculty members at Universities or are established members of industrial research organizations. For both faculty and student participants, a background in programming (preferably in C or PASCAL) is highly desirable and basic knowledge of neurobiology is required. Limited to 20 students. This four-week course presents the basic techniques necessary to study single cells and neural networks from a computational point of view, emphasizing their possible function in information processing. The aim is to enable participants to simulate the functional properties of their particular system of study and to appreciate the advantages and pitfalls of this approach to understanding the nervous system. The first section of the course focuses on simulating the electrical properties of single neurons (compartmental models, active currents, interactions between synapses, calcium dynamics). The second section deals with the numerical and graphical techniques necessary for modeling biological neuronal networks. Examples are drawn from the invertebrate and vertebrate literature (visual system of the fly, learning in Hermissenda, mammalian olfactory and visual cortex). In the final section, more abstract models relevant to perception and learning in the mammalian cortex, as well as network learning algorithms will be analyzed and discussed from a neurobiological point of view. The course includes lectures each morning and a computer laboratory in the afternoons and evenings. The laboratory section is organized around GENESIS, the Neuronal Network simulator developed at the California Institute of Technology, running on 20 state-of-the-art, single-user, graphic color workstations. Students initially work with GENESIS-based tutorials and then are expected to work on a simulation project of their own choosing. Co-Directors: James M. Bower Christof Koch, Computation and Neural System Program California Institute of Technology 1991 summer faculty: Ken Miller UCSF Paul Adams Stony Brook Idan Segev Jerusalem John Rinzel NIH Richard Andersen MIT David Van Essen Caltech Scot Fraser* Caltech Kevin Martin Oxford Eve Marder* Brandis Nancy Kopell Boston U. Avis Cohen Cornell Rudolfo Llinas NYU Terry Sejnowski UCSD/Salk Chuck Stevens* UCSD/Salk Ted Adelson MIT David Zipser* UCSD *tentative Application deadline: May 15, 1991 Applications are evaluated by an admissions committee and individuals are notified of acceptance or non-acceptance by June 1. Tuition: $1,000 (includes room & board). Financial aid is available to qualified applicants. For further information contact: Admissions Coordinator Marine Biological Laboratory Woods Hole, MA 02543 (508) 548-3705, ext. 216 From josh at flash.bellcore.com Wed Jan 2 14:54:54 1991 From: josh at flash.bellcore.com (Joshua Alspector) Date: Wed, 2 Jan 91 14:54:54 -0500 Subject: NN, AI, biology Message-ID: <9101021954.AA05701@flash.bellcore.com> On the question of NN vs. AI, I think one point that needs emphasizing is that much of the difference between these two views of intelligence arises from physical implementation, and the convenience of modeling based on this underlying structure. A computer program has the capability of simulating the behavior of a physical system including neural models but it may take a long time. Traditional symbolic AI is much better suited to the processes that an arithmetic and logic unit in a computer can perform. On the other hand, a computer's logic and memory are implemented using transistors arranged in circuits in such a way that they act digitally but are really fundamentally analog and messy like neurons (well not THAT messy). One could build a digital computer from biological neurons if the technology existed to manipulate them. Since neural systems and symbol-manipulating computers can simulate each other, NN and AI are fundamentally equivalent in their descriptive powers. But each description has its advantages. Because logic circuits can be implemented in a wide variety of technologies (CMOS, bipolar, optical, neural), it is natural to ignore this level of description. One can further abstract away the logical structure (bus width, registers, instruction set) by a compiler that allows us to work in a high-level language. Here, we get into the doctrine of software separability. Software is everything, hardware doesn't matter. Neural networks are much messier, and the levels of description cannot be separated so easily. Slips of the tongue confuse low-level phonetic and articulatory information with high-level linguistic information, something you would have to sully an AI program to do. The computation here reflects the hardware (wetware). Because of the equivalence of NN and AI, NN cannot do anything that AI (computers) can't except to do it faster in a parallel implementation. As has been pointed out, the NN models are not new mathematically. Neural networks are a biological inspiration for how to physically do parallel processing of information. The qualities of NN that are awkward (but not impossible) to model with AI have to do with the physical nature of networks. These include speed in a parallel implementation, natural time scales, network dynamics, and yes, also a functional description of relevance to biological neural networks. Being an experimental neuroscientist is not the only way to understand brains. Suppose we had an intelligent AI system implemented on a digital computer and had no idea how it worked. We would not get far by sticking many oscilloscope probes into it and watching the results as we ask it questions. Josh Alspector josh at bellcore.com From well!mitsu at apple.com Thu Jan 3 04:28:38 1991 From: well!mitsu at apple.com (Mitsuharu Hadeishi) Date: Thu, 3 Jan 91 01:28:38 pst Subject: Connectionism vs. AI Message-ID: <9101030928.AA08049@well.sf.ca.us> I'm not sure how important the "equivalence" between NNs and Turing machines really is, given the fact that the space of all algorithms is hardly exhaustively searchable for any but the most trivial of problems. Obviously NNs impose a particular structure to the space of algorithms which allows systematic searching, whereas traditional AI approaches rely on hand-crafted algorithms. This structure is what makes co\nectionism important; not what is computable in principle, but what is computable because we can find the appropriate algorithm(s) to compute it. Mitsu Hadeishi Open Mind mitsu at well.sf.ca.us apple!well!mitsu From Dave.Touretzky at DST.BOLTZ.CS.CMU.EDU Thu Jan 3 01:36:58 1991 From: Dave.Touretzky at DST.BOLTZ.CS.CMU.EDU (Dave.Touretzky@DST.BOLTZ.CS.CMU.EDU) Date: Thu, 03 Jan 91 01:36:58 EST Subject: Lakoff paper on "Metaphor and War" available Message-ID: <23527.662884618@DST.BOLTZ.CS.CMU.EDU> Some of you, partiicularly those who attend Cognitive Science or participated in the 1988 or 1990 connectionist summer schools, have met George Lakoff, a linguist at Berkeley who is interested in connectionist matters. His work on phonology provided the inspiration for my own research efforts in this area, and his book "Women, Fire, and Dangerous Things" is must reading for people in cognitive science. (When I first heard of it I thought the book was about feminism, but the title actually comes from a true anecdote about a primitive people whose mythology leads them to put the words for women, fire, and dangerous things in the same linguistic class. The sun is female, you see.) Anyway, George works on many interesting problems related to language, including the role of metaphor in our ability to understand and talk about abstract concepts. He has written a paper about the metaphors that underlie our understanding of the current conflict in the Persian Gulf. People who are interested in cognitive science might want to take a look at it. He is particularly eager for people to have a chance to read it before January 15. The paper is too long to post here, and in any case it's too divorced from connectionism to be appropriate for this list, but you can retrieve it from the neuroprose directory at Ohio State. Retrieval instructions appear at the end of this message. A copy has also been posted to comp.ai. Or you can receive a copy by sending email to George at lakoff at cogsci.berkeley.edu. Only the first half of the paper deals with metaphor. The second half is political analysis. It will no doubt be controversial. But I'm warning all readers right now: I will not permit discussions of Persian Gulf politics on this mailing list. Read the paper at your own risk; send mail to George if you like; start a discussion on talk.politics.misc or alt.desert-shield or whatever newsgroup you feel is appropriate, but keep it *off* the CONNECTIONISTS mailing list! I will do unspeakably nasty things to anyone who violates this rule. The only reason I'm permitting the paper to be announced here at all is because of its cognitive science content. -- Dave Touretzky ================ How to retrieve the paper from Neuroprose ================ 1. Open an FTP connection to 128.146.8.62 (cheops.cis.ohio-state.edu) 2. Login as user "anonymous", password "neuron" 3. cd /pub/neuroprose 4. type binary 5. get lakoff.war.ps.Z 6. bye 7. uncompress lakoff.war.ps.Z 8. Then send the file lakoff.war.ps to your local PostScript printer. From simic at kastor.ccsf.caltech.edu Wed Jan 2 20:28:19 1991 From: simic at kastor.ccsf.caltech.edu (Petar Simic) Date: Thu, 3 Jan 91 01:28:19 GMT Subject: There is no use for bad data Message-ID: <1991Jan3.012819.16241@nntp-server.caltech.edu> Connectionism and NN made a bold guess of what should be the zeroth order approximation in modeling neural information processing, and I think that the phrase 'biological plausibility' (or 'neural style') is meant to indicate, (1) fine-grained parallel processing, (2) reasonable although no-doubt crude model-neuron . While I heard that some not-only-IN PRINCIPLE intelligent people think that parallelism is 'JUST a question of speed', I think that so defined biological plausibility is not to be discounted in discussing the differences between the two approaches to modeling intelligence, AI and Connectionism/NN&ALL THAT. Perhaps the phrase 'biological plausibility' should be changed to 'physical plausibility' indicating that Connectionist/NN models have natural implementation in physical hardware. This is not to say that Connectionism/NN&ALL THAT should pose as biology. It doesn't need to ---it has its own subject of study (which in combination with traditional AI makes what one may call 'modern AI') and it need not be theology (J.B) or pure math, providing it is willing to impose on itself some of the hard constraints of either engineering, or natural science (or both). The engineering constraint is that understanding of intelligence should not be too far from building it (in software or physical hardware). This is not the unfamiliar constraint for connectionists, and is also healthy part of the traditional AI, since they both make computational models, and as they move from toy problems to larger scales, concepts and representations they develop should be softly constrained by the plausibility of implementation today, or perhaps tomorrow. The natural science constraint is in traying to reverse-engineer the real brain (J.B), but I would suggest that this constraint is 'softer' then what seem to be suggested by Jim Bower, and that the reason for this is not in our underestimate of the complexity of the brain, but in the theoretical depth of the reverse-engineering problem. I think that at the level of the detailed modeling of specific neural circuits, Connectionism/NN provide a set of tools which may or may not be useful, depending on the problem at hand, and how these tools are applied. The interesting question, therefore, is not how useful is Connectionism/NN to neurobiology ---how useful is sharp pencil or PC for neurobiology? --- but how useful is neurobiology, as practiced today, to the study of the information processing phenomena across the levels, in natural (and, why not, artificial) systems. I would think that the present situation in which Connectionist/NN models are ignoring many of the 'details' should be a source of challenge to theoretically minded neurobiologists ---especially to the ones who think that the theoretical tools needed to describe the transition between between the levels are just the question of reading a chapter from some math textbook ---and that they should come up with computational models and convince everybody that particular detail does matter, in the sense that it is 'visible' at the higher information processing level, and can account for some useful computational phenomena which simplified model can not, or can but in a more complicated way. Modeling firmly rooted in biological fact if to detailed, might not be directly useful as modeling-component at higher level, except as a good starting point for simplification. That simplifications are essential, not DESPITE but BECAUSE the complexity of the brain, should be self-evident to all who believe that an understanding of the brain is possible. What is not self-evident is which details should not be thrown out, and how are they related to the variables useful at the higher information processing level. Even in simple systems such as the ones studied in physics, where one knows exactly the microscopic equations, the question of continuity between the correct theoretical descriptions at two different levels is very deep one. Just "..a cursory look at the brain ..." (J.B.) should not be enough to disqualify simplified phenomenological models which are meant to describe phenomenology at higher (coarser) level, as wrong. For example, if you want to model the heat distribution in some material, you use macroscopic heat-equation, and the basic variable there (the temperature) is related, but in rather unobvious way, to moving particles and their collisions, which are microscopic details of which heat is made of. Yet, the heat equation is the correct phenomenological description of the 'heat processing phenomenology'. This is not to say that the very simplified connectionist/NN models are good, but just warning that the theoretical modeling across the different levels of description is as hard as it is fascinating, it needs to be attacked simultaneously from all ends, and more often then not, one finds that the relationship between the microscopic variables and the variables useful at the next level, is neither obvious nor simple. I would suggest that if we are to understand the brain and the connection between its levels, one should not underestimate the theoretical depth of the problem. While it is definitely a good idea for a theorist to know well the phenomenology he is theorizing about, and it is an imperative for him, in this field, to build computational models, I think that it is somewhat silly to condition the relevance of his work on his ability to do neurobiological experiments. There is no use for bad data. Petar Simic simic at wega.caltech.edu From jbower at smaug.cns.caltech.edu Thu Jan 3 14:49:30 1991 From: jbower at smaug.cns.caltech.edu (Jim Bower) Date: Thu, 3 Jan 91 11:49:30 PST Subject: analogies Message-ID: <9101031949.AA13327@smaug.cns.caltech.edu> I would like to know when I said that one had to be an experimental neurobiologist to do computational neurobiology. It is really unbelievable how defensive the reaction has been to my comments. Just for the record, Christof Koch, Carver Mead, Nancy Kopel, Bard Ermentrout, Wilfrid Rall, Jack Cowan, Idan Segev, and many other NON-experimentalists have made important contributions to computational neurobiology. They have also invested a tremendous amount of time educating themselves about the detailed structure of the nervous system on their own and through interactions with experimental neurobiologists. And I don't mean listening to invited talks at neural net conferences. It is absolutely bizarre that claims of scientific relevance and biological inspiration are made and accepted by a field largely composed of people who know very little about the nervous system, think that it can be ignored or regard it as simply one implementation alternative, and generally appear to have little real interest in the subject. Two other remarks. The computer and the oscilloscope analogy is a terrible one. The point that Josh correctly made in the beginning of his comment is that there is a rather poor mapping between neural network algorithms and digital computer architecture. I think that it can also be argued that a lot of the most interesting work in neural networks is on the side of implementation, i.e. how one constructs hardware that reflects the algorithms of interest. The brain almost certainly has taken this association to an extreme which is probably closely related to its spectacular efficiency and power. The form reflects the function. An electrode in a computer is a mess because the computer is a relatively low level computing device that trades generality for efficiency. The brain is a different situation altogether. For example, we increasingly suspect, based on detailed computational modeling of brain circuits, that principle computational features of a circuit are reflected at all its organizational levels. That is, if a network oscillates at 40 Hz, that periodicity is seen at the network, single cell, and subcellular levels as well as in the structure of the input and output. That means that sticking the electrode anywhere will reveal some aspect of what is probably an important functional property of the network. Second, the standard particle in a box analogy mentioned by Kastor is even worse. Neurons can not be considered particles in a box. This even goes against fundamental assumptions underlying connectionism. This is one of, if not the most difficult problem associated with changing levels of abstraction when modeling the brain. It also means that the best examples of success in theoretical physics may not directly apply to understanding the nervous system. We will see. Finally, with respect to the original AI / NN Connectionist debate. I just received an advertisement from the periodical "AI Expert" that offers with a subscription "five disk-based versions of some of the most important AI programs". You guessed it, "Vector classifier, Adaline/Perceptron, Backpropagation, Outstar Network, and Hopfield Network", presented as "real world examples of working AI systems". I think that settles the issue, NNs etc has now become part of AI marketing. How much closer can you get. Jim Bower From ken at chagall.cns.caltech.edu Fri Jan 4 03:37:54 1991 From: ken at chagall.cns.caltech.edu (Ken Miller) Date: Fri, 4 Jan 91 00:37:54 -0800 Subject: arguments for all seasons Message-ID: <9101040837.AA03168@chagall.cns.caltech.edu> With respect to the great NN/AI/Neurobiology debates: While history serves as a guide to some of the possibilities of the future, it by no means limits them. Furthermore, one can within history find examples to suit most prejudices. Some examples: (1) In the late 1800's some scientists conceived the beautiful idea that the fundamental physical entities were vortices in some medium. It seemed tantalizingly as though such a program could explain all of physics. Working seriously on this idea, a full mathematics of vortices could probably have been developed, albeit one that would always have had some major difficulties in explaining reality. Without experimental studies of atomic physics there was no chance of anyone inventing the real explanation, quantum mechanics, through pure thought. Score one for the neurobiologists. (2) The thermodynamics/stat mech example. An extremely good science of gases, liquids, etc. (thermodynamics) evolved from observation of those things without any knowledge of the underlying atomic and molecular physics. When consideration of atomic level things led eventually to statistical mechanics, it was possible to derive thermodynamics from stat mech; but nobody would ever have found thermodynamics from stat mech alone; knowledge of the laws at the thermodynamic level was needed to find those laws within stat mech. This phenomena of finding the higher-level laws in the lower level ONLY through guidance by prior knowledge of the higher-level laws has occured repeatedly in modern physics. Score one for the NN and AI types --- but only if they are strongly guided by the phenomenological, empirical study of intelligence, perception, motor behavior, or whatever they are modeling. (3) Heredity, like intelligence, was once a great soupy mess. People had lots of complicated, dynamical ideas of how it was accomplished. Detailed study of the biology finally led to a simple structural explanation --- the structure of DNA --- that was largely unanticipated (yes I know about Schrodinger -- who anticipated certain things in an abstract way but not in a way that enabled any useful understanding of heredity). Score one for the neurobiologists. On the other hand, many details of heredity --- i.e. genetics --- were worked out without this molecular-level knowledge, including Barbara McClintock's ``jumping genes" (a discovery that was not widely acknowledged until a molecular-level explanation was found 20 years later, at which time she finally got the Nobel prize). Score one for those studying at a phenomenological level. (4) Einstein took a pre-existing mathematics, essentially differential geometry, and applied it to the invention of general relativity. Similarly, modern field theorists have found the largely preexisting mathematics of knot theory to be crucial to the understanding of superstrings. Score one for those who believe development of mathematical tools in non-neurobiological contexts may aid neurobiologists. (5) Many aspects of modern mathematics were first invented by physicists trying to solve particular physics problems; later they was cleaned up, rigorized, and generalized by the mathematicians. Score one for those who believe the mathematical tools relevant to neurobiology may only be found through attempts to model neurobiology. Although, note that many of these tools were developed in studying "toy" models that at best only caricature one aspect of the real physical problem. So score one for everybody. I could go on and on. We could all go on and on. There's enough history and abstract arguments for everyone. Personally, I hold these truths to be self-evident: (1) Intelligence, like quantum mechanics, is too strange, difficult, complex or what-have-you to be understood by pure thought alone. (2) Insights into intelligence will come both from studying it at its own phenomenological level, and by studying the physical structures (i.e. the brain) that are known to realize it. Personally, I'm putting my bets on studying the brain, but that's just a personal decision. (3) Development of toy models is useful to neurobiologists. Until connectionist models came into being, no one had a solid, non-vague, working model of how a parallel distributed system might represent and transform information. Since experiments are necessarily framed in terms of whatever concepts and metaphors are at hand, connectionist models have had and will continue to have an important influence on systems neurobiology. This does not mean that the toy models are necessarily biological, only that they usefully expand the thinking tools available to the working neurobiologist. [on the other hand, the lack of relevance of the DETAILS of these models to the neurobiologists is suggested by the great lack of working neurobiologists on this net.] (4) Neurobiology is useful to NN/AI types. Again, not too many of the details at any point in time are considered by the NN/AI types, but the overall progress of neurobiology leads to ideas that are important to those trying to engineer or theoretically understand intelligence. (5) Insights and influences will run in all possible directions, and no one can predict for sure what will turn out to be useful to who. We all place our bets by the work we choose to do. (6) None of us have the foggiest idea how the brain, or real intelligence, works. Therefore, we would all be wise to be humble and to listen well. Ken Miller ken at descartes.cns.caltech.edu From GOLDFARB%unb.ca at UNBMVS1.csd.unb.ca Fri Jan 4 01:03:39 1991 From: GOLDFARB%unb.ca at UNBMVS1.csd.unb.ca (GOLDFARB%unb.ca@UNBMVS1.csd.unb.ca) Date: Fri, 04 Jan 91 02:03:39 AST Subject: AI, NN, CNS (central nervous system) In-Reply-To: Message of Fri, 21 Dec 90 01:00:53 EST from Message-ID: Terry : > It should also be > noted that Hartline and Ratliff would not have been able to > develop their model if the mathematics of linear networks had > not already been established by mathematicians, physicists, and > engineers, most of whom were not interested in biological problems. > Without the development of a mathematics of nonlinear dynamical > systems there will be no future models for future Hartlines > and Ratliffs to apply to future biological problems. I find > it encouraging that so many good scientists who are confronting > so many difficult problems in psychology, biology and computation > are begining to at least speak the same mathematics. > > I do not think that anything is going to be settled by > debating ideologies, except who is the better debater. Precious > bandwidth is better spent discussing specific problems. I believe that on closer examination the above remarks disclose an important contradiction, which, in the first place, was an the impetus to the "ideological debate". First, by no means should the AI/NN "debate" be viewed as "ideological", but rather as one about the need for a new mathematical model the importance of which was stressed in the first paragraph. Second, why is it that "without the development of a mathematics of nonlinear dynamical systems there will be no future models for Hartlines and Ratliffs to apply to future biological problems"? Third, I can't see where "many good scientists who . . . are beginning to at least speak the same mathematics" are, and what this mathematics is. Finally, since some of us (perhaps including even yourself) believe that in the absence of "the same mathematics" the "precious bandwidth is better spent" discussing the possible "shape" of the new model, may be you (against yourself) can explain how one can "discuss specific problems" independent of mathematical models, and what these specific problems are. --Lev From GOLDFARB%unb.ca at UNBMVS1.csd.unb.ca Fri Jan 4 02:39:38 1991 From: GOLDFARB%unb.ca at UNBMVS1.csd.unb.ca (GOLDFARB%unb.ca@UNBMVS1.csd.unb.ca) Date: Fri, 04 Jan 91 03:39:38 AST Subject: flag waving In-Reply-To: Message of Wed, 2 Jan 91 13:49:15 EST from Message-ID: Jim, May I take the great responsibility to assure you that it has not "been a waste of time to begin with". But I'm "still unclear about what" you are and, in fact, what all of us are. "First, it is not clear to that an 'analytical model of an intelligent system' is the common goal of the people on this mailing list", then what is the common goal of the people on this mailing list? (see also the von Neumann's quotation in my earlier posting) Second, "one has learned enough about the biology to know that the old models are off the mark". --Lev Goldfarb From chan%unb.ca at UNBMVS1.csd.unb.ca Fri Jan 4 11:26:25 1991 From: chan%unb.ca at UNBMVS1.csd.unb.ca (Tony Chan) Date: Fri, 04 Jan 91 12:26:25 AST Subject: AI vs. NN Message-ID: "NN and AI are fundamentally equivalent in their descriptive powers." So what? Formalisms such as Post systems, Chomsky's type-0 grammars, Lev Goldfarb's transformation systems, McCarthy's Lisp, Wirth's Pascal, RAM, and thousands upon thousands of other formalisms can compute or describe as much as Turing machines can. The more interesting questions, to me, are 1) What are the forces that cause such proliferation? 2) What are the differences among these formalisms? 3) What are the unique properties about each? 4) How do we categorize them? The most important issue for us, I believe, is that we want to have a formalism that simulate learning so that if we use this formalism to describe some "intelligent" (learning) processes, it is relatively easy to express it using this formalism. Also, this formalism cannot be an ad hoc one because its purpose is to model learning and learning is a very general phenomenon. To some extent, the neural net formalism fits the bill because of its limited self-programability and adaptability. Unfortunately, it lacks generality in the sense that it is not well-suited for high-level/ symbolic type of learning. This, partly, is why I believe a more general formalism such as Reconfigurable Learning Machines should be called for or at least debated! From gary at CS.CMU.EDU Fri Jan 4 15:51:21 1991 From: gary at CS.CMU.EDU (gary (Gary Cottrell)) Date: Fri, 4 Jan 91 12:51:21 PST Subject: Correct reference Message-ID: <9101042051.AA27731@desi.ucsd.edu> I get enough requests for this that I feel it is necessary to post it here. Sorry if you don't care! Somehow, the correct reference for the image compression paper we put in Sharkey's editied volume has been scrambled "out there". Here it is: Cottrell, G., Munro, P. and Zipser D. (1989) Image compression by back propagation: A demonstration of extensional programming. In Noel Sharkey (Ed.), \fIModels of Cognition: A review of Cognitive Science, Vol. 1\fP, Norwood: Ablex. From hinton at ai.toronto.edu Fri Jan 4 16:56:48 1991 From: hinton at ai.toronto.edu (Geoffrey Hinton) Date: Fri, 4 Jan 1991 16:56:48 -0500 Subject: A good textbook on neural computation Message-ID: <91Jan4.165653edt.1072@neuron.ai.toronto.edu> Ever since the binding fell apart on my copy of Rumelhart and McClelland I have been looking for a more recent graduate textbook on neural computation (the artificial kind). I imagine that quite a few of the other people on this mailing list have the same problem for their courses. There are a lot of attempts at textbooks out there, and many of the attempts are very good in one respect or another, but (in my opinion) none of them gives good clear coverage of most of the main ideas. However, I just got hold of a really good textbook (in my opinion). It is: Introduction to the Theory of Neural Computation by J. Hertz, A. Krogh and R. Palmer Addison Wesley, 1991. The beginning and end of the book are rather biased towards the physicists view of the world (in my opinion), but the middle covers a lot of the basic material very nicely. Geoff Hinton PS: If you object to getting people's opinions about textbooks, please complain to me, not to the whole mailing list. From simic at kastor.ccsf.caltech.edu Fri Jan 4 18:03:39 1991 From: simic at kastor.ccsf.caltech.edu (Petar Simic) Date: Fri, 4 Jan 91 23:03:39 GMT Subject: ap Message-ID: <1991Jan4.230339.26806@nntp-server.caltech.edu> Apology to all who, due to some problem with local server at CalTech, had to read twice my recent message. I hope that this will not happen again. Petar Simic From tsejnowski at UCSD.EDU Sat Jan 5 14:59:29 1991 From: tsejnowski at UCSD.EDU (Terry Sejnowski) Date: Sat, 5 Jan 91 11:59:29 PST Subject: AI (discrete) moodel and NN (continuous) model Message-ID: <9101051959.AA13566@sdbio2.UCSD.EDU> Regarding my comments about ideologies vs concrete problems, I was objecting to arguments about semantics (what is and what isn't AI) and the right problem to study (biology vs machines) when it is clear that different people are interested in different problems and you arn't going to solve any problems by arguing about taste. Ken Miller made this point much better than I did. Regarding the issue of mathematics vs problems, there are quite a lot of problems that can be attacked with what is already known about nonlinear dynamical systems (including the present generation of recurrent neural nets). If new mathematics is needed someday we can create it, but my own preference is to concentrate on interesting problems and to let the problem, whether it is a computational, psychological, or a biological one, guide you to the right assumptions. The difficulty with simply looking for new mathematics is that the number of models and mathematical formalisms is infinite and without some guidance the chances are you will be studying vortices. I agree here with Jim Bower and Ken Miller. Therefore, debating ideologies is less productive than discussing interesting research problems. Regarding interesting research problems, has anyone made any progress with the Lo problem posed by Feldman, Lakoff et al last year? This is a miniature language acquisition problem that has elements of vision, planning and learning as well as language. As stated, the goal of the problem is to be able to answer simple yes/no questions about pictures of squares, triangles and circles ("Is the circle to the right of the square?"). A variant on this problem would be to include a motor component, that is, to have the system perform simple manipulations of the picture ("Move the circle to the right side of the square.") I suspect that the motor system is an important part of animal cognition that wouldn't be captured by a system that simply answered questions. This problem would also provide interesting comparisons with Winograd's program. It is at the systems level and is the sort that Aaron Sloman was referring to in his long contribution to this topic, started, you will all recall, by Jerry Feldman's original posting asking for achievements in connectionism. I hope that we have contributed more than just this ongoing debate. Terry ----- From GOLDFARB%unb.ca at unbmvs1.csd.unb.ca Sat Jan 5 19:45:12 1991 From: GOLDFARB%unb.ca at unbmvs1.csd.unb.ca (GOLDFARB%unb.ca@unbmvs1.csd.unb.ca) Date: Sat, 05 Jan 91 20:45:12 AST Subject: arguments for all seasons In-Reply-To: Message of Fri, 4 Jan 91 03:37:54 EST from Message-ID: As an explanation of why AI/NN (discrete/continuous)debate is very important, I simply refer you to the similar situations throughout the development of physics: I might remark that history shows us that reconciling inconsistent physical theories is a very good way of making fundamental progress. . . . So, many of the most far reaching advances of the twentieth century have come about because previous theories weren't compatible with one another. History teaches us that reconciling incompatibilities between theories is a good way to make really fundamental progress. (Edward Witten, in Superstrings: A theory of Everything? eds. P.C.W. Davies and J.Brown, Cambridge Univ. Press, 1988, pp.97-8) Since it appears that some of our neurobiological friends feel somewhat left out of the debate, let me explain why I think they "would all be wise to be humble and to listen well." Again, we should turn to physics: The theory of relativity is a fine example of the fundamental character of the modern development of theoretical science. The hypotheses become steadily more abstract and remote from experience. On the other hand, it gets nearer to the grand aim of all science, which is to cover the greatest possible number of empirical facts by logical deduction from the smallest possible number of hypotheses or axioms. Meanwhile, the train of thought leading from the axioms to the empirical facts or verifiable consequences gets steadily longer and more subtle. The theoretical scientist is compelled in an increasing degree to be guided by purely mathematical, formal considerations in his search for a theory, because the physical experience of the experimenter cannot lead him up to the regions of highest abstraction. . . . The theorist who undertakes such a labor should not be caped at as "fanciful"; on the contrary, he should be granted the right to give free reign to his fancy, for *there is no other way to the goal*. (A. Einstein, see the book Ideas and Opinions, by A. Einstein, p.282) I strongly believe that in the study of intelligence we are faced from the very beginning with even more "dramatic" situation: if "the train of thought leading from the axioms to the empirical facts" for the *first* physical theories was relatively short (and that is why a more direct, "hands on" approach was possible), the theory of intelligence, or intelligent (biological) information processing, has not and *cannot* originate with the theories similar in the mathematical structure to the first physical theories, simply because the basic elements of "information" and "intelligence" are much more abstract and are not visible to the naked eye. It is intersting to note that many of the leading physicists would probably agree with this statement. (For one of the most resent opinions see The Emperor's New Mind, by R.Penrose) Besides, any specific biological entity on any "planet" represents an outcome of a particular evolutionary implementation of the intelligence. That is why the "computer and the oscilloscope" analogy is quite appropriate. In conclusion, I strongly believe that, as it is also becoming more and more apparent in physics, the mathematical models of intelligence will strongly lead the neurobiological and perceptual experiments and not the other way around. --Lev Goldfarb From worth at park.bu.edu Sun Jan 6 15:31:10 1991 From: worth at park.bu.edu (Andrew J. Worth) Date: Sun, 6 Jan 91 15:31:10 -0500 Subject: Connectionism vs AI Survey Message-ID: <9101062031.AA21945@park.bu.edu> Since I feel slightly responsible for stoking the current debate on Connectionism vs AI, allow me to propose two surveys which may allow its fruition. Members of the connectionist list have varying backgrounds and interests that lead them to correspondingly various research emphases and various amounts of attention paid to other emphases. Perhaps it would be beneficial to all to find out what we (the collective list) believe is important. Please respond to me (worth at park.bu.edu) with your own personal thoughts on either or both of the following: SURVEY 1: 1) What is your emphasis or interest in this field (i.e. why are you on the connectionist list) and why is this aspect of the field important? 2) If you were King for a day, where would you direct further research? The main question that I am trying to ask here is: Specifically, what aspects of your interests should be important to the rest of us? Or, more generally, what should be the key aspects of Connectionism? SURVEY 2: What SPECIFIC aspects of neural physiology, anatomy, and topology should be considered by Connectionists? PLEASE BE BRIEF. I will summarize, collate, and post your responses (if there is enough interest) in a few weeks. Thanks in advance, Andy. ----------------------------------------------------------------------- Andrew J. Worth worth at park.bu.edu (617) 353-6741 Cognitive & Neural Systems Boston University Center for Adaptive Systems 111 Cummington St. Room 244 (617) 353-7857 Boston, MA 02215 USA From tsejnowski at ucsd.edu Sun Jan 6 18:34:19 1991 From: tsejnowski at ucsd.edu (Terry Sejnowski) Date: Sun, 6 Jan 91 15:34:19 PST Subject: Seminar: Barron on Scaling Message-ID: <9101062334.AA04550@sdbio2.UCSD.EDU> Computer Science Seminar "Approximation Properties of Artificial Neural Networks" Andrew R. Barron University of Illinois Monday, January 7, 4 PM 7421 Applied Physics and Mathematics Building University of California, San Diego Bounds on the approximation error of a class of feed-forward artificial neural network models are presented. A previous result obtained by George Cybenko and by Kurt Hornik, Max Stinchcombe, and Hal White shows that linear combinations of sigmoidal functions are dense in the space of continuous functions on compact subsets in d dimensions. In this talk we examine how the approximation error is related to the number of nodes in the network. We impose the regularity condition that the gradient of the function of d variables has an integrable Fourier transform. In particular, bounds are obtained for the integrated squared error of approximation, where the integration is taken on any given ball and with respect to any probability measure. It is shown that there is a linear combination of n sigmoidal functions such that the integrated squared error is bounded by c/n, where the constant c is depends on the radius of the ball and the integral of the norm of the Fourier transform of the gradient of the function. A sigmoidal function is the composition of a given bounded increasing function of one variable with a linear function of d variables. Such sigmoidal functions comprise a standard artificial neuron model and the linear combination of such functions is a one-layer artificial neural network. The surprising aspect of this result is that an approximation rate is achieved which is independent of the dimension d, using a number of parameters O(nd) which grows only linearly in d. This is in contrast to traditional series expansions which require exponentially many parameters O(n^d) to achieve approximation rates of order O(1/n), under somewhat different hypotheses on the class of functions. We conclude that the "curse of dimensionality" does not apply to the class of functions we examine. From jbower at smaug.cns.caltech.edu Mon Jan 7 00:11:23 1991 From: jbower at smaug.cns.caltech.edu (Jim Bower) Date: Sun, 6 Jan 91 21:11:23 PST Subject: peace and platitudes Message-ID: <9101070511.AA15394@smaug.cns.caltech.edu> In response to Lev Goldfarbs last posting about analytical models: As Terry, Ken, etc., I too think that no one description fits the goals of everyone on this mailing list. Some want to build things, some want to figure out how things are built. Others would like to construct a grand formalism that covers it all. The approach, interests, convictions between and within each group differ. Without question this is our strength. None of these efforts are trivial and it is silly to think that one or the other is the more correct interest to have. Speaking from my own, somewhat underrepresented corner of this effort, I have tried to point out, perhaps a bit too strongly, that these distinctions exist, that they are significant, and that it is important that they be recognized and respected. Jim Bower From sontag at control.rutgers.edu Mon Jan 7 11:37:04 1991 From: sontag at control.rutgers.edu (sontag@control.rutgers.edu) Date: Mon, 7 Jan 91 11:37:04 EST Subject: 4 vs 3 layers -- Tech Report available from connectionists archive Message-ID: <9101071637.AA01072@control.rutgers.edu> REPORT AVAILABLE ON CAPABILITIES OF FOUR-LAYER vs THREE-LAYER NETS At the request of a few people at NIPS, I placed in the connectionists archive the postscript version of my report describing why TWO hidden layers are sometimes necessary when solving function-approximation types of problems, a fact that was mentioned in my poster. (About 1/2 of the report deals with the general question, while the other half is devoted to the application to control that led me to this.) Below are the abstract and instructions on ftp retrieval. I would very much welcome any discussion of the practical implications --if any-- of the result. If you want, send email to me and I can summarize later for the net. Happy palindromic year to all, -eduardo ----------------------------------------------------------------------------- Report SYCON-90-11, Rutgers Center for Systems and Control, October 1990 FEEDBACK STABILIZATION USING TWO-HIDDEN-LAYER NETS This report compares the representational capabilities of three-layer (that is, "one hidden layer") and four-layer ("two hidden layer") nets consisting of feedforward interconnections of linear threshold units. It is remarked that for certain problems four layers are required, contrary to what might be in principle expected from the known approximation theorems. The differences are not based on numerical accuracy or number of units needed, nor on capabilities for feature extraction, but rather on a much more basic classification into "direct" and "inverse" problems. The former correspond to the approximation of continuous functions, while the latter are concerned with approximating one-sided inverses of continuous functions ---and are often encountered in the context of inverse kinematics determination or in control questions. A general result is given showing that nonlinear control systems can be stabilized using four layers, but not in general using three layers. ----------------------------------------------------------------------- To obtain copies of the postscript file, please use Jordan Pollack's service: Example: unix> ftp cheops.cis.ohio-state.edu # (or ftp 128.146.8.62) Name (cheops.cis.ohio-state.edu:): anonymous Password (cheops.cis.ohio-state.edu:anonymous): ftp> cd pub/neuroprose ftp> binary ftp> get (remote-file) sontag.twolayer.ps (local-file) twolayer.ps.Z ftp> quit unix> uncompress twolayer.ps unix> lpr -P(your_local_postscript_printer) twolayer.ps ---------------------------------------------------------------------------- If you have any difficulties with the above, please send e-mail to sontag at hilbert.rutgers.edu. DO NOT "reply" to this message, please. From gmdzi!smieja at relay.EU.net Mon Jan 7 13:40:05 1991 From: gmdzi!smieja at relay.EU.net (Frank Smieja) Date: Mon, 7 Jan 91 17:40:05 -0100 Subject: Vision Message-ID: <9101071640.AA06439@gmdzi.UUCP> Could someone please let me know about the recent objections to Marr's theory of vision, as expounded in his book of the same title? I am particularly interested in current (new) ideas about stereopsis. References to actual vision work are not hard to find, but they are somewhat voluminous, and I would be most grateful just for a brief indication of "where Marr was in error", so that I do not fall into the same trap. If there are fears of starting an unending debate of pro- and anti-Marr arguers, then mail me directly, and I will echo any good articles suggested back to the net. Frank Smieja From kimd at gizmo.usc.edu Mon Jan 7 20:12:13 1991 From: kimd at gizmo.usc.edu (Kim Daugherty) Date: Mon, 7 Jan 1991 17:12:13 PST Subject: Connectionist Simulators Message-ID: Last November, I posted a request for connectionist modeling simulators to the mailing list. I would like to thank those who responded. Following is a list and brief description of several simulators: 1. Genesis - An elaborate X windows simulator that is particularly well suited for modeling biological neural networks. unix> telnet genesis.cns.caltech.edu (or 131.215.135.185) Name: genesis Follow directions there to get a ftp account from which you can ftp 'genesis.tar.Z". This contains genesis source and several tutorial demos. NOTE: There is a fee to become a registered user. 2. PlaNet (AKA SunNet) - A popular connectionist simulator with versions to run under SunTools, X Windows, and non-graphics terminals created by Yoshiro Miyata. The SunTools version is not supported. unix> ftp boulder.colorado.edu (128.138.240.1) Name: anonymous Password: ident ftp> cd pub ftp> binary ftp> get PlaNet5.6.tar.Z ftp> quit unix> zcat PlaNet5.6.tar.Z | tar xf - All you need to do to try it is to type: unix> Tutorial This will install a program appropriate for your environment and start an on-line tutorial. If you don't need a tutorial, just type 'Install' to install the system and then 'source RunNet' to start it. See the file README for more details. The 60-page User's Guide has been split into three separate postscript files so that each can be printed from a printer with limited memory. Print the files doc/PlaNet_doc{1,2,3}.ps from your postscript printer. See the doc/README file for printing the Reference Manual. Enjoy!! And send any questions to miyata at boulder.colorado.edu. 3. CMU Connectionist Archive - There is a lisp backprop simulator in the connectionist archive. unix> ftp b.gp.cs.cmu.edu (or 128.2.242.8) Name: ftpguest Password: cmunix ftp> cd connectionists/archives ftp> get backprop.lisp ftp> quit 4. Cascade Correlation Simulator - There is a LISP and C version of the simulator based on Scott Fahlman's Cascade Correlation algorithm, who also created the LISP version. The C version was created by Scott Crowder. unix> ftp pt.cs.cmu.edu (or 128.2.254.155) Name: anonymous Password: (none) ftp> cd /afs/cs/project/connect/code ftp> get cascor1.lisp ftp> get cascor1.c ftp> quit A technical report descibing the Cascade Correlation algorithm may be obtained as follows: unix> ftp cheops.cis.ohio-state.edu (or 128.146.8.62) Name: anonymous Password: neuron ftp> cd pub/neuroprose ftp> binary ftp> get fahlman.cascor-tr.ps.Z ftp> quit unix> uncompress fahlman.cascor-tr.ps.Z unix> lpr fahlman.cascor-tr.ps 5. Quickprop - A variation of the back-propagation algorithm developed by Scott Fahlman. A LISP and C version can be obtained in the same directory as the cascade correlation simulator above. Kim Daugherty kimd at gizmo.usc.edu From slehar at park.bu.edu Tue Jan 8 08:28:18 1991 From: slehar at park.bu.edu (Steve Lehar) Date: Tue, 8 Jan 91 08:28:18 -0500 Subject: Vision (What's wrong with Marr's model) Message-ID: <9101081328.AA06421@park.bu.edu> Frank Smieja asks about recent objections to Marr's theory of vision. Here is my opinion. David Marr's book is delightfully lucid and beautifully illustrated, and I thoroughly agree with his analysis of the three levels of modeling. Nevertheless I believe that there are two fatal flaws in the philosophy of his vision model. The first fatal flaw is the feedforward nature of this model, from the raw primal sketch through the 2&1/2 D sketch to the 3-D model representation. Decades of "image understanding" and "pattern recognition" research have shown us that such feed-forward processing has a great deal of difficulty with natural imagery. The problem lies in the fact that whenever "feature extraction" or "image enhancement" are performed, they recognize or enhance some features but in the process they inevitably degrade others or introduce artifacts. With successive levels of processing the artifacts accumulate and combine until at the highest levels of processing there is no way to distinguish the real features from the artifacts. Even in our own vision, with all its sophistication, we occasionally see things that are not there. The real problem with such feedforward models is that once a stage of processing is performed, it is never reviewed or reconsidered. Grossberg suggests how nature solves this problem, by use of top-down feedback. Whenever a feature is recognized at any level, a copy of that feature is passed back DOWN the processing hierarchy in an attempt to improve the match at the lower levels. If for instance a set of disconnected edges suggest a larger continuous edge to a higher level, that "hypothesis" is passed down to the local edge detectors to see if they can find supporting evidence for the missing pieces by locally lowering their detection thresholds. If a faint edge is indeed found where expected, it is enhanced by resonant feedback. If however there is strong local opposition to the hypothesis then the enhancement is NOT performed. This is the cooperative / competitive loop of the BCS model which serves to disambiguate the image by simultaneous matching at multiple levels. This explains how, when we occasionally see something that isn't there, we see it in such detail until at a higher level a conflict occurs, at which time the apparition "pops" back to being something more consistant with the global picture. The second fatal flaw in Marr's vision model is related to the first. In the finest tradition of "AI", Marr's 3-D model is an abstract symbolic representation of the visual input, totally divorced from the lower level stimuli which generated it. The great advance of the connectionist perspective is that manipulation of high level symbols is meaningless without regard to the hierarchy of lower level representations to which they are attached. When you look at your grandmother for instance, some high level node (or nodes) must fire in recognition. At the same time however you are very conscious of the low level details of the image, the strands of hair, the wrinkles around the eyes etc. In fact, even in her absence the high level node conjurs up such low level features, without which that node would have no real meaning. It is only because that node rests on the pinacle of a hierarchy of such lower level nodes that it has a meaning of "grandmother". The perfectly gramatical sentence "Grandmother is purple" is only recognized as nonsense when visualized at the lowest level, illustrating that logical processing cannot be separated from low level visualization. Although I recognize Marr's valuable and historic contribution to the understanding of vision, I believe that in this fast moving field we have already progressed to new insights and radically different models. I would be delighted to provide further information by email to interested parties on Grossberg's BCS model, and my own implementation of it for image processing applications. (O)((O))(((O)))((((O))))(((((O)))))(((((O)))))((((O))))(((O)))((O))(O) (O)((O))((( slehar at park.bu.edu )))((O))(O) (O)((O))((( Steve Lehar Boston University Boston MA )))((O))(O) (O)((O))((( (617) 424-7035 (H) (617) 353-6741 (W) )))((O))(O) (O)((O))(((O)))((((O))))(((((O)))))(((((O)))))((((O))))(((O)))((O))(O) From jrs at cs.williams.edu Tue Jan 8 11:04:31 1991 From: jrs at cs.williams.edu (Josh Smith) Date: Tue, 8 Jan 91 11:04:31 EST Subject: C. elegans Message-ID: <9101081604.AA23879@bull> I just read that all 329 (?) neurons in C. elegans, the nematode worm, have been mapped. (That is, the connectivity or wiring pattern for the worm is known.) Even though its nervous system is so simple, the worm apparently has a quite a wide range of behaviors (swimming, following odors, avoiding salt, mating, etc.). Has anyone ever simulated this network? I think such a simulation would be very useful. If it didn't work, that might indicate that the idealized neuron (sum-unit-cum-squashing-function) in use now is too simple. If it did work, connectionists could proceed with more confidence that this idealization is not absurd (if they care). I'm sure this simulation would be interesting in many other ways as well. With the number of neurons in the network, I think it would even be a computationally feasible undertaking. From birnbaum at fido.ils.nwu.edu Tue Jan 8 15:54:23 1991 From: birnbaum at fido.ils.nwu.edu (Lawrence Birnbaum) Date: Tue, 8 Jan 91 14:54:23 CST Subject: Machine learning workshop: Addendum to call for papers Message-ID: <9101082054.AA01130@fido.ils.nwu.edu> ADDENDUM TO CALL FOR PAPERS EIGHTH INTERNATIONAL WORKSHOP ON MACHINE LEARNING NORTHWESTERN UNIVERSITY EVANSTON, ILLINOIS JUNE 27-29, 1991 We wish to clarify the position of ML91 with respect to the issue of multiple publication. In accordance with the consensus expressed at the business meeting at ML90 in Austin, ML91 is considered by its organizers to be a specialized workshop, and thus papers published in its proceedings may overlap substantially with papers published elsewhere, for instance IJCAI or AAAI. The sole exception is with regard to publication in future Machine Learning Conferences. Authors who are concerned by this constraint will be given the option of foregoing publication of their presentation in the ML91 Proceedings. The call for papers contained information concerning seven of the eight individual workshops that will make up ML91. Information concerning the final workshop follows. Larry Birnbaum Gregg Collins Northwestern University The Institute for the Learning Sciences 1890 Maple Avenue Evanston, IL 60201 (708) 491-3500 ------------------------------------------------------------------------------- COMPUTATIONAL MODELS OF HUMAN LEARNING This workshop will foster interaction between researchers concerned with psychological models of learning and those concerned with learning systems developed from a machine learning perspective. We see several ways in which simulations intended to model human learning and algorithms intended to optimize machine learning may be mutually relevant. For example, the way humans learn and the optimal method may turn out to be the same for some tasks. On the other hand, the relation may be more indirect: modeling human behavior may provide task definitions or constraints that are helpful in developing machine learning algorithms; or machine learning algorithms designed for efficiency may mimic human behavior in interesting ways. We invite papers that report on learning algorithms that model or are motivated by learning in humans or animals. We encourage submissions that address any of a variety of learning tasks, including category learning, skill acquisition, learning to plan, and analogical reasoning. In addition, we hope to draw work from a variety of theoretical approaches to learning, including explanation-based learning, empirical learning, connectionist approaches, and genetic algorithims. In all cases, authors should explicitly identify 1) in what ways the system's behavior models human (or animal) behavior, 2) what principles in the algorithm are responsible for this, and 3) the methods for comparing the system's behavior to human behavior and for evaluating the algorithm. A variety of methods have been proposed for computational psychological models; we hope the workshop will lead to a clearer understanding of their relative merits. Progress reports on research projects still in development are appropriate to submit, although more weight will be given to projects that have been implemented and evaluated. Integrative papers providing an analysis of multiple systems or several key issues are also invited. WORKSHOP COMMITTEE Dorrit Billman (Georgia Tech) Randolph Jones (Univ. of Pittsburgh) Michael Pazzani (Univ. of California, Irvine) Jordan Pollack (Ohio State Univ.) Paul Rosenbloom (USC/ISI) Jeff Shrager (Xerox PARC) Richard Sutton (GTE) SUBMISSION DETAILS Papers should be approximately 4000 words in length. Authors should submit seven copies, by March 1, 1991, to: Dorrit Billman School of Psychology Georgia Institute of Technology Atlanta, GA 30332 phone (404) 894-2349 Formats and deadlines for camera-ready copy will be communicated upon acceptance. From aarons at cogs.sussex.ac.uk Tue Jan 8 17:43:46 1991 From: aarons at cogs.sussex.ac.uk (Aaron Sloman) Date: Tue, 8 Jan 91 22:43:46 GMT Subject: Vision (What's wrong with Marr's model) Message-ID: <3206.9101082243@rsuna.cogs.susx.ac.uk> I guess some cynics might respond that connectionists are now talking about vision in a way that's not too far (discounting technical details) from what AI vision researchers were doing before Marr came along and started telling them all how it should be done! Aaron From GOLDFARB%UNB.CA at UNBMVS1.csd.unb.ca Tue Jan 8 19:27:09 1991 From: GOLDFARB%UNB.CA at UNBMVS1.csd.unb.ca (Goldfarb) Date: Tue, 08 Jan 91 20:27:09 AST Subject: continuous vs symbolic: a more concrete problem Message-ID: Over the last two decades it has become reasonably clear that at some stage of information processing (even in vision) it is often convenient to represent objects in a symbolic form. The simplest symbolic representation is a string over a finite alphabet. An immediate question that arises then is: How does one go about constructing a neural network that can recognize a reasonably large (infinite and nontrivial) *class* of formal languages? For example, let us specify a language in the following way: fix 1) some string (e.g. dabbe) and 2) a finite set S of strings (e.g. S={aa, da, cdc}); then the language is formed by all strings that can be obtained from the single fixed string (dabbe) by inserting in any place and in any order any number of strings from set S. Consider now the class of *all* languages that can be specified in this way. It is a subclass of the class of regular languages. If the NN is "the right" model, then the above problem should have a reasonably simple solution. Does anyone know such a solution? By the way, the reason I have chosen the above class of languages is that the new RLM model mentioned in several earlier postings solves the the problem in a very routine manner. --Lev Goldfarb From orjan at thalamus.sans.bion.kth.se Wed Jan 9 08:17:11 1991 From: orjan at thalamus.sans.bion.kth.se (Orjan Ekeberg) Date: Wed, 9 Jan 91 14:17:11 +0100 Subject: C. elegans In-Reply-To: Josh Smith's message of Tue, 8 Jan 91 11:04:31 EST <9101081604.AA23879@bull> Message-ID: <9101091317.AA05479@thalamus> Based on my experience with fairly realistic simulations of the neuronal network producing swimming in the Lamprey (a fish-like lower vertebrate), I would expect that much more information than the wiring pattern is needed to produce a "working" computer model of this nematode. Let me give some examples. Rythmic activity (you mention swimming) in simple invertebrates often depends on single spikes in each cycle. This indicates that a cell model can not rely on an output representing only firing frequency. There might, however, be parts of the system where a squashing-function neuron model is sufficient. Individual cell properties like membrane time constants, localization of synapses, intrinsic pacemaker properties etc. probably plays a crucial role in such a network. Even if such properties are in principle measurable, it is necessary to know what you are looking for beforehand, i.e. you need a theory of the function of each neuron. I believe that computer simulation is an important tool when constructing such theories. Thus, I do not believe that it is possible to follow the strategy of FIRST measuring everything and THEN simulate the final model. Rather, understanding of how a real network operates could gradually be gained in a process involving simulations as one tool. I do agree that, from a computational point of view, simulating 329 interconnected neurons is feasible. We have been simulating about 600 neurons from the Lamprey including a four compartment representation of the dendrites and Na, K, Ca and Ca dependent K channels, transmitter and voltage dependent (NMDA) synapses, etc. Even if the C. elegans system might need further properties added, I think it would still be possible to do simulations even on this level of detail. Orjan Ekeberg +---------------------------------+-----------------------+ + Orjan Ekeberg + O---O---O + + Department of Computing Science + \ /|\ /| Studies of + + Royal Institute of Technology + O-O-O-O Artificial + + S-100 44 Stockholm, Sweden + |/ \ /| Neural + +---------------------------------+ O---O-O Systems + + EMail: orjan at bion.kth.se + SANS-project + +---------------------------------+-----------------------+ From gmdzi!nieters%sphinx at relay.EU.net Wed Jan 9 09:20:00 1991 From: gmdzi!nieters%sphinx at relay.EU.net (Hans Nieters) Date: Wed, 9 Jan 91 15:20+0100 Subject: paper announcement Message-ID: <19910109142040.4.NIETERS@sphinx> The following 15 page paper is now available. It can be ftp-ed from GMD under the name nieters.petri.neural.ps.Z, as shown below, or can be ordered by sending a note to: Hans Nieters GMD-F2G2 W-5205 St. Augustin Postfach 1240 Germany or Email: nit at gmdzi.gmd.de Neural Networks as Predicate Transition systems The relationship between NN and Petri nets is investigated, since both have their merits for seemingly different application areas. For non-recurrent NN (with and without backprop.) a Predicate Transition system (a very high level class of Petri nets) can be given, which is in sense behaviorally equivalent. The aim of this approach is to show that both modeling techniques could gain benefits from each other, if they were put on a common basis: Petrinets allow modeling of concurrency and conflict (missing in NN) whereas NN contributes learning (missing in Petri nets). The proposed technique for transforming NN into PrT systems is demonstrated by examples. The paper is written mainly for people familiar with Petrinets and newcomer in NN, but perhaps some connectionists may find it helpful also, despite of the fact, that the results are very preliminary. unix> ftp gmdzi.gmd.de Name: anonymous Password: state-your-name-please ftp> cd pub/gmd ftp> binary ftp> get nieters.petri.neural.ps.Z (ca 176000 bytes) ftp> quit unix> uncompress nieters.petri.neural.ps.Z unix> lpr nieters.petri.neural.ps The postscript file has been tested and will hopefully print also YOUR printer From slehar at park.bu.edu Wed Jan 9 10:52:33 1991 From: slehar at park.bu.edu (Steve Lehar) Date: Wed, 9 Jan 91 10:52:33 -0500 Subject: Vision (What's wrong with Marr's model) In-Reply-To: Aaron Sloman's message of Tue, 8 Jan 91 22:43:46 GMT <3206.9101082243@rsuna.cogs.susx.ac.uk> Message-ID: <9101091552.AA15053@park.bu.edu> Aaron> I guess some cynics might respond that connectionists are now talking Aaron> about vision in a way that's not too far (discounting technical details) Aaron> from what AI vision researchers were doing before Marr came along and Aaron> started telling them all how it should be done! Aaron> Aaron Tell me about the AI vision researchers before Marr that supported vision models inspired by natural architectures (PDP/connectionist) with intensive feedback mechanisms (as seen in nature) motivated by neurophysiological (single cell recordings) and psychophysical (perceptual illusions) data, making testable hypotheses (reproducing the illusions) about natural vision. I haven't heard about them. From fritz_dg%ncsd.dnet at gte.com Wed Jan 9 12:49:57 1991 From: fritz_dg%ncsd.dnet at gte.com (fritz_dg%ncsd.dnet@gte.com) Date: Wed, 9 Jan 91 12:49:57 -0500 Subject: C. elegans Message-ID: <9101091749.AA02051@bunny.gte.com> Second the motion. If anyone has or is contemplating, a response to the list would be of great interest. fritz_dg%ncsd at gte.com From MURTAGH at SCIVAX.STSCI.EDU Wed Jan 9 13:34:54 1991 From: MURTAGH at SCIVAX.STSCI.EDU (MURTAGH@SCIVAX.STSCI.EDU) Date: Wed, 9 Jan 1991 13:34:54 EST Subject: Workshop, "NNs for Stat. & Econ. Data" Message-ID: <910109133454.20201a55@SCIVAX.STSCI.EDU> Workshop on "Neural Networks for Statistical and Economic Data" This workshop, organized by Munotec Systems, and funded by the Statistical Office of the European Communities, Luxembourg, was held in Dublin, Ireland, on December 10-11, 1990. A proceedings, including abstracts and in many instances papers, will be reproduced and sent to all on the mailing list of the DOSES funding program in the near future. DOSES ("Design of Statistical Expert Systems") is one of the European Community funding programs, and is administered by the Statistical Office. Requests to be included on this mailing list should be addressed to: DOSES, Statistical Office of the European Communities, Batiment Jean Monnet, B.P. 1907, Plateau du Kirchberg, L-2920 Luxembourg. F. Murtagh (murtagh at scivax.stsci.edu, fionn at dgaeso51.bitnet) -------------------------------------------------------------------------------- The following were the talks given at the Dublin meeting: M. Perremans (Stat. Office of EC, Luxembourg) "The European Community statistical research programs." H.-G. Zimmermann (Siemens, Munich) "Neural network features in economics." J. Frain (Central Bank of Ireland, Dublin) "Complex questions in economics and economic statistics." M.B. Priestley (UMIST, Manchester) "Non-linear time series analysis: overview." R. Rohwer (CSTR, Edinburgh) "Neural networks for time-varying data." P. Ormerod and T. Walker (Henley Centre, London) "Neural networks and the monetary base in Switzerland." S. Openshaw and C. Wymer (Univ. of Newcastle upon Tyne) "A neural net classifier system for handling census data." F. Murtagh (Munotec, Dublin; ST-ECF, Munich) "A short survey of neural network approaches for forecasting." D. Wuertz and C. de Groot (ETH, Zrich) "Modeling and forecasting of univariate time series by parsimonious feedforward connectionist nets." J.-C. Fort (Univ. de Paris 1) "Kohonen algorithm and the traveling salesman problem." H.-G. Zimmermann (Siemens, Munich) "Completion of incomplete data." R. Hoptroff and M.J. Bramson (London) "Forecasting the economic cycle." A. Varfis and C. Versino (JRC, Ispra) "Neural networks for economic time series forecasting." D. Mitzman and R. Giovannini (Cerved SpA, Padua) "ActivityNets: A neural classifier of natural language descriptions of economic activities." (Also: demonstration on 386-PC.) C. Doherty (ERC, Dublin) "A comparison between the recurrent cascade-correlation architecture and the Box and Jenkins method on forecasting univariate time series." M. Eaton and B.J. Collins (Univ. of Limerick, Limerick) "Neural network front end to an expert system for decision taking in an uncertain environment." R.J. Henery (Univ. of Strathclyde, Glasgow) "StatLog: Comparative testing of statistical and logical learning algorithms." Ah Chung Tsoi (Univ. of Queensland) "FIR and IIR synapses, a neural network architecture for time series modelling." A. Singer (Thinking Machines, Munich) "Focusing on feature extraction in pattern recognition." R. Rohwer (CSTR, Univ. of Edinburgh) "The 'Moving Targets' algorithm for difficult temporal credit assignment problems." -------------------------------------------------------------------------------- From INS_ATGE%JHUVMS.BITNET at VMA.CC.CMU.EDU Wed Jan 9 13:40:00 1991 From: INS_ATGE%JHUVMS.BITNET at VMA.CC.CMU.EDU (INS_ATGE%JHUVMS.BITNET@VMA.CC.CMU.EDU) Date: Wed, 9 Jan 91 13:40 EST Subject: SSAB questions Message-ID: I am interested in using the Super Self-Adapting Back Propogation (SuperSAB) algorithm recently published in _Neural_Networks_ (T. Tollenaere). The algorithm published appears a bit ambiguous to me. In step four, it says "undo the previous weight update (which caused the change in the gradient sign). This can be done by using [delta weight(i,j,n+1)] = -[delta weight(i,j,n)], instead of calculating the weight-update..." Does this mean undo the previous update of _all_ network weights, or just undo the update of the particular weights which changed sign? Anyway, I am going to try to use SuperSAB to speed up a time-delay neural net (TDNN) of the sort used in Lang, Waibel, and Hinton _Neural_Networks_ 3, p.23 for analysis of multiple-band infrared temporal intensity data. While I am on ther subject, has anyone done a comparison between Quickprop and SuperSAB, or has used SuperSAB in Cascade-Correlation or Time Delay Neural Nets? -Thomas Edwards From gary at cs.UCSD.EDU Wed Jan 9 14:37:38 1991 From: gary at cs.UCSD.EDU (Gary Cottrell) Date: Wed, 9 Jan 91 11:37:38 PST Subject: C. elegans Message-ID: <9101091937.AA03416@desi.ucsd.edu> >I just read that all 329 (?) neurons in C. elegans, the nematode worm, >have been mapped. >Has anyone ever simulated this network? > If it didn't work, >that might indicate that the idealized neuron >(sum-unit-cum-squashing-function) in use now is too simple. It is clear that this version of the neuron is too simple. One of the most common oscillators in neurobiology is the mutually inhibitory pair of neurons. This can't be done with the standard pdp units. You can do it if you have delay or post-inhibitory rebound. Delay can come about through modeling membrane currents more closely. Another aspect of real circuits not addressed by pdp units is electrotonic connections. Fu-Sheng Tsung, Al Selverston, Peter Rowat & I have simulated a 13 unit network in the lobster (the gastric mill of the stomatogastric ganglion), and found pdp units do well at fitting the behavior, but poorly at generalizing to what happens when a cell is removed from the circuit. Peter & Fu-Sheng have developed more realistic models based on differential equations and difference equations that do a better job, although this is still ongoing research. For copies of Peter's paper, write peter at crayfish.ucsd.edu. It will be coming out in Network. Fu-sheng's algorithm is in the connectionist summer school proceedings. gary cottrell From pollack at cis.ohio-state.edu Wed Jan 9 15:23:46 1991 From: pollack at cis.ohio-state.edu (Jordan B Pollack) Date: Wed, 9 Jan 91 15:23:46 -0500 Subject: continuous vs symbolic: a more concrete problem In-Reply-To: Goldfarb's message of Tue, 08 Jan 91 20:27:09 AST Message-ID: <9101092023.AA04952@dendrite.cis.ohio-state.edu> How does one go about constructing a neural network that can recognize a reasonably large (infinite and nontrivial) *class* of formal languages? Lev, Maybe I'm missing something here, but it has been known since Mcculloch and Pitts 1943, and reiterated by Minsky 1967, that the simplest NN's can behave like finite state machines. FSM's can specify regular languages. By trivial construction, almost any NN capable of arbitrary (sum-of-products) boolean functions are able to represent the "infinite, but TRIVIAL *class*" of regular languages, simply by recurrent use of a set of arbitrary boolean functions. The "next state" vector is just a vector boolean function of the "current state" and "input token" vectors. Since McCullogh-Pitts neurons, two layers of linear-threshold units, or two layers of quasi-linear feedforward networks are are all capable of sum-of-product logical forms, they are all capable of simple solutions to regular language recognition. But if the next state is only a first-order function of the current state and input (such as in Elman's SRN), then all even RL's cannot be in general recognized. Consider the regular "odd parity" language, where the next state is the exclusive-or of the current state and input. The more interesting question is how neural-LIKE architectures can deal NATURALLY with *NONTRIVIAL* grammatical systems, such as the context free, indexed CF (thought by many to be the proper category of natural languages), or context sensitive. We can always solve these problems by adding an external stack or a production rule interpreter, but these solutions do not seem very natural. I'd be very interested if your formalism can solve any of these classes in a routine manner. Jordan Pollack Assistant Professor CIS Dept/OSU Laboratory for AI Research 2036 Neil Ave Email: pollack at cis.ohio-state.edu Columbus, OH 43210 Fax/Phone: (614) 292-4890 From ernst at russel.cns.caltech.edu Wed Jan 9 17:01:16 1991 From: ernst at russel.cns.caltech.edu (Ernst Miebur) Date: Wed, 9 Jan 91 14:01:16 PST Subject: C. elegans In-Reply-To: Josh Smith's message of Tue, 8 Jan 91 11:04:31 EST <9101081604.AA23879@bull> Message-ID: <9101092201.AA00426@russel.caltech.edu> Concerning the question whether someone did simulate the nervous system of C. elegans: I did a simulation of the somatic motor system, which controls the movement of all the major muscles of the worm and which comprises about 20% of all neurons (Niebur & Erdos, Computer simulations in networks of electrotonic neurons. In: R. Cotterill (ed.), Computer Simulations in Brain Science. Cambridge Univ. Press,148-163, 1988. This is a preliminary paper which describes some of the methods used. I have some newer papers in preparation that I will send you on request.) To the best of my knowledge, this is the only detailed simulation of the nervous system of a nematode (and I am pretty sure that it is the largest fraction of ANY nervous system ever simulated in detail). I agree with you that nematodes are a fascinating model system. In particular, if one takes into account that there are many species of very different sizes but with similar structure of their nervous sytems. This makes possible complementary experiments, like determining the ultrastructure at the synaptic level in small species (like C. elegans) and doing electrophysiology in large species (like Ascaris lumbricoides). In fact, the news is even better: Not only all the C. elegans neurons have been mapped, but also ALL the other cells! And: This is the case in ALL stages of the development, from the fertilized zygote to the adult. In this sense, C.elegans is certainly the best known of all animals. If you want any further references to this work, I will be happy to provide them. You are wondering whether the connectionist approach can get any justification from a simulation of C. elegans. The answer is very clear: NO WAY! The connectionist model ("sum-unit-cum-squashing-function") is a lousy model for this system. I wouldn't even call it a model, any simulation based on this model would be too far away from anything reasonable in this system. Draw from this whatever conclusion you want for other systems, but I think that in this system, no serious worker would dispute Jim Bowers view that a detailed knowledge of Biology is important. Ernst Niebur ernst at descartes.cns.caltech.edu PS The number of neurons (in the wild-type hermaphodite) is not 329 but 302. From GOLDFARB%unb.ca at UNBMVS1.csd.unb.ca Wed Jan 9 18:53:57 1991 From: GOLDFARB%unb.ca at UNBMVS1.csd.unb.ca (GOLDFARB%unb.ca@UNBMVS1.csd.unb.ca) Date: Wed, 09 Jan 91 19:53:57 AST Subject: continuous vs symbolic: a more concrete problem Message-ID: Let me state the problem again: Given a nontrivial infinite family of languages, how does one go about constructing a reasonably efficient NN that can learn to recognize any language from this family? -- Lev Goldfarb From inesc!lba at relay.EU.net Thu Jan 10 15:58:50 1991 From: inesc!lba at relay.EU.net (Luis Borges de Almeida) Date: Thu, 10 Jan 91 15:58:50 EST Subject: SSAB questions In-Reply-To: INS_ATGE%JHUVMS.BITNET@VMA.CC.CMU.EDU's message of Wed, 9 Jan 91 13:40 EST <0663505911@PTIFM.IFM.RCCN.PT> Message-ID: <9101101558.AA06580@alf.inesc.pt> Richard Rohwer has presented at last NIPS conference, a comparison among a number of acceleration techniques, in various problems. Among these techniques, is one which is quite similar to SuperSAB. This technique was developed by a colleague and myself, independently from Tollenaere's work (see references below; reprints can be sent to anyone interested). I don't recall seeing tests on Quickprop, but Richard had tests on Le Cun's diagonal second-order method, which I believe to be similar, and perhaps a bit faster, than Quickprop. Richard's data showed better results for Le Cun's method than for ours in many problems, but we found out, while talking to Richard, that he had missed a (probably important) step of the algorithm. I think he may have gone to the work of redoing the tests, you might want to contact him directly. In short, the main difference between our algorithm and Tollenaere's, is that we only undo a weight update if it has caused an increase in the objective function (the quadratic error accumulated over all outputs and all trainig patterns). Luis B. Almeida INESC Phone: +351-1-544607 Apartado 10105 Fax: +351-1-525843 P-1017 Lisboa Codex Portugal lba at inesc.inesc.pt lba at inesc.uucp (if you have access to uucp) --------------------- REFERENCES F. M. Silva and L. B. Almeida, "Acceleration Techniques for the Backpropagation Algorithm", in L. B. Almeida and C. J. Wellekens (eds), Neural Networks, Proc. 1990 EURASIP Workshop, Sesimbra, Portugal, Feb. 1990, New York: Springer-Verlag (Lecture Notes in Computer Science series). F. M. Silva and L. B. Almeida, "Speeding up Backpropagation", in R. Eckmiller (ed), Advanced Neural Computers, Amsterdam: Elsevier Science Publishers, 1990. From GOLDFARB%unb.ca at UNBMVS1.csd.unb.ca Thu Jan 10 14:31:26 1991 From: GOLDFARB%unb.ca at UNBMVS1.csd.unb.ca (GOLDFARB%unb.ca@UNBMVS1.csd.unb.ca) Date: Thu, 10 Jan 91 15:31:26 AST Subject: continuous vs symbolic: a more concrete problem Message-ID: One (but not the only) important point, which is independent of the continuous/discrete issue, my example of the infinite learning environment attempts to clarify is the following: No finite number of *static* NNs can operate successfully in a reasonably complex infinite environment. All interesting real environments are essentially infinite. -- Lev Goldfarb From LAMBERTB at UIUCVMD Thu Jan 10 15:00:22 1991 From: LAMBERTB at UIUCVMD (LAMBERTB@UIUCVMD) Date: 10 January 1991 14:00:22 CST Subject: Request Message-ID: To whom it may concern, Please add my name to your list. Thanks in advance. - B. Lambert From GOLDFARB%unb.ca at UNBMVS1.csd.unb.ca Thu Jan 10 15:15:05 1991 From: GOLDFARB%unb.ca at UNBMVS1.csd.unb.ca (GOLDFARB%unb.ca@UNBMVS1.csd.unb.ca) Date: Thu, 10 Jan 91 16:15:05 AST Subject: Re(corrected): continuous vs symbolic: a more concrete problem Message-ID: One (but not the only) important point, which is independent of the continuous/discrete issue, my example of the infinite learning environment attempts to clarify is the following: No finite number of *static* NNs can operate successfully in a reasonably complex infinite environment (with infinite number of classes). All interesting real environments are essentially infinite. -- Lev Goldfarb From peterc at chaos.cs.brandeis.edu Thu Jan 10 22:06:26 1991 From: peterc at chaos.cs.brandeis.edu (Peter Cariani) Date: Thu, 10 Jan 91 22:06:26 est Subject: Vision (What's wrong with Marr's model) Message-ID: <9101110306.AA28003@chaos.cs.brandeis.edu> Stephen Lehar was asking: "Tell me about the AI vision researchers before Marr that supported vision models inspired by natural architectures (PDP/connectionist) with intensive feedback mechanisms (as seen in nature) motivated by neurophysiological (single cell recordings) and psychophysical (perceptual illusions) data, making testable hypotheses (reproducing the illusions) about natural vision. I haven't heard about them." I think the 1947 paper of Walter Pitts and Warren McCulloch "How we know universals: the preception of auditory and visual forms." in the Bulletin of Mathematical Biophysics (9:127-147) easily fits all of the criteria a full two decades before Marr. I would wager there are many more of these earlier models which have been forgotten by the current connectionist discussions. Looking back on the cybernetics literature of the 1940's and 50's (particularly the Macie conferences), I always have the feeling that they seriously considered many more different types of neural mechanisms (analog, temporal codes as well as digital ones) than we do. Just because our research communities have short memory spans doesn't mean that alot of deep thinking (and modeling) didn't happen before the late 1960's. Peter Cariani From uhr at cs.wisc.edu Fri Jan 11 16:11:35 1991 From: uhr at cs.wisc.edu (Leonard Uhr) Date: Fri, 11 Jan 91 15:11:35 -0600 Subject: On blurring the gap between NN and AI Message-ID: <9101112111.AA19891@thor.cs.wisc.edu> If you take - as I and many people do - the "formal model for AI" and the "formal model for NN" to be Post productions-Turing machines (discretely approximating continua, as is almost always the case in both NN and AI), then they clearly are the same and anything that can be accomplished in one can be done in the other. So the differences boil down to differences in styles and tendencies - e.g., serial, lists, lisp vs. parallel, simple processes, learning. Unfortunately traditional AI has largely ignored learning, but from Samuel, Kochen, Hunt, etc. on, through the more recent Similarity-based, Explanation- based, etc. approaches to learning there has always been a good bit. I personally find the differences more (roughly) analogous to the differences between people who swear by Lisp vs. C vs. Smalltalk. If I'm missing something, please explain how you define NN and AI in such a way as to make them differ. (This was written in bemusement after a number of notes, especially from Lev Goldfarb). Len Uhr From GOLDFARB%unb.ca at UNBMVS1.csd.unb.ca Fri Jan 11 19:32:05 1991 From: GOLDFARB%unb.ca at UNBMVS1.csd.unb.ca (GOLDFARB%unb.ca@UNBMVS1.csd.unb.ca) Date: Fri, 11 Jan 91 20:32:05 AST Subject: On blurring the gap between NN and AI In-Reply-To: Message of Fri, 11 Jan 91 17:11:57 AST from Message-ID: I would have liked to be amused by the separation of several areas, such as AI, Pattern Recognition, NN, if the cost to the taxpayers and especially to our science would not have been so high. Having said that, one still has to find a new mathematical model that addresses the situation described by Scott Fahlman in his posting of Dec. 27 as follows: ". . . connectionism and traditional AI are attacking the same huge problem, but beginning at opposite ends of the problems and using very different tools." In other words, a fundamentally new mathematical model is needed that, on the one hand, would remove any apparent and not so apparent analytical tentions between the two existing mathematical "tools" (production systems/vector space) and, on the other hand, would clearly demonstrate that the "opposite ends of the problem" are intrinsically connected. The reason why I allow myself to be very "philosophical" about the situation is that, as was mentioned earlier on the number of occasions, we believe that a new quite satisfactory model has been found. Although I have already outlined the model and mentioned the first original reference (see, for example, the posting of Sept.27), since I was repeatedly asked about the model, I will shortly outline it again to the extent to which there is an interest in it. -- Lev Goldfarb From uhr at cs.wisc.edu Fri Jan 11 15:46:42 1991 From: uhr at cs.wisc.edu (Leonard Uhr) Date: Fri, 11 Jan 91 14:46:42 -0600 Subject: ontogenesis and synaptogenesis (constructing, generating) Message-ID: <9101112046.AA19867@thor.cs.wisc.edu> An alternative to adding physical nodes (and/or links) whenever constructive algorithms (or "generations") need them, is to have the system generate the physical substrate with whatever vigor it can muster (hopefully, so there will always be as-yet-unused resources available), and also when needed free up (degenerate) resources to make new space. Then the structures of processes can, as needed, be embedded where appropriate. This raises several interesting unsolved problems. A rather general topology is needed, one into which the particular structures that are actually generated will fit with reasonable efficiency. I'm describing this in a way that brings out the similarities to the problem of finding good topologies for massively parallel multi-computers, so that a variety of different structures of processes can be embedded and executed efficiently. The major architectures used today are 2-D arrays, trees, and N-cubes; each accepts some embeddings reasonably well, but not others. One pervasive problem is that their small number of links (typically 2 to 12) can easily lead to bottlenecks - which NN with e.g. 100 or so links might almost always overcome. And there are many many other possible graphs, including interesting hybrids. There are several other issues, but I hope this is enough detail to make the following point: If the physical substrate forms a graph whose topology is reasonably close-to-isomorphic to a variety of structures that combine many smaller graphs, the problem can be viewed as one of finding graphs for the physical into which rich enough sets of graphs of processes can be embedded to step serially through the (usually small) increments that good learning algorithms will generate. To the extent that generation builds relatively small, local graphs this probably becomes easier. I don't mean to solve a problem by using the result of yet another unsolved problem. Just as is done with today's multi-computers, we can use mediocre topologies with poor embeddings and slower-than-optimal processing (and even call these super- and ultra-computers, since they may well be the fastest and most powerful around today), and at the same time try to improve upon them. There's another type of learning that almost certainly needs just this kind of thing. Consider how much we remember when we're told the plot of a novel or some gossip, or shown some pictures and then asked which ones we were shown. The brain is able to make large amounts of already-present physical substrate available, whether for temporary or permanent remembered new information, including processes. As Scott points out, the hardware processors do NOT have to be generated exactly when and because the functions they execute are. Len Uhr From ernst at russel.cns.caltech.edu Sat Jan 12 16:19:28 1991 From: ernst at russel.cns.caltech.edu (Ernst Niebur) Date: Sat, 12 Jan 91 13:19:28 PST Subject: Nematodes: references Message-ID: <9101122119.AA01484@russel.caltech.edu> I have received many requests for references of work on the nematode nervous system. Someone asked me to send "all references on nematodes" - I am sorry, but that would go a little too far. The number of papers only on C. elegans (ONE of an estimated 500,000 nematode species) is in the hundreds per year. I will focus on a few articles which are of special interest for neural modelers. The paper in which the complete nervous system of C. elegans is described (actually, not quite complete: the pharynx part is in Albertson & Thomson, Phil. Trans. Roy. Soc. London B275, 299, 1976) is White, J.G., Southgate, E., Thomson, J.N., Brenner, S. The structure of the nervous system of the nematode C. elegans. Phil. Trans. Roy. Soc. London B 314, 1, 1986. The function of an interesting subcircuit of C. elegans is studied by Laser ablation experiments (i.e., by killing identified neurons in a worm and comparing the behavior of this animal with that of untreated worms) in Chalfie, M., Sulston, J.E., White, J.G, Thomson, J.N and Brenner, S. The neural circuit for touch sensitivity in C. elegans. J. Neurosci. 5(4), 956, 1984. A very well written review paper on the electrophysiological (and some other) work in the large nematode Ascaris lumbricoides is Stretton, A.O.W., Davis, R.E., Angstadt, J.D, Donmoyer, J.E. and Johnson, C.D. Neural control of behavior in Ascaris. Trends in Neuroscience, 294, June 1985. Some more recent results on this system are described in the following papers, which also contain useful references to other work of the Stretton group: Angstadt, J.D, and Stretton, A.O.W. Slow active potentials in ventral inhibitory motor neurons of the nematode Ascaris. J. Comp. Physiol. A 166, 165, 1989. Davis, R.E. and Stretton, A.O.W. Passive membrane properties of motorneurons and their role in long-distance signaling in the nematode Ascaris. J. Neurosci. 9(2), 403, 1989. Davis, R.E. and Stretton, A.O.W. Signaling properties of Ascaris motorneurons. Graded active responses, graded synaptic transmission and tonic transmitter release. J. Neurosci. 9(2), 415, 1989. I will be happy to provide references to more specific topics, but I think these are the ones that might be of potential interest for a larger number of people on this mailing list. Ernst Niebur ernst at descartes.cns.caltech.edu From GOLDFARB%UNB.CA at UNBMVS1.csd.unb.ca Sun Jan 13 21:31:23 1991 From: GOLDFARB%UNB.CA at UNBMVS1.csd.unb.ca (Lev Goldfarb) Date: Sun, 13 Jan 91 22:31:23 AST Subject: Reconfigurable Learning Machines (RLM): motivations Message-ID: The model that I will outline in the next posting was motivated, on the one hand, by the informal considerations similar to those that motivated connectionism, and on the other hand, by the formal considerations that are fundamentally different. The formal considerations can be stated informally as follows: each object (event) in the environment, depending on the specific recognition goal, can belong to to an unbounded number of possible categories (classes). Since these categories are not and can not be learned simultaneously and especially because their recognition requires a DYNAMIC (EVOLVING) concept of object similarity, it is very useful to think about the learning process as the process of computation of the SIMILARITY FIELD, or metric field, induced in the environment by the corresponding learning task. From shawn at helmholtz.sdsc.edu Mon Jan 14 17:53:24 1991 From: shawn at helmholtz.sdsc.edu (Shawn Lockery) Date: Mon, 14 Jan 91 14:53:24 PST Subject: No subject Message-ID: <9101142253.AA29240@helmholtz.sdsc.edu> I am a neurobiologist interested in training neural networks to perform chemotaxis, and other feats of simple animal navigation. I'd be very interested to know what has been done by connectionists in this area. The only things I have found so far are: Mozer and Bachrach (1990) Discovering the Structure of a Reacative nvironment by Exploration, and Nolfi et al. (1990) Learning and Evolution in Neural Networks Many thanks, Shawn Lockery CNL Salk Institute Box 85800 San Diego, CA 92186-5800 (619) 453-4100 x527 shawn at helmholtz.sdsc.edu From shawn at helmholtz.sdsc.edu Mon Jan 14 17:59:54 1991 From: shawn at helmholtz.sdsc.edu (Shawn Lockery) Date: Mon, 14 Jan 91 14:59:54 PST Subject: No subject Message-ID: <9101142259.AA29243@helmholtz.sdsc.edu> Several months ago I asked about canned backprop simulators. At long last, here is the result of my query: ------------------------------------------------------------------------------- Barak Pearlmutter has written a dynamical backprop simulator. A version of his program that solves a toy problem and that is readily modifiable is available by anonymous ftp from helmholtz.sdsc.edu. The directory is pub/ and the filename is pearlmutter.tar ------------------------------------------------------------------------------- Yoshiro Miyata (miyata at dendrite.colorado.edu) has written an excellent public domain connectionist simulator with a nice X windows or Sun View interface. It is called SunNet. He provides a pretty easy to learn "general" definition language so a user can experiment with quite varied back-prop and non-conventional architectures. Examples are provided of backpropagation, boltzmann learning, and others. Source code is available by anonymous ftp from boulder. Look for SunNet5.5.tar.Z at boulder.colorado.edu. ------------------------------------------------------------------------------- Yan Le Cun (Department of Computer Science, University of Toronto, Toronto, Ontario, M5S 1A4, Canado) has written a commercial simulator called SN /2 that is powerful and well documented. ------------------------------------------------------------------------------ The Rochester Connectionist Simulator (RCS) is obtainable by anonymous ftp from cs.rochester.edu. You will find the code in the directory pub/simulator. -------------------------------------------------------------------------------- The speech group at Oregon Graduate Institute has written a conjugate-gradient optimization program called OPT to train fully connected feed-forward networks. It is available by anonymous ftp from cse.ogi.edu. The code is in the directory pub/speech. Copy the file opt.tar. You will need to use the unix "tar" command to process the file once you have it on your computer. --------------------------------------------------------------------------------- For the Macintosh, there is the commercial program called MacBrain (Neuronics, Inc., ! Kendall Square #2200, Cambridge, MA 02139). It has the usual Macintosh bells and whitsles and costs $400. --------------------------------------------------------------------------------- For the Macintosh, there is a public domain program called Mactivation. Mactivation version 3.3 is available via anonymous ftp on alumni.Colorado.EDU (internet address 128.138.240.32) The file is in /pub and is called mactivation.3.3.sit.hqx Mactivation is an introductory neural network simulator which runs on all Apple Macintosh computers. A graphical interface provides direct access to units, connections, and patterns. Basic concepts of network operations can be explored, with many low level parameters available for modification. Back-propagation is not supported (coming in 4.0) A user's manual containing an introduction to connectionist networks and program documentation is included. The ftp version includes a plain text file and an MS Word version with nice graphics and footnotes. The program may be freely copied, including for classroom distribution. for version 4.0. You can also get a copy by mail. Send $5 to Mike Kranzdorf, Box 1379, Nederland, C0 80466-1379. --------------------------------------------------------------------------------- For 386 based PC's, you may purchase ExploreNet from HNC, 5501 Oberlin Drive, San Diego, CA 92121. You don't get source code for your $750, but it's powerful and flexible. --------------------------------------------------------------------------------- For IBM PC's, there is a disk that comes along with the third volume of the PDP books (Parallel Distributed Processing, Rumelhart, McClelland and the PDP Research Group, MIT Press, 1986 . You get lots of source code, and the third volume itself is a nice manual. --------------------------------------------------------------------------------- From apache!weil at uunet.UU.NET Mon Jan 14 17:09:52 1991 From: apache!weil at uunet.UU.NET (wei li) Date: Mon, 14 Jan 91 17:09:52 EST Subject: rejection, adaptive learning. Message-ID: <9101142209.AA13377@cmdsun> Hi, we are doing text classification using a feedforword neural network. Through our experiments, we found two problems: 1) in our class definition, we have texts which are not belong to any classes. We threshold the output, if the output is below the threshold, the input is considered as rejected. It did not seem to work well for the patterns which sould be rejected. 2) When some patterns can not be correctly recognized, we have to retrain the system including these new patterns. We wonder if there is way to gradually adapt the system without having to retrain the old correctly learned patterns too. We have tried RCE network for adaptive learning too, but it seems that if we do not retraining the old patterns, some previously correctly learned patterns will become wrong. Any comments on approaches that could reject patterns and adapt to new patterns? Wei Li uunet!apache!weil or weil%apache at uunet.uu.net From honavar at iastate.edu Mon Jan 14 14:01:47 1991 From: honavar at iastate.edu (honavar@iastate.edu) Date: Mon, 14 Jan 91 13:01:47 CST Subject: tech report available by ftp Message-ID: <9101141901.AA03738@iastate.edu> The following technical report is available in postscript form by anonymous ftp (courtesy Jordan Pollack of Ohio State Univ). ---------------------------------------------------------------------- Generative Learning Structures and Processes for Generalized Connectionist Networks Vasant Honavar Leonard Uhr Department of Computer Science Computer Sciences Department Iowa State University University of Wisconsin-Madison Technical Report #91-02, January 1991 Department of Computer Science Iowa State University, Ames, IA 50011 Abstract Massively parallel networks of relatively simple computing elements offer an attractive and versatile framework for exploring a variety of learning structures and processes for intelligent systems. This paper briefly summarizes the popular learning structures and processes used in such networks. It outlines a range of potentially more powerful alternatives for pattern-directed inductive learning in such systems. It motivates and develops a class of new learning algorithms for massively parallel networks of simple computing elements. We call this class of learning processes \fIgenerative\fR for they offer a set of mechanisms for constructive and adaptive determination of the network architecture - the number of processing elements and the connectivity among them - as a function of experience. Such generative learning algorithms attempt to overcome some of the limitations of some approaches to learning in networks that rely on modification of \fIweights\fR on the links within an otherwise fixed network topology e.g., rather slow learning and the need for an a-priori choice of a network architecture. Several alternative designs, extensions and refinements of generative learning algorithms, as well as a range of control structures and processes which can be used to regulate the form and content of internal representations learned by such networks are examined. ______________________________________________________________________________ You will need a POSTSCRIPT printer to print the file. To obtain a copy of the report, use anonymous ftp from cheops.cis.ohio-state.edu (here is what the transaction looks like): % ftp ftp> open cheops.cis.ohio-state.edu Connected to cheops.cis.ohio-state.edu. 220 cheops.cis.ohio-state.edu FTP server (Version blah blah) ready. Name (cheops.cis.ohio-state.edu:yourname): anonymous 331 Guest login ok, send ident as password. Password: anything 230 Guest login ok, access restrictions apply. ftp> cd pub/neuroprose 250 CWD command successful. ftp> bin 200 Type set to I. ftp> get honavar.generate.ps.Z 200 PORT command successful. 150 Opening BINARY mode data connection for honavar.generate.ps.Z (55121 bytes). 226 Transfer complete. local: honavar.generate.ps.Z remote: honavar.generate.ps.Z 55121 bytes received in 1.8 seconds (30 Kbytes/s) ftp> quit 221 Goodbye. % uncompress honavar.generate.ps.Z % lpr honavar.generate.ps From Connectionists-Request at CS.CMU.EDU Tue Jan 15 12:48:37 1991 From: Connectionists-Request at CS.CMU.EDU (Connectionists-Request@CS.CMU.EDU) Date: Tue, 15 Jan 91 12:48:37 EST Subject: Bi-monthly Reminder Message-ID: <21200.663961717@B.GP.CS.CMU.EDU> CONNECTIONISTS is getting rather large (~1000 subscribers) and we have wasted too much bandwidth in the last six months arguing over what is and is not appropriate for posting to CONNECTIONISTS. To combat this problem we are going to start to posting list guidelines on a bi-monthly basis. The text of the bi-monthly posting follows. If you have comments on this post please direct them to me at Connectionists-Request at cs.cmu.edu. Do NOT reply to the entire list. If there is sufficient interest, I will summarize the comments for the rest of the list. Thanks Scott Crowder Connectionists-Request at cs.cmu.edu (ARPAnet) -------------------------------------------- This is an automatically posted bi-monthly reminder about how the CONNECTIONISTS list works and how to access various online resources. CONNECTIONISTS is not an edited forum like the Neuron Digest, or a free-for-all newsgroup like comp.ai.neural-nets. It's somewhere in between, relying on the self-restraint of its subscribers. Membership in CONNECTIONISTS is restricted to persons actively involved in neural net research. The following posting guidelines are designed to reduce the amount of irrelevant messages sent to the list. Before you post, please remember that this list is distributed to over a thousand busy people who don't want their time wasted on trivia. Also, many subscribers pay cash for each kbyte; they shouldn't be forced to pay for junk mail. Happy hacking. -- Dave Touretzky & Scott Crowder --------------------------------------------------------------------- What to post to CONNECTIONISTS ------------------------------ - The list is primarily intended to support the discussion of technical issues relating to neural computation. - We encourage people to post the abstracts of their latest papers and tech reports. - Conferences and workshops may be announced on this list AT MOST twice: once to send out a call for papers, and once to remind non-authors about the registration deadline. A flood of repetitive announcements about the same conference is not welcome here. - Requests for ADDITIONAL references. This has been a particularly sensitive subject lately. Please try to (a) demonstrate that you have already pursued the quick, obvious routes to finding the information you desire, and (b) give people something back in return for bothering them. The easiest way to do both these things is to FIRST do the library work to find the basic references, then POST these as part of your query. Here's an example: WRONG WAY: "Can someone please mail me all references to cascade correlation?" RIGHT WAY: "I'm looking for references to work on cascade correlation. I've already read Fahlman's paper in NIPS 2, his NIPS 3 abstract, and found the code in the nn-bench archive. Is anyone aware of additional work with this algorithm? I'll summarize and post results to the list." - Announcements of job openings related to neural computation. - Short reviews of new text books related to neural computation. To send mail to everyone on the list, address it to Connectionists at CS.CMU.EDU ------------------------------------------------------------------- What NOT to post to CONNECTIONISTS: ----------------------------------- - Requests for addition to the list, change of address and other administrative matters should be sent to: "Connectionists-Request at cs.cmu.edu" (note the exact spelling: many "connectionists", one "request"). If you mention our mailing list to someone who may apply to be added to it, please make sure they use the above and NOT "Connectionists at cs.cmu.edu". - Requests for e-mail addresses of people who are believed to subscribe to CONNECTIONISTS should be sent to postmaster at appropriate-site. If the site address is unknown, send your request to Connectionists-Request at cs.cmu.edu and we'll do our best to help. A phone call to the appropriate institution may sometimes be simpler and faster. - Note that in many mail programs a reply to a message is automatically "CC"-ed to all the addresses on the "To" and "CC" lines of the original message. If the mailer you use has this property, please make sure your personal response (request for a Tech Report etc.) is NOT broadcast over the net. - Do NOT tell a friend about Connectionists at cs.cmu.edu. Tell him or her only about Connectionists-Request at cs.cmu.edu. This will save your friend from public embarrassment if she/he tries to subscribe. ------------------------------------------------------------------------------- The CONNECTIONISTS Archive: --------------------------- All e-mail messages sent to "Connectionists at cs.cmu.edu" starting 27-Feb-88 are now available for public perusal. A separate file exists for each month. The files' names are: arch.yymm where yymm stand for the obvious thing. Thus the earliest available data are in the file: arch.8802 Files ending with .Z are compressed using the standard unix compress program. To browse through these files (as well as through other files, see below) you must FTP them to your local machine. ------------------------------------------------------------------------------- How to FTP Files from the CONNECTIONISTS Archive ------------------------------------------------ 1. Open an FTP connection to host B.GP.CS.CMU.EDU (Internet address 128.2.242.8). 2. Login as user anonymous with password your username. 3. 'cd' directly to one of the following directories: /usr/connect/connectionists/archives /usr/connect/connectionists/bibliographies 4. The archives and bibliographies directories are the ONLY ones you can access. You can't even find out whether any other directories exist. If you are using the 'cd' command you must cd DIRECTLY into one of these two directories. Access will be denied to any others, including their parent directory. 5. The archives subdirectory contains back issues of the mailing list. Some bibliographies are in the bibliographies subdirectory. Problems? - contact us at "Connectionists-Request at cs.cmu.edu". ------------------------------------------------------------------------------- How to FTP Files from the Neuroprose Archive -------------------------------------------- Host cheops.cis.ohio-state.edu (128.146.8.62) directory pub/neuroprose This directory contains technical reports as a public service to the connectionist and neural networks scientific community. Researchers may place electronic versions of their preprints or articles in this directory, announce availability, and other interested researchers can rapidly retrieve and print the postscripts. To place a file, put it in the Inbox subdirectory, and send mail to pollack at cis.ohio-state.edu. Your mail should include: 1) filename 2) way to contact author 3) single sentence abstract Current naming convention is author.title.filetype[.Z] where title is enough to discriminate among the files of the same author. The filetype is usually "ps" for postscript, our desired universal printing format, but may be tex, which requires more local software than a spooler. Very large files (e.g. over 200k) should be squashed (with either a sigmoid function:) or the standard unix "compress" utility, which results in the .Z affix. To place or retrieve .Z files, make sure to issue the FTP command "BINARY" before transferring files. After retrieval, call the standard unix "uncompress" utility, which removes the .Z affix. A shell script called Getps is available which will automatically perform the necessary operations. The file can be retrieved from the CONNECTIONISTS archive (see above). For further questions contact: Jordan Pollack Email: pollack at cis.ohio-state.edu ------------------------------------------------------------------------ How to FTP Files from the NN-Bench Collection --------------------------------------------- 1. Create an FTP connection from wherever you are to machine "pt.cs.cmu.edu" (128.2.254.155). 2. Log in as user "anonymous" with password your username. 3. Change remote directory to "/afs/cs/project/connect/bench". Any subdirectories of this one should also be accessible. Parent directories should not be. 4. At this point FTP should be able to get a listing of files in this directory and fetch the ones you want. Problems? - contact us at "nn-bench-request at cs.cmu.edu". From GOLDFARB%UNB.CA at UNBMVS1.csd.unb.ca Tue Jan 15 13:34:35 1991 From: GOLDFARB%UNB.CA at UNBMVS1.csd.unb.ca (Lev Goldfarb) Date: Tue, 15 Jan 91 14:34:35 AST Subject: Reconfigurable Learning Machines (RLM) Message-ID: Here is an informal outline of the model proposed in On the Foundations of Intelligent Processes I: An Evolving Model for Pattern Learning, Pattern Recognition, Vol.23, No.6, 1990. Think of an object representation as of a "molecule" (vectors and strings are special types of such molecules). Let O denotes the set of all objects from the environment, and let S = {Si} denotes the set of BASIC SUBSTITUTION OPERATIONS, where each operation can transform one object into another by removing a piece of the molecule and replacing it by another small molecule. For string molecules these could be the operations of letter deletions/insertions. In addition, a small FIXED set CR of COMPOSITION RULES for forming new operations from the existing operations is also given. Think of these rules as specifications for gluing several operations together into one operation. The intrinsic distance D between two objects is defined as the minimum number of operations from S that are necessary to transform one molecule into the other. D depends on the set S of operations. A larger set S can only reduce some of the distances. This idea of distance embodies a very important, perhaps the most important, physical principle -- the least- action principle, which was characterized by Max Planck as follows: Amid the more or less general laws which mark the achievements of physical science during the course of the last centuries, the principle of least action is perhaps that which, as regards form and content, may claim to come nearest to that ideal final aim of theoretical research. In a vector setting, D is a city-block distance between two vectors. In a non-vector setting, however, even small sets of patterns (4-5) with such distances cannot be represented in a Euclidean vector space of ANY dimension. The adjective "intrinsic" in the above definition refers to the fact that the distance D does not reflect any empirical knowledge about the role of the substitution operations. Thus, we are led to the most natural extension of this concept obtained by allowing different substitution operations to have different weights associated with them: assign to each operation Si nonnegative weight w^i subject to one restriction that their sum is 1. The latter constraint is necessary to ensure that during learning the operations are forced to cooperatively compete for the weights. The new weighted distance WD is defined similarly to the above distance D, but replacing the minimum number of operations by the shortest weighted path between the two molecules. In the basic learning process, i.e. that of learning to recognize one class, the RLM is presented with two finite sets C^+ (of positive training patterns) and C^- (of negative training patterns). The main objective of the learning process is to produce, if necessary, an expanded set S of operations and at least one corresponding weight vector w*, such that with the help of the distance WD(w*) (which induces in the set O the corresponding similarity field) the RLM can classify new patterns as positive or negative. The basic step in the learning process is optimization of the function F(w)=F1(w)/c+F2(w) where F1 is the smallest WD(w) distance between C^+ and C^-, F2 is the average WD(w) distance in C^+, and c is a small positive constant to prevent the overflow (when the values of F2 approach 0). One can show that the above continuous optimization problem can be reduced to the discrete one. During the learning process the new operations to be added to the set S of current operations are chosen among the compositions of the "optimum" current operations. The addition of such new operations "improves" the value of F, and therefore the learning process is guaranteed to converge. The concept of non-probabilistic class entropy, or complexity, w.r.t. the (current) state of the RLM can also be introduced. During the learning process this entropy DECREASES. Within the proposed model it is not difficult to see the relations between the learning process and the propositional class description. Moreover, most of the extensive psychological observation related, for example, to object perception (Object Perception: Structure and Process, eds. B.E. Shepp and S. Ballesteros, Lawrence Erlbaum Associates, 1989) can naturally be explained. --Lev Goldfarb From rsun at chaos.cs.brandeis.edu Tue Jan 15 17:12:08 1991 From: rsun at chaos.cs.brandeis.edu (Ron Sun) Date: Tue, 15 Jan 91 17:12:08 est Subject: No subject Message-ID: <9101152212.AA03681@chaos.cs.brandeis.edu> -------------------Technical Report available ------------- Integrating Rules and Connectionism for Robust Reasoning} Technical Report TR-CS-90-154 Ron Sun Brandeis University Computer Science Department rsun at cs.brandeis.edu Abstract A connectionist model for robust reasoning, CONSYDERR, is proposed to account for some common reasoning patterns found in commonsense reasoning and to remedy the brittleness problem. A dual representation scheme is devised, which utilizes both localist representation and distributed representation with features. We explore the synergy resulted from the interaction between these two types of representations, which helps to deal with problems such as partial information, no exact match, property inheritance, rule interaction, etc. Because of this, the CONSYDERR system is capable of accounting for many difficult patterns in commonsense reasoning. This work also shows that connectionist models of reasoning are not just an ``implementation" of their symbolic counterparts, but better computational models of common sense reasoning, taking into consideration of the approximate, evidential and adaptive nature of reasoning, and accounting for the spontaneity and parallelism in reasoning processes. +++ comments and suggestions are especially welcome +++ ------------ FTP procedures --------- ftp cheops.cis.ohio-state.edu >name: anonymous >passwork: neuron >binary >cd pub/neuroprose >get sun.integrate.ps.Z >quit uncompress sun.integrate.ps.Z lpr sun.integrate.ps From choukri at capsogeti.fr Wed Jan 16 09:26:23 1991 From: choukri at capsogeti.fr (Khalid Choukri) Date: Wed, 16 Jan 91 14:26:23 GMT Subject: neural sessions /13th IMACS World Congress Message-ID: <9101161426.AA17825@thor> --------------------------------------------------------- 13th IMACS World Congress on Computation and Applied Mathematics July 22-26,1991, Trinity college, Dublin, Ireland Neural Computing sessions Preliminary announcement and call for papers ----------------------------------------------------- In the scope of the 13th IMACS World Congress on Computation and Applied Mathematics that will be held on July 22-26, 1991 at Trinity college, Dublin, Ireland, several sessions will be devoted to Neural computing and Applied Mathematics. A typical session consists of six 20-minutes papers. Invited papers (tutorials ~ 1-hour) are welcome. Contributions from all fields related to neuro-computing techniques are welcome. Including applications to pattern recognition and classification, optimization problems, etc. Information and a non-exclusive list of topics may be obtained from the session organizer or the Congress Secretariat. Proceedings will be available at the Congress. A more formal Transactions will be available at a later date. Submission procedure : --------------------- Authors are solicited to submit proposals consisting of an abstract (one page, 500 words maximum) which must clearly state the purpose of the work, the specific original results obtained and their significance. The final paper length is two pages (IEEE two-column format). A first page of the proposal should contain the following information in the order shown: - Title. - Authors' names and affiliation. - Contact information (name, postal address, phone, fax and email address) - Domain area and key words: one or more terms describing the problem domain area. AUTHORS ARE ENCOURAGED to submit a preliminary version of the complete paper in addition to the abstract. Calendar: -------- Deadline for submission : February, 15, 1990 Notification of acceptance : March , 15 , 1991 Camera ready paper : April, 5, 1991 Three copies should be sent directly to the technical chairman of these sessions at the following address: Dr. Khalid Choukri Cap GEMINI Innovation 118, Rue de Tocqueville 75017, Paris, France Phone: (+33-1) 40 54 66 28 Fax: (+33-1) 42 67 41 39 e-mail choukri at capsogeti.fr For further information about the IMACS Congress in general, contact Post: IMACS '91 Congress Secretariat 26 Temple Lane Dublin 2 IRELAND Fax: (+353-1) 451739 Phone: (+353-1) 452081 From tenorio at ecn.purdue.edu Wed Jan 16 09:41:37 1991 From: tenorio at ecn.purdue.edu (Manoel F Tenorio) Date: Wed, 16 Jan 91 09:41:37 -0500 Subject: report: optimal NN size for classifiers Message-ID: <9101161441.AA27296@dynamo.ecn.purdue.edu> This report addresses the analysis of a new criterion for optimal classifier design. In particular we study the effects of the sizing ot the hidden layers and the optimal predicted value by this criterion. Resquest should be sent to: jld at ecn.purdue.edu TR-EE 91-5 There is a fee for requests outside USA,Canada and Mexico. On Optimal Adaptive Classifier Design Criterion - How many hidden units are necessary for an optimal neural network classifier? Wei-Tsih Lee Manoel Fernando Tenorio Parallel Distributed Structures Lab. Parallel Distributed Structures Lab. School of Electrical Engineering School of Electrical Engineering Purdue University Purdue University West Lafayette, IN 47907 West Lafayette, IN 47907 lwt at ecn.purdue.edu tenorio at ecn.purdue.edu Abstract A central problem in classifier design is the estimation of classification error. The difficulty in classifier design arises in situations where the sample distribution is unknown and the number of training samples available is limited. In this paper, we present a new approach for solving this problem. In our model, there are two types of classification error: approximation and generalization error. The former is due to the imperfect knowledge of the underlying sample distribution, while the latter is mainly the result of inaccuracies in parameter estimation, which is a consequence of the small number of training samples. We therefore propose a criterion for optimal classifier selection, called the Generalized Minimum Empirical Criterion (GMEE). The GMEE criterion consists of two terms, corresponding to the estimates of two types of error. The first term is the empirical error, which is the classification error observed for the training samples. The second is an estimate of the generalization error, which is related to the classifier complexity. In this paper we consider the Vapnik-Chervonenkis dimension (VCdim) as a measure of classifier complexity. Hence, the classifier which minimizes the criterion is the one with minimal error probability. Bayes consistency of the GMEE criterion has been proven. As an application, the criterion is used to design the optimal neural network classifier. A corollary to the Bayes optimality of neural network-based classifiers has been proven. Thus, our approach provides a theoretic foundation for the connectionist approach to optimal classifier design. Experimental results are given to validate the approach, followed by discussions and suggestions for future research. From bms at dcs.leeds.ac.uk Wed Jan 16 11:06:40 1991 From: bms at dcs.leeds.ac.uk (B M Smith) Date: Wed, 16 Jan 91 16:06:40 GMT Subject: Item for Distribution Message-ID: <21087.9101161606@csunb0.dcs.leeds.ac.uk> PRELIMINARY CALL FOR PARTICIPATION ================================== AISB91 University of Leeds 16-19 April 1991 Interested to know what is happening at the forefront of current AI research? Tired of going to AI conferences where you hear nothing but talk about applications? Bored at big AI conferences where there are so many parallel sessions that you don't know where to go? Saturated with small workshops that focus only on one narrow topic in AI? ==> the 1991 AISB conference may be just the thing for you ! AISB91 is organized by the Society for the Study of Artificial Intelligence and Simulation of Behaviour. It is not only the oldest regular conference in Europe on AI - which spawned the ECAI conferences in 1982 - but it is also the conference that has a tradition of focusing on research as opposed to applications. The 1991 edition of the conference is no different in this respect. The conference has a single session and covers the full spectrum of AI work, from robotics to knowledge systems. It is designed for researchers active in AI who want to follow the complete field. Papers were selected that are representative for ongoing research, particularly for research topics that promise new exciting avenues into a deeper understanding of intelligence. There will be a tutorial programme on Tuesday 16 April, followed by the technical programme from Wednesday 17 to Friday 19 April. The conference will be held at Bodington Hall, University of Leeds, a large student residence and conference centre. Bodington Hall is 4 miles from the centre of Leeds and set in 14 acres of private grounds. Leeds/Bradford airport is 6 miles away, with frequent flights from London Heathrow, Amsterdam and Paris. Leeds itself is easily accessible by rail (2 and a half hours from London) and the motorway network. The Yorkshire Dales National Park is close by, and the historic city of York is only 30 minutes away by rail. TECHNICAL PROGRAMME Wednesday 17 - Friday 19 April 1991 ======================================================== The technical programme sessions are organized around problem areas, not around approaches. This means sessions show how different schools of AI - knowledge-based approaches, logic based approaches, and neural networks - address the fundamental problems of AI. The technical programme lasts 2 and a half days. Each day has a morning session focusing on a particular area of AI. The first day this area is distributed AI, the second day new modes of reasoning, and the third day theorem proving and machine learning. The afternoon is devoted to research topics which are at the forefront of current research. On the first afternoon this topic is emergent functionality and autonomous agents. It presents the new stream of ideas for building autonomous agents featuring concepts like situatedness, physical symbol grounding, reactive systems, and emergence. On the second day the topic is knowledge level expert systems research. It reflects the paradigm shift currently experienced in knowledge based systems away from the symbol level and towards the knowledge level, both for design and knowledge acquisition. Each session has first a series of accepted papers, then two papers which treat the main theme from a principled point of view, and finally a panel. In addition the conference features three exciting invited speakers: Andy Clark who talks about the philosophical foundations of AI, Rolf Pfeifer who reflects on AI and emotion, and Tony Cohn who looks at the formal modeling of common sense. The conference is closed by the Programme Chairman, Luc Steels, who speculates on the role of consciousness in Artificial Intelligence. Here is a more detailed description of the various sessions and the papers contained in them: Distributed Intelligent Agents ============================== Research in distributed AI is concerned with the problem of how multiple agents and societies of agents can be organized to co-operate and collectively solve a problem. The first paper by Chakravarty (MIT) focuses on the problem of evolving agents in the context of Minsky's society of mind theory. It addresses the question how new agents can be formed by transforming existing ones and illustrates the theory with an example from game playing. Smieja (GMD, Germany) focuses on the problem of organizing networks of agents which consist internally of neural networks. Smieja builds upon the seminal work of Selfridge in the late fifties on the Pandemonium system. Bond (University of California) addresses the problem of regulating co-operation between agents. He seeks inspiration in sociological theory and proposes a framework based on negotiation. Finally Mamede and Martins (Technical University of Lisbon) address the problem of resource-bounded reasoning within the context of logical inference. Situatedness and emergence in autonomous agents =============================================== Research on robots and autonomous agents used to be focused strongly on low level mechanisms. As such there were few connections with the core problems of AI. Recently, there has been a shift of emphasis towards the construction of complete agents. This has lead to a review of some traditional concepts, such as the hierarchical decomposition of an agent into a perception module, a decision module and an action module and it has returned robotics research to the front of the AI stage. This session testifies to the renewed interest in the area. It starts with a paper by Bersini (Free University of Brussels) which is strongly within the new perspective of emphasizing situatedness and non-symbolic relations between perception and action. It discusses the trade-offs between reactive systems and goal-oriented systems. Seel (STC Technology, Harlow, UK) provides some of the formal foundations for understanding and building reactive systems. Jackson and Sharkey (University of Exeter) address the problem of symbol grounding: how signals can be related to concepts. They use a connectionist mechanism to relate spatial descriptions with results from perception. Cliff (University of Sussex) discusses an experiment in computational neuroethology. The next paper is from the Edinburgh Really Useful Robot project which has built up a strong tradition in building autonomous mobile robots. The paper will be given by Hallam (University of Edinburgh) and discusses an experiment in real-time control using toy cars. The final paper is by Kaelbling (Teleos Research, Palo Alto, California) who elaborates her proposals for principled programming of autonomous agents based on logical specifications. The panel which ends the session tries to put the current work on autonomous agents into the broader perspective of AI. The panel includes Smithers (University of Edinburgh), Kaelbling, Connah (Philips Research, UK), and Agre (University of Sussex). Following this session, on Wednesday evening, the conference dinner will be held at the National Museum of Photography, film and Television at Bradford. The evening will include a special showing in the IMAX auditorium, which has the largest cinema screen in Britain. New modes of reasoning ====================== Reasoning remains one of the core topics of AI. This session explores some of the current work to find new forms of reasoning. The first paper by Hendler and Dickens (University of Maryland) looks at the integration of neural networks and symbolic AI in the context of a concrete example involving an underwater robot. Euzenat and Maesano (CEDIAG/Bull, Louveciennes, France) address the problem of forgetting. Pfahringer (University of Vienna) builds further on research in constraint propagation in qualitative modelling. He proposes a mechanism to improve efficiency through domain variables. Ghassem-Sani and Steel (University of Essex) extend the arsenal of methods for non-recursive planning by introducing a method derived from mathematical induction. The knowledge level perspective =============================== Knowledge systems (also known as expert systems or knowledge-based systems) continue to be the most successful area of AI application. The conference does not focus on applications but on foundational principles for building knowledge systems. Recently there has been an important shift of emphasis from symbol level considerations (which focus on the formalism in which a system is implemented) to knowledge level considerations. The session highlights this shift in emphasis. The first paper by Pierret-Golbreich and Delouis (Universite Paris Sud) is related to work on the generic task architectures. It proposes a framework including support tools for performing analysis of the task structure of the knowledge system. Reichgelt and Shadbolt (University of Nottingham) apply the knowledge level perspective to the problem of knowledge acquisition. Wetter and Schmidt (IBM Germany) focus on the formalization of the KADS interpretation models which is one of the major frameworks for doing knowledge level design. Finally Lackinger and Haselbock (University of Vienna) focus on domain models in knowledge systems, particularly qualitative models for simulation and control of dynamic systems. Then there are two papers which directly address foundational issues. The first one by Van de Velde (VUB AI Lab, Brussels) clarifies the (difficult) concepts involved in knowledge level discussions of expert systems, particularly the principle of rationality. Schreiber, Akkermans and Wielinga (University of Amsterdam) critically examine the suitability of the knowledge level for expert system design. The panel involves Leitch (Heriot Watt University, Edinburgh), Wielinga, Van de Velde, Sticklen (Michigan State University), and Pfeifer (University of Zurich). Theorem proving and Machine learning =============== ================ The final set of papers focuses on recent work in theorem proving and in machine learning. The first paper by Giunchiglia (IRST Trento, Italy) and Walsh (University of Edinburgh) discusses how abstraction can be used in theorem proving and presents solid evidence to show that it is useful. Steel (University of Essex) proposes a new inference scheme for modal logic. Then there are two papers which represent current work on machine learning. The first one by Churchill and Young (University of Cambridge) reports on an experiment using SOAR concerned with modelling representations of device knowledge. The second paper by Elliott and Scott (University of Essex) compares instance-based and generalization-based learning procedures. TUTORIAL PROGRAMME - Tuesday 16 April 1991 ========================================== Six full-day tutorials will be offered on 16 April (subject to sufficient registrations for each.) Tutorial 1 Knowledge Base Coherence Checking ---------- Professor Jean-Pierre LAURENT University of Savoie FRANCE Like conventional software, AI Systems also need validation tools. Some of these tools must be specific, especially for validating Knowledge-Based Systems, and in particular for checking the coherence of a Knowledge Base (KB). In the introduction to this tutorial we will clarify the distinctions to be made between Validation, Verification, Static Analysis and Testing. We will present methods which try to check exhaustively for the coherence of a knowledge Base. Then we will present a pragmatic approach in which, instead of trying to assert the global coherence of a KB, it is proposed to check heuristically whether it contains incoherences. This approach is illustrated by the SACCO System, dealing with KBs which contain classes and objects, and furthermore rules with variables. Tutorial 2 Advanced Constraint Techniques ---------- Dr. Hans Werner Guesgen and Dr. Joachim Hertzberg German National Centre for Computer Science (GMD) Sankt Augustin, GERMANY This tutorial will present a coherent overview of the more recent concepts and approaches to constraint reasoning. It presents the concept of dynamic constraints as a formalism subsuming classical constraint satisfaction, constraint manipulation and relaxation, bearing a relationship to reflective systems; moreover, the tutorial presents approaches to parallel implementations of constraint satisfaction in general and dynamic constraints in particular. Tutorial 3 Functional Representation and Modeling ---------- Prof. Jon Sticklen and Dr. Dean Allemang* Michigan State University USA * Universitaet Zurich, SWITZERLAND A growing body of AI research centres on using the known functions of a device as indices to causal understanding of how the device "works". The results of functional representation and modeling have typically used this organization of causal understanding to produce tractable solutions to inherently complex modelling problems. In this tutorial, the fundamentals of functional representation and reasoning will be explained. Liberal use of examples throughout will illustrate the representational concepts underlying the functional approach. Contacts with other model based reasoning (MBR) techniques will be made whenever appropriate. Sufficient background will be covered to make this suitable for both those unacquainted with the MBR field, and for more experienced individuals who may be working now in MBR research. A general familiarity with AI is assumed. Participants should send in with their registration materials a one page description of a modeling problem which they face in their domain. Tutorial 4 Intelligent Pattern Recognition and Applications ---------- Prof. Patrick Wang M.I.T. Artificial Intelligence Laboratory and Northeastern University, Boston USA The core of pattern recognition, including "learning techniques" and "inference" plays an important and central role in AI. On the other hand, the methods in AI such as knowledge representation, semantic networks, and heuristic searching algorithms can also be applied to improve the pattern representation and matching techniques in many pattern recognition problems - leading to "smart" pattern recognition. Moreover, the recognition and understanding of sensory data like speech or images, which are major concerns in pattern recognition, have always been considered as important subfields of AI. This tutorial includes overviews of pattern recognition and articifical intelligence; including recent developments at MIT. The focus of the tutorial will be on the overlap and interplay between these fields. Tutorial 5 SILICON SOULS - Philosophical foundations of computing and AI ---------- Prof. Aaron Sloman University of Birmingham This will not be a technical tutorial. Rather the tutor will introduce a collection of philosophical questions about the nature of computation, the aims of AI, connectionist and non-connectionist approaches to AI, the relevance of computation to the study of mind, varieties of mechanism, consciousness, and the nature of emotions and other affective states. Considerable time will be provided for discussion by participants. Prof. Sloman has provided a list of pertinent questions, these will be sent to participants upon registration. Tutorial 6 Knowledge Acquisition -------- Dr. Nigel Shadbolt Nottingham University Practical methods for acquiring knowledge from experts. The methods described have been shown to be effective through the pioneering research at Nottingham which compared common and less common methods for eliciting knowledge from experts. This tutorial is an updated version of the knowledge acquisition tutorial given at AISB'89 which was well-attended and enthusiastically received. ======================================================================== For further information on the tutorials, mail tutorials at hplb.hpl.hp.com or tutorials at hplb.lb.hp.co.uk or tutorials%hplb.uucp at ukc.ac.uk For a conference programme and registration form, or general information about the conference, mail aisb91 at ai.leeds.ac.uk or write to: Barbara Smith AISB91 Local Organizer School of Computer Studies University of Leeds Leeds LS2 9JT U.K. 9 9 From uhr at cs.wisc.edu Wed Jan 16 20:11:04 1991 From: uhr at cs.wisc.edu (Leonard Uhr) Date: Wed, 16 Jan 91 19:11:04 -0600 Subject: Vision (What's wrong with Marr's model) Message-ID: <9101170111.AA07455@thor.cs.wisc.edu> Expanding on Peter Cariani's reply to Stephen Lehar about earlier brain-like vision models, I published a Psych Review paper around 1961-63 that briefly summarized a lot of them, and a book, "Pattern Recognition, Learning and Thought" (Prentice-Hall, about 1973) that concentrated on one approach. The Minsky-Papert book, as people have mentioned, inspired an almost lemming-like flight away from networks, and traditional AI has indeed been largely heuristic search through symbolic domains. But there has always been non-trad- itional work. Probably the type of computer vision research that comes closest to bridging the AI-network gap (which really doesn't exist) is the pyramid approach I and a number of others have been taking (e.g., see books edited by Tanimoto and Klinger, Acad. Press, 1980; by Rosenfeld around 1986; and by Uhr, 1988, Acad. Press). Len Uhr From millan at lsi.upc.es Thu Jan 17 06:55:00 1991 From: millan at lsi.upc.es (Jose del R. MILLAN) Date: 17 Jan 91 12:55 +0100 Subject: Reply to Shawn Lockery Message-ID: <166*millan@lsi.upc.es> Regarding Shawn Lockery's question about connectionist approaches to navigation, we have developed a reinforcement learning system that tackles the robot path-finding problem. In order to study the feasibility of our approach, a first version (Millan & Torras 1990a, 1990b) was required to generate a ``quasi-optimal path'' between two fixed configurations in a given workspace with obstacles. The fact that the robot is a point, allows us to concentrate on the capabilities of reinforcement learning for the problem at hand. The third reference describes a review of existing connectionist approaches to the problem. The abstracts of the first two papers can be found at the end of this message. A forthcoming paper will describe the second version of the system. The new system combines reinforcement with supervised and unsupervised learning techniques to solve a more complex instance of the problem, namely, to generate quasi-optimal paths from any initial configuration to a fixed goal in a given workspace with obstacles. Currently, we are extending the system to deal with dimensional mobile robots. References Millan, J. del R. & Torras, C. (1990a). Reinforcement connectionist learning for robot path finding: a comparative study. Proc. COGNITIVA-90, 123--131. Millan, J. del R. & Torras, C. (1990b). Reinforcement learning: discovering stable solutions in the robot path finding domain. Proc. 9th European Conference on Artificial Intelligence, 219--221. Millan, J. del R. & Torras, C. (To appear). Connectionist approaches to robot path finding. In O.M. Omidvar (ed.) Progress in Neural Networks Series, Vol. 3. Ablex Publishing Corporation. ------------------------------------------------------------------------------- Reinforcement Connectionist Learning for Robot Path Finding: A Comparative Study ABSTRACT. A connectionist approach to robot path finding has been devised so as to avoid some of the bottlenecks of algebraic and heuristic approaches, namely construction of configuration space, discretization and step predetermination. The approach relies on reinforcement learning. In order to identify the learning rule within this paradigm that best suits the problem at hand, an experimental comparison of 13 different such rules has been carried out. The most promising rule has been shown to be that using predicted comparison as reinforcement baseline, and the Hebbian formula as eligibility factor. ------------------------------------------------------------------------------- Reinforcement Learning: Discovering Stable Solutions in the Robot Path Finding Domain ABSTRACT. After outlining the drawbacks of classical approaches to robot path finding, a prototypical system overcoming some of them and demonstrating the feasibility of reinforcement connectionist learning approaches to the problem is presented. The way in which the information is codified and the computational model used allow to avoid both the explicit construction of configuration space required by algebraic approaches as well as the discretization and step homogeneity demands of heuristic search algorithms. In addition, the simulations show that finding feasible paths is not as computational expensive as it is usually assumed for a reinforcement learning system. Finally, a mechanism is incorporated into the system to ``stabilise'' learning once an acceptable path has been found. ------------------------------------------------------------------------------- Jose del R. MILLAN Institute for System Engineering and Informatics Commission of the European Communities. Joint Research Centre Building A36. 21020 ISPRA (VA). ITALY e-mail: j_millan at cen.jrc.it (try this one first, please), or millan at lsi.upc.es From XIN at PHYS4.anu.edu.au Fri Jan 18 12:24:28 1991 From: XIN at PHYS4.anu.edu.au (Xin Yao) Date: Fri, 18 Jan 91 12:24:28 EST Subject: Dear researchers, Message-ID: <9101180100.AA19697@anu.anu.edu.au> I'm going to write a survey report on the applications of evolutionary search procedures (like genetic algorithm) to neural networks (including any hybrids of evolutionary and connectionist learning). Any comments or references in this area are greatly welcomed. I will certainly post the bibliographies or the report if there are enough people interested in it. Thank you for your help. Xin Yao Email: xin at cslab.anu.edu.au Computer Sciences Laboratory Tel: (+616)/(06)2495097 (O) Research School of Physical Sciences (+616)/(06)2512662 (H) Australian National University Fax: (+616)/(06)2491884 GPO Box 4, Canberra, ACT 2601 AUSTRALIA From P.Refenes at cs.ucl.ac.uk Fri Jan 18 12:49:39 1991 From: P.Refenes at cs.ucl.ac.uk (P.Refenes@cs.ucl.ac.uk) Date: Fri, 18 Jan 91 17:49:39 +0000 Subject: Dear researchers, In-Reply-To: Your message of Fri, 18 Jan 91 12:24:28 -0500. <9101180100.AA19697@anu.anu.edu.au> Message-ID: YOU MAY BE INTERESTED IN THIS: CONSTRUCTIVE LEARNING by SPECIALISATION A. N. REFENES & S. VITHLANI Department of Computer Science University College London Gower Street London WC1 6BT, UK ABSTRACT This paper describes and evaluates a procedure for constructing and training multi-layered perceptrons by adding units to the network dynamically during training. The ultimate aim of the learning procedure is to construct an architecture which is sufficiently large to learn the problem but necessarily small to generalise well. Units are added as they are needed. By showing that the newly added unit makes fewer mistakes than before, and by training the unit not to disturb the earlier dynamics, eventual convergence to zero-errors is guaranteed. Other techniques operate in the opposite direction by pruning the network and removing "redundant" connections. Simulations suggest that this method is efficient in terms of learning speed, the number of free parameters in the network (i.e. the number of free parameters in the network (i.e. connections), and it compares well to other methods in terms of generalisation performance. From ST401843 at brownvm.brown.edu Fri Jan 18 22:57:17 1991 From: ST401843 at brownvm.brown.edu (thanasis kehagias) Date: Fri, 18 Jan 91 22:57:17 EST Subject: Another Point of View on what Connectionism REALLY is ... Message-ID: I am sorry that I show up a little late in this "what connectionism really is (should be)?" debate - I want to present another point of view and this really is a different discussion. But it got started in my mind by reading the latest debate and at the same time writing a paper. It is my impression that historically connectionism started as something and developed into something quite different. Researchers in the late 50's to (even) early 80's were mostly focusing on building intelligent/thinking networks. However, for many reasons, not least among them being the proverbial bandwagon effect and the increase of funds available for connectionist research, many "computationally" oriented researchers started conducting connectionist research. I repeat that all this is personal opinion. I should also say that I consider myself to be one of the more "computationally" oriented researchers. I will now describe why neural nets are an interesting subject to me, I suspect this is the case for other researchers. My interest in neural nets is NOT biologically motivated. It is an interesting conjecture that systems that look "like" the brain will behave "intelligently", but I do not feel qualified to pursue this. It is a good thing that there are people pursuing this direction, as well as (other ?) people trying to model actual brains. But neural nets are also parallel, distributed computers. Some advantages of PDP are obvious, e.g. faster computing, and other somewhat not so obvious, e.g. robustness of computing properties with respect to local damage. This is what I find really interesting in connectionism, and I believe some people agree with me. This point of view has an important implication. We no longer need to insist that a network develops its own internal representations without any external guidance. It's OK to try to incorporate in the network/algorithm as much knowledge and design as we can. It's also OK to look into other disciplines (statistics, stochastic control) to see what methods they have developed and try to incorporate them into our networks. What really counts is to make the algorithm fast and efficient. Is this really connectionism? Is it any different from, say, the art of parallel algorithms? I believe the answer is yes to both questions. The Book has "Parallel Distributed Computing" in its title and we are all looking at parallel and distributed networks. And, even if boundaries are blurry, connectionism has the distinct flavor of building things up from small, simple units, in a way that is different from, say, a parallel implementation of Kalman filtering. In conclusion, I repeat that the boundaries between connectionism and a number of other disiplines, e.g. parallel algorithms, cellular automata, statistics etc. are blurry. This is not a disadvantage, on the contrary, I think, we would benefit from taking a look at these disciplines and comparing their point of view at problems similar to the ones we are examining. From der%psych at Forsythe.Stanford.EDU Sat Jan 19 15:46:22 1991 From: der%psych at Forsythe.Stanford.EDU (Dave Rumelhart) Date: Sat, 19 Jan 91 12:46:22 PST Subject: Job Opportunity at Stanford University Message-ID: <9101192046.AA02884@psych> The Psychology Department at Stanford University currently has two job openings at least one of which may be appropriate for a connectionist. I enclose a copy of the advertisement which appeared in several publications. If you feel you may be appropriate or know of someone who may be appropriate please apply for the position. Note from the ad that we are open to people at any level and with a variety of interests. This means, in short, we are interested in the best person we can attract within reasonably broad guidelines. I personally hope that this person has connectionist interests. David Rumelhart Chair of the Search Committee Stanford University Psychology Department. The Department of Psychology plans two tenure-track appointments in the Sensory/Perceptual and/or Cognitive Sciences (including the biological basis of cognition) beginning in the academic year 1991. Appointments may be either at the tenured or non-tenured (assistant professor) level. Outstanding scientists who have strong research records in sensory/perceptual processes, cognitive neuroscience and/or computational/mathematical models of cognitive processes are encouraged to apply. Applicants should send a current curriculum vitae, copies of their most important scholarly papers, and letters of recommendation to: The Cognitive Sciences Search Committee, c/o Ms. Frances Martin, Department of Psychology, Bldg. 420, Stanford University, Stanford, California, 94305. The deadline for application is February 18, 1991, but applicants are encouraged to submit their materials as soon as possible. Stanford University is an Equal Opportunity Employer. From tds at ai.mit.edu Sat Jan 19 16:33:00 1991 From: tds at ai.mit.edu (Terence D. Sanger) Date: Sat, 19 Jan 91 16:33:00 EST Subject: Nips90 Preprint available from neuroprose archive Message-ID: <9101192133.AA04425@gelatinosa> The following preprint is available, and will appear in the Nips'90 proceedings: ------------------------------------------------------------------------------ Basis-Function Trees as a Generalization of Local Variable Selection Methods for Function Approximation Terence D. Sanger Local variable selection has proven to be a powerful technique for approximating functions in high-dimensional spaces. It is used in several statistical methods, including CART, ID3, C4, MARS, and others (see the bibliography for references to these algorithms). In this paper I present a tree-structured network which is a generalization of these techniques. The network provides a framework for understanding the behavior of such algorithms and for modifying them to suit particular applications. ------------------------------------------------------------------------------ Bibtex entry: @INCOLLECTION{sanger91, AUTHOR = {Terence D. Sanger}, TITLE = {Basis-Function Trees as a Generalization of Local Variable Selection Methods for Function Approximation}, BOOKTITLE = {Advances in Neural Information Processing Systems 3}, PUBLISHER = {Morgan Kaufmann}, YEAR = {1991}, EDITOR = {Richard P. Lippmann and John Moody and David S. Touretzky}, NOTE = {Proc. NIPS'90, Denver CO} } This paper can be obtained by anonymous ftp from the neuroprose database: unix> ftp cheops.cis.ohio-state.edu # (or ftp 128.146.8.62) Name (cheops.cis.ohio-state.edu:): anonymous Password (cheops.cis.ohio-state.edu:anonymous): ftp> cd pub/neuroprose ftp> binary ftp> get sanger.trees.ps.Z ftp> quit unix> uncompress sanger.trees.ps unix> lpr -P(your_local_postscript_printer) sanger.trees.ps # in some cases you will need to use the -s switch to lpr. Terry Sanger MIT, E25-534 Cambridge, MA 02139 USA tds at ai.mit.edu From ga1013 at sdcc6.UCSD.EDU Sun Jan 20 17:46:12 1991 From: ga1013 at sdcc6.UCSD.EDU (ga1013) Date: Sun, 20 Jan 91 14:46:12 PST Subject: Job Opportunity at Stanford University Message-ID: <9101202246.AA18825@sdcc6.UCSD.EDU> hangup From crr%shum.huji.ac.il at BITNET.CC.CMU.EDU Thu Jan 17 11:31:12 1991 From: crr%shum.huji.ac.il at BITNET.CC.CMU.EDU (crr%shum.huji.ac.il@BITNET.CC.CMU.EDU) Date: Thu, 17 Jan 91 18:31:12 +0200 Subject: Description vs Explanation Message-ID: <9101171631.AA25120@shum.huji.ac.il> John von Neumann via the ever-present Lev Goldfarb: The sciences do not try to explain, they hardly even try to interpret, they mainly make models. By a model is meant a mathematical constract which, with the addition of certain verbal interpretations, describes observed phenomena. The justification of such a mathematical construct is solely and precisely that it is expected to work. I don't know where this quote comes from, but I disagree. Science also has the goal of explaining phenomena, not merely describing them. For a descriptive model, simple descriptive (mathematical) adequacy is the goal, whereas an explanation purports to go further and account for "the way things really are." It may be worthwhile for people doing work in connectionist models in particular to at least think about the distinction. Sometimes I feel that we modelers are not always clear as to which kind of model we are putting forth. Charlie Rosenberg From GOLDFARB%unb.ca at UNBMVS1.csd.unb.ca Mon Jan 21 09:22:03 1991 From: GOLDFARB%unb.ca at UNBMVS1.csd.unb.ca (GOLDFARB%unb.ca@UNBMVS1.csd.unb.ca) Date: Mon, 21 Jan 91 10:22:03 AST Subject: Description vs Explanation In-Reply-To: Message of Thu, 17 Jan 91 11:31:12 EST from <@BITNET.CC.CMU.EDU:crr@shum.huj Message-ID: > Sometimes I feel that we modelers are not always clear as to > which kind of model we are putting forth. > > Charlie Rosenberg It might be that "sometimes" is "most of the time". -- Lev Goldfarb From der%psych at Forsythe.Stanford.EDU Mon Jan 21 15:22:00 1991 From: der%psych at Forsythe.Stanford.EDU (Dave Rumelhart) Date: Mon, 21 Jan 91 12:22:00 PST Subject: IJCNN-91-SEATTLE Message-ID: <9101212022.AA15402@psych> In my role as conference chairman of the International Joint Conference on Neural Networks to be held this summer (July 8-12) in Seattle, Washington, I would like to remind readers of this mailing list that the deadline for paper submissions is February 1, 1991. I would encourage submissions. The quality of a conference is largely determined by the quality of the submitted papers. As a further reminder, or in case you haven't seen a formal call for papers, I provide some of the details below. Papers may be submitted in the areas of neurobiology, optical and electronic implementations, image processing, vision, speech, network dynamics, optimization, robotics and control, learning and generalization, neural network architectures, applications and other areas in neural networks. Papers must be submitted in English (1 original and seven copies) maximum six pages, camera-ready on 8 1/2" x 11" white paper with 1" margins on all sides and un-numbered. Centered at the top of the first page should be the complete title, author name(s), affiliation(s) and mailing address(es). This is followed by a blank space and then the abstract up to 15 lines, followed by the text. A cover letter including the corresponding author's name, mailing address, telephone and fax number, technical area, oral or poster presentation preference. Send papers to IJCNN-91-SEATTLE, University of Washington, Conference Management, Attn: Sarah Eck, MS/GH-22, 5001 25th Ave. N.E., Seattle WA 98195. The program planning for this meeting is outstanding. The site of the meeting will, I think, be outstanding. A major contribution to the success of the meeting (and, I think, the success of the field) will be made by each quality paper submitted. I look forward to an exciting meeting and hope to see a strong contribution from participants on the connectionist mailing list. Thank you for your consideration. David E. Rumelhart, Conference Chair, IJCNN-91-SEATTLE From MURRE%rulfsw.LeidenUniv.nl at BITNET.CC.CMU.EDU Tue Jan 22 11:52:00 1991 From: MURRE%rulfsw.LeidenUniv.nl at BITNET.CC.CMU.EDU (MURRE%rulfsw.LeidenUniv.nl@BITNET.CC.CMU.EDU) Date: Tue, 22 Jan 91 11:52 MET Subject: Pointers to papers on the effect of implementation precision Message-ID: Dear connectionist researchers, We are in the process of designing a new neurocomputer. An important design consideration is precision: Should we use 1-bit, 4-bit, 8-bit, etc. representations for weights, activations, and other parameters? We are scaling-up our present neurocomputer, the BSP400 (Brain Style Processor with 400 processors), which uses 8-bit internal representations for activations and weights, but activations are exchanged as single bits (using partial time-coding induced by floating thresholds). This scheme does not scale well. Though we have tracked down scattered remarks in the literature on precision, we have not been able to find many systematic studies on this subject. Does anyone know of systematic simulations or analytical results of the effect of implementation precision on the performance of a neural network? In particular we are interested in the question of how (and to what extent) limited precision (i.e., 8-bits) implementations deviate from, say, 8-byte, double precision implementations. The only systematic studies we have been able to find so far deal with fault tolerance, which is only of indirect relevance to our problem: Brause, R. (1988). Pattern recognition and fault tolerance in non-linear neural networks. Biological Cybernetics, 58, 129-139. Jou, J., & J.A. Abraham (1986). Fault-tolerant matrix arithmetic and signal processing on highly concurrent computing structures. Proceedings of the IEEE, 74, 732-741. Moore, W.R. (1988). Conventional fault-tolerance and neural computers. In: R. Eckmiller, & C. Von der Malsburg (Eds.). Neural Computers. NATO ASI Series, F41, (Berling: Springer-Verlag), 29-37. Nijhuis, J., & L. Spaanenburg (1989). Fault tolerance of neural associative memories. IEE Proceedings, 136, 389-394. Thanks! From gary at cs.UCSD.EDU Tue Jan 22 13:51:45 1991 From: gary at cs.UCSD.EDU (Gary Cottrell) Date: Tue, 22 Jan 91 10:51:45 PST Subject: IJCNN-91-SEATTLE Message-ID: <9101221851.AA18040@desi.ucsd.edu> I am interested to know who picks the categories for submission. It has always seemed odd to me that there is no Natural Language processing or Cognitive Modeling categories. These must be relegated to "Other" by people who submit in these categories. g. From hwang at uw-isdl.ee.washington.edu Tue Jan 22 14:33:11 1991 From: hwang at uw-isdl.ee.washington.edu ( J. N. Hwang) Date: Tue, 22 Jan 91 11:33:11 PST Subject: Pointers to papers on the effect of implementation precision Message-ID: <9101221933.AA11885@uw-isdl.ee.washington.edu> We are in the process of finishing up a paper which gives a theoretical (systematic) derivation of the finite precision neural network computation. The idea is a nonlinear extension of "general compound operators" widely used for error analysis of linear computation. We derive several mathematical formula for both retrieving and learning of neural networks. The finite precision error in the retrieving phase can be written as a function of several parameters, e.g., number of bits of weights, number of bits for multiplication and accumlation, size of nonlinear table-look-up, truncation/rounding or jamming approaches, and etc. Then we are able to extend this retrieving phase error analysis to iterative learning to predict the necessary number of bits. This can be shown using a ratio between the finite precision error and the (floating point) back-propagated error. Simulations have been conducted and matched the theoretical prediction quite well. Hopefully, we can have a final version of this paper available to you soon. Jordan L. Holt and Jenq-Neng Hwang ,"Finite Precision Error Analysis of Neural Network Hardware Implementation," University of Washington, FT-10, Seattle, WA 98195 Best Regards, Jenq-Neng From tenorio at ecn.purdue.edu Tue Jan 22 15:15:41 1991 From: tenorio at ecn.purdue.edu (Manoel F Tenorio) Date: Tue, 22 Jan 91 15:15:41 -0500 Subject: TR EE 90-63 , Short term memory and Hysterisis Message-ID: <9101222015.AA04814@dynamo.ecn.purdue.edu> Subject: TR-EE 90-63: The Hystery Unit - short term memory Bcc: tenorio -------- The task of performing recognition of patterns on spatio-temporal signals is not an easy one, primarily due to the time structure of the signal. Classical methods of handling this problem have proven themselves unsatisfactory, and they range from "projecting out" the time axis, to "memorizing" the entire sequence before a decision can be made. In particular, the latter can be very difficult if no a priori information about signal length is present, if the signal can suffer compression and extension, or if the entire pattern is massively large, as in the case of time varying imagery. Neural Network models to solve this problem have either been based on the classical approach or on recursive loops within the network which can make learning algorithms numerically unstable. It is clear that for all the spatio-temporal processing, done by biological systems, some kind of short term memory is needed, and has been long conjectured. In this report, we have taken the first step at the design of a spatio-temporal system that deals naturally with the problems present in this type of processing. In particular we investigate the exchange of the simple sigmoid function, commonly used, by a hysterisis function. Later, with the addition of an integrator which represents the neuron membrane effect, we construct a simple computational device to perform spatio-pattern recognition tasks. The results are that for bipolar input sequence, this device remaps the entire sequence into a real number. Knowing the output of the device suffices for knowing the sequence. For trajectories embbeded in noise, the device shows superior recognition to other techniques. Furthermore, properties of the device allows the designer to determine the memory length, and explain with simple circuits sensitization and habituation phenomena. The report below deals with the device and its mathematical properties. Other forthcoming papers will concentrate on other aspects of circuits constructed with this device. ---------------------------------------------------------------------- Requests from within US, Canada, and Mexico: The technical report with figures has been/will soon be placed in the account kindly provided by Ohio State. Here is the instruction to get the files: ftp cheops.cis.ohio-state.edu (or, ftp 128.146.8.62) Name: anonymous Password: neuron ftp> cd pub/neuroprose ftp> mget tom.hystery* (type y and hit return) ftp> quit unix> uncompress tom.hystery*.Z unix> lpr -P(your_postscript_printer) tom.hystery.ps unix> lpr -P(your_Mac_laserwriter) tom.hystery_figs.ps Please contact mdtom at ecn.purdue.edu for technical difficulties. ---------------------------------------------------------------------- Requests from outside North America: The technical report is available at a cost of US$22.39 per copy, postage included. Please make checks payable to Purdue University in US dollars. You may send your requests, checks, and full first class mail address to: J. L. Dixon School of Electrical Engineering Purdue University West Lafayette, Indiana 47907 USA Please mention the technical report number: TR-EE 90-63. ---------------------------------------------------------------------- The Hystery Unit - A Short Term Memory Model for Computational Neurons M. Daniel Tom Manoel Fernando Tenorio Parallel Distributed Structures Laboratory School of Electrical Engineering Purdue University West Lafayette, Indiana 47907, USA December, 1990 Abstract: In this paper, a model of short term memory is introduced. This model is inspired by the transient behavior of neurons and magnetic storage as memory. The transient response of a neuron is hypothesized to be a combination of a pair of sigmoids, and a relation is drawn to the hysteresis loop found in magnetic materials. A model is created as a composition of two coupled families of curves. Two theorems are derived regarding the asymptotic convergence behavior of the model. Another conjecture claims that the model retains full memory of all past unit step inputs. From tenorio at ecn.purdue.edu Tue Jan 22 16:36:15 1991 From: tenorio at ecn.purdue.edu (Manoel F Tenorio) Date: Tue, 22 Jan 91 16:36:15 -0500 Subject: TR EE Short Term Memory - Hysterisis Message-ID: <9101222136.AA09950@dynamo.ecn.purdue.edu> Subject: TR-EE 90-63: The Hystery Unit - short term memory Bcc: tenorio -------- The task of performing recognition of patterns on spatio-temporal signals is not an easy one, primarily due to the time structure of the signal. Classical methods of handling this problem have proven themselves unsatisfactory, and they range from "projecting out" the time axis, to "memorizing" the entire sequence before a decision can be made. In particular, the latter can be very difficult if no a priori information about signal length is present, if the signal can suffer compression and extension, or if the entire pattern is massively large, as in the case of time varying imagery. Neural Network models to solve this problem have either been based on the classical approach or on recursive loops within the network which can make learning algorithms numerically unstable. It is clear that for all the spatio-temporal processing, done by biological systems, some kind of short term memory is needed, and has been long conjectured. In this report, we have taken the first step at the design of a spatio-temporal system that deals naturally with the problems present in this type of processing. In particular we investigate the exchange of the simple sigmoid function, commonly used, by a hysterisis function. Later, with the addition of an integrator which represents the neuron membrane effect, we construct a simple computational device to perform spatio-pattern recognition tasks. The results are that for bipolar input sequence, this device remaps the entire sequence into a real number. Knowing the output of the device suffices for knowing the sequence. For trajectories embbeded in noise, the device shows superior recognition to other techniques. Furthermore, properties of the device allows the designer to determine the memory length, and explain with simple circuits sensitization and habituation phenomena. The report below deals with the device and its mathematical properties. Other forthcoming papers will concentrate on other aspects of circuits constructed with this device. ---------------------------------------------------------------------- Requests from within US, Canada, and Mexico: The technical report with figures has been/will soon be placed in the account kindly provided by Ohio State. Here is the instruction to get the files: ftp cheops.cis.ohio-state.edu (or, ftp 128.146.8.62) Name: anonymous Password: neuron ftp> cd pub/neuroprose ftp> mget tom.hystery* (type y and hit return) ftp> quit unix> uncompress tom.hystery*.Z unix> lpr -P(your_postscript_printer) tom.hystery.ps unix> lpr -P(your_Mac_laserwriter) tom.hystery_figs.ps Please contact mdtom at ecn.purdue.edu for technical difficulties. ---------------------------------------------------------------------- Requests from outside North America: The technical report is available at a cost of US$22.39 per copy, postage included. Please make checks payable to Purdue University in US dollars. You may send your requests, checks, and full first class mail address to: J. L. Dixon School of Electrical Engineering Purdue University West Lafayette, Indiana 47907 USA Please mention the technical report number: TR-EE 90-63. ---------------------------------------------------------------------- The Hystery Unit - A Short Term Memory Model for Computational Neurons M. Daniel Tom Manoel Fernando Tenorio Parallel Distributed Structures Laboratory School of Electrical Engineering Purdue University West Lafayette, Indiana 47907, USA December, 1990 Abstract: In this paper, a model of short term memory is introduced. This model is inspired by the transient behavior of neurons and magnetic storage as memory. The transient response of a neuron is hypothesized to be a combination of a pair of sigmoids, and a relation is drawn to the hysteresis loop found in magnetic materials. A model is created as a composition of two coupled families of curves. Two theorems are derived regarding the asymptotic convergence behavior of the model. Another conjecture claims that the model retains full memory of all past unit step inputs. From erol at ehei.ehei.fr Tue Jan 22 10:25:13 1991 From: erol at ehei.ehei.fr (Erol Gelenbe) Date: Tue, 22 Jan 91 15:27:13 +2 Subject: Charlie Rosenberg's message Message-ID: <9101231128.AA11600@inria.inria.fr> I fully agree with him. Our purpose is to understand, and then to be able to explain to others, rather than simply to represent and manipulate models or equations. Erol Gelenbe From weber at icsib Wed Jan 23 11:36:47 1991 From: weber at icsib (Susan Weber) Date: Wed, 23 Jan 91 08:36:47 PST Subject: bank credit and neural nets Message-ID: <9101231636.AA02813@icsib> From bradley at ivy.Princeton.EDU Wed Jan 23 13:14:09 1991 From: bradley at ivy.Princeton.EDU (Bradley Dickinson) Date: Wed, 23 Jan 91 13:14:09 EST Subject: Neural Network Council Awards Message-ID: <9101231814.AA07390@ivy.Princeton.EDU> Nominations Sought for IEEE Neural Networks Council Awards The IEEE Neural Networks Council is soliciting nominations for two new awards. Pending final approval the IEEE, it is planned to present these awards for the first time at the July 1991 International Joint Conference on Neural Networks. Nominations for these awards should be submitted in writing according to the instructions given below. IEEE Transactions on Neural Networks Outstanding Paper Award This is an award of $500 for the outstanding paper published in the IEEE Transactions on Neural Networks in the previous two-year period. For 1991, all papers published in 1990 (Volume 1) in the IEEE Transactions on Neural Networks are eligible. For a paper with multiple authors, the award will be shared by the coauthors. Nominations must include a written statement describing the outstanding characteristics of the paper. The deadline for receipt of nominations is March 31, 1991. Nominations should be sent to Prof. Bradley W. Dickinson, NNC Awards Chair, Dept. of Electrical Engineering, Princeton University, Princeton, NJ 08544-5263. IEEE Neural Networks Council Pioneer Award This award has been established to recognize and honor the vision of those people whose efforts resulted in significant contributions to the early concepts and developments in the neural networks field. Up to three awards may be presented annually to outstanding individuals whose main contribution has been made at least fifteen years earlier. The recognition is engraved on the Neural Networks Pioneer Medal specially struck for the Council. Selection of Pioneer Medalists will be based on nomination letters received by the Pioneer Awards Committee. All who meet the contribution requirements are eligible, and anyone can nominate. The award is not approved posthumously. Written nomination letters must include a detailed description of the nominee's contributions and must be accompanied by full supporting documentation. For the 1991 Pioneer Award, nominations must be received by March 1, 1991. Nominations should be sent to Prof. Bradley W. Dickinson, NNC Pioneer Award Chair, Department of Electrical Engineering, Princeton University, Princeton, NJ 08544-5263. Questions and preliminary inquiries about the above awards should be directed to Prof. Bradley W. Dickinson, NNC Awards Chair; telephone: (609)-258-4644, electronic mail: bradley at ivy.princeton.edu From mclennan at cs.utk.edu Wed Jan 23 16:28:54 1991 From: mclennan at cs.utk.edu (mclennan@cs.utk.edu) Date: Wed, 23 Jan 91 16:28:54 -0500 Subject: tech report: continuous spatial automata Message-ID: <9101232128.AA04465@maclennan.cs.utk.edu> The following technical report is now available: Continuous Spatial Automata B. J. MacLennan Department of Computer Science University of Tennessee Knoxville, TN 37996-1301 maclennan at cs.utk.edu CS-90-121 November 26, 1990 ABSTRACT A _continuous_spatial_automaton_ is analogous to a cellular auto- maton, except that the cells form a continuum, as do the possible states of the cells. After an informal mathematical description of spatial automata, we describe in detail a continuous analog of Conway's ``Life,'' and show how the automaton can be implemented using the basic operations of field computation. Typically a cellular automaton has a finite (sometimes denu- merably infinite) set of cells, often arranged in a one or two dimensional array. Each cell can be in one of a number of states. In contrast, a continuous spatial automaton has a one, two or higher dimensional continuum of _loci_ (corresponding to cells), each of which has a state drawn from a continuum (typically [0,1]). The state is required to vary continuously with the locus. In a cellular automaton there is a transition function that determines the state of a cell at the next time step based on the state of it and a finite number of neighbors at the current time step. A discrete-time spatial automaton is very similar: the future state of a locus is a continuous function of the states of the loci in a (closed or open) bounded neighborhood of the given locus. The report is available as a compressed postscript file in the pub/neuroprose subdirectory; it may be obtained with the Getps script: Getps maclennan.csa.ps.Z For HARDCOPY send your address to: library at cs.utk.edu For other correspondence: Bruce MacLennan Department of Computer Science 107 Ayres Hall The University of Tennessee Knoxville, TN 37996-1301 (615)974-0994/5067 maclennan at cs.utk.edu From mdtom at ecn.purdue.edu Thu Jan 24 11:28:41 1991 From: mdtom at ecn.purdue.edu (M Daniel Tom) Date: Thu, 24 Jan 91 11:28:41 -0500 Subject: TR-EE 90-63 figs fix Message-ID: <9101241628.AA07433@transform.ecn.purdue.edu> Dear Connectionists, Following the instructions in Mac2ps (found in the neuroprose database) I created the file tom.hystery_figs.ps.Z (using command-k from a Mactintosh application). However people have trouble printing it. Steve, talking below, reports the problem to me, and helps solve the problem. I have followed Steps 1 and 2 below, and put the new file in neuroprose/Inbox/tom.hystery_figs.ps.Z, which may be moved to neuroprose later. I hope you have success following Step 3 below. Daniel Tom > Date: Wed, 23 Jan 91 16:13:04 -0500 > From: bradtke at envy.cs.umass.edu > To: mdtom at ecn.purdue.edu > Subject: printing the figures for the hystery unit paper > > > Well, we finally got the figure pages to print. > We had to play with the file retrieved from neuroprose a bit > first, though. > > step 1: change all \r in the file to > > step 2: remove the Apple ProcSet header > > step 3: use SendPS to print it on a Mac LaserWriter > (This wouldn't work till steps 1 and 2 were done.) > > > I also got it to print on a unix system on a dec ln03 -ps by > changing step 3 to > cat lprep68.pro tom.hystery_figs.ps graphics-restore.ps | lpr > where the file lprep68.pro is the Apple Mac PostScript prolog, and > the file graphics-restore.ps just does a showpage. The only problem > with this second solution is that all of the figures printed on just > one page. > > That's all I know. > > Steve > daniel From gyen at steinbeck.helios.nd.edu Thu Jan 24 13:42:05 1991 From: gyen at steinbeck.helios.nd.edu (Gune Yen) Date: Thu, 24 Jan 91 13:42:05 EST Subject: reference for nonsymmetric interconnection networks Message-ID: <9101241842.AA00823@steinbeck.helios.nd.edu> Hi, Synthesis techniques for associative memories via artificial feedback neural networks have been studied for many years. Among them, I am very interested in any design method results in neural networks with non-symmetric interconnecting structure. Obviously, the requirement to have symmetric interconnection weights will pose difficulties in implementation and may result in spurious states as well. I am aware of the work given below: A. Lapedes and R. Farber, "A Self-Optimizing Nonsymmetrical Neural Net for Content Addressable Memory and Pattern Recognition", Physica 22D, 1986, pp. 247-259. J. A. Farrell and A. N. Michel, "Analysis and Synthesis Techniques for Hopfield Type Synchronous Discrete Neural Networks with Application to Associative Memory", IEEE Transactions on Circuits and Systems, Vol. 37, No. 11, November 1990, pp. 1356-1366. Could anyone who familiar in this type of problem provide any references and comments in terms of symmetric/non-symmetric interconnection weights, with/without self-feedback term, difficulties in analog/digital implementation, and etc. Thank you very much. Gary at gyen at steinbeck.helios.nd.edu University of Notre Dame From sontag at hilbert.rutgers.edu Thu Jan 24 16:16:17 1991 From: sontag at hilbert.rutgers.edu (Eduardo Sontag) Date: Thu, 24 Jan 91 16:16:17 EST Subject: TR available from neuroprose; feedforward nets Message-ID: <9101242116.AA27780@hilbert.rutgers.edu> I have deposited in the neuroprose archive the extended version of my NIPS-90 Proceedings paper. The title is: "FEEDFORWARD NETS FOR INTERPOLATION AND CLASSIFICATION" and the abstract is: "This paper deals with single-hidden-layer feedforward nets, studying various measures of classification power and interpolation capability. Results are given showing that direct input to output connections in threshold nets double the recognition but not the interpolation power, while using sigmoids rather than thresholds allows (at least) doubling both." (NOTE: This is closely related to report SYCON-90-03, which was put in the archive last year under the title "sontag.capabilities.ps.Z". No point in retrieving unless you found the other paper of interest. The current paper besically adds a few results on interpolation.) -eduardo ----------------------------------------------------------------------------- To obtain copies of the postscript file, please use Jordan Pollack's service: Example: unix> ftp cheops.cis.ohio-state.edu # (or ftp 128.146.8.62) Name (cheops.cis.ohio-state.edu:): anonymous Password (cheops.cis.ohio-state.edu:anonymous): ftp> cd pub/neuroprose ftp> binary ftp> get (remote-file) sontag.nips90.ps.Z (local-file) sontag.nips90.ps.Z ftp> quit unix> uncompress sontag.nips90.ps.Z unix> lpr -P(your_local_postscript_printer) sontag.nips90.ps ---------------------------------------------------------------------------- If you have any difficulties with the above, please send e-mail to sontag at hilbert.rutgers.edu. DO NOT "reply" to this message, please. NOTES about FTP'ing, etc: (1) The last time I posted something, I forgot to include the ".Z" in the file name in the above "remote-file" line, and I received many messages telling me that FTP didn't find the file. Sorry for that. Please note that most files in the archive are compressed, and people may forget to mention the ".Z". (2) I also received some email (and saw much discussion in a bboard) concerning the printer errors with the file. Please note that postscript files sometimes require a fair amount of memory from the printer, especially if they contain illustrations, and many smaller printers do not have enough memory. This may result on some pages not being printed, or the print job not being done at all. If you experience this problem with papers you retrieve (mine or from others), I suggest that you ask the author to email you a source file (e.g. LaTex) or a postscript file sans figures. Also, some postscript files are "nonconforming", and this may cause problems with certain printers. From lacher at lambda.cs.fsu.edu Thu Jan 24 16:16:45 1991 From: lacher at lambda.cs.fsu.edu (Chris Lacher) Date: Thu, 24 Jan 91 16:16:45 -0500 Subject: Abstract Message-ID: <9101242116.AA19172@lambda.cs.fsu.edu> Backpropagation Learning in Expert Networks by R. C. Lacher, Susan I. Hruska, and David C. Kuncicky Department of Computer Science Florida State University ABSTRACT. Expert networks are event-driven, acyclic networks of neural objects derived from expert systems. The neural objects process information through a non-linear combining function that is different from, and more complex than, typical neural network node processors. We develop backpropagation learning for acyclic, event-driven nets in general and derive a specific algorithm for learning in EMYCIN-derived expert networks. The algorithm combines backpropagation learning with other features of expert nets, including calculation of gradients of the non-linear combining functions and the hypercube nature of the knowledge space. Results of testing the learning algorithm with a medium-scale (97 node) expert network are presented. For a copy of this preprint send an email request with your (snail)MAIL ADDRESS and the TITLE of the preprint to: santan at nu.cs.fsu.edu --- Chris Lacher From prowat at UCSD.EDU Thu Jan 24 19:05:13 1991 From: prowat at UCSD.EDU (Peter Rowat) Date: Thu, 24 Jan 91 16:05:13 PST Subject: Learning with "realistic" neurons Message-ID: <9101250005.AA20022@crayfish.UCSD.EDU> Gary Cottrell recently referred to work I am doing with models of the gastric mill network in the lobster's stomatogastric ganglion. This work is not published, but I do have a related paper which generalizes BP to arbitrarily complex models (amongst other things), and which is now available by ftp from the neuroprose archive. Namely: Peter Rowat and Allen Selverston (1990). Learning algorithms for oscillatory networks with gap junctions and membrane currents. To appear in: NETWORK: Computation in Neural systems, Volume 2, Issue 1, February 1991. Abstract: We view the problem of parameter adjustment in oscillatory neural networks as the minimization of the difference between two limit cycles. Backpropagation is described as the application of gradient descent to an error function that computes this difference. A mathematical formulation is given that is applicable to any type of network model, and applied to several models. By considering a neuron equivalent circuit, the standard connectionist model of a neuron is extended to allow gap junctions between cells and to include membrane currents. Learning algorithms are derived for a two cell network with a single gap junction, and for a pair of mutually inhibitory neurons each having a simplified membrane current. For example, when learning in a network in which all cells have a common, adjustable, bias current, the value of the bias is adjusted at a rate proportional to the difference between the sum of the target outputs and the sum of the actual outputs. When learning in a network of n cells where a target output is given for every cell, the learning algorithm splits into n independent learning algorithms, one per cell. For networks containing gap junctions, a gap junction is modelled as a conductance times the potential difference between the two adjacent cells. The requirement that a conductance g must be positive is enforced by replacing g by a function pos(g*) whose value is always positive, for example exp(0.1 g*), and deriving an algorithm that adjusts the parameter g* in place of g. When target output is specified for every cell in a network with gap junctions, the learning algorithm splits into fewer independent components, one for each gap-connected subset of the network. The learning algorithm for a gap-connected set of cells cannot be parallelized further. As a final example, a learning algorithm is derived for a mutually inhibitory two-cell network in which each cell has a membrane current. This generalized approach to backpropagation allows one to derive a learning algorithm for almost any model neural network given in terms of differential equations. It is one solution to the problem of parameter adjustment in small but complex network models. - --------------------------------------------------------------------------- Copies of the postscript file rowat.learn-osc.ps.Z may be obtained from the pub/neuroprose directory in cheops.cis.ohio-state.edu. Either use the Getps script or do this: unix-1> ftp cheops.cis.ohio-state.edu # (or ftp 128.146.8.62) Connected to cheops.cis.ohio-state.edu. Name (cheops.cis.ohio-state.edu:): anonymous 331 Guest login ok, sent ident as password. Password: neuron 230 Guest login ok, access restrictions apply. ftp> cd pub/neuroprose ftp> binary ftp> get rowat.learn-osc.ps.Z ftp> quit unix-2> uncompress rowat.learn-osc.ps.Z unix-3> lpr -P(your_local_postscript_printer) rowat.learn-osc.ps (The file starts with 7 bitmapped figures which are slow to print.) From FEGROSS%WEIZMANN.BITNET at bitnet.cc.CMU.EDU Fri Jan 25 04:31:44 1991 From: FEGROSS%WEIZMANN.BITNET at bitnet.cc.CMU.EDU (Tal Grossman) Date: Fri, 25 Jan 91 11:31:44 +0200 Subject: Anti SCAD NNs Message-ID: <8A161BC8400001B6@BITNET.CC.CMU.EDU> An artificial neural network for the recognition and deception of SCAD missiles is urgently needed. Assign all your Back-Props and ARTs to that mission. A huge training set is constantly presented (in a cyclic order) by the CNN... Yours faithfully - Tal Grossman (somewhere near Tel Aviv). P.S. Only Neural that can perform well with a gas mask should be considered. Nervous neural networks are out of the question. From eniac!lba at relay.EU.net Fri Jan 25 10:22:53 1991 From: eniac!lba at relay.EU.net (Luis Borges de Almeida) Date: Fri, 25 Jan 91 15:22:53 GMT Subject: reference for nonsymmetric interconnection networks In-Reply-To: Gune Yen's message of Thu, 24 Jan 91 13:42:05 EST <9101241842.AA00823@steinbeck.helios.nd.edu> Message-ID: <9101251522.AA09062@eniac.inesc.pt> The paper referenced below (reprints can be sent to those interested), gives results of some experiments on the training of small Hopfield-style networks by recurrent backpropagation. The examples given are all for symmetric networks, but the procedure can be extended to the nonsymmetric case, by just not imposing weight symmetry during the training. We performed a few tests for that situation, with good results, though in that case there is no guarantee that the training will always result in a stable network. Those tests are not reported in the paper, because we wanted to limit ourselves to the more Hopfield-like case. The tests reported in the paper all use the self-feedback term. On the other hand, the paper reports two very simple experiments on other extensions of Hopfield networks: use of hidden units, and storage of analog-valued patterns. Luis B. Almeida INESC Phone: +351-1-544607 Apartado 10105 Fax: +351-1-525843 P-1017 Lisboa Codex Portugal lba at inesc.inesc.pt lba at inesc.uucp (if you have access to uucp) --------------------------------------- REFERENCE: Luis B. Almeida and Joao P. Neto, "Recurrent Backpropagation and Hopfield Networks", in F.Fogelman-Soulie and J. Herault (eds.), Proc. of the NATO ARW on Neurocomputing, Algorithms, Architectures and Implementations, Les Arcs, France, Feb/Mar 1989, Springer-Verlag (1990). From lwyse at park.bu.edu Fri Jan 25 14:12:22 1991 From: lwyse at park.bu.edu (lwyse@park.bu.edu) Date: Fri, 25 Jan 91 14:12:22 -0500 Subject: reference for nonsymmetric interconnection networks In-Reply-To: connectionists@c.cs.cmu.edu's message of 25 Jan 91 02:33:42 GM Message-ID: <9101251912.AA06868@kendall.bu.edu> The ART architectures of Carpenter and Grosberg are all associative memories using non-symmetric interconnections. There is a good paper on ART II in Applied Optics 26:23 (December, '87) pg 4919. - lonce XXX XXX Lonce Wyse | X X Center for Adaptive Systems \ | / X X Boston University \ / 111 Cummington St. Boston,MA 02215 ---- ---- X X X X "The best things in life / \ XXX XXX are emergent." / | \ | From haussler at saturn.ucsc.edu Fri Jan 25 20:31:37 1991 From: haussler at saturn.ucsc.edu (David Haussler) Date: Fri, 25 Jan 91 17:31:37 -0800 Subject: tech rep on overfitting, decision theory, PAC learning, and... Message-ID: <9101260131.AA05685@saturn.ucsc.edu> TECHNICAL REPORT AVAILABLE ---------------------------- Decision Theoretic Generalizations of the PAC Model for Neural Net and Other Learning Applications David Haussler UCSC-CRL-91-02 September, 1989 Revised: December, 1990 haussler at saturn.ucsc.edu Baskin Center for Computer Engineering and Information Sciences University of California, Santa Cruz, CA 95064 Abstract: We describe a generalization of the PAC learning model that is based on statistical decision theory. In this model the learner receives randomly drawn examples, each example consisting of an instance $x \in X$ and an outcome $y \in Y$, and tries to find a hypothesis $h : X \rightarrow A$, where $h \in \cH$, that specifies the appropriate action $a \in A$ to take for each instance $x$, in order to minimize the expectation of a loss $\L(y,a)$. Here $X$, $Y$, and $A$ are arbitrary sets, $\L$ is a real-valued function, and examples are generated according to an arbitrary joint distribution on $X \times Y$. Special cases include the problem of learning a function from $X$ into $Y$, the problem of learning the conditional probability distribution on $Y$ given $X$ (regression), and the problem of learning a distribution on $X$ (density estimation). We give theorems on the uniform convergence of empirical loss estimates to true expected loss rates for certain hypothesis spaces $\cH$, and show how this implies learnability with bounded sample size, disregarding computational complexity. As an application, we give distribution-independent upper bounds on the sample size needed for learning with feedforward neural networks. Our theorems use a generalized notion of VC dimension that applies to classes of real-valued functions, adapted from Pollard's work, and a notion of {\em capacity} and {\em metric dimension} for classes of functions that map into a bounded metric space. The report can be retrieved by anonymous ftp from the UCSC Tech report library. An example follows: unix> ftp midgard.ucsc.edu # (or ftp 128.114.134.15) Connected ... Name (...): anonymous Password: yourname at cs.anyuniversity.edu (i.e. your email address) (Please use your email address so we can correspond with you.) Guest login ok, access restrictions apply. ftp> cd pub/tr ftp> binary ftp> get ucsc-crl-91-02.ps.Z 200 PORT command successful. 150 Opening BINARY mode data connection for ucsc-crl-91-02.ps.Z (576429 bytes). 226 Transfer complete. local: ucsc-crl-91-02.ps.Z remote: ucsc-crl-91-02.ps.Z 576429 bytes received in 10 seconds (70 Kbytes/s) ftp> quit unix> uncompress ucsc-crl-91-02.ps.Z unix> lpr -P(your_local_postscript_printer) ucsc-crl-91-02.ps (Note: you will need a printer with a large memory.) (also: some other UCSC tech reports are available as well and more will be added soon. ftp the file INDEX to see what's there.) If you have any difficulties with the above, please send e-mail to jean at cis.ucsc.edu. DO NOT "reply" to this message, please. -David From N.E.Sharkey at cs.exeter.ac.uk Fri Jan 25 09:54:38 1991 From: N.E.Sharkey at cs.exeter.ac.uk (Noel Sharkey) Date: Fri, 25 Jan 91 14:54:38 GMT Subject: IJCNN-91-SEATTLE In-Reply-To: Gary Cottrell's message of Tue, 22 Jan 91 10:51:45 PST <9101221851.AA18040@desi.ucsd.edu> Message-ID: <16315.9101251454@entropy.cs.exeter.ac.uk> I agree with gary about CNLP (and cognition). Natural language does get ignored at many of the connectionist (or neural net) meetings. It may be that up until fairly recently there wasn't much research in that area. But there is certainly quite a lot now. - (You can mail lyn at my address below for a potted history and references up to 1990). There has been an explosion of work since 1988 and much of it is related to cognitive areas (e.g. whether or not connectionist research offers a new theory of representation). CNLP also offers much of interest to general neuro-computing such as the nature of connectionist compositionality and the encoding and recognition of recursive structures. CNLP research can provide input into some of the big questions about the relationship between neural net and mind (i use the more neutral term neural net rather than brain), though of course i am broadening the field here. Obviously, i would like to see this area represented more at the major conferences. Do others feel that this field is being backgrounded? and if so why do you think it is? Perhaps it is felt that this is an inappropriate area for connectionist research - that it is too high level, or the it should be left to the symbol grinders. who knows? From barto at envy.cs.umass.edu Sat Jan 26 09:55:38 1991 From: barto at envy.cs.umass.edu (Andy Barto) Date: Sat, 26 Jan 91 09:55:38 EST Subject: reference for nonsymmetric interconnection networks Message-ID: <9101261455.AA16863@envy.cs.umass.edu> From VAINA at buenga.bu.edu Sat Jan 26 21:20:00 1991 From: VAINA at buenga.bu.edu (VAINA@buenga.bu.edu) Date: Sat, 26 Jan 91 21:20 EST Subject: The COMPUTING BRAIN LECTURE AT BU-ENGINEERING Message-ID: From: BUENGA::CORTEX 26-JAN-1991 18:58 To: @CORTEX-NEW,IN%CORTEX-IN-DISTRIBUTION CC: CORTEX Subj: COMPUTING BRAIN LECTURE SERIES - John Hopfield *************************************************************************** THE COMPUTING BRAIN LECTURE SERIES *************************************************************************** " THE DYNAMICS OF COMPUTING " Professor John Hopfield California Institute of Technology Wednesday, February 27, 1991 at 5 pm Old Engineering Building - Room 150 110 Cummington Street, Boston, Ma Tea at 4 pm in Room 129 (same address as above) Lecture open to all For further information contact: Professor Lucia M. Vaina 353-2455 or vaina at buenga.bu.edu *************************************************************************** From hertz at nordita.dk Mon Jan 28 06:04:05 1991 From: hertz at nordita.dk (hertz@nordita.dk) Date: Mon, 28 Jan 91 11:04:05 GMT Subject: preprint available Message-ID: <9101281104.AA01456@thor.dk> The following technical report has been placed in the neuroprose archives at Ohio State University: Dynamics of Generalization in Linear Perceptrons Anders Krogh John Hertz Niels Bohr Institut Nordita Abstract: We study the evolution of the generalization ability of a simple linear perceptron with N inputs which learns to imitate a ``teacher perceptron''. The system is trained on p = \alpha N binary example inputs and the generalization ability measured by testing for agreement with the teacher on all 2^N possible binary input patterns. The dynamics may be solved analytically and exhibits a phase transition from imperfect to perfect generalization at \alpha = 1. Except at this point the generalization ability approaches its asymptotic value exponentially, with critical slowing down near the transition; the relaxation time is \propto (1-\sqrt{\alpha})^{-2}. Right at the critical point, the approach to perfect generalization follows a power law \propto t^{-1/2}. In the presence of noise, the generalization ability is degraded by an amount \propto (\sqrt{\alpha}-1)^{-1} just above \alpha = 1. This paper will appear in the NIPS-90 proceedings. To retrieve it by anonymous ftp, do the following: unix> ftp cheops.cis.ohio-state.edu # (or ftp 128.146.8.62) Name (cheops.cis.ohio-state.edu:): anonymous Password (cheops.cis.ohio-state.edu:anonymous): ftp> cd pub/neuroprose ftp> binary ftp> get krogh.generalization.ps.Z ftp> quit unix> uncompress krogh.generalization.ps unix> lpr -P(your_local_postscript_printer) krogh.generalization.ps An old-fashioned paper preprint version is also available -- send requests to hertz at nordita.dk or John Hertz Nordita Blegdamsvej 17 DK-2100 Copenhagen Denmark From pjh at compsci.stirling.ac.uk Mon Jan 28 11:07:12 1991 From: pjh at compsci.stirling.ac.uk (Peter J.B. Hancock) Date: 28 Jan 91 16:07:12 GMT (Mon) Subject: MSc in NEURAL COMPUTATION Message-ID: <9101281607.AA01507@uk.ac.stir.cs.lira> M.Sc. in NEURAL COMPUTATION: A one-year full time course at the University of Stirling, Scotland, offered by the Centre for Cognitive and Computational Neuroscience (CCCN), and the Departments of Psychology, Computing Science, and Applied Mathematics. Aims and context: The course is designed for students entering the field from any of a variety of disciplines, e.g. Computing, Psychology, Biology, Engineering, Mathematics, Physics. It aims to combine extensive practical experience with a concern for basic principles. The study of neural computation in general is combined with an in-depth analysis of vision. The first few weeks of the course form an integrated crash course in the basic techniques and ideas. During the autumn semester lectures, seminars, and specified practical exercises predominate. In the spring and summer work based on each student's own interests and abilities predominates. This culminates in a research project that can be submitted anytime between July 1 and September 1. Where work on the M. Sc. has been of a sufficiently high standard it can be converted into the first year of a Ph. D. program. Courses: Autumn: 1. Principles of neural computation. 2. Principles of vision. 3. Cognitive Neuroscience. 4. Computational and Mathematical techniques. Spring and summer: 1. Advanced practical courses, including e.g. properties, design and use of neurocomputational systems, image processing, visual psychophysics. 2. Advanced topics in neural computation, vision, and cognitive neuroscience. 3. Research project. CCCN: The CCCN is a broadly-based interdisciplinary research centre. It has a well established reputation for work on vision, neural nets, and neuropsychology. The atmosphere is informal, friendly, and enthusiastic. Research meetings are held once or twice a week during semester. Students, research staff, and teaching staff work closely together. The centre has excellent lab and office space overlooking lakes and mountains. The university is located in the most beautiful landscaped campus in Europe. It has excellent sporting facilities. Some of the most striking regions of the Scottish highlands are within easy reach. Eligibility: Applicants should have a first degree, e.g. B.A., B.Sc., in any of a variety of disciplines, e.g. Computing, Psychology, Biology, Mathematics Engineering, Physics. For further information and application forms contact: School Office, School of Human Sciences, Stirling University, Stirling FK9 4LA, SCOTLAND Specific enquiries to: Dr W A Phillips, CCCN, Psychology, Stirling University, Scotland e-mail: WAP at UK.AC.STIRLING.CS No deadline for applications is specified. From jose at learning.siemens.com Mon Jan 28 08:46:06 1991 From: jose at learning.siemens.com (Steve Hanson) Date: Mon, 28 Jan 91 08:46:06 EST Subject: IJCNN-91-SEATTLE Message-ID: <9101281346.AA28682@learning.siemens.com.siemens.com> NIPS*91 is certainly interested in Cognitive Science and Natural Language research. And I would like to point out that NIPS*90 had an 2*fold increase in submissions in that area from the year before. Steve Hanson NIPS*91 Program Chair From andercha at grieg.CS.ColoState.EDU Mon Jan 28 13:46:28 1991 From: andercha at grieg.CS.ColoState.EDU (charles anderson) Date: Mon, 28 Jan 91 11:46:28 MST Subject: Call for Papers for 1991 Machine Learning Workshop Message-ID: <9101281846.AA04012@grieg.CS.ColoState.Edu> CALL FOR PAPERS 1991 MACHINE LEARNING WORKSHOP Northwestern University June 27-29, 1991 CONSTRUCTIVE INDUCTION Selection of an appropriate representation is critical to the success of most learning systems. In difficult learning problems (e.g., protein folding, word pronunciation, relation learning), considerable human effort is often required to identify the basic terms of the representation language. Constructive induction offers a partial solution to this problem by automatically introducing new terms into the representation as needed. Automatically constructing new terms is difficult because the environment or teacher usually provides only indirect feedback, thus raising the issue of credit assignment. However, as learning systems face tasks of greater autonomy and complexity, effective methods for constructive induction are becoming increasingly important. The objective of this workshop is to provide a forum for the interchange of ideas among researchers actively working on constructive induction issues. It is intended to identify commonalities and differences among various existing and emerging approaches such as knowledge-based term construction, relation learning, theory revision in analytic systems, learning of hidden-units in multi-layer neural networks, rule-creation in classifier systems, inverse resolution, and qualitative-law discovery. Submissions are encouraged in the following topic areas: o Empirical approaches and the use of inductive biases o Use of domain knowledge in the construction and evaluation of new terms o Construction of or from relational predicates o Theory revision in analytic-learning systems o Unsupervised learning and credit assignment in constructive induction o Interpreting hidden units as constructed features o Constructive induction in human learning o Techniques for handling noise and uncertainty o Experimental studies of constructive induction systems o Theoretical proofs, frameworks, and comparative analyses o Comparison of techniques from empirical learning, analytical learning, classifier systems, and neural networks Send six copies of paper submissions (3000 word maximum) to Christopher Matheus, GTE Laboratories, 40 Sylvan Road, MS-45, Waltham MA 02254 (matheus at gte.com). Submissions must be received by February 1, 1991. Include a cover page with authors' names, addresses, phone numbers, electronic mail addresses, paper title, and a 300 (maximum) word abstract. Do not indicate or allude to authorship anywhere within the paper. Acceptance notification will be mailed by March 31, 1991. Accepted papers will be allotted four two-column pages for publication in the Proceedings of the 1991 Machine Learning Workshop. Organizing Committee: Program Committee: Christopher Matheus, GTE Laboratories Chuck Anderson, Colorado State George Drastal, Siemens Corp. Research Gunar Liepins, Oak Ridge National Lab. Larry Rendell, University of Illinois Douglas Medin, University of Michigan Paul Utgoff, University of Massachusetts From N.E.Sharkey at cs.exeter.ac.uk Mon Jan 28 13:37:58 1991 From: N.E.Sharkey at cs.exeter.ac.uk (Noel Sharkey) Date: Mon, 28 Jan 91 18:37:58 GMT Subject: references to CNLP Message-ID: <259.9101281837@entropy.cs.exeter.ac.uk> I said that people could obtain a copy of references to CNLP by writing to lyn at my address. But i forgot to enclose the address. I have had several messages now pointing out difficulties. My apologies to all. noel sharkey The addresses are: JANET: lyn at uk.ac.exeter.cs UUCP: lyn at expya.uucp BITNET: lyn at cs.exeter.ac.uk@UKACRL From andercha at grieg.CS.ColoState.EDU Mon Jan 28 17:06:28 1991 From: andercha at grieg.CS.ColoState.EDU (charles anderson) Date: Mon, 28 Jan 91 15:06:28 MST Subject: CFP Constructive Induction Workshop, Due March 1st Message-ID: <9101282206.AA04326@grieg.CS.ColoState.Edu> CALL FOR PAPERS 1991 MACHINE LEARNING WORKSHOP Northwestern University June 27-29, 1991 CONSTRUCTIVE INDUCTION Selection of an appropriate representation is critical to the success of most learning systems. In difficult learning problems (e.g., protein folding, word pronunciation, relation learning), considerable human effort is often required to identify the basic terms of the representation language. Constructive induction offers a partial solution to this problem by automatically introducing new terms into the representation as needed. Automatically constructing new terms is difficult because the environment or teacher usually provides only indirect feedback, thus raising the issue of credit assignment. However, as learning systems face tasks of greater autonomy and complexity, effective methods for constructive induction are becoming increasingly important. The objective of this workshop is to provide a forum for the interchange of ideas among researchers actively working on constructive induction issues. It is intended to identify commonalities and differences among various existing and emerging approaches such as knowledge-based term construction, relation learning, theory revision in analytic systems, learning of hidden-units in multi-layer neural networks, rule-creation in classifier systems, inverse resolution, and qualitative-law discovery. Submissions are encouraged in the following topic areas: o Empirical approaches and the use of inductive biases o Use of domain knowledge in the construction and evaluation of new terms o Construction of or from relational predicates o Theory revision in analytic-learning systems o Unsupervised learning and credit assignment in constructive induction o Interpreting hidden units as constructed features o Constructive induction in human learning o Techniques for handling noise and uncertainty o Experimental studies of constructive induction systems o Theoretical proofs, frameworks, and comparative analyses o Comparison of techniques from empirical learning, analytical learning, classifier systems, and neural networks Send six copies of paper submissions (4000 word maximum) to Christopher Matheus, GTE Laboratories, 40 Sylvan Road, MS-45, Waltham MA 02254 (matheus at gte.com). Submissions must be received by March 1, 1991. Include a cover page with authors' names, addresses, phone numbers, electronic mail addresses, paper title, and a 300 (maximum) word abstract. Do not indicate or allude to authorship anywhere within the paper. Acceptance notification will be mailed by April 30, 1991. Accepted papers will be allotted four two-column pages for publication in the Proceedings of the 1991 Machine Learning Workshop. Organizing Committee: Program Committee: Christopher Matheus, GTE Laboratories Chuck Anderson, Colorado State George Drastal, Siemens Corp. Research Gunar Liepins, Oak Ridge National Lab. Larry Rendell, University of Illinois Douglas Medin, University of Michigan Paul Utgoff, University of Massachusetts From smieja at gmdzi.uucp Mon Jan 28 10:01:13 1991 From: smieja at gmdzi.uucp (Frank Smieja) Date: Mon, 28 Jan 91 14:01:13 -0100 Subject: Vision (What's wrong with Marr's model) Message-ID: <9101281301.AA26852@gmdzi.gmd.de> I received the following suggestions for alternative (or more modern) views on the vision problem, as opposed to the Marr viewpoint. Thanks very much to those who replied. Young, D.S. (1990). Quantitative ecological optics. In Ph. Jorrand & V. Sgurev (Eds.), Artificial Intelligence IV: Methodology, Systems, Applications (pp. 423-431). Amsterdam: North-Holland ----------- A. Sloman `On designing a visual system: Towards a Gibsonian computational model of vision' Journal of Experimental and Theoretical AI 1,4, 1989 It is concerned with fairly high level design requirements for a visual system that needs to be able to cope with real-time constraints, optical information of variable quality, a multitude of different uses for vision and, and many different links between the vision sub-system and various other sub-systems. Aaron Sloman, School of Cognitive and Computing Sciences, Univ of Sussex, Brighton, BN1 9QH, England EMAIL aarons at cogs.sussex.ac.uk or: aarons%uk.ac.sussex.cogs at nsfnet-relay.ac.uk --------- D. Weinshal and S. Edelman, "Computational Vision: A Critical Review", MIT-AI Technical Report #??? (sorry!), 1989, and @article{Hild87, author = "E. C. Hildreth and C. Koch", year = "1987", journal = "Annual Reviews of Neuroscience", volume = "10", pages = "477-533", title = "The Analysis of Visual Motion: From Computation Theory to Neuronal Mechanisms" } --------- I have written two papers that call into doubt Marr & Nishihara's proposals about shape recognition being accomplished by matching object-centered, viewpoint-independent shape representations. They are: Tarr, M. J. & Pinker, S. (1989) Mental rotation and orientation-dependence in shape recognition. @i[Cognitive Psychology, 21], 233-282. Tarr, M. J. & Pinker, S. (1990) When does human object recognition use a viewer-centered recognition frame? @i[Psychological Science, 1], 253-256. --Steve Pinker ---------- From haussler at saturn.ucsc.edu Tue Jan 29 20:10:26 1991 From: haussler at saturn.ucsc.edu (David Haussler) Date: Tue, 29 Jan 91 17:10:26 -0800 Subject: problems ftping large UCSC tech report fixed (we hope) Message-ID: <9101300110.AA11994@saturn.ucsc.edu> Several people had problems printing the tech report Decision Theoretic Generalizations of the PAC Model for Neural Net and Other Learning Applications UCSC-CRL-91-02 which I announced a few days ago. We have now split it into 2 parts so that it can be printed on most printers. revised ftp instructions follow: unix> ftp midgard.ucsc.edu # (or ftp 128.114.134.15) Connected ... Name (...): anonymous Password: yourname at cs.anyuniversity.edu (i.e. your email address) (Please use your email address so we can correspond with you.) Guest login ok, access restrictions apply. ftp> cd pub/tr ftp> binary ftp> get ucsc-crl-91-02.part1.ps.Z ... 310226 bytes received in 4.4 seconds (70 Kbytes/s) ftp> get ucsc-crl-91-02.part2.ps.Z ... 277165 bytes received in 4.4 seconds (70 Kbytes/s) ftp> quit unix> uncompress ucsc-crl-91-02.part1.ps.Z unix> uncompress ucsc-crl-91-02.part2.ps.Z unix> lpr -P(your_local_postscript_printer) ucsc-crl-91-02.part1.ps unix> lpr -P(your_local_postscript_printer) ucsc-crl-91-02.part2.ps (Some other UCSC tech reports are also available. We fixed some problems that we were having with these earlier as well, notably 90-16. ftp the file INDEX to see what's there.) If you have any difficulties with the above, please send e-mail to jean at cis.ucsc.edu. DO NOT "reply" to this message, please. -David From pjh at compsci.stirling.ac.uk Tue Jan 29 10:31:50 1991 From: pjh at compsci.stirling.ac.uk (Peter J.B. Hancock) Date: 29 Jan 91 15:31:50 GMT (Tue) Subject: re MSc in Neural Computation Message-ID: <9101291531.AA05715@uk.ac.stir.cs.lira> Due to vagaries in the UK email service and the fact that 'cs' is the country code for Czechoslovakia, the email address given on the recent announcement of an MSc in neural computation at Stirling was incorrect. It should be: wap at uk.ac.stirling.compsci However, most users will probably at least have to reverse this: wap at compsci.stirling.ac.uk Apologies for any confusion and the wasted bandwidth. Peter Hancock From reggia at cs.UMD.EDU Thu Jan 31 10:56:20 1991 From: reggia at cs.UMD.EDU (James A. Reggia) Date: Thu, 31 Jan 91 10:56:20 -0500 Subject: CALL FOR PAPERS: CONNECTIONIST MODELS IN BIOMEDICINE Message-ID: <9101311556.AA12211@mimsy.UMD.EDU> CALL FOR PAPERS: The 15th Symposium on Computer Applications in Medical Care will include a Program Area Track on Connectionism, Simulation and Modeling. Submission of papers is welcomed. Papers are solicited which report on original research, system development or survey the state of the art in an aspect of this wide- ranging field. Papers in previous years have addressed such topics as modelling invertebrate nervous systems, modelling disorders of higher cortical functions, development of high-level languages for building connectionist models, and systems for medical diagnosis, among other topics. Deadline for receipt of manuscripts is April 1, 1991. The conference will be held November 17-20, 1991 in Washington, DC. For submittal forms please write: Paul D. Clayton, PhD SCAMC Program Chair, 1991 AMIA 11140 Rockville Pike Box 324 Rockville, MD 20852 or contact Gail Mutnik at mutnik at lhc.nlm.nih.gov by email. If you have questions about whether your paper would be appropriate for this conference please contact me at: Stan Tuhrim SSTMS at CUNYVM.CUNY.EDU From ff at sun8.lri.fr Thu Jan 31 12:45:46 1991 From: ff at sun8.lri.fr (ff@sun8.lri.fr) Date: Thu, 31 Jan 91 18:45:46 +0100 Subject: change in receiver name Message-ID: <9101311745.AA06011@sun3a.lri.fr> Please, could you change the subscription destination to: ----------------------------------- conx at FRLRI61.BITNET@CUNYVM.CUNY.EDU ----------------------------------- (instead of: ff at FRLRI61.BITNET@CUNYVM.CUNY.EDU ) best regards Francoise Fogelman From tsejnowski at UCSD.EDU Tue Jan 1 16:38:04 1991 From: tsejnowski at UCSD.EDU (Terry Sejnowski) Date: Tue, 1 Jan 91 13:38:04 PST Subject: NEURAL COMPUTATION 2:4 Message-ID: <9101012138.AA20315@sdbio2.UCSD.EDU> NEURAL COMPUTATION Volume 2 Issue 4 NOTES Leonid Kruglyak How to Solve the N Bit Encoder Problem with Just Two Hidden Units Zoran Obradovic and Peiyuan Yan Small Depth Polynomial Size Neural Networks LETTERS Steven D. Whitehead and Dana H. Ballard Active Perception and Reinforcement Learning Lucia M. Vaina, Norberto M. Grzywacz, and Marjorie LeMay Structure from Motion with Impaired Local-Speed and Global Motion-Field Computations Pierre Baldi and Ronny Meir Computing with Arrays of Coupled Oscillators An Application to Preattentive Texture Discrimination Reza Shadmehr Learning Virtual Equilibrium Trajectories for Control of a Robot Arm Michael C. Mozer and Jonathan Bachrach Discovering the Structure of a Reactive Environment by Exploration Alan J. Katz, Michael T. Gately, and Dean R. Collin Robust Classifiers without Robust Features William G. Baxt Use of an Artificial Neural Network for Data Analysis in Clinical Decision-Making: the Diagnosis of Acute Coronary Occlusion Ronald J. Williams and Jing Peng An Efficient Gradient-Based Algorithm for On-Line Training of Recurrent Network Trajectories Michael Georgiopoulos, Gregory L. Heileman, and Juxin Huang Convergence Properties of Learning in ART1 Eric B. Baum A Polynomial Time Algorithm That Learns Two Hidden Unit Nets James A. Reggia and Mark Edwards Phase Transitions in Connectionist Models Having Rapidly Varying Connection Strengths R. Bijjani and P. Das An M-ary Neural Network Model Mark J. Brady Guaranteed Learning Algorithm for Network with Units Having Periodic Threshold Output Function SUBSCRIPTIONS: ***** Last opportunity to obtain all issues in Volume 2 ***** ______ $35 Student ______ $50 Individual ______ $100 Institution Add $12. for postage outside USA and Canada surface mail. Add $18. for air mail. (Back issues of volume 1 are available for $27 each.) MIT Press Journals, 55 Hayward Street, Cambridge, MA 02142. (617) 253-2889. ----- From GOLDFARB%UNB.CA at UNBMVS1.csd.unb.ca Tue Jan 1 22:35:28 1991 From: GOLDFARB%UNB.CA at UNBMVS1.csd.unb.ca (Lev Goldfarb) Date: Tue, 01 Jan 91 23:35:28 AST Subject: On the shape of things to come Message-ID: In view of the multidisciplinary background of members of this mailing list and, more particularly, in view of some resent remarks (which I don't want to quote since they may appear to have been taken out of the context) about the specific ways of proceeding to our common goal -- an analytical model of an "intelligent" system -- I feel compelled to draw your attention to the following apparent paradox, if we are to accept a more "direct" or "hands on" approach to the modeling of the brain. Most of us would probably agree that to model "intelligent" (biological, irreversible) processes a fundamentally new mathematical models are necessary. Is it possible then to proceed to construct these models in a piecemeal fashion using "pieces" of the old mathematical models (that were constructed for entirely different purposes) as one is compelled to do under the "direct approach" philosophy? If it has been impossible to do this within the confines of the same science, physics, it will prove to be even less possible to do so for the biological processes. I for one would be much less sure of the feasibility of the new model proposed by me if one could show that it is reducible to one of the existing mathematical models. Incidentally, the NN models are not new mathematical models. --Lev Goldfarb From VAINA at buenga.bu.edu Wed Jan 2 11:13:00 1991 From: VAINA at buenga.bu.edu (VAINA@buenga.bu.edu) Date: Wed, 2 Jan 91 11:13 EST Subject: The computing Brain lecture series-at BU: Engineering Message-ID: From: BUENGA::CORTEX 20-DEC-1990 16:02 To: @CORTEX-NEW,IN%CORTEX-IN-DISTRIBUTION CC: CORTEX Subj: COMPUTING BRAIN LECTURE SERIES - Marvin Minsky *************************************************************************** THE COMPUTING BRAIN LECTURE SERIES *************************************************************************** " SOCIETY OF MIND II " MARVIN MINSKY Toshiba Professor of Media Arts and Sciences Massachusetts Institute of Technology Wednesday, January 9, 1991 at 4 pm Old Engineering Building - Room 150 110 Cummington Street, Boston, Ma Tea at 3 pm in Room 129 (same address as above) Lecture open to all For further information contact: Professor Lucia M. Vaina 353-2455 or vaina at buenga.bu.edu *************************************************************************** From jbower at smaug.cns.caltech.edu Wed Jan 2 13:49:15 1991 From: jbower at smaug.cns.caltech.edu (Jim Bower) Date: Wed, 2 Jan 91 10:49:15 PST Subject: flag waving Message-ID: <9101021849.AA09749@smaug.cns.caltech.edu> I am extremely loath to continue to participate in this debate in that it HAS become too redundant, may very well have been a waste of time to begin with, and is fast becoming an exercise in patriotic flag waving (Paul Munro's ridiculous equivalence for example). If anyone is still unclear about what I am, or am not trying to say, or wants to take personal exception to some comment I have made, please communicate with me directly. But I would like to make two brief comments with respect to Lev Goldfarb's posting. First, it is not clear to me that an "analytical model of an intelligent system" is the common goal of the people on this mailing list. Second, the assumption that a "direct approach" to modeling the brain is limited to old tools is simply not correct. Both Terry Sejnowski's example from history, or the current work of Nancy Kopel (BU) and Bard Ermentrout (UPitt) are examples to the contrary. It may even be that taking a direct approach applies more pressure to come up with new mathematical tools, especially if one has learned enough about the biology to know that the old models are off the mark. Jim Bower From simic at kastor.ccsf.caltech.edu Wed Jan 2 14:02:55 1991 From: simic at kastor.ccsf.caltech.edu (Petar Simic) Date: Wed, 2 Jan 91 19:02:55 GMT Subject: There is no use for bad data Message-ID: <1991Jan2.190255.6920@nntp-server.caltech.edu> Connectionism and NN made a bold guess of what should be the zeroth order approximation in modeling neural information processing, and I think that the phrase 'biological plausibility' (or 'neural style') is meant to indicate, (1) fine-grained parallel processing, (2) reasonable although no-doubt crude model-neuron . While I heard that some not-only-IN PRINCIPLE intelligent people think that parallelism is 'JUST a question of speed', I think that so defined biological plausibility is not to be discounted in discussing the differences between the two approaches to modeling intelligence, AI and Connectionism/NN&ALL THAT. Perhaps the phrase 'biological plausibility' should be changed to 'physical plausibility' indicating that Connectionist/NN models have natural implementation in physical hardware. This is not to say that Connectionism/NN&ALL THAT should pose as biology. It doesn't need to ---it has its own subject of study (which in combination with traditional AI makes what one may call 'modern AI') and it need not be theology (J.B) or pure math, providing it is willing to impose on itself some of the hard constraints of either engineering, or natural science (or both). The engineering constraint is that understanding of intelligence should not be too far from building it (in software or physical hardware). This is not the unfamiliar constraint for connectionists, and is also healthy part of the traditional AI, since they both make computational models, and as they move from toy problems to larger scales, concepts and representations they develop should be softly constrained by the plausibility of implementation today, or perhaps tomorrow. The natural science constraint is in traying to reverse-engineer the real brain (J.B), but I would suggest that this constraint is 'softer' then what seem to be suggested by Jim Bower, and that the reason for this is not in our underestimate of the complexity of the brain, but in the theoretical depth of the reverse-engineering problem. I think that at the level of the detailed modeling of specific neural circuits, Connectionism/NN provide a set of tools which may or may not be useful, depending on the problem at hand, and how these tools are applied. The interesting question, therefore, is not how useful is Connectionism/NN to neurobiology ---how useful is sharp pencil or PC for neurobiology? --- but how useful is neurobiology, as practiced today, to the study of the information processing phenomena across the levels, in natural (and, why not, artificial) systems. I would think that the present situation in which Connectionist/NN models are ignoring many of the 'details' should be a source of challenge to theoretically minded neurobiologists ---especially to the ones who think that the theoretical tools needed to describe the transition between between the levels are just the question of reading a chapter from some math textbook ---and that they should come up with computational models and convince everybody that particular detail does matter, in the sense that it is 'visible' at the higher information processing level, and can account for some useful computational phenomena which simplified model can not, or can but in a more complicated way. Modeling firmly rooted in biological fact if to detailed, might not be directly useful as modeling-component at higher level, except as a good starting point for simplification. That simplifications are essential, not DESPITE but BECAUSE the complexity of the brain, should be self-evident to all who believe that an understanding of the brain is possible. What is not self-evident is which details should not be thrown out, and how are they related to the variables useful at the higher information processing level. Even in simple systems such as the ones studied in physics, where one knows exactly the microscopic equations, the question of continuity between the correct theoretical descriptions at two different levels is very deep one. Just "..a cursory look at the brain ..." (J.B.) should not be enough to disqualify simplified phenomenological models which are meant to describe phenomenology at higher (coarser) level, as wrong. For example, if you want to model the heat distribution in some material, you use macroscopic heat-equation, and the basic variable there (the temperature) is related, but in rather unobvious way, to moving particles and their collisions, which are microscopic details of which heat is made of. Yet, the heat equation is the correct phenomenological description of the 'heat processing phenomenology'. This is not to say that the very simplified connectionist/NN models are good, but just warning that the theoretical modeling across the different levels of description is as hard as it is fascinating, it needs to be attacked simultaneously from all ends, and more often then not, one finds that the relationship between the microscopic variables and the variables useful at the next level, is neither obvious nor simple. I would suggest that if we are to understand the brain and the connection between its levels, one should not underestimate the theoretical depth of the problem. While it is definitely a good idea for a theorist to know well the phenomenology he is theorizing about, and it is an imperative for him, in this field, to build computational models, I think that it is somewhat silly to condition the relevance of his work on his ability to do neurobiological experiments. There is no use for bad data. From clee at ICSI.Berkeley.EDU Wed Jan 2 14:32:50 1991 From: clee at ICSI.Berkeley.EDU (Connie Lee) Date: Wed, 2 Jan 91 11:32:50 PST Subject: TR request Message-ID: <9101021932.AA04164@icsid.Berkeley.EDU> From Mailer-Daemon at icsib.Berkeley.EDU Wed Jan 2 14:30:33 1991 From: Mailer-Daemon at icsib.Berkeley.EDU (Mail Delivery Subsystem) Date: Wed, 2 Jan 91 11:30:33 PST Subject: Returned mail: User unknown Message-ID: <9101021930.AB20207@icsib> ----- Transcript of session follows ----- Connected to cs.cmu.edu: >>> RCPT To: <<< 550-(USER) Unknown user name in "connectionist at cs.cmu.edu" <<< 550-Some of the nearest matches are: <<< 550- <<< 550- connectionists-archive (connectionists-archive) <<< 550- Connectionists (connectionists) <<< 550- connectionists-request-list (connectionists-request-list) <<< 550- connectionist-request (connectionist-request) <<< 550- connectionist-requests (connectionist-requests) <<< 550- Connectionists-Requests (connectionists-requests) <<< 550- Connectionists-Request (Connectionists-Request) <<< 550 550 ... User unknown ----- Unsent message follows ----- From clee at ICSI.Berkeley.EDU Wed Jan 2 14:31:06 1991 From: clee at ICSI.Berkeley.EDU (Connie Lee) Date: Wed, 2 Jan 91 11:31:06 PST Subject: TR request Message-ID: <9101021931.AA04142@icsid.Berkeley.EDU> Please send Report FKI-140-90 "Beyond Hebb Synapses: Biological Building Blocks for Unsupervised Learning in Artifician Neural Networks by Patrick V. Thomas to: Dr. Jerome A. Feldman Director International Computer Science Institute 1947 Center Street Berkeley, CA 94704 Thank you, Connie Lee Admin. Assist. From jbower at smaug.cns.caltech.edu Wed Jan 2 15:27:39 1991 From: jbower at smaug.cns.caltech.edu (Jim Bower) Date: Wed, 2 Jan 91 12:27:39 PST Subject: Summer course Message-ID: <9101022027.AA09884@smaug.cns.caltech.edu> Summer Course Announcement Methods in Computational Neurobiology August 4th - August 31st Marine Biological Laboratory Woods Hole, MA This course is for advanced graduate students and postdoctoral fellows in neurobiology, physics, electrical engineering, computer science and psychology with an interest in "Computational Neuroscience." 20 such students will be enrolled. In addition, this coming summer the course has allocated 5 additional positions for participants who are currently faculty members at Universities or are established members of industrial research organizations. For both faculty and student participants, a background in programming (preferably in C or PASCAL) is highly desirable and basic knowledge of neurobiology is required. Limited to 20 students. This four-week course presents the basic techniques necessary to study single cells and neural networks from a computational point of view, emphasizing their possible function in information processing. The aim is to enable participants to simulate the functional properties of their particular system of study and to appreciate the advantages and pitfalls of this approach to understanding the nervous system. The first section of the course focuses on simulating the electrical properties of single neurons (compartmental models, active currents, interactions between synapses, calcium dynamics). The second section deals with the numerical and graphical techniques necessary for modeling biological neuronal networks. Examples are drawn from the invertebrate and vertebrate literature (visual system of the fly, learning in Hermissenda, mammalian olfactory and visual cortex). In the final section, more abstract models relevant to perception and learning in the mammalian cortex, as well as network learning algorithms will be analyzed and discussed from a neurobiological point of view. The course includes lectures each morning and a computer laboratory in the afternoons and evenings. The laboratory section is organized around GENESIS, the Neuronal Network simulator developed at the California Institute of Technology, running on 20 state-of-the-art, single-user, graphic color workstations. Students initially work with GENESIS-based tutorials and then are expected to work on a simulation project of their own choosing. Co-Directors: James M. Bower Christof Koch, Computation and Neural System Program California Institute of Technology 1991 summer faculty: Ken Miller UCSF Paul Adams Stony Brook Idan Segev Jerusalem John Rinzel NIH Richard Andersen MIT David Van Essen Caltech Scot Fraser* Caltech Kevin Martin Oxford Eve Marder* Brandis Nancy Kopell Boston U. Avis Cohen Cornell Rudolfo Llinas NYU Terry Sejnowski UCSD/Salk Chuck Stevens* UCSD/Salk Ted Adelson MIT David Zipser* UCSD *tentative Application deadline: May 15, 1991 Applications are evaluated by an admissions committee and individuals are notified of acceptance or non-acceptance by June 1. Tuition: $1,000 (includes room & board). Financial aid is available to qualified applicants. For further information contact: Admissions Coordinator Marine Biological Laboratory Woods Hole, MA 02543 (508) 548-3705, ext. 216 From josh at flash.bellcore.com Wed Jan 2 14:54:54 1991 From: josh at flash.bellcore.com (Joshua Alspector) Date: Wed, 2 Jan 91 14:54:54 -0500 Subject: NN, AI, biology Message-ID: <9101021954.AA05701@flash.bellcore.com> On the question of NN vs. AI, I think one point that needs emphasizing is that much of the difference between these two views of intelligence arises from physical implementation, and the convenience of modeling based on this underlying structure. A computer program has the capability of simulating the behavior of a physical system including neural models but it may take a long time. Traditional symbolic AI is much better suited to the processes that an arithmetic and logic unit in a computer can perform. On the other hand, a computer's logic and memory are implemented using transistors arranged in circuits in such a way that they act digitally but are really fundamentally analog and messy like neurons (well not THAT messy). One could build a digital computer from biological neurons if the technology existed to manipulate them. Since neural systems and symbol-manipulating computers can simulate each other, NN and AI are fundamentally equivalent in their descriptive powers. But each description has its advantages. Because logic circuits can be implemented in a wide variety of technologies (CMOS, bipolar, optical, neural), it is natural to ignore this level of description. One can further abstract away the logical structure (bus width, registers, instruction set) by a compiler that allows us to work in a high-level language. Here, we get into the doctrine of software separability. Software is everything, hardware doesn't matter. Neural networks are much messier, and the levels of description cannot be separated so easily. Slips of the tongue confuse low-level phonetic and articulatory information with high-level linguistic information, something you would have to sully an AI program to do. The computation here reflects the hardware (wetware). Because of the equivalence of NN and AI, NN cannot do anything that AI (computers) can't except to do it faster in a parallel implementation. As has been pointed out, the NN models are not new mathematically. Neural networks are a biological inspiration for how to physically do parallel processing of information. The qualities of NN that are awkward (but not impossible) to model with AI have to do with the physical nature of networks. These include speed in a parallel implementation, natural time scales, network dynamics, and yes, also a functional description of relevance to biological neural networks. Being an experimental neuroscientist is not the only way to understand brains. Suppose we had an intelligent AI system implemented on a digital computer and had no idea how it worked. We would not get far by sticking many oscilloscope probes into it and watching the results as we ask it questions. Josh Alspector josh at bellcore.com From well!mitsu at apple.com Thu Jan 3 04:28:38 1991 From: well!mitsu at apple.com (Mitsuharu Hadeishi) Date: Thu, 3 Jan 91 01:28:38 pst Subject: Connectionism vs. AI Message-ID: <9101030928.AA08049@well.sf.ca.us> I'm not sure how important the "equivalence" between NNs and Turing machines really is, given the fact that the space of all algorithms is hardly exhaustively searchable for any but the most trivial of problems. Obviously NNs impose a particular structure to the space of algorithms which allows systematic searching, whereas traditional AI approaches rely on hand-crafted algorithms. This structure is what makes co\nectionism important; not what is computable in principle, but what is computable because we can find the appropriate algorithm(s) to compute it. Mitsu Hadeishi Open Mind mitsu at well.sf.ca.us apple!well!mitsu From Dave.Touretzky at DST.BOLTZ.CS.CMU.EDU Thu Jan 3 01:36:58 1991 From: Dave.Touretzky at DST.BOLTZ.CS.CMU.EDU (Dave.Touretzky@DST.BOLTZ.CS.CMU.EDU) Date: Thu, 03 Jan 91 01:36:58 EST Subject: Lakoff paper on "Metaphor and War" available Message-ID: <23527.662884618@DST.BOLTZ.CS.CMU.EDU> Some of you, partiicularly those who attend Cognitive Science or participated in the 1988 or 1990 connectionist summer schools, have met George Lakoff, a linguist at Berkeley who is interested in connectionist matters. His work on phonology provided the inspiration for my own research efforts in this area, and his book "Women, Fire, and Dangerous Things" is must reading for people in cognitive science. (When I first heard of it I thought the book was about feminism, but the title actually comes from a true anecdote about a primitive people whose mythology leads them to put the words for women, fire, and dangerous things in the same linguistic class. The sun is female, you see.) Anyway, George works on many interesting problems related to language, including the role of metaphor in our ability to understand and talk about abstract concepts. He has written a paper about the metaphors that underlie our understanding of the current conflict in the Persian Gulf. People who are interested in cognitive science might want to take a look at it. He is particularly eager for people to have a chance to read it before January 15. The paper is too long to post here, and in any case it's too divorced from connectionism to be appropriate for this list, but you can retrieve it from the neuroprose directory at Ohio State. Retrieval instructions appear at the end of this message. A copy has also been posted to comp.ai. Or you can receive a copy by sending email to George at lakoff at cogsci.berkeley.edu. Only the first half of the paper deals with metaphor. The second half is political analysis. It will no doubt be controversial. But I'm warning all readers right now: I will not permit discussions of Persian Gulf politics on this mailing list. Read the paper at your own risk; send mail to George if you like; start a discussion on talk.politics.misc or alt.desert-shield or whatever newsgroup you feel is appropriate, but keep it *off* the CONNECTIONISTS mailing list! I will do unspeakably nasty things to anyone who violates this rule. The only reason I'm permitting the paper to be announced here at all is because of its cognitive science content. -- Dave Touretzky ================ How to retrieve the paper from Neuroprose ================ 1. Open an FTP connection to 128.146.8.62 (cheops.cis.ohio-state.edu) 2. Login as user "anonymous", password "neuron" 3. cd /pub/neuroprose 4. type binary 5. get lakoff.war.ps.Z 6. bye 7. uncompress lakoff.war.ps.Z 8. Then send the file lakoff.war.ps to your local PostScript printer. From simic at kastor.ccsf.caltech.edu Wed Jan 2 20:28:19 1991 From: simic at kastor.ccsf.caltech.edu (Petar Simic) Date: Thu, 3 Jan 91 01:28:19 GMT Subject: There is no use for bad data Message-ID: <1991Jan3.012819.16241@nntp-server.caltech.edu> Connectionism and NN made a bold guess of what should be the zeroth order approximation in modeling neural information processing, and I think that the phrase 'biological plausibility' (or 'neural style') is meant to indicate, (1) fine-grained parallel processing, (2) reasonable although no-doubt crude model-neuron . While I heard that some not-only-IN PRINCIPLE intelligent people think that parallelism is 'JUST a question of speed', I think that so defined biological plausibility is not to be discounted in discussing the differences between the two approaches to modeling intelligence, AI and Connectionism/NN&ALL THAT. Perhaps the phrase 'biological plausibility' should be changed to 'physical plausibility' indicating that Connectionist/NN models have natural implementation in physical hardware. This is not to say that Connectionism/NN&ALL THAT should pose as biology. It doesn't need to ---it has its own subject of study (which in combination with traditional AI makes what one may call 'modern AI') and it need not be theology (J.B) or pure math, providing it is willing to impose on itself some of the hard constraints of either engineering, or natural science (or both). The engineering constraint is that understanding of intelligence should not be too far from building it (in software or physical hardware). This is not the unfamiliar constraint for connectionists, and is also healthy part of the traditional AI, since they both make computational models, and as they move from toy problems to larger scales, concepts and representations they develop should be softly constrained by the plausibility of implementation today, or perhaps tomorrow. The natural science constraint is in traying to reverse-engineer the real brain (J.B), but I would suggest that this constraint is 'softer' then what seem to be suggested by Jim Bower, and that the reason for this is not in our underestimate of the complexity of the brain, but in the theoretical depth of the reverse-engineering problem. I think that at the level of the detailed modeling of specific neural circuits, Connectionism/NN provide a set of tools which may or may not be useful, depending on the problem at hand, and how these tools are applied. The interesting question, therefore, is not how useful is Connectionism/NN to neurobiology ---how useful is sharp pencil or PC for neurobiology? --- but how useful is neurobiology, as practiced today, to the study of the information processing phenomena across the levels, in natural (and, why not, artificial) systems. I would think that the present situation in which Connectionist/NN models are ignoring many of the 'details' should be a source of challenge to theoretically minded neurobiologists ---especially to the ones who think that the theoretical tools needed to describe the transition between between the levels are just the question of reading a chapter from some math textbook ---and that they should come up with computational models and convince everybody that particular detail does matter, in the sense that it is 'visible' at the higher information processing level, and can account for some useful computational phenomena which simplified model can not, or can but in a more complicated way. Modeling firmly rooted in biological fact if to detailed, might not be directly useful as modeling-component at higher level, except as a good starting point for simplification. That simplifications are essential, not DESPITE but BECAUSE the complexity of the brain, should be self-evident to all who believe that an understanding of the brain is possible. What is not self-evident is which details should not be thrown out, and how are they related to the variables useful at the higher information processing level. Even in simple systems such as the ones studied in physics, where one knows exactly the microscopic equations, the question of continuity between the correct theoretical descriptions at two different levels is very deep one. Just "..a cursory look at the brain ..." (J.B.) should not be enough to disqualify simplified phenomenological models which are meant to describe phenomenology at higher (coarser) level, as wrong. For example, if you want to model the heat distribution in some material, you use macroscopic heat-equation, and the basic variable there (the temperature) is related, but in rather unobvious way, to moving particles and their collisions, which are microscopic details of which heat is made of. Yet, the heat equation is the correct phenomenological description of the 'heat processing phenomenology'. This is not to say that the very simplified connectionist/NN models are good, but just warning that the theoretical modeling across the different levels of description is as hard as it is fascinating, it needs to be attacked simultaneously from all ends, and more often then not, one finds that the relationship between the microscopic variables and the variables useful at the next level, is neither obvious nor simple. I would suggest that if we are to understand the brain and the connection between its levels, one should not underestimate the theoretical depth of the problem. While it is definitely a good idea for a theorist to know well the phenomenology he is theorizing about, and it is an imperative for him, in this field, to build computational models, I think that it is somewhat silly to condition the relevance of his work on his ability to do neurobiological experiments. There is no use for bad data. Petar Simic simic at wega.caltech.edu From jbower at smaug.cns.caltech.edu Thu Jan 3 14:49:30 1991 From: jbower at smaug.cns.caltech.edu (Jim Bower) Date: Thu, 3 Jan 91 11:49:30 PST Subject: analogies Message-ID: <9101031949.AA13327@smaug.cns.caltech.edu> I would like to know when I said that one had to be an experimental neurobiologist to do computational neurobiology. It is really unbelievable how defensive the reaction has been to my comments. Just for the record, Christof Koch, Carver Mead, Nancy Kopel, Bard Ermentrout, Wilfrid Rall, Jack Cowan, Idan Segev, and many other NON-experimentalists have made important contributions to computational neurobiology. They have also invested a tremendous amount of time educating themselves about the detailed structure of the nervous system on their own and through interactions with experimental neurobiologists. And I don't mean listening to invited talks at neural net conferences. It is absolutely bizarre that claims of scientific relevance and biological inspiration are made and accepted by a field largely composed of people who know very little about the nervous system, think that it can be ignored or regard it as simply one implementation alternative, and generally appear to have little real interest in the subject. Two other remarks. The computer and the oscilloscope analogy is a terrible one. The point that Josh correctly made in the beginning of his comment is that there is a rather poor mapping between neural network algorithms and digital computer architecture. I think that it can also be argued that a lot of the most interesting work in neural networks is on the side of implementation, i.e. how one constructs hardware that reflects the algorithms of interest. The brain almost certainly has taken this association to an extreme which is probably closely related to its spectacular efficiency and power. The form reflects the function. An electrode in a computer is a mess because the computer is a relatively low level computing device that trades generality for efficiency. The brain is a different situation altogether. For example, we increasingly suspect, based on detailed computational modeling of brain circuits, that principle computational features of a circuit are reflected at all its organizational levels. That is, if a network oscillates at 40 Hz, that periodicity is seen at the network, single cell, and subcellular levels as well as in the structure of the input and output. That means that sticking the electrode anywhere will reveal some aspect of what is probably an important functional property of the network. Second, the standard particle in a box analogy mentioned by Kastor is even worse. Neurons can not be considered particles in a box. This even goes against fundamental assumptions underlying connectionism. This is one of, if not the most difficult problem associated with changing levels of abstraction when modeling the brain. It also means that the best examples of success in theoretical physics may not directly apply to understanding the nervous system. We will see. Finally, with respect to the original AI / NN Connectionist debate. I just received an advertisement from the periodical "AI Expert" that offers with a subscription "five disk-based versions of some of the most important AI programs". You guessed it, "Vector classifier, Adaline/Perceptron, Backpropagation, Outstar Network, and Hopfield Network", presented as "real world examples of working AI systems". I think that settles the issue, NNs etc has now become part of AI marketing. How much closer can you get. Jim Bower From ken at chagall.cns.caltech.edu Fri Jan 4 03:37:54 1991 From: ken at chagall.cns.caltech.edu (Ken Miller) Date: Fri, 4 Jan 91 00:37:54 -0800 Subject: arguments for all seasons Message-ID: <9101040837.AA03168@chagall.cns.caltech.edu> With respect to the great NN/AI/Neurobiology debates: While history serves as a guide to some of the possibilities of the future, it by no means limits them. Furthermore, one can within history find examples to suit most prejudices. Some examples: (1) In the late 1800's some scientists conceived the beautiful idea that the fundamental physical entities were vortices in some medium. It seemed tantalizingly as though such a program could explain all of physics. Working seriously on this idea, a full mathematics of vortices could probably have been developed, albeit one that would always have had some major difficulties in explaining reality. Without experimental studies of atomic physics there was no chance of anyone inventing the real explanation, quantum mechanics, through pure thought. Score one for the neurobiologists. (2) The thermodynamics/stat mech example. An extremely good science of gases, liquids, etc. (thermodynamics) evolved from observation of those things without any knowledge of the underlying atomic and molecular physics. When consideration of atomic level things led eventually to statistical mechanics, it was possible to derive thermodynamics from stat mech; but nobody would ever have found thermodynamics from stat mech alone; knowledge of the laws at the thermodynamic level was needed to find those laws within stat mech. This phenomena of finding the higher-level laws in the lower level ONLY through guidance by prior knowledge of the higher-level laws has occured repeatedly in modern physics. Score one for the NN and AI types --- but only if they are strongly guided by the phenomenological, empirical study of intelligence, perception, motor behavior, or whatever they are modeling. (3) Heredity, like intelligence, was once a great soupy mess. People had lots of complicated, dynamical ideas of how it was accomplished. Detailed study of the biology finally led to a simple structural explanation --- the structure of DNA --- that was largely unanticipated (yes I know about Schrodinger -- who anticipated certain things in an abstract way but not in a way that enabled any useful understanding of heredity). Score one for the neurobiologists. On the other hand, many details of heredity --- i.e. genetics --- were worked out without this molecular-level knowledge, including Barbara McClintock's ``jumping genes" (a discovery that was not widely acknowledged until a molecular-level explanation was found 20 years later, at which time she finally got the Nobel prize). Score one for those studying at a phenomenological level. (4) Einstein took a pre-existing mathematics, essentially differential geometry, and applied it to the invention of general relativity. Similarly, modern field theorists have found the largely preexisting mathematics of knot theory to be crucial to the understanding of superstrings. Score one for those who believe development of mathematical tools in non-neurobiological contexts may aid neurobiologists. (5) Many aspects of modern mathematics were first invented by physicists trying to solve particular physics problems; later they was cleaned up, rigorized, and generalized by the mathematicians. Score one for those who believe the mathematical tools relevant to neurobiology may only be found through attempts to model neurobiology. Although, note that many of these tools were developed in studying "toy" models that at best only caricature one aspect of the real physical problem. So score one for everybody. I could go on and on. We could all go on and on. There's enough history and abstract arguments for everyone. Personally, I hold these truths to be self-evident: (1) Intelligence, like quantum mechanics, is too strange, difficult, complex or what-have-you to be understood by pure thought alone. (2) Insights into intelligence will come both from studying it at its own phenomenological level, and by studying the physical structures (i.e. the brain) that are known to realize it. Personally, I'm putting my bets on studying the brain, but that's just a personal decision. (3) Development of toy models is useful to neurobiologists. Until connectionist models came into being, no one had a solid, non-vague, working model of how a parallel distributed system might represent and transform information. Since experiments are necessarily framed in terms of whatever concepts and metaphors are at hand, connectionist models have had and will continue to have an important influence on systems neurobiology. This does not mean that the toy models are necessarily biological, only that they usefully expand the thinking tools available to the working neurobiologist. [on the other hand, the lack of relevance of the DETAILS of these models to the neurobiologists is suggested by the great lack of working neurobiologists on this net.] (4) Neurobiology is useful to NN/AI types. Again, not too many of the details at any point in time are considered by the NN/AI types, but the overall progress of neurobiology leads to ideas that are important to those trying to engineer or theoretically understand intelligence. (5) Insights and influences will run in all possible directions, and no one can predict for sure what will turn out to be useful to who. We all place our bets by the work we choose to do. (6) None of us have the foggiest idea how the brain, or real intelligence, works. Therefore, we would all be wise to be humble and to listen well. Ken Miller ken at descartes.cns.caltech.edu From GOLDFARB%unb.ca at UNBMVS1.csd.unb.ca Fri Jan 4 01:03:39 1991 From: GOLDFARB%unb.ca at UNBMVS1.csd.unb.ca (GOLDFARB%unb.ca@UNBMVS1.csd.unb.ca) Date: Fri, 04 Jan 91 02:03:39 AST Subject: AI, NN, CNS (central nervous system) In-Reply-To: Message of Fri, 21 Dec 90 01:00:53 EST from Message-ID: Terry : > It should also be > noted that Hartline and Ratliff would not have been able to > develop their model if the mathematics of linear networks had > not already been established by mathematicians, physicists, and > engineers, most of whom were not interested in biological problems. > Without the development of a mathematics of nonlinear dynamical > systems there will be no future models for future Hartlines > and Ratliffs to apply to future biological problems. I find > it encouraging that so many good scientists who are confronting > so many difficult problems in psychology, biology and computation > are begining to at least speak the same mathematics. > > I do not think that anything is going to be settled by > debating ideologies, except who is the better debater. Precious > bandwidth is better spent discussing specific problems. I believe that on closer examination the above remarks disclose an important contradiction, which, in the first place, was an the impetus to the "ideological debate". First, by no means should the AI/NN "debate" be viewed as "ideological", but rather as one about the need for a new mathematical model the importance of which was stressed in the first paragraph. Second, why is it that "without the development of a mathematics of nonlinear dynamical systems there will be no future models for Hartlines and Ratliffs to apply to future biological problems"? Third, I can't see where "many good scientists who . . . are beginning to at least speak the same mathematics" are, and what this mathematics is. Finally, since some of us (perhaps including even yourself) believe that in the absence of "the same mathematics" the "precious bandwidth is better spent" discussing the possible "shape" of the new model, may be you (against yourself) can explain how one can "discuss specific problems" independent of mathematical models, and what these specific problems are. --Lev From GOLDFARB%unb.ca at UNBMVS1.csd.unb.ca Fri Jan 4 02:39:38 1991 From: GOLDFARB%unb.ca at UNBMVS1.csd.unb.ca (GOLDFARB%unb.ca@UNBMVS1.csd.unb.ca) Date: Fri, 04 Jan 91 03:39:38 AST Subject: flag waving In-Reply-To: Message of Wed, 2 Jan 91 13:49:15 EST from Message-ID: Jim, May I take the great responsibility to assure you that it has not "been a waste of time to begin with". But I'm "still unclear about what" you are and, in fact, what all of us are. "First, it is not clear to that an 'analytical model of an intelligent system' is the common goal of the people on this mailing list", then what is the common goal of the people on this mailing list? (see also the von Neumann's quotation in my earlier posting) Second, "one has learned enough about the biology to know that the old models are off the mark". --Lev Goldfarb From chan%unb.ca at UNBMVS1.csd.unb.ca Fri Jan 4 11:26:25 1991 From: chan%unb.ca at UNBMVS1.csd.unb.ca (Tony Chan) Date: Fri, 04 Jan 91 12:26:25 AST Subject: AI vs. NN Message-ID: "NN and AI are fundamentally equivalent in their descriptive powers." So what? Formalisms such as Post systems, Chomsky's type-0 grammars, Lev Goldfarb's transformation systems, McCarthy's Lisp, Wirth's Pascal, RAM, and thousands upon thousands of other formalisms can compute or describe as much as Turing machines can. The more interesting questions, to me, are 1) What are the forces that cause such proliferation? 2) What are the differences among these formalisms? 3) What are the unique properties about each? 4) How do we categorize them? The most important issue for us, I believe, is that we want to have a formalism that simulate learning so that if we use this formalism to describe some "intelligent" (learning) processes, it is relatively easy to express it using this formalism. Also, this formalism cannot be an ad hoc one because its purpose is to model learning and learning is a very general phenomenon. To some extent, the neural net formalism fits the bill because of its limited self-programability and adaptability. Unfortunately, it lacks generality in the sense that it is not well-suited for high-level/ symbolic type of learning. This, partly, is why I believe a more general formalism such as Reconfigurable Learning Machines should be called for or at least debated! From gary at CS.CMU.EDU Fri Jan 4 15:51:21 1991 From: gary at CS.CMU.EDU (gary (Gary Cottrell)) Date: Fri, 4 Jan 91 12:51:21 PST Subject: Correct reference Message-ID: <9101042051.AA27731@desi.ucsd.edu> I get enough requests for this that I feel it is necessary to post it here. Sorry if you don't care! Somehow, the correct reference for the image compression paper we put in Sharkey's editied volume has been scrambled "out there". Here it is: Cottrell, G., Munro, P. and Zipser D. (1989) Image compression by back propagation: A demonstration of extensional programming. In Noel Sharkey (Ed.), \fIModels of Cognition: A review of Cognitive Science, Vol. 1\fP, Norwood: Ablex. From hinton at ai.toronto.edu Fri Jan 4 16:56:48 1991 From: hinton at ai.toronto.edu (Geoffrey Hinton) Date: Fri, 4 Jan 1991 16:56:48 -0500 Subject: A good textbook on neural computation Message-ID: <91Jan4.165653edt.1072@neuron.ai.toronto.edu> Ever since the binding fell apart on my copy of Rumelhart and McClelland I have been looking for a more recent graduate textbook on neural computation (the artificial kind). I imagine that quite a few of the other people on this mailing list have the same problem for their courses. There are a lot of attempts at textbooks out there, and many of the attempts are very good in one respect or another, but (in my opinion) none of them gives good clear coverage of most of the main ideas. However, I just got hold of a really good textbook (in my opinion). It is: Introduction to the Theory of Neural Computation by J. Hertz, A. Krogh and R. Palmer Addison Wesley, 1991. The beginning and end of the book are rather biased towards the physicists view of the world (in my opinion), but the middle covers a lot of the basic material very nicely. Geoff Hinton PS: If you object to getting people's opinions about textbooks, please complain to me, not to the whole mailing list. From simic at kastor.ccsf.caltech.edu Fri Jan 4 18:03:39 1991 From: simic at kastor.ccsf.caltech.edu (Petar Simic) Date: Fri, 4 Jan 91 23:03:39 GMT Subject: ap Message-ID: <1991Jan4.230339.26806@nntp-server.caltech.edu> Apology to all who, due to some problem with local server at CalTech, had to read twice my recent message. I hope that this will not happen again. Petar Simic From tsejnowski at UCSD.EDU Sat Jan 5 14:59:29 1991 From: tsejnowski at UCSD.EDU (Terry Sejnowski) Date: Sat, 5 Jan 91 11:59:29 PST Subject: AI (discrete) moodel and NN (continuous) model Message-ID: <9101051959.AA13566@sdbio2.UCSD.EDU> Regarding my comments about ideologies vs concrete problems, I was objecting to arguments about semantics (what is and what isn't AI) and the right problem to study (biology vs machines) when it is clear that different people are interested in different problems and you arn't going to solve any problems by arguing about taste. Ken Miller made this point much better than I did. Regarding the issue of mathematics vs problems, there are quite a lot of problems that can be attacked with what is already known about nonlinear dynamical systems (including the present generation of recurrent neural nets). If new mathematics is needed someday we can create it, but my own preference is to concentrate on interesting problems and to let the problem, whether it is a computational, psychological, or a biological one, guide you to the right assumptions. The difficulty with simply looking for new mathematics is that the number of models and mathematical formalisms is infinite and without some guidance the chances are you will be studying vortices. I agree here with Jim Bower and Ken Miller. Therefore, debating ideologies is less productive than discussing interesting research problems. Regarding interesting research problems, has anyone made any progress with the Lo problem posed by Feldman, Lakoff et al last year? This is a miniature language acquisition problem that has elements of vision, planning and learning as well as language. As stated, the goal of the problem is to be able to answer simple yes/no questions about pictures of squares, triangles and circles ("Is the circle to the right of the square?"). A variant on this problem would be to include a motor component, that is, to have the system perform simple manipulations of the picture ("Move the circle to the right side of the square.") I suspect that the motor system is an important part of animal cognition that wouldn't be captured by a system that simply answered questions. This problem would also provide interesting comparisons with Winograd's program. It is at the systems level and is the sort that Aaron Sloman was referring to in his long contribution to this topic, started, you will all recall, by Jerry Feldman's original posting asking for achievements in connectionism. I hope that we have contributed more than just this ongoing debate. Terry ----- From GOLDFARB%unb.ca at unbmvs1.csd.unb.ca Sat Jan 5 19:45:12 1991 From: GOLDFARB%unb.ca at unbmvs1.csd.unb.ca (GOLDFARB%unb.ca@unbmvs1.csd.unb.ca) Date: Sat, 05 Jan 91 20:45:12 AST Subject: arguments for all seasons In-Reply-To: Message of Fri, 4 Jan 91 03:37:54 EST from Message-ID: As an explanation of why AI/NN (discrete/continuous)debate is very important, I simply refer you to the similar situations throughout the development of physics: I might remark that history shows us that reconciling inconsistent physical theories is a very good way of making fundamental progress. . . . So, many of the most far reaching advances of the twentieth century have come about because previous theories weren't compatible with one another. History teaches us that reconciling incompatibilities between theories is a good way to make really fundamental progress. (Edward Witten, in Superstrings: A theory of Everything? eds. P.C.W. Davies and J.Brown, Cambridge Univ. Press, 1988, pp.97-8) Since it appears that some of our neurobiological friends feel somewhat left out of the debate, let me explain why I think they "would all be wise to be humble and to listen well." Again, we should turn to physics: The theory of relativity is a fine example of the fundamental character of the modern development of theoretical science. The hypotheses become steadily more abstract and remote from experience. On the other hand, it gets nearer to the grand aim of all science, which is to cover the greatest possible number of empirical facts by logical deduction from the smallest possible number of hypotheses or axioms. Meanwhile, the train of thought leading from the axioms to the empirical facts or verifiable consequences gets steadily longer and more subtle. The theoretical scientist is compelled in an increasing degree to be guided by purely mathematical, formal considerations in his search for a theory, because the physical experience of the experimenter cannot lead him up to the regions of highest abstraction. . . . The theorist who undertakes such a labor should not be caped at as "fanciful"; on the contrary, he should be granted the right to give free reign to his fancy, for *there is no other way to the goal*. (A. Einstein, see the book Ideas and Opinions, by A. Einstein, p.282) I strongly believe that in the study of intelligence we are faced from the very beginning with even more "dramatic" situation: if "the train of thought leading from the axioms to the empirical facts" for the *first* physical theories was relatively short (and that is why a more direct, "hands on" approach was possible), the theory of intelligence, or intelligent (biological) information processing, has not and *cannot* originate with the theories similar in the mathematical structure to the first physical theories, simply because the basic elements of "information" and "intelligence" are much more abstract and are not visible to the naked eye. It is intersting to note that many of the leading physicists would probably agree with this statement. (For one of the most resent opinions see The Emperor's New Mind, by R.Penrose) Besides, any specific biological entity on any "planet" represents an outcome of a particular evolutionary implementation of the intelligence. That is why the "computer and the oscilloscope" analogy is quite appropriate. In conclusion, I strongly believe that, as it is also becoming more and more apparent in physics, the mathematical models of intelligence will strongly lead the neurobiological and perceptual experiments and not the other way around. --Lev Goldfarb From worth at park.bu.edu Sun Jan 6 15:31:10 1991 From: worth at park.bu.edu (Andrew J. Worth) Date: Sun, 6 Jan 91 15:31:10 -0500 Subject: Connectionism vs AI Survey Message-ID: <9101062031.AA21945@park.bu.edu> Since I feel slightly responsible for stoking the current debate on Connectionism vs AI, allow me to propose two surveys which may allow its fruition. Members of the connectionist list have varying backgrounds and interests that lead them to correspondingly various research emphases and various amounts of attention paid to other emphases. Perhaps it would be beneficial to all to find out what we (the collective list) believe is important. Please respond to me (worth at park.bu.edu) with your own personal thoughts on either or both of the following: SURVEY 1: 1) What is your emphasis or interest in this field (i.e. why are you on the connectionist list) and why is this aspect of the field important? 2) If you were King for a day, where would you direct further research? The main question that I am trying to ask here is: Specifically, what aspects of your interests should be important to the rest of us? Or, more generally, what should be the key aspects of Connectionism? SURVEY 2: What SPECIFIC aspects of neural physiology, anatomy, and topology should be considered by Connectionists? PLEASE BE BRIEF. I will summarize, collate, and post your responses (if there is enough interest) in a few weeks. Thanks in advance, Andy. ----------------------------------------------------------------------- Andrew J. Worth worth at park.bu.edu (617) 353-6741 Cognitive & Neural Systems Boston University Center for Adaptive Systems 111 Cummington St. Room 244 (617) 353-7857 Boston, MA 02215 USA From tsejnowski at ucsd.edu Sun Jan 6 18:34:19 1991 From: tsejnowski at ucsd.edu (Terry Sejnowski) Date: Sun, 6 Jan 91 15:34:19 PST Subject: Seminar: Barron on Scaling Message-ID: <9101062334.AA04550@sdbio2.UCSD.EDU> Computer Science Seminar "Approximation Properties of Artificial Neural Networks" Andrew R. Barron University of Illinois Monday, January 7, 4 PM 7421 Applied Physics and Mathematics Building University of California, San Diego Bounds on the approximation error of a class of feed-forward artificial neural network models are presented. A previous result obtained by George Cybenko and by Kurt Hornik, Max Stinchcombe, and Hal White shows that linear combinations of sigmoidal functions are dense in the space of continuous functions on compact subsets in d dimensions. In this talk we examine how the approximation error is related to the number of nodes in the network. We impose the regularity condition that the gradient of the function of d variables has an integrable Fourier transform. In particular, bounds are obtained for the integrated squared error of approximation, where the integration is taken on any given ball and with respect to any probability measure. It is shown that there is a linear combination of n sigmoidal functions such that the integrated squared error is bounded by c/n, where the constant c is depends on the radius of the ball and the integral of the norm of the Fourier transform of the gradient of the function. A sigmoidal function is the composition of a given bounded increasing function of one variable with a linear function of d variables. Such sigmoidal functions comprise a standard artificial neuron model and the linear combination of such functions is a one-layer artificial neural network. The surprising aspect of this result is that an approximation rate is achieved which is independent of the dimension d, using a number of parameters O(nd) which grows only linearly in d. This is in contrast to traditional series expansions which require exponentially many parameters O(n^d) to achieve approximation rates of order O(1/n), under somewhat different hypotheses on the class of functions. We conclude that the "curse of dimensionality" does not apply to the class of functions we examine. From jbower at smaug.cns.caltech.edu Mon Jan 7 00:11:23 1991 From: jbower at smaug.cns.caltech.edu (Jim Bower) Date: Sun, 6 Jan 91 21:11:23 PST Subject: peace and platitudes Message-ID: <9101070511.AA15394@smaug.cns.caltech.edu> In response to Lev Goldfarbs last posting about analytical models: As Terry, Ken, etc., I too think that no one description fits the goals of everyone on this mailing list. Some want to build things, some want to figure out how things are built. Others would like to construct a grand formalism that covers it all. The approach, interests, convictions between and within each group differ. Without question this is our strength. None of these efforts are trivial and it is silly to think that one or the other is the more correct interest to have. Speaking from my own, somewhat underrepresented corner of this effort, I have tried to point out, perhaps a bit too strongly, that these distinctions exist, that they are significant, and that it is important that they be recognized and respected. Jim Bower From sontag at control.rutgers.edu Mon Jan 7 11:37:04 1991 From: sontag at control.rutgers.edu (sontag@control.rutgers.edu) Date: Mon, 7 Jan 91 11:37:04 EST Subject: 4 vs 3 layers -- Tech Report available from connectionists archive Message-ID: <9101071637.AA01072@control.rutgers.edu> REPORT AVAILABLE ON CAPABILITIES OF FOUR-LAYER vs THREE-LAYER NETS At the request of a few people at NIPS, I placed in the connectionists archive the postscript version of my report describing why TWO hidden layers are sometimes necessary when solving function-approximation types of problems, a fact that was mentioned in my poster. (About 1/2 of the report deals with the general question, while the other half is devoted to the application to control that led me to this.) Below are the abstract and instructions on ftp retrieval. I would very much welcome any discussion of the practical implications --if any-- of the result. If you want, send email to me and I can summarize later for the net. Happy palindromic year to all, -eduardo ----------------------------------------------------------------------------- Report SYCON-90-11, Rutgers Center for Systems and Control, October 1990 FEEDBACK STABILIZATION USING TWO-HIDDEN-LAYER NETS This report compares the representational capabilities of three-layer (that is, "one hidden layer") and four-layer ("two hidden layer") nets consisting of feedforward interconnections of linear threshold units. It is remarked that for certain problems four layers are required, contrary to what might be in principle expected from the known approximation theorems. The differences are not based on numerical accuracy or number of units needed, nor on capabilities for feature extraction, but rather on a much more basic classification into "direct" and "inverse" problems. The former correspond to the approximation of continuous functions, while the latter are concerned with approximating one-sided inverses of continuous functions ---and are often encountered in the context of inverse kinematics determination or in control questions. A general result is given showing that nonlinear control systems can be stabilized using four layers, but not in general using three layers. ----------------------------------------------------------------------- To obtain copies of the postscript file, please use Jordan Pollack's service: Example: unix> ftp cheops.cis.ohio-state.edu # (or ftp 128.146.8.62) Name (cheops.cis.ohio-state.edu:): anonymous Password (cheops.cis.ohio-state.edu:anonymous): ftp> cd pub/neuroprose ftp> binary ftp> get (remote-file) sontag.twolayer.ps (local-file) twolayer.ps.Z ftp> quit unix> uncompress twolayer.ps unix> lpr -P(your_local_postscript_printer) twolayer.ps ---------------------------------------------------------------------------- If you have any difficulties with the above, please send e-mail to sontag at hilbert.rutgers.edu. DO NOT "reply" to this message, please. From gmdzi!smieja at relay.EU.net Mon Jan 7 13:40:05 1991 From: gmdzi!smieja at relay.EU.net (Frank Smieja) Date: Mon, 7 Jan 91 17:40:05 -0100 Subject: Vision Message-ID: <9101071640.AA06439@gmdzi.UUCP> Could someone please let me know about the recent objections to Marr's theory of vision, as expounded in his book of the same title? I am particularly interested in current (new) ideas about stereopsis. References to actual vision work are not hard to find, but they are somewhat voluminous, and I would be most grateful just for a brief indication of "where Marr was in error", so that I do not fall into the same trap. If there are fears of starting an unending debate of pro- and anti-Marr arguers, then mail me directly, and I will echo any good articles suggested back to the net. Frank Smieja From kimd at gizmo.usc.edu Mon Jan 7 20:12:13 1991 From: kimd at gizmo.usc.edu (Kim Daugherty) Date: Mon, 7 Jan 1991 17:12:13 PST Subject: Connectionist Simulators Message-ID: Last November, I posted a request for connectionist modeling simulators to the mailing list. I would like to thank those who responded. Following is a list and brief description of several simulators: 1. Genesis - An elaborate X windows simulator that is particularly well suited for modeling biological neural networks. unix> telnet genesis.cns.caltech.edu (or 131.215.135.185) Name: genesis Follow directions there to get a ftp account from which you can ftp 'genesis.tar.Z". This contains genesis source and several tutorial demos. NOTE: There is a fee to become a registered user. 2. PlaNet (AKA SunNet) - A popular connectionist simulator with versions to run under SunTools, X Windows, and non-graphics terminals created by Yoshiro Miyata. The SunTools version is not supported. unix> ftp boulder.colorado.edu (128.138.240.1) Name: anonymous Password: ident ftp> cd pub ftp> binary ftp> get PlaNet5.6.tar.Z ftp> quit unix> zcat PlaNet5.6.tar.Z | tar xf - All you need to do to try it is to type: unix> Tutorial This will install a program appropriate for your environment and start an on-line tutorial. If you don't need a tutorial, just type 'Install' to install the system and then 'source RunNet' to start it. See the file README for more details. The 60-page User's Guide has been split into three separate postscript files so that each can be printed from a printer with limited memory. Print the files doc/PlaNet_doc{1,2,3}.ps from your postscript printer. See the doc/README file for printing the Reference Manual. Enjoy!! And send any questions to miyata at boulder.colorado.edu. 3. CMU Connectionist Archive - There is a lisp backprop simulator in the connectionist archive. unix> ftp b.gp.cs.cmu.edu (or 128.2.242.8) Name: ftpguest Password: cmunix ftp> cd connectionists/archives ftp> get backprop.lisp ftp> quit 4. Cascade Correlation Simulator - There is a LISP and C version of the simulator based on Scott Fahlman's Cascade Correlation algorithm, who also created the LISP version. The C version was created by Scott Crowder. unix> ftp pt.cs.cmu.edu (or 128.2.254.155) Name: anonymous Password: (none) ftp> cd /afs/cs/project/connect/code ftp> get cascor1.lisp ftp> get cascor1.c ftp> quit A technical report descibing the Cascade Correlation algorithm may be obtained as follows: unix> ftp cheops.cis.ohio-state.edu (or 128.146.8.62) Name: anonymous Password: neuron ftp> cd pub/neuroprose ftp> binary ftp> get fahlman.cascor-tr.ps.Z ftp> quit unix> uncompress fahlman.cascor-tr.ps.Z unix> lpr fahlman.cascor-tr.ps 5. Quickprop - A variation of the back-propagation algorithm developed by Scott Fahlman. A LISP and C version can be obtained in the same directory as the cascade correlation simulator above. Kim Daugherty kimd at gizmo.usc.edu From slehar at park.bu.edu Tue Jan 8 08:28:18 1991 From: slehar at park.bu.edu (Steve Lehar) Date: Tue, 8 Jan 91 08:28:18 -0500 Subject: Vision (What's wrong with Marr's model) Message-ID: <9101081328.AA06421@park.bu.edu> Frank Smieja asks about recent objections to Marr's theory of vision. Here is my opinion. David Marr's book is delightfully lucid and beautifully illustrated, and I thoroughly agree with his analysis of the three levels of modeling. Nevertheless I believe that there are two fatal flaws in the philosophy of his vision model. The first fatal flaw is the feedforward nature of this model, from the raw primal sketch through the 2&1/2 D sketch to the 3-D model representation. Decades of "image understanding" and "pattern recognition" research have shown us that such feed-forward processing has a great deal of difficulty with natural imagery. The problem lies in the fact that whenever "feature extraction" or "image enhancement" are performed, they recognize or enhance some features but in the process they inevitably degrade others or introduce artifacts. With successive levels of processing the artifacts accumulate and combine until at the highest levels of processing there is no way to distinguish the real features from the artifacts. Even in our own vision, with all its sophistication, we occasionally see things that are not there. The real problem with such feedforward models is that once a stage of processing is performed, it is never reviewed or reconsidered. Grossberg suggests how nature solves this problem, by use of top-down feedback. Whenever a feature is recognized at any level, a copy of that feature is passed back DOWN the processing hierarchy in an attempt to improve the match at the lower levels. If for instance a set of disconnected edges suggest a larger continuous edge to a higher level, that "hypothesis" is passed down to the local edge detectors to see if they can find supporting evidence for the missing pieces by locally lowering their detection thresholds. If a faint edge is indeed found where expected, it is enhanced by resonant feedback. If however there is strong local opposition to the hypothesis then the enhancement is NOT performed. This is the cooperative / competitive loop of the BCS model which serves to disambiguate the image by simultaneous matching at multiple levels. This explains how, when we occasionally see something that isn't there, we see it in such detail until at a higher level a conflict occurs, at which time the apparition "pops" back to being something more consistant with the global picture. The second fatal flaw in Marr's vision model is related to the first. In the finest tradition of "AI", Marr's 3-D model is an abstract symbolic representation of the visual input, totally divorced from the lower level stimuli which generated it. The great advance of the connectionist perspective is that manipulation of high level symbols is meaningless without regard to the hierarchy of lower level representations to which they are attached. When you look at your grandmother for instance, some high level node (or nodes) must fire in recognition. At the same time however you are very conscious of the low level details of the image, the strands of hair, the wrinkles around the eyes etc. In fact, even in her absence the high level node conjurs up such low level features, without which that node would have no real meaning. It is only because that node rests on the pinacle of a hierarchy of such lower level nodes that it has a meaning of "grandmother". The perfectly gramatical sentence "Grandmother is purple" is only recognized as nonsense when visualized at the lowest level, illustrating that logical processing cannot be separated from low level visualization. Although I recognize Marr's valuable and historic contribution to the understanding of vision, I believe that in this fast moving field we have already progressed to new insights and radically different models. I would be delighted to provide further information by email to interested parties on Grossberg's BCS model, and my own implementation of it for image processing applications. (O)((O))(((O)))((((O))))(((((O)))))(((((O)))))((((O))))(((O)))((O))(O) (O)((O))((( slehar at park.bu.edu )))((O))(O) (O)((O))((( Steve Lehar Boston University Boston MA )))((O))(O) (O)((O))((( (617) 424-7035 (H) (617) 353-6741 (W) )))((O))(O) (O)((O))(((O)))((((O))))(((((O)))))(((((O)))))((((O))))(((O)))((O))(O) From jrs at cs.williams.edu Tue Jan 8 11:04:31 1991 From: jrs at cs.williams.edu (Josh Smith) Date: Tue, 8 Jan 91 11:04:31 EST Subject: C. elegans Message-ID: <9101081604.AA23879@bull> I just read that all 329 (?) neurons in C. elegans, the nematode worm, have been mapped. (That is, the connectivity or wiring pattern for the worm is known.) Even though its nervous system is so simple, the worm apparently has a quite a wide range of behaviors (swimming, following odors, avoiding salt, mating, etc.). Has anyone ever simulated this network? I think such a simulation would be very useful. If it didn't work, that might indicate that the idealized neuron (sum-unit-cum-squashing-function) in use now is too simple. If it did work, connectionists could proceed with more confidence that this idealization is not absurd (if they care). I'm sure this simulation would be interesting in many other ways as well. With the number of neurons in the network, I think it would even be a computationally feasible undertaking. From birnbaum at fido.ils.nwu.edu Tue Jan 8 15:54:23 1991 From: birnbaum at fido.ils.nwu.edu (Lawrence Birnbaum) Date: Tue, 8 Jan 91 14:54:23 CST Subject: Machine learning workshop: Addendum to call for papers Message-ID: <9101082054.AA01130@fido.ils.nwu.edu> ADDENDUM TO CALL FOR PAPERS EIGHTH INTERNATIONAL WORKSHOP ON MACHINE LEARNING NORTHWESTERN UNIVERSITY EVANSTON, ILLINOIS JUNE 27-29, 1991 We wish to clarify the position of ML91 with respect to the issue of multiple publication. In accordance with the consensus expressed at the business meeting at ML90 in Austin, ML91 is considered by its organizers to be a specialized workshop, and thus papers published in its proceedings may overlap substantially with papers published elsewhere, for instance IJCAI or AAAI. The sole exception is with regard to publication in future Machine Learning Conferences. Authors who are concerned by this constraint will be given the option of foregoing publication of their presentation in the ML91 Proceedings. The call for papers contained information concerning seven of the eight individual workshops that will make up ML91. Information concerning the final workshop follows. Larry Birnbaum Gregg Collins Northwestern University The Institute for the Learning Sciences 1890 Maple Avenue Evanston, IL 60201 (708) 491-3500 ------------------------------------------------------------------------------- COMPUTATIONAL MODELS OF HUMAN LEARNING This workshop will foster interaction between researchers concerned with psychological models of learning and those concerned with learning systems developed from a machine learning perspective. We see several ways in which simulations intended to model human learning and algorithms intended to optimize machine learning may be mutually relevant. For example, the way humans learn and the optimal method may turn out to be the same for some tasks. On the other hand, the relation may be more indirect: modeling human behavior may provide task definitions or constraints that are helpful in developing machine learning algorithms; or machine learning algorithms designed for efficiency may mimic human behavior in interesting ways. We invite papers that report on learning algorithms that model or are motivated by learning in humans or animals. We encourage submissions that address any of a variety of learning tasks, including category learning, skill acquisition, learning to plan, and analogical reasoning. In addition, we hope to draw work from a variety of theoretical approaches to learning, including explanation-based learning, empirical learning, connectionist approaches, and genetic algorithims. In all cases, authors should explicitly identify 1) in what ways the system's behavior models human (or animal) behavior, 2) what principles in the algorithm are responsible for this, and 3) the methods for comparing the system's behavior to human behavior and for evaluating the algorithm. A variety of methods have been proposed for computational psychological models; we hope the workshop will lead to a clearer understanding of their relative merits. Progress reports on research projects still in development are appropriate to submit, although more weight will be given to projects that have been implemented and evaluated. Integrative papers providing an analysis of multiple systems or several key issues are also invited. WORKSHOP COMMITTEE Dorrit Billman (Georgia Tech) Randolph Jones (Univ. of Pittsburgh) Michael Pazzani (Univ. of California, Irvine) Jordan Pollack (Ohio State Univ.) Paul Rosenbloom (USC/ISI) Jeff Shrager (Xerox PARC) Richard Sutton (GTE) SUBMISSION DETAILS Papers should be approximately 4000 words in length. Authors should submit seven copies, by March 1, 1991, to: Dorrit Billman School of Psychology Georgia Institute of Technology Atlanta, GA 30332 phone (404) 894-2349 Formats and deadlines for camera-ready copy will be communicated upon acceptance. From aarons at cogs.sussex.ac.uk Tue Jan 8 17:43:46 1991 From: aarons at cogs.sussex.ac.uk (Aaron Sloman) Date: Tue, 8 Jan 91 22:43:46 GMT Subject: Vision (What's wrong with Marr's model) Message-ID: <3206.9101082243@rsuna.cogs.susx.ac.uk> I guess some cynics might respond that connectionists are now talking about vision in a way that's not too far (discounting technical details) from what AI vision researchers were doing before Marr came along and started telling them all how it should be done! Aaron From GOLDFARB%UNB.CA at UNBMVS1.csd.unb.ca Tue Jan 8 19:27:09 1991 From: GOLDFARB%UNB.CA at UNBMVS1.csd.unb.ca (Goldfarb) Date: Tue, 08 Jan 91 20:27:09 AST Subject: continuous vs symbolic: a more concrete problem Message-ID: Over the last two decades it has become reasonably clear that at some stage of information processing (even in vision) it is often convenient to represent objects in a symbolic form. The simplest symbolic representation is a string over a finite alphabet. An immediate question that arises then is: How does one go about constructing a neural network that can recognize a reasonably large (infinite and nontrivial) *class* of formal languages? For example, let us specify a language in the following way: fix 1) some string (e.g. dabbe) and 2) a finite set S of strings (e.g. S={aa, da, cdc}); then the language is formed by all strings that can be obtained from the single fixed string (dabbe) by inserting in any place and in any order any number of strings from set S. Consider now the class of *all* languages that can be specified in this way. It is a subclass of the class of regular languages. If the NN is "the right" model, then the above problem should have a reasonably simple solution. Does anyone know such a solution? By the way, the reason I have chosen the above class of languages is that the new RLM model mentioned in several earlier postings solves the the problem in a very routine manner. --Lev Goldfarb From orjan at thalamus.sans.bion.kth.se Wed Jan 9 08:17:11 1991 From: orjan at thalamus.sans.bion.kth.se (Orjan Ekeberg) Date: Wed, 9 Jan 91 14:17:11 +0100 Subject: C. elegans In-Reply-To: Josh Smith's message of Tue, 8 Jan 91 11:04:31 EST <9101081604.AA23879@bull> Message-ID: <9101091317.AA05479@thalamus> Based on my experience with fairly realistic simulations of the neuronal network producing swimming in the Lamprey (a fish-like lower vertebrate), I would expect that much more information than the wiring pattern is needed to produce a "working" computer model of this nematode. Let me give some examples. Rythmic activity (you mention swimming) in simple invertebrates often depends on single spikes in each cycle. This indicates that a cell model can not rely on an output representing only firing frequency. There might, however, be parts of the system where a squashing-function neuron model is sufficient. Individual cell properties like membrane time constants, localization of synapses, intrinsic pacemaker properties etc. probably plays a crucial role in such a network. Even if such properties are in principle measurable, it is necessary to know what you are looking for beforehand, i.e. you need a theory of the function of each neuron. I believe that computer simulation is an important tool when constructing such theories. Thus, I do not believe that it is possible to follow the strategy of FIRST measuring everything and THEN simulate the final model. Rather, understanding of how a real network operates could gradually be gained in a process involving simulations as one tool. I do agree that, from a computational point of view, simulating 329 interconnected neurons is feasible. We have been simulating about 600 neurons from the Lamprey including a four compartment representation of the dendrites and Na, K, Ca and Ca dependent K channels, transmitter and voltage dependent (NMDA) synapses, etc. Even if the C. elegans system might need further properties added, I think it would still be possible to do simulations even on this level of detail. Orjan Ekeberg +---------------------------------+-----------------------+ + Orjan Ekeberg + O---O---O + + Department of Computing Science + \ /|\ /| Studies of + + Royal Institute of Technology + O-O-O-O Artificial + + S-100 44 Stockholm, Sweden + |/ \ /| Neural + +---------------------------------+ O---O-O Systems + + EMail: orjan at bion.kth.se + SANS-project + +---------------------------------+-----------------------+ From gmdzi!nieters%sphinx at relay.EU.net Wed Jan 9 09:20:00 1991 From: gmdzi!nieters%sphinx at relay.EU.net (Hans Nieters) Date: Wed, 9 Jan 91 15:20+0100 Subject: paper announcement Message-ID: <19910109142040.4.NIETERS@sphinx> The following 15 page paper is now available. It can be ftp-ed from GMD under the name nieters.petri.neural.ps.Z, as shown below, or can be ordered by sending a note to: Hans Nieters GMD-F2G2 W-5205 St. Augustin Postfach 1240 Germany or Email: nit at gmdzi.gmd.de Neural Networks as Predicate Transition systems The relationship between NN and Petri nets is investigated, since both have their merits for seemingly different application areas. For non-recurrent NN (with and without backprop.) a Predicate Transition system (a very high level class of Petri nets) can be given, which is in sense behaviorally equivalent. The aim of this approach is to show that both modeling techniques could gain benefits from each other, if they were put on a common basis: Petrinets allow modeling of concurrency and conflict (missing in NN) whereas NN contributes learning (missing in Petri nets). The proposed technique for transforming NN into PrT systems is demonstrated by examples. The paper is written mainly for people familiar with Petrinets and newcomer in NN, but perhaps some connectionists may find it helpful also, despite of the fact, that the results are very preliminary. unix> ftp gmdzi.gmd.de Name: anonymous Password: state-your-name-please ftp> cd pub/gmd ftp> binary ftp> get nieters.petri.neural.ps.Z (ca 176000 bytes) ftp> quit unix> uncompress nieters.petri.neural.ps.Z unix> lpr nieters.petri.neural.ps The postscript file has been tested and will hopefully print also YOUR printer From slehar at park.bu.edu Wed Jan 9 10:52:33 1991 From: slehar at park.bu.edu (Steve Lehar) Date: Wed, 9 Jan 91 10:52:33 -0500 Subject: Vision (What's wrong with Marr's model) In-Reply-To: Aaron Sloman's message of Tue, 8 Jan 91 22:43:46 GMT <3206.9101082243@rsuna.cogs.susx.ac.uk> Message-ID: <9101091552.AA15053@park.bu.edu> Aaron> I guess some cynics might respond that connectionists are now talking Aaron> about vision in a way that's not too far (discounting technical details) Aaron> from what AI vision researchers were doing before Marr came along and Aaron> started telling them all how it should be done! Aaron> Aaron Tell me about the AI vision researchers before Marr that supported vision models inspired by natural architectures (PDP/connectionist) with intensive feedback mechanisms (as seen in nature) motivated by neurophysiological (single cell recordings) and psychophysical (perceptual illusions) data, making testable hypotheses (reproducing the illusions) about natural vision. I haven't heard about them. From fritz_dg%ncsd.dnet at gte.com Wed Jan 9 12:49:57 1991 From: fritz_dg%ncsd.dnet at gte.com (fritz_dg%ncsd.dnet@gte.com) Date: Wed, 9 Jan 91 12:49:57 -0500 Subject: C. elegans Message-ID: <9101091749.AA02051@bunny.gte.com> Second the motion. If anyone has or is contemplating, a response to the list would be of great interest. fritz_dg%ncsd at gte.com From MURTAGH at SCIVAX.STSCI.EDU Wed Jan 9 13:34:54 1991 From: MURTAGH at SCIVAX.STSCI.EDU (MURTAGH@SCIVAX.STSCI.EDU) Date: Wed, 9 Jan 1991 13:34:54 EST Subject: Workshop, "NNs for Stat. & Econ. Data" Message-ID: <910109133454.20201a55@SCIVAX.STSCI.EDU> Workshop on "Neural Networks for Statistical and Economic Data" This workshop, organized by Munotec Systems, and funded by the Statistical Office of the European Communities, Luxembourg, was held in Dublin, Ireland, on December 10-11, 1990. A proceedings, including abstracts and in many instances papers, will be reproduced and sent to all on the mailing list of the DOSES funding program in the near future. DOSES ("Design of Statistical Expert Systems") is one of the European Community funding programs, and is administered by the Statistical Office. Requests to be included on this mailing list should be addressed to: DOSES, Statistical Office of the European Communities, Batiment Jean Monnet, B.P. 1907, Plateau du Kirchberg, L-2920 Luxembourg. F. Murtagh (murtagh at scivax.stsci.edu, fionn at dgaeso51.bitnet) -------------------------------------------------------------------------------- The following were the talks given at the Dublin meeting: M. Perremans (Stat. Office of EC, Luxembourg) "The European Community statistical research programs." H.-G. Zimmermann (Siemens, Munich) "Neural network features in economics." J. Frain (Central Bank of Ireland, Dublin) "Complex questions in economics and economic statistics." M.B. Priestley (UMIST, Manchester) "Non-linear time series analysis: overview." R. Rohwer (CSTR, Edinburgh) "Neural networks for time-varying data." P. Ormerod and T. Walker (Henley Centre, London) "Neural networks and the monetary base in Switzerland." S. Openshaw and C. Wymer (Univ. of Newcastle upon Tyne) "A neural net classifier system for handling census data." F. Murtagh (Munotec, Dublin; ST-ECF, Munich) "A short survey of neural network approaches for forecasting." D. Wuertz and C. de Groot (ETH, Zrich) "Modeling and forecasting of univariate time series by parsimonious feedforward connectionist nets." J.-C. Fort (Univ. de Paris 1) "Kohonen algorithm and the traveling salesman problem." H.-G. Zimmermann (Siemens, Munich) "Completion of incomplete data." R. Hoptroff and M.J. Bramson (London) "Forecasting the economic cycle." A. Varfis and C. Versino (JRC, Ispra) "Neural networks for economic time series forecasting." D. Mitzman and R. Giovannini (Cerved SpA, Padua) "ActivityNets: A neural classifier of natural language descriptions of economic activities." (Also: demonstration on 386-PC.) C. Doherty (ERC, Dublin) "A comparison between the recurrent cascade-correlation architecture and the Box and Jenkins method on forecasting univariate time series." M. Eaton and B.J. Collins (Univ. of Limerick, Limerick) "Neural network front end to an expert system for decision taking in an uncertain environment." R.J. Henery (Univ. of Strathclyde, Glasgow) "StatLog: Comparative testing of statistical and logical learning algorithms." Ah Chung Tsoi (Univ. of Queensland) "FIR and IIR synapses, a neural network architecture for time series modelling." A. Singer (Thinking Machines, Munich) "Focusing on feature extraction in pattern recognition." R. Rohwer (CSTR, Univ. of Edinburgh) "The 'Moving Targets' algorithm for difficult temporal credit assignment problems." -------------------------------------------------------------------------------- From INS_ATGE%JHUVMS.BITNET at VMA.CC.CMU.EDU Wed Jan 9 13:40:00 1991 From: INS_ATGE%JHUVMS.BITNET at VMA.CC.CMU.EDU (INS_ATGE%JHUVMS.BITNET@VMA.CC.CMU.EDU) Date: Wed, 9 Jan 91 13:40 EST Subject: SSAB questions Message-ID: I am interested in using the Super Self-Adapting Back Propogation (SuperSAB) algorithm recently published in _Neural_Networks_ (T. Tollenaere). The algorithm published appears a bit ambiguous to me. In step four, it says "undo the previous weight update (which caused the change in the gradient sign). This can be done by using [delta weight(i,j,n+1)] = -[delta weight(i,j,n)], instead of calculating the weight-update..." Does this mean undo the previous update of _all_ network weights, or just undo the update of the particular weights which changed sign? Anyway, I am going to try to use SuperSAB to speed up a time-delay neural net (TDNN) of the sort used in Lang, Waibel, and Hinton _Neural_Networks_ 3, p.23 for analysis of multiple-band infrared temporal intensity data. While I am on ther subject, has anyone done a comparison between Quickprop and SuperSAB, or has used SuperSAB in Cascade-Correlation or Time Delay Neural Nets? -Thomas Edwards From gary at cs.UCSD.EDU Wed Jan 9 14:37:38 1991 From: gary at cs.UCSD.EDU (Gary Cottrell) Date: Wed, 9 Jan 91 11:37:38 PST Subject: C. elegans Message-ID: <9101091937.AA03416@desi.ucsd.edu> >I just read that all 329 (?) neurons in C. elegans, the nematode worm, >have been mapped. >Has anyone ever simulated this network? > If it didn't work, >that might indicate that the idealized neuron >(sum-unit-cum-squashing-function) in use now is too simple. It is clear that this version of the neuron is too simple. One of the most common oscillators in neurobiology is the mutually inhibitory pair of neurons. This can't be done with the standard pdp units. You can do it if you have delay or post-inhibitory rebound. Delay can come about through modeling membrane currents more closely. Another aspect of real circuits not addressed by pdp units is electrotonic connections. Fu-Sheng Tsung, Al Selverston, Peter Rowat & I have simulated a 13 unit network in the lobster (the gastric mill of the stomatogastric ganglion), and found pdp units do well at fitting the behavior, but poorly at generalizing to what happens when a cell is removed from the circuit. Peter & Fu-Sheng have developed more realistic models based on differential equations and difference equations that do a better job, although this is still ongoing research. For copies of Peter's paper, write peter at crayfish.ucsd.edu. It will be coming out in Network. Fu-sheng's algorithm is in the connectionist summer school proceedings. gary cottrell From pollack at cis.ohio-state.edu Wed Jan 9 15:23:46 1991 From: pollack at cis.ohio-state.edu (Jordan B Pollack) Date: Wed, 9 Jan 91 15:23:46 -0500 Subject: continuous vs symbolic: a more concrete problem In-Reply-To: Goldfarb's message of Tue, 08 Jan 91 20:27:09 AST Message-ID: <9101092023.AA04952@dendrite.cis.ohio-state.edu> How does one go about constructing a neural network that can recognize a reasonably large (infinite and nontrivial) *class* of formal languages? Lev, Maybe I'm missing something here, but it has been known since Mcculloch and Pitts 1943, and reiterated by Minsky 1967, that the simplest NN's can behave like finite state machines. FSM's can specify regular languages. By trivial construction, almost any NN capable of arbitrary (sum-of-products) boolean functions are able to represent the "infinite, but TRIVIAL *class*" of regular languages, simply by recurrent use of a set of arbitrary boolean functions. The "next state" vector is just a vector boolean function of the "current state" and "input token" vectors. Since McCullogh-Pitts neurons, two layers of linear-threshold units, or two layers of quasi-linear feedforward networks are are all capable of sum-of-product logical forms, they are all capable of simple solutions to regular language recognition. But if the next state is only a first-order function of the current state and input (such as in Elman's SRN), then all even RL's cannot be in general recognized. Consider the regular "odd parity" language, where the next state is the exclusive-or of the current state and input. The more interesting question is how neural-LIKE architectures can deal NATURALLY with *NONTRIVIAL* grammatical systems, such as the context free, indexed CF (thought by many to be the proper category of natural languages), or context sensitive. We can always solve these problems by adding an external stack or a production rule interpreter, but these solutions do not seem very natural. I'd be very interested if your formalism can solve any of these classes in a routine manner. Jordan Pollack Assistant Professor CIS Dept/OSU Laboratory for AI Research 2036 Neil Ave Email: pollack at cis.ohio-state.edu Columbus, OH 43210 Fax/Phone: (614) 292-4890 From ernst at russel.cns.caltech.edu Wed Jan 9 17:01:16 1991 From: ernst at russel.cns.caltech.edu (Ernst Miebur) Date: Wed, 9 Jan 91 14:01:16 PST Subject: C. elegans In-Reply-To: Josh Smith's message of Tue, 8 Jan 91 11:04:31 EST <9101081604.AA23879@bull> Message-ID: <9101092201.AA00426@russel.caltech.edu> Concerning the question whether someone did simulate the nervous system of C. elegans: I did a simulation of the somatic motor system, which controls the movement of all the major muscles of the worm and which comprises about 20% of all neurons (Niebur & Erdos, Computer simulations in networks of electrotonic neurons. In: R. Cotterill (ed.), Computer Simulations in Brain Science. Cambridge Univ. Press,148-163, 1988. This is a preliminary paper which describes some of the methods used. I have some newer papers in preparation that I will send you on request.) To the best of my knowledge, this is the only detailed simulation of the nervous system of a nematode (and I am pretty sure that it is the largest fraction of ANY nervous system ever simulated in detail). I agree with you that nematodes are a fascinating model system. In particular, if one takes into account that there are many species of very different sizes but with similar structure of their nervous sytems. This makes possible complementary experiments, like determining the ultrastructure at the synaptic level in small species (like C. elegans) and doing electrophysiology in large species (like Ascaris lumbricoides). In fact, the news is even better: Not only all the C. elegans neurons have been mapped, but also ALL the other cells! And: This is the case in ALL stages of the development, from the fertilized zygote to the adult. In this sense, C.elegans is certainly the best known of all animals. If you want any further references to this work, I will be happy to provide them. You are wondering whether the connectionist approach can get any justification from a simulation of C. elegans. The answer is very clear: NO WAY! The connectionist model ("sum-unit-cum-squashing-function") is a lousy model for this system. I wouldn't even call it a model, any simulation based on this model would be too far away from anything reasonable in this system. Draw from this whatever conclusion you want for other systems, but I think that in this system, no serious worker would dispute Jim Bowers view that a detailed knowledge of Biology is important. Ernst Niebur ernst at descartes.cns.caltech.edu PS The number of neurons (in the wild-type hermaphodite) is not 329 but 302. From GOLDFARB%unb.ca at UNBMVS1.csd.unb.ca Wed Jan 9 18:53:57 1991 From: GOLDFARB%unb.ca at UNBMVS1.csd.unb.ca (GOLDFARB%unb.ca@UNBMVS1.csd.unb.ca) Date: Wed, 09 Jan 91 19:53:57 AST Subject: continuous vs symbolic: a more concrete problem Message-ID: Let me state the problem again: Given a nontrivial infinite family of languages, how does one go about constructing a reasonably efficient NN that can learn to recognize any language from this family? -- Lev Goldfarb From inesc!lba at relay.EU.net Thu Jan 10 15:58:50 1991 From: inesc!lba at relay.EU.net (Luis Borges de Almeida) Date: Thu, 10 Jan 91 15:58:50 EST Subject: SSAB questions In-Reply-To: INS_ATGE%JHUVMS.BITNET@VMA.CC.CMU.EDU's message of Wed, 9 Jan 91 13:40 EST <0663505911@PTIFM.IFM.RCCN.PT> Message-ID: <9101101558.AA06580@alf.inesc.pt> Richard Rohwer has presented at last NIPS conference, a comparison among a number of acceleration techniques, in various problems. Among these techniques, is one which is quite similar to SuperSAB. This technique was developed by a colleague and myself, independently from Tollenaere's work (see references below; reprints can be sent to anyone interested). I don't recall seeing tests on Quickprop, but Richard had tests on Le Cun's diagonal second-order method, which I believe to be similar, and perhaps a bit faster, than Quickprop. Richard's data showed better results for Le Cun's method than for ours in many problems, but we found out, while talking to Richard, that he had missed a (probably important) step of the algorithm. I think he may have gone to the work of redoing the tests, you might want to contact him directly. In short, the main difference between our algorithm and Tollenaere's, is that we only undo a weight update if it has caused an increase in the objective function (the quadratic error accumulated over all outputs and all trainig patterns). Luis B. Almeida INESC Phone: +351-1-544607 Apartado 10105 Fax: +351-1-525843 P-1017 Lisboa Codex Portugal lba at inesc.inesc.pt lba at inesc.uucp (if you have access to uucp) --------------------- REFERENCES F. M. Silva and L. B. Almeida, "Acceleration Techniques for the Backpropagation Algorithm", in L. B. Almeida and C. J. Wellekens (eds), Neural Networks, Proc. 1990 EURASIP Workshop, Sesimbra, Portugal, Feb. 1990, New York: Springer-Verlag (Lecture Notes in Computer Science series). F. M. Silva and L. B. Almeida, "Speeding up Backpropagation", in R. Eckmiller (ed), Advanced Neural Computers, Amsterdam: Elsevier Science Publishers, 1990. From GOLDFARB%unb.ca at UNBMVS1.csd.unb.ca Thu Jan 10 14:31:26 1991 From: GOLDFARB%unb.ca at UNBMVS1.csd.unb.ca (GOLDFARB%unb.ca@UNBMVS1.csd.unb.ca) Date: Thu, 10 Jan 91 15:31:26 AST Subject: continuous vs symbolic: a more concrete problem Message-ID: One (but not the only) important point, which is independent of the continuous/discrete issue, my example of the infinite learning environment attempts to clarify is the following: No finite number of *static* NNs can operate successfully in a reasonably complex infinite environment. All interesting real environments are essentially infinite. -- Lev Goldfarb From LAMBERTB at UIUCVMD Thu Jan 10 15:00:22 1991 From: LAMBERTB at UIUCVMD (LAMBERTB@UIUCVMD) Date: 10 January 1991 14:00:22 CST Subject: Request Message-ID: To whom it may concern, Please add my name to your list. Thanks in advance. - B. Lambert From GOLDFARB%unb.ca at UNBMVS1.csd.unb.ca Thu Jan 10 15:15:05 1991 From: GOLDFARB%unb.ca at UNBMVS1.csd.unb.ca (GOLDFARB%unb.ca@UNBMVS1.csd.unb.ca) Date: Thu, 10 Jan 91 16:15:05 AST Subject: Re(corrected): continuous vs symbolic: a more concrete problem Message-ID: One (but not the only) important point, which is independent of the continuous/discrete issue, my example of the infinite learning environment attempts to clarify is the following: No finite number of *static* NNs can operate successfully in a reasonably complex infinite environment (with infinite number of classes). All interesting real environments are essentially infinite. -- Lev Goldfarb From peterc at chaos.cs.brandeis.edu Thu Jan 10 22:06:26 1991 From: peterc at chaos.cs.brandeis.edu (Peter Cariani) Date: Thu, 10 Jan 91 22:06:26 est Subject: Vision (What's wrong with Marr's model) Message-ID: <9101110306.AA28003@chaos.cs.brandeis.edu> Stephen Lehar was asking: "Tell me about the AI vision researchers before Marr that supported vision models inspired by natural architectures (PDP/connectionist) with intensive feedback mechanisms (as seen in nature) motivated by neurophysiological (single cell recordings) and psychophysical (perceptual illusions) data, making testable hypotheses (reproducing the illusions) about natural vision. I haven't heard about them." I think the 1947 paper of Walter Pitts and Warren McCulloch "How we know universals: the preception of auditory and visual forms." in the Bulletin of Mathematical Biophysics (9:127-147) easily fits all of the criteria a full two decades before Marr. I would wager there are many more of these earlier models which have been forgotten by the current connectionist discussions. Looking back on the cybernetics literature of the 1940's and 50's (particularly the Macie conferences), I always have the feeling that they seriously considered many more different types of neural mechanisms (analog, temporal codes as well as digital ones) than we do. Just because our research communities have short memory spans doesn't mean that alot of deep thinking (and modeling) didn't happen before the late 1960's. Peter Cariani From uhr at cs.wisc.edu Fri Jan 11 16:11:35 1991 From: uhr at cs.wisc.edu (Leonard Uhr) Date: Fri, 11 Jan 91 15:11:35 -0600 Subject: On blurring the gap between NN and AI Message-ID: <9101112111.AA19891@thor.cs.wisc.edu> If you take - as I and many people do - the "formal model for AI" and the "formal model for NN" to be Post productions-Turing machines (discretely approximating continua, as is almost always the case in both NN and AI), then they clearly are the same and anything that can be accomplished in one can be done in the other. So the differences boil down to differences in styles and tendencies - e.g., serial, lists, lisp vs. parallel, simple processes, learning. Unfortunately traditional AI has largely ignored learning, but from Samuel, Kochen, Hunt, etc. on, through the more recent Similarity-based, Explanation- based, etc. approaches to learning there has always been a good bit. I personally find the differences more (roughly) analogous to the differences between people who swear by Lisp vs. C vs. Smalltalk. If I'm missing something, please explain how you define NN and AI in such a way as to make them differ. (This was written in bemusement after a number of notes, especially from Lev Goldfarb). Len Uhr From GOLDFARB%unb.ca at UNBMVS1.csd.unb.ca Fri Jan 11 19:32:05 1991 From: GOLDFARB%unb.ca at UNBMVS1.csd.unb.ca (GOLDFARB%unb.ca@UNBMVS1.csd.unb.ca) Date: Fri, 11 Jan 91 20:32:05 AST Subject: On blurring the gap between NN and AI In-Reply-To: Message of Fri, 11 Jan 91 17:11:57 AST from Message-ID: I would have liked to be amused by the separation of several areas, such as AI, Pattern Recognition, NN, if the cost to the taxpayers and especially to our science would not have been so high. Having said that, one still has to find a new mathematical model that addresses the situation described by Scott Fahlman in his posting of Dec. 27 as follows: ". . . connectionism and traditional AI are attacking the same huge problem, but beginning at opposite ends of the problems and using very different tools." In other words, a fundamentally new mathematical model is needed that, on the one hand, would remove any apparent and not so apparent analytical tentions between the two existing mathematical "tools" (production systems/vector space) and, on the other hand, would clearly demonstrate that the "opposite ends of the problem" are intrinsically connected. The reason why I allow myself to be very "philosophical" about the situation is that, as was mentioned earlier on the number of occasions, we believe that a new quite satisfactory model has been found. Although I have already outlined the model and mentioned the first original reference (see, for example, the posting of Sept.27), since I was repeatedly asked about the model, I will shortly outline it again to the extent to which there is an interest in it. -- Lev Goldfarb From uhr at cs.wisc.edu Fri Jan 11 15:46:42 1991 From: uhr at cs.wisc.edu (Leonard Uhr) Date: Fri, 11 Jan 91 14:46:42 -0600 Subject: ontogenesis and synaptogenesis (constructing, generating) Message-ID: <9101112046.AA19867@thor.cs.wisc.edu> An alternative to adding physical nodes (and/or links) whenever constructive algorithms (or "generations") need them, is to have the system generate the physical substrate with whatever vigor it can muster (hopefully, so there will always be as-yet-unused resources available), and also when needed free up (degenerate) resources to make new space. Then the structures of processes can, as needed, be embedded where appropriate. This raises several interesting unsolved problems. A rather general topology is needed, one into which the particular structures that are actually generated will fit with reasonable efficiency. I'm describing this in a way that brings out the similarities to the problem of finding good topologies for massively parallel multi-computers, so that a variety of different structures of processes can be embedded and executed efficiently. The major architectures used today are 2-D arrays, trees, and N-cubes; each accepts some embeddings reasonably well, but not others. One pervasive problem is that their small number of links (typically 2 to 12) can easily lead to bottlenecks - which NN with e.g. 100 or so links might almost always overcome. And there are many many other possible graphs, including interesting hybrids. There are several other issues, but I hope this is enough detail to make the following point: If the physical substrate forms a graph whose topology is reasonably close-to-isomorphic to a variety of structures that combine many smaller graphs, the problem can be viewed as one of finding graphs for the physical into which rich enough sets of graphs of processes can be embedded to step serially through the (usually small) increments that good learning algorithms will generate. To the extent that generation builds relatively small, local graphs this probably becomes easier. I don't mean to solve a problem by using the result of yet another unsolved problem. Just as is done with today's multi-computers, we can use mediocre topologies with poor embeddings and slower-than-optimal processing (and even call these super- and ultra-computers, since they may well be the fastest and most powerful around today), and at the same time try to improve upon them. There's another type of learning that almost certainly needs just this kind of thing. Consider how much we remember when we're told the plot of a novel or some gossip, or shown some pictures and then asked which ones we were shown. The brain is able to make large amounts of already-present physical substrate available, whether for temporary or permanent remembered new information, including processes. As Scott points out, the hardware processors do NOT have to be generated exactly when and because the functions they execute are. Len Uhr From ernst at russel.cns.caltech.edu Sat Jan 12 16:19:28 1991 From: ernst at russel.cns.caltech.edu (Ernst Niebur) Date: Sat, 12 Jan 91 13:19:28 PST Subject: Nematodes: references Message-ID: <9101122119.AA01484@russel.caltech.edu> I have received many requests for references of work on the nematode nervous system. Someone asked me to send "all references on nematodes" - I am sorry, but that would go a little too far. The number of papers only on C. elegans (ONE of an estimated 500,000 nematode species) is in the hundreds per year. I will focus on a few articles which are of special interest for neural modelers. The paper in which the complete nervous system of C. elegans is described (actually, not quite complete: the pharynx part is in Albertson & Thomson, Phil. Trans. Roy. Soc. London B275, 299, 1976) is White, J.G., Southgate, E., Thomson, J.N., Brenner, S. The structure of the nervous system of the nematode C. elegans. Phil. Trans. Roy. Soc. London B 314, 1, 1986. The function of an interesting subcircuit of C. elegans is studied by Laser ablation experiments (i.e., by killing identified neurons in a worm and comparing the behavior of this animal with that of untreated worms) in Chalfie, M., Sulston, J.E., White, J.G, Thomson, J.N and Brenner, S. The neural circuit for touch sensitivity in C. elegans. J. Neurosci. 5(4), 956, 1984. A very well written review paper on the electrophysiological (and some other) work in the large nematode Ascaris lumbricoides is Stretton, A.O.W., Davis, R.E., Angstadt, J.D, Donmoyer, J.E. and Johnson, C.D. Neural control of behavior in Ascaris. Trends in Neuroscience, 294, June 1985. Some more recent results on this system are described in the following papers, which also contain useful references to other work of the Stretton group: Angstadt, J.D, and Stretton, A.O.W. Slow active potentials in ventral inhibitory motor neurons of the nematode Ascaris. J. Comp. Physiol. A 166, 165, 1989. Davis, R.E. and Stretton, A.O.W. Passive membrane properties of motorneurons and their role in long-distance signaling in the nematode Ascaris. J. Neurosci. 9(2), 403, 1989. Davis, R.E. and Stretton, A.O.W. Signaling properties of Ascaris motorneurons. Graded active responses, graded synaptic transmission and tonic transmitter release. J. Neurosci. 9(2), 415, 1989. I will be happy to provide references to more specific topics, but I think these are the ones that might be of potential interest for a larger number of people on this mailing list. Ernst Niebur ernst at descartes.cns.caltech.edu From GOLDFARB%UNB.CA at UNBMVS1.csd.unb.ca Sun Jan 13 21:31:23 1991 From: GOLDFARB%UNB.CA at UNBMVS1.csd.unb.ca (Lev Goldfarb) Date: Sun, 13 Jan 91 22:31:23 AST Subject: Reconfigurable Learning Machines (RLM): motivations Message-ID: The model that I will outline in the next posting was motivated, on the one hand, by the informal considerations similar to those that motivated connectionism, and on the other hand, by the formal considerations that are fundamentally different. The formal considerations can be stated informally as follows: each object (event) in the environment, depending on the specific recognition goal, can belong to to an unbounded number of possible categories (classes). Since these categories are not and can not be learned simultaneously and especially because their recognition requires a DYNAMIC (EVOLVING) concept of object similarity, it is very useful to think about the learning process as the process of computation of the SIMILARITY FIELD, or metric field, induced in the environment by the corresponding learning task. From shawn at helmholtz.sdsc.edu Mon Jan 14 17:53:24 1991 From: shawn at helmholtz.sdsc.edu (Shawn Lockery) Date: Mon, 14 Jan 91 14:53:24 PST Subject: No subject Message-ID: <9101142253.AA29240@helmholtz.sdsc.edu> I am a neurobiologist interested in training neural networks to perform chemotaxis, and other feats of simple animal navigation. I'd be very interested to know what has been done by connectionists in this area. The only things I have found so far are: Mozer and Bachrach (1990) Discovering the Structure of a Reacative nvironment by Exploration, and Nolfi et al. (1990) Learning and Evolution in Neural Networks Many thanks, Shawn Lockery CNL Salk Institute Box 85800 San Diego, CA 92186-5800 (619) 453-4100 x527 shawn at helmholtz.sdsc.edu From shawn at helmholtz.sdsc.edu Mon Jan 14 17:59:54 1991 From: shawn at helmholtz.sdsc.edu (Shawn Lockery) Date: Mon, 14 Jan 91 14:59:54 PST Subject: No subject Message-ID: <9101142259.AA29243@helmholtz.sdsc.edu> Several months ago I asked about canned backprop simulators. At long last, here is the result of my query: ------------------------------------------------------------------------------- Barak Pearlmutter has written a dynamical backprop simulator. A version of his program that solves a toy problem and that is readily modifiable is available by anonymous ftp from helmholtz.sdsc.edu. The directory is pub/ and the filename is pearlmutter.tar ------------------------------------------------------------------------------- Yoshiro Miyata (miyata at dendrite.colorado.edu) has written an excellent public domain connectionist simulator with a nice X windows or Sun View interface. It is called SunNet. He provides a pretty easy to learn "general" definition language so a user can experiment with quite varied back-prop and non-conventional architectures. Examples are provided of backpropagation, boltzmann learning, and others. Source code is available by anonymous ftp from boulder. Look for SunNet5.5.tar.Z at boulder.colorado.edu. ------------------------------------------------------------------------------- Yan Le Cun (Department of Computer Science, University of Toronto, Toronto, Ontario, M5S 1A4, Canado) has written a commercial simulator called SN /2 that is powerful and well documented. ------------------------------------------------------------------------------ The Rochester Connectionist Simulator (RCS) is obtainable by anonymous ftp from cs.rochester.edu. You will find the code in the directory pub/simulator. -------------------------------------------------------------------------------- The speech group at Oregon Graduate Institute has written a conjugate-gradient optimization program called OPT to train fully connected feed-forward networks. It is available by anonymous ftp from cse.ogi.edu. The code is in the directory pub/speech. Copy the file opt.tar. You will need to use the unix "tar" command to process the file once you have it on your computer. --------------------------------------------------------------------------------- For the Macintosh, there is the commercial program called MacBrain (Neuronics, Inc., ! Kendall Square #2200, Cambridge, MA 02139). It has the usual Macintosh bells and whitsles and costs $400. --------------------------------------------------------------------------------- For the Macintosh, there is a public domain program called Mactivation. Mactivation version 3.3 is available via anonymous ftp on alumni.Colorado.EDU (internet address 128.138.240.32) The file is in /pub and is called mactivation.3.3.sit.hqx Mactivation is an introductory neural network simulator which runs on all Apple Macintosh computers. A graphical interface provides direct access to units, connections, and patterns. Basic concepts of network operations can be explored, with many low level parameters available for modification. Back-propagation is not supported (coming in 4.0) A user's manual containing an introduction to connectionist networks and program documentation is included. The ftp version includes a plain text file and an MS Word version with nice graphics and footnotes. The program may be freely copied, including for classroom distribution. for version 4.0. You can also get a copy by mail. Send $5 to Mike Kranzdorf, Box 1379, Nederland, C0 80466-1379. --------------------------------------------------------------------------------- For 386 based PC's, you may purchase ExploreNet from HNC, 5501 Oberlin Drive, San Diego, CA 92121. You don't get source code for your $750, but it's powerful and flexible. --------------------------------------------------------------------------------- For IBM PC's, there is a disk that comes along with the third volume of the PDP books (Parallel Distributed Processing, Rumelhart, McClelland and the PDP Research Group, MIT Press, 1986 . You get lots of source code, and the third volume itself is a nice manual. --------------------------------------------------------------------------------- From apache!weil at uunet.UU.NET Mon Jan 14 17:09:52 1991 From: apache!weil at uunet.UU.NET (wei li) Date: Mon, 14 Jan 91 17:09:52 EST Subject: rejection, adaptive learning. Message-ID: <9101142209.AA13377@cmdsun> Hi, we are doing text classification using a feedforword neural network. Through our experiments, we found two problems: 1) in our class definition, we have texts which are not belong to any classes. We threshold the output, if the output is below the threshold, the input is considered as rejected. It did not seem to work well for the patterns which sould be rejected. 2) When some patterns can not be correctly recognized, we have to retrain the system including these new patterns. We wonder if there is way to gradually adapt the system without having to retrain the old correctly learned patterns too. We have tried RCE network for adaptive learning too, but it seems that if we do not retraining the old patterns, some previously correctly learned patterns will become wrong. Any comments on approaches that could reject patterns and adapt to new patterns? Wei Li uunet!apache!weil or weil%apache at uunet.uu.net From honavar at iastate.edu Mon Jan 14 14:01:47 1991 From: honavar at iastate.edu (honavar@iastate.edu) Date: Mon, 14 Jan 91 13:01:47 CST Subject: tech report available by ftp Message-ID: <9101141901.AA03738@iastate.edu> The following technical report is available in postscript form by anonymous ftp (courtesy Jordan Pollack of Ohio State Univ). ---------------------------------------------------------------------- Generative Learning Structures and Processes for Generalized Connectionist Networks Vasant Honavar Leonard Uhr Department of Computer Science Computer Sciences Department Iowa State University University of Wisconsin-Madison Technical Report #91-02, January 1991 Department of Computer Science Iowa State University, Ames, IA 50011 Abstract Massively parallel networks of relatively simple computing elements offer an attractive and versatile framework for exploring a variety of learning structures and processes for intelligent systems. This paper briefly summarizes the popular learning structures and processes used in such networks. It outlines a range of potentially more powerful alternatives for pattern-directed inductive learning in such systems. It motivates and develops a class of new learning algorithms for massively parallel networks of simple computing elements. We call this class of learning processes \fIgenerative\fR for they offer a set of mechanisms for constructive and adaptive determination of the network architecture - the number of processing elements and the connectivity among them - as a function of experience. Such generative learning algorithms attempt to overcome some of the limitations of some approaches to learning in networks that rely on modification of \fIweights\fR on the links within an otherwise fixed network topology e.g., rather slow learning and the need for an a-priori choice of a network architecture. Several alternative designs, extensions and refinements of generative learning algorithms, as well as a range of control structures and processes which can be used to regulate the form and content of internal representations learned by such networks are examined. ______________________________________________________________________________ You will need a POSTSCRIPT printer to print the file. To obtain a copy of the report, use anonymous ftp from cheops.cis.ohio-state.edu (here is what the transaction looks like): % ftp ftp> open cheops.cis.ohio-state.edu Connected to cheops.cis.ohio-state.edu. 220 cheops.cis.ohio-state.edu FTP server (Version blah blah) ready. Name (cheops.cis.ohio-state.edu:yourname): anonymous 331 Guest login ok, send ident as password. Password: anything 230 Guest login ok, access restrictions apply. ftp> cd pub/neuroprose 250 CWD command successful. ftp> bin 200 Type set to I. ftp> get honavar.generate.ps.Z 200 PORT command successful. 150 Opening BINARY mode data connection for honavar.generate.ps.Z (55121 bytes). 226 Transfer complete. local: honavar.generate.ps.Z remote: honavar.generate.ps.Z 55121 bytes received in 1.8 seconds (30 Kbytes/s) ftp> quit 221 Goodbye. % uncompress honavar.generate.ps.Z % lpr honavar.generate.ps From Connectionists-Request at CS.CMU.EDU Tue Jan 15 12:48:37 1991 From: Connectionists-Request at CS.CMU.EDU (Connectionists-Request@CS.CMU.EDU) Date: Tue, 15 Jan 91 12:48:37 EST Subject: Bi-monthly Reminder Message-ID: <21200.663961717@B.GP.CS.CMU.EDU> CONNECTIONISTS is getting rather large (~1000 subscribers) and we have wasted too much bandwidth in the last six months arguing over what is and is not appropriate for posting to CONNECTIONISTS. To combat this problem we are going to start to posting list guidelines on a bi-monthly basis. The text of the bi-monthly posting follows. If you have comments on this post please direct them to me at Connectionists-Request at cs.cmu.edu. Do NOT reply to the entire list. If there is sufficient interest, I will summarize the comments for the rest of the list. Thanks Scott Crowder Connectionists-Request at cs.cmu.edu (ARPAnet) -------------------------------------------- This is an automatically posted bi-monthly reminder about how the CONNECTIONISTS list works and how to access various online resources. CONNECTIONISTS is not an edited forum like the Neuron Digest, or a free-for-all newsgroup like comp.ai.neural-nets. It's somewhere in between, relying on the self-restraint of its subscribers. Membership in CONNECTIONISTS is restricted to persons actively involved in neural net research. The following posting guidelines are designed to reduce the amount of irrelevant messages sent to the list. Before you post, please remember that this list is distributed to over a thousand busy people who don't want their time wasted on trivia. Also, many subscribers pay cash for each kbyte; they shouldn't be forced to pay for junk mail. Happy hacking. -- Dave Touretzky & Scott Crowder --------------------------------------------------------------------- What to post to CONNECTIONISTS ------------------------------ - The list is primarily intended to support the discussion of technical issues relating to neural computation. - We encourage people to post the abstracts of their latest papers and tech reports. - Conferences and workshops may be announced on this list AT MOST twice: once to send out a call for papers, and once to remind non-authors about the registration deadline. A flood of repetitive announcements about the same conference is not welcome here. - Requests for ADDITIONAL references. This has been a particularly sensitive subject lately. Please try to (a) demonstrate that you have already pursued the quick, obvious routes to finding the information you desire, and (b) give people something back in return for bothering them. The easiest way to do both these things is to FIRST do the library work to find the basic references, then POST these as part of your query. Here's an example: WRONG WAY: "Can someone please mail me all references to cascade correlation?" RIGHT WAY: "I'm looking for references to work on cascade correlation. I've already read Fahlman's paper in NIPS 2, his NIPS 3 abstract, and found the code in the nn-bench archive. Is anyone aware of additional work with this algorithm? I'll summarize and post results to the list." - Announcements of job openings related to neural computation. - Short reviews of new text books related to neural computation. To send mail to everyone on the list, address it to Connectionists at CS.CMU.EDU ------------------------------------------------------------------- What NOT to post to CONNECTIONISTS: ----------------------------------- - Requests for addition to the list, change of address and other administrative matters should be sent to: "Connectionists-Request at cs.cmu.edu" (note the exact spelling: many "connectionists", one "request"). If you mention our mailing list to someone who may apply to be added to it, please make sure they use the above and NOT "Connectionists at cs.cmu.edu". - Requests for e-mail addresses of people who are believed to subscribe to CONNECTIONISTS should be sent to postmaster at appropriate-site. If the site address is unknown, send your request to Connectionists-Request at cs.cmu.edu and we'll do our best to help. A phone call to the appropriate institution may sometimes be simpler and faster. - Note that in many mail programs a reply to a message is automatically "CC"-ed to all the addresses on the "To" and "CC" lines of the original message. If the mailer you use has this property, please make sure your personal response (request for a Tech Report etc.) is NOT broadcast over the net. - Do NOT tell a friend about Connectionists at cs.cmu.edu. Tell him or her only about Connectionists-Request at cs.cmu.edu. This will save your friend from public embarrassment if she/he tries to subscribe. ------------------------------------------------------------------------------- The CONNECTIONISTS Archive: --------------------------- All e-mail messages sent to "Connectionists at cs.cmu.edu" starting 27-Feb-88 are now available for public perusal. A separate file exists for each month. The files' names are: arch.yymm where yymm stand for the obvious thing. Thus the earliest available data are in the file: arch.8802 Files ending with .Z are compressed using the standard unix compress program. To browse through these files (as well as through other files, see below) you must FTP them to your local machine. ------------------------------------------------------------------------------- How to FTP Files from the CONNECTIONISTS Archive ------------------------------------------------ 1. Open an FTP connection to host B.GP.CS.CMU.EDU (Internet address 128.2.242.8). 2. Login as user anonymous with password your username. 3. 'cd' directly to one of the following directories: /usr/connect/connectionists/archives /usr/connect/connectionists/bibliographies 4. The archives and bibliographies directories are the ONLY ones you can access. You can't even find out whether any other directories exist. If you are using the 'cd' command you must cd DIRECTLY into one of these two directories. Access will be denied to any others, including their parent directory. 5. The archives subdirectory contains back issues of the mailing list. Some bibliographies are in the bibliographies subdirectory. Problems? - contact us at "Connectionists-Request at cs.cmu.edu". ------------------------------------------------------------------------------- How to FTP Files from the Neuroprose Archive -------------------------------------------- Host cheops.cis.ohio-state.edu (128.146.8.62) directory pub/neuroprose This directory contains technical reports as a public service to the connectionist and neural networks scientific community. Researchers may place electronic versions of their preprints or articles in this directory, announce availability, and other interested researchers can rapidly retrieve and print the postscripts. To place a file, put it in the Inbox subdirectory, and send mail to pollack at cis.ohio-state.edu. Your mail should include: 1) filename 2) way to contact author 3) single sentence abstract Current naming convention is author.title.filetype[.Z] where title is enough to discriminate among the files of the same author. The filetype is usually "ps" for postscript, our desired universal printing format, but may be tex, which requires more local software than a spooler. Very large files (e.g. over 200k) should be squashed (with either a sigmoid function:) or the standard unix "compress" utility, which results in the .Z affix. To place or retrieve .Z files, make sure to issue the FTP command "BINARY" before transferring files. After retrieval, call the standard unix "uncompress" utility, which removes the .Z affix. A shell script called Getps is available which will automatically perform the necessary operations. The file can be retrieved from the CONNECTIONISTS archive (see above). For further questions contact: Jordan Pollack Email: pollack at cis.ohio-state.edu ------------------------------------------------------------------------ How to FTP Files from the NN-Bench Collection --------------------------------------------- 1. Create an FTP connection from wherever you are to machine "pt.cs.cmu.edu" (128.2.254.155). 2. Log in as user "anonymous" with password your username. 3. Change remote directory to "/afs/cs/project/connect/bench". Any subdirectories of this one should also be accessible. Parent directories should not be. 4. At this point FTP should be able to get a listing of files in this directory and fetch the ones you want. Problems? - contact us at "nn-bench-request at cs.cmu.edu". From GOLDFARB%UNB.CA at UNBMVS1.csd.unb.ca Tue Jan 15 13:34:35 1991 From: GOLDFARB%UNB.CA at UNBMVS1.csd.unb.ca (Lev Goldfarb) Date: Tue, 15 Jan 91 14:34:35 AST Subject: Reconfigurable Learning Machines (RLM) Message-ID: Here is an informal outline of the model proposed in On the Foundations of Intelligent Processes I: An Evolving Model for Pattern Learning, Pattern Recognition, Vol.23, No.6, 1990. Think of an object representation as of a "molecule" (vectors and strings are special types of such molecules). Let O denotes the set of all objects from the environment, and let S = {Si} denotes the set of BASIC SUBSTITUTION OPERATIONS, where each operation can transform one object into another by removing a piece of the molecule and replacing it by another small molecule. For string molecules these could be the operations of letter deletions/insertions. In addition, a small FIXED set CR of COMPOSITION RULES for forming new operations from the existing operations is also given. Think of these rules as specifications for gluing several operations together into one operation. The intrinsic distance D between two objects is defined as the minimum number of operations from S that are necessary to transform one molecule into the other. D depends on the set S of operations. A larger set S can only reduce some of the distances. This idea of distance embodies a very important, perhaps the most important, physical principle -- the least- action principle, which was characterized by Max Planck as follows: Amid the more or less general laws which mark the achievements of physical science during the course of the last centuries, the principle of least action is perhaps that which, as regards form and content, may claim to come nearest to that ideal final aim of theoretical research. In a vector setting, D is a city-block distance between two vectors. In a non-vector setting, however, even small sets of patterns (4-5) with such distances cannot be represented in a Euclidean vector space of ANY dimension. The adjective "intrinsic" in the above definition refers to the fact that the distance D does not reflect any empirical knowledge about the role of the substitution operations. Thus, we are led to the most natural extension of this concept obtained by allowing different substitution operations to have different weights associated with them: assign to each operation Si nonnegative weight w^i subject to one restriction that their sum is 1. The latter constraint is necessary to ensure that during learning the operations are forced to cooperatively compete for the weights. The new weighted distance WD is defined similarly to the above distance D, but replacing the minimum number of operations by the shortest weighted path between the two molecules. In the basic learning process, i.e. that of learning to recognize one class, the RLM is presented with two finite sets C^+ (of positive training patterns) and C^- (of negative training patterns). The main objective of the learning process is to produce, if necessary, an expanded set S of operations and at least one corresponding weight vector w*, such that with the help of the distance WD(w*) (which induces in the set O the corresponding similarity field) the RLM can classify new patterns as positive or negative. The basic step in the learning process is optimization of the function F(w)=F1(w)/c+F2(w) where F1 is the smallest WD(w) distance between C^+ and C^-, F2 is the average WD(w) distance in C^+, and c is a small positive constant to prevent the overflow (when the values of F2 approach 0). One can show that the above continuous optimization problem can be reduced to the discrete one. During the learning process the new operations to be added to the set S of current operations are chosen among the compositions of the "optimum" current operations. The addition of such new operations "improves" the value of F, and therefore the learning process is guaranteed to converge. The concept of non-probabilistic class entropy, or complexity, w.r.t. the (current) state of the RLM can also be introduced. During the learning process this entropy DECREASES. Within the proposed model it is not difficult to see the relations between the learning process and the propositional class description. Moreover, most of the extensive psychological observation related, for example, to object perception (Object Perception: Structure and Process, eds. B.E. Shepp and S. Ballesteros, Lawrence Erlbaum Associates, 1989) can naturally be explained. --Lev Goldfarb From rsun at chaos.cs.brandeis.edu Tue Jan 15 17:12:08 1991 From: rsun at chaos.cs.brandeis.edu (Ron Sun) Date: Tue, 15 Jan 91 17:12:08 est Subject: No subject Message-ID: <9101152212.AA03681@chaos.cs.brandeis.edu> -------------------Technical Report available ------------- Integrating Rules and Connectionism for Robust Reasoning} Technical Report TR-CS-90-154 Ron Sun Brandeis University Computer Science Department rsun at cs.brandeis.edu Abstract A connectionist model for robust reasoning, CONSYDERR, is proposed to account for some common reasoning patterns found in commonsense reasoning and to remedy the brittleness problem. A dual representation scheme is devised, which utilizes both localist representation and distributed representation with features. We explore the synergy resulted from the interaction between these two types of representations, which helps to deal with problems such as partial information, no exact match, property inheritance, rule interaction, etc. Because of this, the CONSYDERR system is capable of accounting for many difficult patterns in commonsense reasoning. This work also shows that connectionist models of reasoning are not just an ``implementation" of their symbolic counterparts, but better computational models of common sense reasoning, taking into consideration of the approximate, evidential and adaptive nature of reasoning, and accounting for the spontaneity and parallelism in reasoning processes. +++ comments and suggestions are especially welcome +++ ------------ FTP procedures --------- ftp cheops.cis.ohio-state.edu >name: anonymous >passwork: neuron >binary >cd pub/neuroprose >get sun.integrate.ps.Z >quit uncompress sun.integrate.ps.Z lpr sun.integrate.ps From choukri at capsogeti.fr Wed Jan 16 09:26:23 1991 From: choukri at capsogeti.fr (Khalid Choukri) Date: Wed, 16 Jan 91 14:26:23 GMT Subject: neural sessions /13th IMACS World Congress Message-ID: <9101161426.AA17825@thor> --------------------------------------------------------- 13th IMACS World Congress on Computation and Applied Mathematics July 22-26,1991, Trinity college, Dublin, Ireland Neural Computing sessions Preliminary announcement and call for papers ----------------------------------------------------- In the scope of the 13th IMACS World Congress on Computation and Applied Mathematics that will be held on July 22-26, 1991 at Trinity college, Dublin, Ireland, several sessions will be devoted to Neural computing and Applied Mathematics. A typical session consists of six 20-minutes papers. Invited papers (tutorials ~ 1-hour) are welcome. Contributions from all fields related to neuro-computing techniques are welcome. Including applications to pattern recognition and classification, optimization problems, etc. Information and a non-exclusive list of topics may be obtained from the session organizer or the Congress Secretariat. Proceedings will be available at the Congress. A more formal Transactions will be available at a later date. Submission procedure : --------------------- Authors are solicited to submit proposals consisting of an abstract (one page, 500 words maximum) which must clearly state the purpose of the work, the specific original results obtained and their significance. The final paper length is two pages (IEEE two-column format). A first page of the proposal should contain the following information in the order shown: - Title. - Authors' names and affiliation. - Contact information (name, postal address, phone, fax and email address) - Domain area and key words: one or more terms describing the problem domain area. AUTHORS ARE ENCOURAGED to submit a preliminary version of the complete paper in addition to the abstract. Calendar: -------- Deadline for submission : February, 15, 1990 Notification of acceptance : March , 15 , 1991 Camera ready paper : April, 5, 1991 Three copies should be sent directly to the technical chairman of these sessions at the following address: Dr. Khalid Choukri Cap GEMINI Innovation 118, Rue de Tocqueville 75017, Paris, France Phone: (+33-1) 40 54 66 28 Fax: (+33-1) 42 67 41 39 e-mail choukri at capsogeti.fr For further information about the IMACS Congress in general, contact Post: IMACS '91 Congress Secretariat 26 Temple Lane Dublin 2 IRELAND Fax: (+353-1) 451739 Phone: (+353-1) 452081 From tenorio at ecn.purdue.edu Wed Jan 16 09:41:37 1991 From: tenorio at ecn.purdue.edu (Manoel F Tenorio) Date: Wed, 16 Jan 91 09:41:37 -0500 Subject: report: optimal NN size for classifiers Message-ID: <9101161441.AA27296@dynamo.ecn.purdue.edu> This report addresses the analysis of a new criterion for optimal classifier design. In particular we study the effects of the sizing ot the hidden layers and the optimal predicted value by this criterion. Resquest should be sent to: jld at ecn.purdue.edu TR-EE 91-5 There is a fee for requests outside USA,Canada and Mexico. On Optimal Adaptive Classifier Design Criterion - How many hidden units are necessary for an optimal neural network classifier? Wei-Tsih Lee Manoel Fernando Tenorio Parallel Distributed Structures Lab. Parallel Distributed Structures Lab. School of Electrical Engineering School of Electrical Engineering Purdue University Purdue University West Lafayette, IN 47907 West Lafayette, IN 47907 lwt at ecn.purdue.edu tenorio at ecn.purdue.edu Abstract A central problem in classifier design is the estimation of classification error. The difficulty in classifier design arises in situations where the sample distribution is unknown and the number of training samples available is limited. In this paper, we present a new approach for solving this problem. In our model, there are two types of classification error: approximation and generalization error. The former is due to the imperfect knowledge of the underlying sample distribution, while the latter is mainly the result of inaccuracies in parameter estimation, which is a consequence of the small number of training samples. We therefore propose a criterion for optimal classifier selection, called the Generalized Minimum Empirical Criterion (GMEE). The GMEE criterion consists of two terms, corresponding to the estimates of two types of error. The first term is the empirical error, which is the classification error observed for the training samples. The second is an estimate of the generalization error, which is related to the classifier complexity. In this paper we consider the Vapnik-Chervonenkis dimension (VCdim) as a measure of classifier complexity. Hence, the classifier which minimizes the criterion is the one with minimal error probability. Bayes consistency of the GMEE criterion has been proven. As an application, the criterion is used to design the optimal neural network classifier. A corollary to the Bayes optimality of neural network-based classifiers has been proven. Thus, our approach provides a theoretic foundation for the connectionist approach to optimal classifier design. Experimental results are given to validate the approach, followed by discussions and suggestions for future research. From bms at dcs.leeds.ac.uk Wed Jan 16 11:06:40 1991 From: bms at dcs.leeds.ac.uk (B M Smith) Date: Wed, 16 Jan 91 16:06:40 GMT Subject: Item for Distribution Message-ID: <21087.9101161606@csunb0.dcs.leeds.ac.uk> PRELIMINARY CALL FOR PARTICIPATION ================================== AISB91 University of Leeds 16-19 April 1991 Interested to know what is happening at the forefront of current AI research? Tired of going to AI conferences where you hear nothing but talk about applications? Bored at big AI conferences where there are so many parallel sessions that you don't know where to go? Saturated with small workshops that focus only on one narrow topic in AI? ==> the 1991 AISB conference may be just the thing for you ! AISB91 is organized by the Society for the Study of Artificial Intelligence and Simulation of Behaviour. It is not only the oldest regular conference in Europe on AI - which spawned the ECAI conferences in 1982 - but it is also the conference that has a tradition of focusing on research as opposed to applications. The 1991 edition of the conference is no different in this respect. The conference has a single session and covers the full spectrum of AI work, from robotics to knowledge systems. It is designed for researchers active in AI who want to follow the complete field. Papers were selected that are representative for ongoing research, particularly for research topics that promise new exciting avenues into a deeper understanding of intelligence. There will be a tutorial programme on Tuesday 16 April, followed by the technical programme from Wednesday 17 to Friday 19 April. The conference will be held at Bodington Hall, University of Leeds, a large student residence and conference centre. Bodington Hall is 4 miles from the centre of Leeds and set in 14 acres of private grounds. Leeds/Bradford airport is 6 miles away, with frequent flights from London Heathrow, Amsterdam and Paris. Leeds itself is easily accessible by rail (2 and a half hours from London) and the motorway network. The Yorkshire Dales National Park is close by, and the historic city of York is only 30 minutes away by rail. TECHNICAL PROGRAMME Wednesday 17 - Friday 19 April 1991 ======================================================== The technical programme sessions are organized around problem areas, not around approaches. This means sessions show how different schools of AI - knowledge-based approaches, logic based approaches, and neural networks - address the fundamental problems of AI. The technical programme lasts 2 and a half days. Each day has a morning session focusing on a particular area of AI. The first day this area is distributed AI, the second day new modes of reasoning, and the third day theorem proving and machine learning. The afternoon is devoted to research topics which are at the forefront of current research. On the first afternoon this topic is emergent functionality and autonomous agents. It presents the new stream of ideas for building autonomous agents featuring concepts like situatedness, physical symbol grounding, reactive systems, and emergence. On the second day the topic is knowledge level expert systems research. It reflects the paradigm shift currently experienced in knowledge based systems away from the symbol level and towards the knowledge level, both for design and knowledge acquisition. Each session has first a series of accepted papers, then two papers which treat the main theme from a principled point of view, and finally a panel. In addition the conference features three exciting invited speakers: Andy Clark who talks about the philosophical foundations of AI, Rolf Pfeifer who reflects on AI and emotion, and Tony Cohn who looks at the formal modeling of common sense. The conference is closed by the Programme Chairman, Luc Steels, who speculates on the role of consciousness in Artificial Intelligence. Here is a more detailed description of the various sessions and the papers contained in them: Distributed Intelligent Agents ============================== Research in distributed AI is concerned with the problem of how multiple agents and societies of agents can be organized to co-operate and collectively solve a problem. The first paper by Chakravarty (MIT) focuses on the problem of evolving agents in the context of Minsky's society of mind theory. It addresses the question how new agents can be formed by transforming existing ones and illustrates the theory with an example from game playing. Smieja (GMD, Germany) focuses on the problem of organizing networks of agents which consist internally of neural networks. Smieja builds upon the seminal work of Selfridge in the late fifties on the Pandemonium system. Bond (University of California) addresses the problem of regulating co-operation between agents. He seeks inspiration in sociological theory and proposes a framework based on negotiation. Finally Mamede and Martins (Technical University of Lisbon) address the problem of resource-bounded reasoning within the context of logical inference. Situatedness and emergence in autonomous agents =============================================== Research on robots and autonomous agents used to be focused strongly on low level mechanisms. As such there were few connections with the core problems of AI. Recently, there has been a shift of emphasis towards the construction of complete agents. This has lead to a review of some traditional concepts, such as the hierarchical decomposition of an agent into a perception module, a decision module and an action module and it has returned robotics research to the front of the AI stage. This session testifies to the renewed interest in the area. It starts with a paper by Bersini (Free University of Brussels) which is strongly within the new perspective of emphasizing situatedness and non-symbolic relations between perception and action. It discusses the trade-offs between reactive systems and goal-oriented systems. Seel (STC Technology, Harlow, UK) provides some of the formal foundations for understanding and building reactive systems. Jackson and Sharkey (University of Exeter) address the problem of symbol grounding: how signals can be related to concepts. They use a connectionist mechanism to relate spatial descriptions with results from perception. Cliff (University of Sussex) discusses an experiment in computational neuroethology. The next paper is from the Edinburgh Really Useful Robot project which has built up a strong tradition in building autonomous mobile robots. The paper will be given by Hallam (University of Edinburgh) and discusses an experiment in real-time control using toy cars. The final paper is by Kaelbling (Teleos Research, Palo Alto, California) who elaborates her proposals for principled programming of autonomous agents based on logical specifications. The panel which ends the session tries to put the current work on autonomous agents into the broader perspective of AI. The panel includes Smithers (University of Edinburgh), Kaelbling, Connah (Philips Research, UK), and Agre (University of Sussex). Following this session, on Wednesday evening, the conference dinner will be held at the National Museum of Photography, film and Television at Bradford. The evening will include a special showing in the IMAX auditorium, which has the largest cinema screen in Britain. New modes of reasoning ====================== Reasoning remains one of the core topics of AI. This session explores some of the current work to find new forms of reasoning. The first paper by Hendler and Dickens (University of Maryland) looks at the integration of neural networks and symbolic AI in the context of a concrete example involving an underwater robot. Euzenat and Maesano (CEDIAG/Bull, Louveciennes, France) address the problem of forgetting. Pfahringer (University of Vienna) builds further on research in constraint propagation in qualitative modelling. He proposes a mechanism to improve efficiency through domain variables. Ghassem-Sani and Steel (University of Essex) extend the arsenal of methods for non-recursive planning by introducing a method derived from mathematical induction. The knowledge level perspective =============================== Knowledge systems (also known as expert systems or knowledge-based systems) continue to be the most successful area of AI application. The conference does not focus on applications but on foundational principles for building knowledge systems. Recently there has been an important shift of emphasis from symbol level considerations (which focus on the formalism in which a system is implemented) to knowledge level considerations. The session highlights this shift in emphasis. The first paper by Pierret-Golbreich and Delouis (Universite Paris Sud) is related to work on the generic task architectures. It proposes a framework including support tools for performing analysis of the task structure of the knowledge system. Reichgelt and Shadbolt (University of Nottingham) apply the knowledge level perspective to the problem of knowledge acquisition. Wetter and Schmidt (IBM Germany) focus on the formalization of the KADS interpretation models which is one of the major frameworks for doing knowledge level design. Finally Lackinger and Haselbock (University of Vienna) focus on domain models in knowledge systems, particularly qualitative models for simulation and control of dynamic systems. Then there are two papers which directly address foundational issues. The first one by Van de Velde (VUB AI Lab, Brussels) clarifies the (difficult) concepts involved in knowledge level discussions of expert systems, particularly the principle of rationality. Schreiber, Akkermans and Wielinga (University of Amsterdam) critically examine the suitability of the knowledge level for expert system design. The panel involves Leitch (Heriot Watt University, Edinburgh), Wielinga, Van de Velde, Sticklen (Michigan State University), and Pfeifer (University of Zurich). Theorem proving and Machine learning =============== ================ The final set of papers focuses on recent work in theorem proving and in machine learning. The first paper by Giunchiglia (IRST Trento, Italy) and Walsh (University of Edinburgh) discusses how abstraction can be used in theorem proving and presents solid evidence to show that it is useful. Steel (University of Essex) proposes a new inference scheme for modal logic. Then there are two papers which represent current work on machine learning. The first one by Churchill and Young (University of Cambridge) reports on an experiment using SOAR concerned with modelling representations of device knowledge. The second paper by Elliott and Scott (University of Essex) compares instance-based and generalization-based learning procedures. TUTORIAL PROGRAMME - Tuesday 16 April 1991 ========================================== Six full-day tutorials will be offered on 16 April (subject to sufficient registrations for each.) Tutorial 1 Knowledge Base Coherence Checking ---------- Professor Jean-Pierre LAURENT University of Savoie FRANCE Like conventional software, AI Systems also need validation tools. Some of these tools must be specific, especially for validating Knowledge-Based Systems, and in particular for checking the coherence of a Knowledge Base (KB). In the introduction to this tutorial we will clarify the distinctions to be made between Validation, Verification, Static Analysis and Testing. We will present methods which try to check exhaustively for the coherence of a knowledge Base. Then we will present a pragmatic approach in which, instead of trying to assert the global coherence of a KB, it is proposed to check heuristically whether it contains incoherences. This approach is illustrated by the SACCO System, dealing with KBs which contain classes and objects, and furthermore rules with variables. Tutorial 2 Advanced Constraint Techniques ---------- Dr. Hans Werner Guesgen and Dr. Joachim Hertzberg German National Centre for Computer Science (GMD) Sankt Augustin, GERMANY This tutorial will present a coherent overview of the more recent concepts and approaches to constraint reasoning. It presents the concept of dynamic constraints as a formalism subsuming classical constraint satisfaction, constraint manipulation and relaxation, bearing a relationship to reflective systems; moreover, the tutorial presents approaches to parallel implementations of constraint satisfaction in general and dynamic constraints in particular. Tutorial 3 Functional Representation and Modeling ---------- Prof. Jon Sticklen and Dr. Dean Allemang* Michigan State University USA * Universitaet Zurich, SWITZERLAND A growing body of AI research centres on using the known functions of a device as indices to causal understanding of how the device "works". The results of functional representation and modeling have typically used this organization of causal understanding to produce tractable solutions to inherently complex modelling problems. In this tutorial, the fundamentals of functional representation and reasoning will be explained. Liberal use of examples throughout will illustrate the representational concepts underlying the functional approach. Contacts with other model based reasoning (MBR) techniques will be made whenever appropriate. Sufficient background will be covered to make this suitable for both those unacquainted with the MBR field, and for more experienced individuals who may be working now in MBR research. A general familiarity with AI is assumed. Participants should send in with their registration materials a one page description of a modeling problem which they face in their domain. Tutorial 4 Intelligent Pattern Recognition and Applications ---------- Prof. Patrick Wang M.I.T. Artificial Intelligence Laboratory and Northeastern University, Boston USA The core of pattern recognition, including "learning techniques" and "inference" plays an important and central role in AI. On the other hand, the methods in AI such as knowledge representation, semantic networks, and heuristic searching algorithms can also be applied to improve the pattern representation and matching techniques in many pattern recognition problems - leading to "smart" pattern recognition. Moreover, the recognition and understanding of sensory data like speech or images, which are major concerns in pattern recognition, have always been considered as important subfields of AI. This tutorial includes overviews of pattern recognition and articifical intelligence; including recent developments at MIT. The focus of the tutorial will be on the overlap and interplay between these fields. Tutorial 5 SILICON SOULS - Philosophical foundations of computing and AI ---------- Prof. Aaron Sloman University of Birmingham This will not be a technical tutorial. Rather the tutor will introduce a collection of philosophical questions about the nature of computation, the aims of AI, connectionist and non-connectionist approaches to AI, the relevance of computation to the study of mind, varieties of mechanism, consciousness, and the nature of emotions and other affective states. Considerable time will be provided for discussion by participants. Prof. Sloman has provided a list of pertinent questions, these will be sent to participants upon registration. Tutorial 6 Knowledge Acquisition -------- Dr. Nigel Shadbolt Nottingham University Practical methods for acquiring knowledge from experts. The methods described have been shown to be effective through the pioneering research at Nottingham which compared common and less common methods for eliciting knowledge from experts. This tutorial is an updated version of the knowledge acquisition tutorial given at AISB'89 which was well-attended and enthusiastically received. ======================================================================== For further information on the tutorials, mail tutorials at hplb.hpl.hp.com or tutorials at hplb.lb.hp.co.uk or tutorials%hplb.uucp at ukc.ac.uk For a conference programme and registration form, or general information about the conference, mail aisb91 at ai.leeds.ac.uk or write to: Barbara Smith AISB91 Local Organizer School of Computer Studies University of Leeds Leeds LS2 9JT U.K. 9 9 From uhr at cs.wisc.edu Wed Jan 16 20:11:04 1991 From: uhr at cs.wisc.edu (Leonard Uhr) Date: Wed, 16 Jan 91 19:11:04 -0600 Subject: Vision (What's wrong with Marr's model) Message-ID: <9101170111.AA07455@thor.cs.wisc.edu> Expanding on Peter Cariani's reply to Stephen Lehar about earlier brain-like vision models, I published a Psych Review paper around 1961-63 that briefly summarized a lot of them, and a book, "Pattern Recognition, Learning and Thought" (Prentice-Hall, about 1973) that concentrated on one approach. The Minsky-Papert book, as people have mentioned, inspired an almost lemming-like flight away from networks, and traditional AI has indeed been largely heuristic search through symbolic domains. But there has always been non-trad- itional work. Probably the type of computer vision research that comes closest to bridging the AI-network gap (which really doesn't exist) is the pyramid approach I and a number of others have been taking (e.g., see books edited by Tanimoto and Klinger, Acad. Press, 1980; by Rosenfeld around 1986; and by Uhr, 1988, Acad. Press). Len Uhr From millan at lsi.upc.es Thu Jan 17 06:55:00 1991 From: millan at lsi.upc.es (Jose del R. MILLAN) Date: 17 Jan 91 12:55 +0100 Subject: Reply to Shawn Lockery Message-ID: <166*millan@lsi.upc.es> Regarding Shawn Lockery's question about connectionist approaches to navigation, we have developed a reinforcement learning system that tackles the robot path-finding problem. In order to study the feasibility of our approach, a first version (Millan & Torras 1990a, 1990b) was required to generate a ``quasi-optimal path'' between two fixed configurations in a given workspace with obstacles. The fact that the robot is a point, allows us to concentrate on the capabilities of reinforcement learning for the problem at hand. The third reference describes a review of existing connectionist approaches to the problem. The abstracts of the first two papers can be found at the end of this message. A forthcoming paper will describe the second version of the system. The new system combines reinforcement with supervised and unsupervised learning techniques to solve a more complex instance of the problem, namely, to generate quasi-optimal paths from any initial configuration to a fixed goal in a given workspace with obstacles. Currently, we are extending the system to deal with dimensional mobile robots. References Millan, J. del R. & Torras, C. (1990a). Reinforcement connectionist learning for robot path finding: a comparative study. Proc. COGNITIVA-90, 123--131. Millan, J. del R. & Torras, C. (1990b). Reinforcement learning: discovering stable solutions in the robot path finding domain. Proc. 9th European Conference on Artificial Intelligence, 219--221. Millan, J. del R. & Torras, C. (To appear). Connectionist approaches to robot path finding. In O.M. Omidvar (ed.) Progress in Neural Networks Series, Vol. 3. Ablex Publishing Corporation. ------------------------------------------------------------------------------- Reinforcement Connectionist Learning for Robot Path Finding: A Comparative Study ABSTRACT. A connectionist approach to robot path finding has been devised so as to avoid some of the bottlenecks of algebraic and heuristic approaches, namely construction of configuration space, discretization and step predetermination. The approach relies on reinforcement learning. In order to identify the learning rule within this paradigm that best suits the problem at hand, an experimental comparison of 13 different such rules has been carried out. The most promising rule has been shown to be that using predicted comparison as reinforcement baseline, and the Hebbian formula as eligibility factor. ------------------------------------------------------------------------------- Reinforcement Learning: Discovering Stable Solutions in the Robot Path Finding Domain ABSTRACT. After outlining the drawbacks of classical approaches to robot path finding, a prototypical system overcoming some of them and demonstrating the feasibility of reinforcement connectionist learning approaches to the problem is presented. The way in which the information is codified and the computational model used allow to avoid both the explicit construction of configuration space required by algebraic approaches as well as the discretization and step homogeneity demands of heuristic search algorithms. In addition, the simulations show that finding feasible paths is not as computational expensive as it is usually assumed for a reinforcement learning system. Finally, a mechanism is incorporated into the system to ``stabilise'' learning once an acceptable path has been found. ------------------------------------------------------------------------------- Jose del R. MILLAN Institute for System Engineering and Informatics Commission of the European Communities. Joint Research Centre Building A36. 21020 ISPRA (VA). ITALY e-mail: j_millan at cen.jrc.it (try this one first, please), or millan at lsi.upc.es From XIN at PHYS4.anu.edu.au Fri Jan 18 12:24:28 1991 From: XIN at PHYS4.anu.edu.au (Xin Yao) Date: Fri, 18 Jan 91 12:24:28 EST Subject: Dear researchers, Message-ID: <9101180100.AA19697@anu.anu.edu.au> I'm going to write a survey report on the applications of evolutionary search procedures (like genetic algorithm) to neural networks (including any hybrids of evolutionary and connectionist learning). Any comments or references in this area are greatly welcomed. I will certainly post the bibliographies or the report if there are enough people interested in it. Thank you for your help. Xin Yao Email: xin at cslab.anu.edu.au Computer Sciences Laboratory Tel: (+616)/(06)2495097 (O) Research School of Physical Sciences (+616)/(06)2512662 (H) Australian National University Fax: (+616)/(06)2491884 GPO Box 4, Canberra, ACT 2601 AUSTRALIA From P.Refenes at cs.ucl.ac.uk Fri Jan 18 12:49:39 1991 From: P.Refenes at cs.ucl.ac.uk (P.Refenes@cs.ucl.ac.uk) Date: Fri, 18 Jan 91 17:49:39 +0000 Subject: Dear researchers, In-Reply-To: Your message of Fri, 18 Jan 91 12:24:28 -0500. <9101180100.AA19697@anu.anu.edu.au> Message-ID: YOU MAY BE INTERESTED IN THIS: CONSTRUCTIVE LEARNING by SPECIALISATION A. N. REFENES & S. VITHLANI Department of Computer Science University College London Gower Street London WC1 6BT, UK ABSTRACT This paper describes and evaluates a procedure for constructing and training multi-layered perceptrons by adding units to the network dynamically during training. The ultimate aim of the learning procedure is to construct an architecture which is sufficiently large to learn the problem but necessarily small to generalise well. Units are added as they are needed. By showing that the newly added unit makes fewer mistakes than before, and by training the unit not to disturb the earlier dynamics, eventual convergence to zero-errors is guaranteed. Other techniques operate in the opposite direction by pruning the network and removing "redundant" connections. Simulations suggest that this method is efficient in terms of learning speed, the number of free parameters in the network (i.e. the number of free parameters in the network (i.e. connections), and it compares well to other methods in terms of generalisation performance. From ST401843 at brownvm.brown.edu Fri Jan 18 22:57:17 1991 From: ST401843 at brownvm.brown.edu (thanasis kehagias) Date: Fri, 18 Jan 91 22:57:17 EST Subject: Another Point of View on what Connectionism REALLY is ... Message-ID: I am sorry that I show up a little late in this "what connectionism really is (should be)?" debate - I want to present another point of view and this really is a different discussion. But it got started in my mind by reading the latest debate and at the same time writing a paper. It is my impression that historically connectionism started as something and developed into something quite different. Researchers in the late 50's to (even) early 80's were mostly focusing on building intelligent/thinking networks. However, for many reasons, not least among them being the proverbial bandwagon effect and the increase of funds available for connectionist research, many "computationally" oriented researchers started conducting connectionist research. I repeat that all this is personal opinion. I should also say that I consider myself to be one of the more "computationally" oriented researchers. I will now describe why neural nets are an interesting subject to me, I suspect this is the case for other researchers. My interest in neural nets is NOT biologically motivated. It is an interesting conjecture that systems that look "like" the brain will behave "intelligently", but I do not feel qualified to pursue this. It is a good thing that there are people pursuing this direction, as well as (other ?) people trying to model actual brains. But neural nets are also parallel, distributed computers. Some advantages of PDP are obvious, e.g. faster computing, and other somewhat not so obvious, e.g. robustness of computing properties with respect to local damage. This is what I find really interesting in connectionism, and I believe some people agree with me. This point of view has an important implication. We no longer need to insist that a network develops its own internal representations without any external guidance. It's OK to try to incorporate in the network/algorithm as much knowledge and design as we can. It's also OK to look into other disciplines (statistics, stochastic control) to see what methods they have developed and try to incorporate them into our networks. What really counts is to make the algorithm fast and efficient. Is this really connectionism? Is it any different from, say, the art of parallel algorithms? I believe the answer is yes to both questions. The Book has "Parallel Distributed Computing" in its title and we are all looking at parallel and distributed networks. And, even if boundaries are blurry, connectionism has the distinct flavor of building things up from small, simple units, in a way that is different from, say, a parallel implementation of Kalman filtering. In conclusion, I repeat that the boundaries between connectionism and a number of other disiplines, e.g. parallel algorithms, cellular automata, statistics etc. are blurry. This is not a disadvantage, on the contrary, I think, we would benefit from taking a look at these disciplines and comparing their point of view at problems similar to the ones we are examining. From der%psych at Forsythe.Stanford.EDU Sat Jan 19 15:46:22 1991 From: der%psych at Forsythe.Stanford.EDU (Dave Rumelhart) Date: Sat, 19 Jan 91 12:46:22 PST Subject: Job Opportunity at Stanford University Message-ID: <9101192046.AA02884@psych> The Psychology Department at Stanford University currently has two job openings at least one of which may be appropriate for a connectionist. I enclose a copy of the advertisement which appeared in several publications. If you feel you may be appropriate or know of someone who may be appropriate please apply for the position. Note from the ad that we are open to people at any level and with a variety of interests. This means, in short, we are interested in the best person we can attract within reasonably broad guidelines. I personally hope that this person has connectionist interests. David Rumelhart Chair of the Search Committee Stanford University Psychology Department. The Department of Psychology plans two tenure-track appointments in the Sensory/Perceptual and/or Cognitive Sciences (including the biological basis of cognition) beginning in the academic year 1991. Appointments may be either at the tenured or non-tenured (assistant professor) level. Outstanding scientists who have strong research records in sensory/perceptual processes, cognitive neuroscience and/or computational/mathematical models of cognitive processes are encouraged to apply. Applicants should send a current curriculum vitae, copies of their most important scholarly papers, and letters of recommendation to: The Cognitive Sciences Search Committee, c/o Ms. Frances Martin, Department of Psychology, Bldg. 420, Stanford University, Stanford, California, 94305. The deadline for application is February 18, 1991, but applicants are encouraged to submit their materials as soon as possible. Stanford University is an Equal Opportunity Employer. From tds at ai.mit.edu Sat Jan 19 16:33:00 1991 From: tds at ai.mit.edu (Terence D. Sanger) Date: Sat, 19 Jan 91 16:33:00 EST Subject: Nips90 Preprint available from neuroprose archive Message-ID: <9101192133.AA04425@gelatinosa> The following preprint is available, and will appear in the Nips'90 proceedings: ------------------------------------------------------------------------------ Basis-Function Trees as a Generalization of Local Variable Selection Methods for Function Approximation Terence D. Sanger Local variable selection has proven to be a powerful technique for approximating functions in high-dimensional spaces. It is used in several statistical methods, including CART, ID3, C4, MARS, and others (see the bibliography for references to these algorithms). In this paper I present a tree-structured network which is a generalization of these techniques. The network provides a framework for understanding the behavior of such algorithms and for modifying them to suit particular applications. ------------------------------------------------------------------------------ Bibtex entry: @INCOLLECTION{sanger91, AUTHOR = {Terence D. Sanger}, TITLE = {Basis-Function Trees as a Generalization of Local Variable Selection Methods for Function Approximation}, BOOKTITLE = {Advances in Neural Information Processing Systems 3}, PUBLISHER = {Morgan Kaufmann}, YEAR = {1991}, EDITOR = {Richard P. Lippmann and John Moody and David S. Touretzky}, NOTE = {Proc. NIPS'90, Denver CO} } This paper can be obtained by anonymous ftp from the neuroprose database: unix> ftp cheops.cis.ohio-state.edu # (or ftp 128.146.8.62) Name (cheops.cis.ohio-state.edu:): anonymous Password (cheops.cis.ohio-state.edu:anonymous): ftp> cd pub/neuroprose ftp> binary ftp> get sanger.trees.ps.Z ftp> quit unix> uncompress sanger.trees.ps unix> lpr -P(your_local_postscript_printer) sanger.trees.ps # in some cases you will need to use the -s switch to lpr. Terry Sanger MIT, E25-534 Cambridge, MA 02139 USA tds at ai.mit.edu From ga1013 at sdcc6.UCSD.EDU Sun Jan 20 17:46:12 1991 From: ga1013 at sdcc6.UCSD.EDU (ga1013) Date: Sun, 20 Jan 91 14:46:12 PST Subject: Job Opportunity at Stanford University Message-ID: <9101202246.AA18825@sdcc6.UCSD.EDU> hangup From crr%shum.huji.ac.il at BITNET.CC.CMU.EDU Thu Jan 17 11:31:12 1991 From: crr%shum.huji.ac.il at BITNET.CC.CMU.EDU (crr%shum.huji.ac.il@BITNET.CC.CMU.EDU) Date: Thu, 17 Jan 91 18:31:12 +0200 Subject: Description vs Explanation Message-ID: <9101171631.AA25120@shum.huji.ac.il> John von Neumann via the ever-present Lev Goldfarb: The sciences do not try to explain, they hardly even try to interpret, they mainly make models. By a model is meant a mathematical constract which, with the addition of certain verbal interpretations, describes observed phenomena. The justification of such a mathematical construct is solely and precisely that it is expected to work. I don't know where this quote comes from, but I disagree. Science also has the goal of explaining phenomena, not merely describing them. For a descriptive model, simple descriptive (mathematical) adequacy is the goal, whereas an explanation purports to go further and account for "the way things really are." It may be worthwhile for people doing work in connectionist models in particular to at least think about the distinction. Sometimes I feel that we modelers are not always clear as to which kind of model we are putting forth. Charlie Rosenberg From GOLDFARB%unb.ca at UNBMVS1.csd.unb.ca Mon Jan 21 09:22:03 1991 From: GOLDFARB%unb.ca at UNBMVS1.csd.unb.ca (GOLDFARB%unb.ca@UNBMVS1.csd.unb.ca) Date: Mon, 21 Jan 91 10:22:03 AST Subject: Description vs Explanation In-Reply-To: Message of Thu, 17 Jan 91 11:31:12 EST from <@BITNET.CC.CMU.EDU:crr@shum.huj Message-ID: > Sometimes I feel that we modelers are not always clear as to > which kind of model we are putting forth. > > Charlie Rosenberg It might be that "sometimes" is "most of the time". -- Lev Goldfarb From der%psych at Forsythe.Stanford.EDU Mon Jan 21 15:22:00 1991 From: der%psych at Forsythe.Stanford.EDU (Dave Rumelhart) Date: Mon, 21 Jan 91 12:22:00 PST Subject: IJCNN-91-SEATTLE Message-ID: <9101212022.AA15402@psych> In my role as conference chairman of the International Joint Conference on Neural Networks to be held this summer (July 8-12) in Seattle, Washington, I would like to remind readers of this mailing list that the deadline for paper submissions is February 1, 1991. I would encourage submissions. The quality of a conference is largely determined by the quality of the submitted papers. As a further reminder, or in case you haven't seen a formal call for papers, I provide some of the details below. Papers may be submitted in the areas of neurobiology, optical and electronic implementations, image processing, vision, speech, network dynamics, optimization, robotics and control, learning and generalization, neural network architectures, applications and other areas in neural networks. Papers must be submitted in English (1 original and seven copies) maximum six pages, camera-ready on 8 1/2" x 11" white paper with 1" margins on all sides and un-numbered. Centered at the top of the first page should be the complete title, author name(s), affiliation(s) and mailing address(es). This is followed by a blank space and then the abstract up to 15 lines, followed by the text. A cover letter including the corresponding author's name, mailing address, telephone and fax number, technical area, oral or poster presentation preference. Send papers to IJCNN-91-SEATTLE, University of Washington, Conference Management, Attn: Sarah Eck, MS/GH-22, 5001 25th Ave. N.E., Seattle WA 98195. The program planning for this meeting is outstanding. The site of the meeting will, I think, be outstanding. A major contribution to the success of the meeting (and, I think, the success of the field) will be made by each quality paper submitted. I look forward to an exciting meeting and hope to see a strong contribution from participants on the connectionist mailing list. Thank you for your consideration. David E. Rumelhart, Conference Chair, IJCNN-91-SEATTLE From MURRE%rulfsw.LeidenUniv.nl at BITNET.CC.CMU.EDU Tue Jan 22 11:52:00 1991 From: MURRE%rulfsw.LeidenUniv.nl at BITNET.CC.CMU.EDU (MURRE%rulfsw.LeidenUniv.nl@BITNET.CC.CMU.EDU) Date: Tue, 22 Jan 91 11:52 MET Subject: Pointers to papers on the effect of implementation precision Message-ID: Dear connectionist researchers, We are in the process of designing a new neurocomputer. An important design consideration is precision: Should we use 1-bit, 4-bit, 8-bit, etc. representations for weights, activations, and other parameters? We are scaling-up our present neurocomputer, the BSP400 (Brain Style Processor with 400 processors), which uses 8-bit internal representations for activations and weights, but activations are exchanged as single bits (using partial time-coding induced by floating thresholds). This scheme does not scale well. Though we have tracked down scattered remarks in the literature on precision, we have not been able to find many systematic studies on this subject. Does anyone know of systematic simulations or analytical results of the effect of implementation precision on the performance of a neural network? In particular we are interested in the question of how (and to what extent) limited precision (i.e., 8-bits) implementations deviate from, say, 8-byte, double precision implementations. The only systematic studies we have been able to find so far deal with fault tolerance, which is only of indirect relevance to our problem: Brause, R. (1988). Pattern recognition and fault tolerance in non-linear neural networks. Biological Cybernetics, 58, 129-139. Jou, J., & J.A. Abraham (1986). Fault-tolerant matrix arithmetic and signal processing on highly concurrent computing structures. Proceedings of the IEEE, 74, 732-741. Moore, W.R. (1988). Conventional fault-tolerance and neural computers. In: R. Eckmiller, & C. Von der Malsburg (Eds.). Neural Computers. NATO ASI Series, F41, (Berling: Springer-Verlag), 29-37. Nijhuis, J., & L. Spaanenburg (1989). Fault tolerance of neural associative memories. IEE Proceedings, 136, 389-394. Thanks! From gary at cs.UCSD.EDU Tue Jan 22 13:51:45 1991 From: gary at cs.UCSD.EDU (Gary Cottrell) Date: Tue, 22 Jan 91 10:51:45 PST Subject: IJCNN-91-SEATTLE Message-ID: <9101221851.AA18040@desi.ucsd.edu> I am interested to know who picks the categories for submission. It has always seemed odd to me that there is no Natural Language processing or Cognitive Modeling categories. These must be relegated to "Other" by people who submit in these categories. g. From hwang at uw-isdl.ee.washington.edu Tue Jan 22 14:33:11 1991 From: hwang at uw-isdl.ee.washington.edu ( J. N. Hwang) Date: Tue, 22 Jan 91 11:33:11 PST Subject: Pointers to papers on the effect of implementation precision Message-ID: <9101221933.AA11885@uw-isdl.ee.washington.edu> We are in the process of finishing up a paper which gives a theoretical (systematic) derivation of the finite precision neural network computation. The idea is a nonlinear extension of "general compound operators" widely used for error analysis of linear computation. We derive several mathematical formula for both retrieving and learning of neural networks. The finite precision error in the retrieving phase can be written as a function of several parameters, e.g., number of bits of weights, number of bits for multiplication and accumlation, size of nonlinear table-look-up, truncation/rounding or jamming approaches, and etc. Then we are able to extend this retrieving phase error analysis to iterative learning to predict the necessary number of bits. This can be shown using a ratio between the finite precision error and the (floating point) back-propagated error. Simulations have been conducted and matched the theoretical prediction quite well. Hopefully, we can have a final version of this paper available to you soon. Jordan L. Holt and Jenq-Neng Hwang ,"Finite Precision Error Analysis of Neural Network Hardware Implementation," University of Washington, FT-10, Seattle, WA 98195 Best Regards, Jenq-Neng From tenorio at ecn.purdue.edu Tue Jan 22 15:15:41 1991 From: tenorio at ecn.purdue.edu (Manoel F Tenorio) Date: Tue, 22 Jan 91 15:15:41 -0500 Subject: TR EE 90-63 , Short term memory and Hysterisis Message-ID: <9101222015.AA04814@dynamo.ecn.purdue.edu> Subject: TR-EE 90-63: The Hystery Unit - short term memory Bcc: tenorio -------- The task of performing recognition of patterns on spatio-temporal signals is not an easy one, primarily due to the time structure of the signal. Classical methods of handling this problem have proven themselves unsatisfactory, and they range from "projecting out" the time axis, to "memorizing" the entire sequence before a decision can be made. In particular, the latter can be very difficult if no a priori information about signal length is present, if the signal can suffer compression and extension, or if the entire pattern is massively large, as in the case of time varying imagery. Neural Network models to solve this problem have either been based on the classical approach or on recursive loops within the network which can make learning algorithms numerically unstable. It is clear that for all the spatio-temporal processing, done by biological systems, some kind of short term memory is needed, and has been long conjectured. In this report, we have taken the first step at the design of a spatio-temporal system that deals naturally with the problems present in this type of processing. In particular we investigate the exchange of the simple sigmoid function, commonly used, by a hysterisis function. Later, with the addition of an integrator which represents the neuron membrane effect, we construct a simple computational device to perform spatio-pattern recognition tasks. The results are that for bipolar input sequence, this device remaps the entire sequence into a real number. Knowing the output of the device suffices for knowing the sequence. For trajectories embbeded in noise, the device shows superior recognition to other techniques. Furthermore, properties of the device allows the designer to determine the memory length, and explain with simple circuits sensitization and habituation phenomena. The report below deals with the device and its mathematical properties. Other forthcoming papers will concentrate on other aspects of circuits constructed with this device. ---------------------------------------------------------------------- Requests from within US, Canada, and Mexico: The technical report with figures has been/will soon be placed in the account kindly provided by Ohio State. Here is the instruction to get the files: ftp cheops.cis.ohio-state.edu (or, ftp 128.146.8.62) Name: anonymous Password: neuron ftp> cd pub/neuroprose ftp> mget tom.hystery* (type y and hit return) ftp> quit unix> uncompress tom.hystery*.Z unix> lpr -P(your_postscript_printer) tom.hystery.ps unix> lpr -P(your_Mac_laserwriter) tom.hystery_figs.ps Please contact mdtom at ecn.purdue.edu for technical difficulties. ---------------------------------------------------------------------- Requests from outside North America: The technical report is available at a cost of US$22.39 per copy, postage included. Please make checks payable to Purdue University in US dollars. You may send your requests, checks, and full first class mail address to: J. L. Dixon School of Electrical Engineering Purdue University West Lafayette, Indiana 47907 USA Please mention the technical report number: TR-EE 90-63. ---------------------------------------------------------------------- The Hystery Unit - A Short Term Memory Model for Computational Neurons M. Daniel Tom Manoel Fernando Tenorio Parallel Distributed Structures Laboratory School of Electrical Engineering Purdue University West Lafayette, Indiana 47907, USA December, 1990 Abstract: In this paper, a model of short term memory is introduced. This model is inspired by the transient behavior of neurons and magnetic storage as memory. The transient response of a neuron is hypothesized to be a combination of a pair of sigmoids, and a relation is drawn to the hysteresis loop found in magnetic materials. A model is created as a composition of two coupled families of curves. Two theorems are derived regarding the asymptotic convergence behavior of the model. Another conjecture claims that the model retains full memory of all past unit step inputs. From tenorio at ecn.purdue.edu Tue Jan 22 16:36:15 1991 From: tenorio at ecn.purdue.edu (Manoel F Tenorio) Date: Tue, 22 Jan 91 16:36:15 -0500 Subject: TR EE Short Term Memory - Hysterisis Message-ID: <9101222136.AA09950@dynamo.ecn.purdue.edu> Subject: TR-EE 90-63: The Hystery Unit - short term memory Bcc: tenorio -------- The task of performing recognition of patterns on spatio-temporal signals is not an easy one, primarily due to the time structure of the signal. Classical methods of handling this problem have proven themselves unsatisfactory, and they range from "projecting out" the time axis, to "memorizing" the entire sequence before a decision can be made. In particular, the latter can be very difficult if no a priori information about signal length is present, if the signal can suffer compression and extension, or if the entire pattern is massively large, as in the case of time varying imagery. Neural Network models to solve this problem have either been based on the classical approach or on recursive loops within the network which can make learning algorithms numerically unstable. It is clear that for all the spatio-temporal processing, done by biological systems, some kind of short term memory is needed, and has been long conjectured. In this report, we have taken the first step at the design of a spatio-temporal system that deals naturally with the problems present in this type of processing. In particular we investigate the exchange of the simple sigmoid function, commonly used, by a hysterisis function. Later, with the addition of an integrator which represents the neuron membrane effect, we construct a simple computational device to perform spatio-pattern recognition tasks. The results are that for bipolar input sequence, this device remaps the entire sequence into a real number. Knowing the output of the device suffices for knowing the sequence. For trajectories embbeded in noise, the device shows superior recognition to other techniques. Furthermore, properties of the device allows the designer to determine the memory length, and explain with simple circuits sensitization and habituation phenomena. The report below deals with the device and its mathematical properties. Other forthcoming papers will concentrate on other aspects of circuits constructed with this device. ---------------------------------------------------------------------- Requests from within US, Canada, and Mexico: The technical report with figures has been/will soon be placed in the account kindly provided by Ohio State. Here is the instruction to get the files: ftp cheops.cis.ohio-state.edu (or, ftp 128.146.8.62) Name: anonymous Password: neuron ftp> cd pub/neuroprose ftp> mget tom.hystery* (type y and hit return) ftp> quit unix> uncompress tom.hystery*.Z unix> lpr -P(your_postscript_printer) tom.hystery.ps unix> lpr -P(your_Mac_laserwriter) tom.hystery_figs.ps Please contact mdtom at ecn.purdue.edu for technical difficulties. ---------------------------------------------------------------------- Requests from outside North America: The technical report is available at a cost of US$22.39 per copy, postage included. Please make checks payable to Purdue University in US dollars. You may send your requests, checks, and full first class mail address to: J. L. Dixon School of Electrical Engineering Purdue University West Lafayette, Indiana 47907 USA Please mention the technical report number: TR-EE 90-63. ---------------------------------------------------------------------- The Hystery Unit - A Short Term Memory Model for Computational Neurons M. Daniel Tom Manoel Fernando Tenorio Parallel Distributed Structures Laboratory School of Electrical Engineering Purdue University West Lafayette, Indiana 47907, USA December, 1990 Abstract: In this paper, a model of short term memory is introduced. This model is inspired by the transient behavior of neurons and magnetic storage as memory. The transient response of a neuron is hypothesized to be a combination of a pair of sigmoids, and a relation is drawn to the hysteresis loop found in magnetic materials. A model is created as a composition of two coupled families of curves. Two theorems are derived regarding the asymptotic convergence behavior of the model. Another conjecture claims that the model retains full memory of all past unit step inputs. From erol at ehei.ehei.fr Tue Jan 22 10:25:13 1991 From: erol at ehei.ehei.fr (Erol Gelenbe) Date: Tue, 22 Jan 91 15:27:13 +2 Subject: Charlie Rosenberg's message Message-ID: <9101231128.AA11600@inria.inria.fr> I fully agree with him. Our purpose is to understand, and then to be able to explain to others, rather than simply to represent and manipulate models or equations. Erol Gelenbe From weber at icsib Wed Jan 23 11:36:47 1991 From: weber at icsib (Susan Weber) Date: Wed, 23 Jan 91 08:36:47 PST Subject: bank credit and neural nets Message-ID: <9101231636.AA02813@icsib> From bradley at ivy.Princeton.EDU Wed Jan 23 13:14:09 1991 From: bradley at ivy.Princeton.EDU (Bradley Dickinson) Date: Wed, 23 Jan 91 13:14:09 EST Subject: Neural Network Council Awards Message-ID: <9101231814.AA07390@ivy.Princeton.EDU> Nominations Sought for IEEE Neural Networks Council Awards The IEEE Neural Networks Council is soliciting nominations for two new awards. Pending final approval the IEEE, it is planned to present these awards for the first time at the July 1991 International Joint Conference on Neural Networks. Nominations for these awards should be submitted in writing according to the instructions given below. IEEE Transactions on Neural Networks Outstanding Paper Award This is an award of $500 for the outstanding paper published in the IEEE Transactions on Neural Networks in the previous two-year period. For 1991, all papers published in 1990 (Volume 1) in the IEEE Transactions on Neural Networks are eligible. For a paper with multiple authors, the award will be shared by the coauthors. Nominations must include a written statement describing the outstanding characteristics of the paper. The deadline for receipt of nominations is March 31, 1991. Nominations should be sent to Prof. Bradley W. Dickinson, NNC Awards Chair, Dept. of Electrical Engineering, Princeton University, Princeton, NJ 08544-5263. IEEE Neural Networks Council Pioneer Award This award has been established to recognize and honor the vision of those people whose efforts resulted in significant contributions to the early concepts and developments in the neural networks field. Up to three awards may be presented annually to outstanding individuals whose main contribution has been made at least fifteen years earlier. The recognition is engraved on the Neural Networks Pioneer Medal specially struck for the Council. Selection of Pioneer Medalists will be based on nomination letters received by the Pioneer Awards Committee. All who meet the contribution requirements are eligible, and anyone can nominate. The award is not approved posthumously. Written nomination letters must include a detailed description of the nominee's contributions and must be accompanied by full supporting documentation. For the 1991 Pioneer Award, nominations must be received by March 1, 1991. Nominations should be sent to Prof. Bradley W. Dickinson, NNC Pioneer Award Chair, Department of Electrical Engineering, Princeton University, Princeton, NJ 08544-5263. Questions and preliminary inquiries about the above awards should be directed to Prof. Bradley W. Dickinson, NNC Awards Chair; telephone: (609)-258-4644, electronic mail: bradley at ivy.princeton.edu From mclennan at cs.utk.edu Wed Jan 23 16:28:54 1991 From: mclennan at cs.utk.edu (mclennan@cs.utk.edu) Date: Wed, 23 Jan 91 16:28:54 -0500 Subject: tech report: continuous spatial automata Message-ID: <9101232128.AA04465@maclennan.cs.utk.edu> The following technical report is now available: Continuous Spatial Automata B. J. MacLennan Department of Computer Science University of Tennessee Knoxville, TN 37996-1301 maclennan at cs.utk.edu CS-90-121 November 26, 1990 ABSTRACT A _continuous_spatial_automaton_ is analogous to a cellular auto- maton, except that the cells form a continuum, as do the possible states of the cells. After an informal mathematical description of spatial automata, we describe in detail a continuous analog of Conway's ``Life,'' and show how the automaton can be implemented using the basic operations of field computation. Typically a cellular automaton has a finite (sometimes denu- merably infinite) set of cells, often arranged in a one or two dimensional array. Each cell can be in one of a number of states. In contrast, a continuous spatial automaton has a one, two or higher dimensional continuum of _loci_ (corresponding to cells), each of which has a state drawn from a continuum (typically [0,1]). The state is required to vary continuously with the locus. In a cellular automaton there is a transition function that determines the state of a cell at the next time step based on the state of it and a finite number of neighbors at the current time step. A discrete-time spatial automaton is very similar: the future state of a locus is a continuous function of the states of the loci in a (closed or open) bounded neighborhood of the given locus. The report is available as a compressed postscript file in the pub/neuroprose subdirectory; it may be obtained with the Getps script: Getps maclennan.csa.ps.Z For HARDCOPY send your address to: library at cs.utk.edu For other correspondence: Bruce MacLennan Department of Computer Science 107 Ayres Hall The University of Tennessee Knoxville, TN 37996-1301 (615)974-0994/5067 maclennan at cs.utk.edu From mdtom at ecn.purdue.edu Thu Jan 24 11:28:41 1991 From: mdtom at ecn.purdue.edu (M Daniel Tom) Date: Thu, 24 Jan 91 11:28:41 -0500 Subject: TR-EE 90-63 figs fix Message-ID: <9101241628.AA07433@transform.ecn.purdue.edu> Dear Connectionists, Following the instructions in Mac2ps (found in the neuroprose database) I created the file tom.hystery_figs.ps.Z (using command-k from a Mactintosh application). However people have trouble printing it. Steve, talking below, reports the problem to me, and helps solve the problem. I have followed Steps 1 and 2 below, and put the new file in neuroprose/Inbox/tom.hystery_figs.ps.Z, which may be moved to neuroprose later. I hope you have success following Step 3 below. Daniel Tom > Date: Wed, 23 Jan 91 16:13:04 -0500 > From: bradtke at envy.cs.umass.edu > To: mdtom at ecn.purdue.edu > Subject: printing the figures for the hystery unit paper > > > Well, we finally got the figure pages to print. > We had to play with the file retrieved from neuroprose a bit > first, though. > > step 1: change all \r in the file to > > step 2: remove the Apple ProcSet header > > step 3: use SendPS to print it on a Mac LaserWriter > (This wouldn't work till steps 1 and 2 were done.) > > > I also got it to print on a unix system on a dec ln03 -ps by > changing step 3 to > cat lprep68.pro tom.hystery_figs.ps graphics-restore.ps | lpr > where the file lprep68.pro is the Apple Mac PostScript prolog, and > the file graphics-restore.ps just does a showpage. The only problem > with this second solution is that all of the figures printed on just > one page. > > That's all I know. > > Steve > daniel From gyen at steinbeck.helios.nd.edu Thu Jan 24 13:42:05 1991 From: gyen at steinbeck.helios.nd.edu (Gune Yen) Date: Thu, 24 Jan 91 13:42:05 EST Subject: reference for nonsymmetric interconnection networks Message-ID: <9101241842.AA00823@steinbeck.helios.nd.edu> Hi, Synthesis techniques for associative memories via artificial feedback neural networks have been studied for many years. Among them, I am very interested in any design method results in neural networks with non-symmetric interconnecting structure. Obviously, the requirement to have symmetric interconnection weights will pose difficulties in implementation and may result in spurious states as well. I am aware of the work given below: A. Lapedes and R. Farber, "A Self-Optimizing Nonsymmetrical Neural Net for Content Addressable Memory and Pattern Recognition", Physica 22D, 1986, pp. 247-259. J. A. Farrell and A. N. Michel, "Analysis and Synthesis Techniques for Hopfield Type Synchronous Discrete Neural Networks with Application to Associative Memory", IEEE Transactions on Circuits and Systems, Vol. 37, No. 11, November 1990, pp. 1356-1366. Could anyone who familiar in this type of problem provide any references and comments in terms of symmetric/non-symmetric interconnection weights, with/without self-feedback term, difficulties in analog/digital implementation, and etc. Thank you very much. Gary at gyen at steinbeck.helios.nd.edu University of Notre Dame From sontag at hilbert.rutgers.edu Thu Jan 24 16:16:17 1991 From: sontag at hilbert.rutgers.edu (Eduardo Sontag) Date: Thu, 24 Jan 91 16:16:17 EST Subject: TR available from neuroprose; feedforward nets Message-ID: <9101242116.AA27780@hilbert.rutgers.edu> I have deposited in the neuroprose archive the extended version of my NIPS-90 Proceedings paper. The title is: "FEEDFORWARD NETS FOR INTERPOLATION AND CLASSIFICATION" and the abstract is: "This paper deals with single-hidden-layer feedforward nets, studying various measures of classification power and interpolation capability. Results are given showing that direct input to output connections in threshold nets double the recognition but not the interpolation power, while using sigmoids rather than thresholds allows (at least) doubling both." (NOTE: This is closely related to report SYCON-90-03, which was put in the archive last year under the title "sontag.capabilities.ps.Z". No point in retrieving unless you found the other paper of interest. The current paper besically adds a few results on interpolation.) -eduardo ----------------------------------------------------------------------------- To obtain copies of the postscript file, please use Jordan Pollack's service: Example: unix> ftp cheops.cis.ohio-state.edu # (or ftp 128.146.8.62) Name (cheops.cis.ohio-state.edu:): anonymous Password (cheops.cis.ohio-state.edu:anonymous): ftp> cd pub/neuroprose ftp> binary ftp> get (remote-file) sontag.nips90.ps.Z (local-file) sontag.nips90.ps.Z ftp> quit unix> uncompress sontag.nips90.ps.Z unix> lpr -P(your_local_postscript_printer) sontag.nips90.ps ---------------------------------------------------------------------------- If you have any difficulties with the above, please send e-mail to sontag at hilbert.rutgers.edu. DO NOT "reply" to this message, please. NOTES about FTP'ing, etc: (1) The last time I posted something, I forgot to include the ".Z" in the file name in the above "remote-file" line, and I received many messages telling me that FTP didn't find the file. Sorry for that. Please note that most files in the archive are compressed, and people may forget to mention the ".Z". (2) I also received some email (and saw much discussion in a bboard) concerning the printer errors with the file. Please note that postscript files sometimes require a fair amount of memory from the printer, especially if they contain illustrations, and many smaller printers do not have enough memory. This may result on some pages not being printed, or the print job not being done at all. If you experience this problem with papers you retrieve (mine or from others), I suggest that you ask the author to email you a source file (e.g. LaTex) or a postscript file sans figures. Also, some postscript files are "nonconforming", and this may cause problems with certain printers. From lacher at lambda.cs.fsu.edu Thu Jan 24 16:16:45 1991 From: lacher at lambda.cs.fsu.edu (Chris Lacher) Date: Thu, 24 Jan 91 16:16:45 -0500 Subject: Abstract Message-ID: <9101242116.AA19172@lambda.cs.fsu.edu> Backpropagation Learning in Expert Networks by R. C. Lacher, Susan I. Hruska, and David C. Kuncicky Department of Computer Science Florida State University ABSTRACT. Expert networks are event-driven, acyclic networks of neural objects derived from expert systems. The neural objects process information through a non-linear combining function that is different from, and more complex than, typical neural network node processors. We develop backpropagation learning for acyclic, event-driven nets in general and derive a specific algorithm for learning in EMYCIN-derived expert networks. The algorithm combines backpropagation learning with other features of expert nets, including calculation of gradients of the non-linear combining functions and the hypercube nature of the knowledge space. Results of testing the learning algorithm with a medium-scale (97 node) expert network are presented. For a copy of this preprint send an email request with your (snail)MAIL ADDRESS and the TITLE of the preprint to: santan at nu.cs.fsu.edu --- Chris Lacher From prowat at UCSD.EDU Thu Jan 24 19:05:13 1991 From: prowat at UCSD.EDU (Peter Rowat) Date: Thu, 24 Jan 91 16:05:13 PST Subject: Learning with "realistic" neurons Message-ID: <9101250005.AA20022@crayfish.UCSD.EDU> Gary Cottrell recently referred to work I am doing with models of the gastric mill network in the lobster's stomatogastric ganglion. This work is not published, but I do have a related paper which generalizes BP to arbitrarily complex models (amongst other things), and which is now available by ftp from the neuroprose archive. Namely: Peter Rowat and Allen Selverston (1990). Learning algorithms for oscillatory networks with gap junctions and membrane currents. To appear in: NETWORK: Computation in Neural systems, Volume 2, Issue 1, February 1991. Abstract: We view the problem of parameter adjustment in oscillatory neural networks as the minimization of the difference between two limit cycles. Backpropagation is described as the application of gradient descent to an error function that computes this difference. A mathematical formulation is given that is applicable to any type of network model, and applied to several models. By considering a neuron equivalent circuit, the standard connectionist model of a neuron is extended to allow gap junctions between cells and to include membrane currents. Learning algorithms are derived for a two cell network with a single gap junction, and for a pair of mutually inhibitory neurons each having a simplified membrane current. For example, when learning in a network in which all cells have a common, adjustable, bias current, the value of the bias is adjusted at a rate proportional to the difference between the sum of the target outputs and the sum of the actual outputs. When learning in a network of n cells where a target output is given for every cell, the learning algorithm splits into n independent learning algorithms, one per cell. For networks containing gap junctions, a gap junction is modelled as a conductance times the potential difference between the two adjacent cells. The requirement that a conductance g must be positive is enforced by replacing g by a function pos(g*) whose value is always positive, for example exp(0.1 g*), and deriving an algorithm that adjusts the parameter g* in place of g. When target output is specified for every cell in a network with gap junctions, the learning algorithm splits into fewer independent components, one for each gap-connected subset of the network. The learning algorithm for a gap-connected set of cells cannot be parallelized further. As a final example, a learning algorithm is derived for a mutually inhibitory two-cell network in which each cell has a membrane current. This generalized approach to backpropagation allows one to derive a learning algorithm for almost any model neural network given in terms of differential equations. It is one solution to the problem of parameter adjustment in small but complex network models. - --------------------------------------------------------------------------- Copies of the postscript file rowat.learn-osc.ps.Z may be obtained from the pub/neuroprose directory in cheops.cis.ohio-state.edu. Either use the Getps script or do this: unix-1> ftp cheops.cis.ohio-state.edu # (or ftp 128.146.8.62) Connected to cheops.cis.ohio-state.edu. Name (cheops.cis.ohio-state.edu:): anonymous 331 Guest login ok, sent ident as password. Password: neuron 230 Guest login ok, access restrictions apply. ftp> cd pub/neuroprose ftp> binary ftp> get rowat.learn-osc.ps.Z ftp> quit unix-2> uncompress rowat.learn-osc.ps.Z unix-3> lpr -P(your_local_postscript_printer) rowat.learn-osc.ps (The file starts with 7 bitmapped figures which are slow to print.) From FEGROSS%WEIZMANN.BITNET at bitnet.cc.CMU.EDU Fri Jan 25 04:31:44 1991 From: FEGROSS%WEIZMANN.BITNET at bitnet.cc.CMU.EDU (Tal Grossman) Date: Fri, 25 Jan 91 11:31:44 +0200 Subject: Anti SCAD NNs Message-ID: <8A161BC8400001B6@BITNET.CC.CMU.EDU> An artificial neural network for the recognition and deception of SCAD missiles is urgently needed. Assign all your Back-Props and ARTs to that mission. A huge training set is constantly presented (in a cyclic order) by the CNN... Yours faithfully - Tal Grossman (somewhere near Tel Aviv). P.S. Only Neural that can perform well with a gas mask should be considered. Nervous neural networks are out of the question. From eniac!lba at relay.EU.net Fri Jan 25 10:22:53 1991 From: eniac!lba at relay.EU.net (Luis Borges de Almeida) Date: Fri, 25 Jan 91 15:22:53 GMT Subject: reference for nonsymmetric interconnection networks In-Reply-To: Gune Yen's message of Thu, 24 Jan 91 13:42:05 EST <9101241842.AA00823@steinbeck.helios.nd.edu> Message-ID: <9101251522.AA09062@eniac.inesc.pt> The paper referenced below (reprints can be sent to those interested), gives results of some experiments on the training of small Hopfield-style networks by recurrent backpropagation. The examples given are all for symmetric networks, but the procedure can be extended to the nonsymmetric case, by just not imposing weight symmetry during the training. We performed a few tests for that situation, with good results, though in that case there is no guarantee that the training will always result in a stable network. Those tests are not reported in the paper, because we wanted to limit ourselves to the more Hopfield-like case. The tests reported in the paper all use the self-feedback term. On the other hand, the paper reports two very simple experiments on other extensions of Hopfield networks: use of hidden units, and storage of analog-valued patterns. Luis B. Almeida INESC Phone: +351-1-544607 Apartado 10105 Fax: +351-1-525843 P-1017 Lisboa Codex Portugal lba at inesc.inesc.pt lba at inesc.uucp (if you have access to uucp) --------------------------------------- REFERENCE: Luis B. Almeida and Joao P. Neto, "Recurrent Backpropagation and Hopfield Networks", in F.Fogelman-Soulie and J. Herault (eds.), Proc. of the NATO ARW on Neurocomputing, Algorithms, Architectures and Implementations, Les Arcs, France, Feb/Mar 1989, Springer-Verlag (1990). From lwyse at park.bu.edu Fri Jan 25 14:12:22 1991 From: lwyse at park.bu.edu (lwyse@park.bu.edu) Date: Fri, 25 Jan 91 14:12:22 -0500 Subject: reference for nonsymmetric interconnection networks In-Reply-To: connectionists@c.cs.cmu.edu's message of 25 Jan 91 02:33:42 GM Message-ID: <9101251912.AA06868@kendall.bu.edu> The ART architectures of Carpenter and Grosberg are all associative memories using non-symmetric interconnections. There is a good paper on ART II in Applied Optics 26:23 (December, '87) pg 4919. - lonce XXX XXX Lonce Wyse | X X Center for Adaptive Systems \ | / X X Boston University \ / 111 Cummington St. Boston,MA 02215 ---- ---- X X X X "The best things in life / \ XXX XXX are emergent." / | \ | From haussler at saturn.ucsc.edu Fri Jan 25 20:31:37 1991 From: haussler at saturn.ucsc.edu (David Haussler) Date: Fri, 25 Jan 91 17:31:37 -0800 Subject: tech rep on overfitting, decision theory, PAC learning, and... Message-ID: <9101260131.AA05685@saturn.ucsc.edu> TECHNICAL REPORT AVAILABLE ---------------------------- Decision Theoretic Generalizations of the PAC Model for Neural Net and Other Learning Applications David Haussler UCSC-CRL-91-02 September, 1989 Revised: December, 1990 haussler at saturn.ucsc.edu Baskin Center for Computer Engineering and Information Sciences University of California, Santa Cruz, CA 95064 Abstract: We describe a generalization of the PAC learning model that is based on statistical decision theory. In this model the learner receives randomly drawn examples, each example consisting of an instance $x \in X$ and an outcome $y \in Y$, and tries to find a hypothesis $h : X \rightarrow A$, where $h \in \cH$, that specifies the appropriate action $a \in A$ to take for each instance $x$, in order to minimize the expectation of a loss $\L(y,a)$. Here $X$, $Y$, and $A$ are arbitrary sets, $\L$ is a real-valued function, and examples are generated according to an arbitrary joint distribution on $X \times Y$. Special cases include the problem of learning a function from $X$ into $Y$, the problem of learning the conditional probability distribution on $Y$ given $X$ (regression), and the problem of learning a distribution on $X$ (density estimation). We give theorems on the uniform convergence of empirical loss estimates to true expected loss rates for certain hypothesis spaces $\cH$, and show how this implies learnability with bounded sample size, disregarding computational complexity. As an application, we give distribution-independent upper bounds on the sample size needed for learning with feedforward neural networks. Our theorems use a generalized notion of VC dimension that applies to classes of real-valued functions, adapted from Pollard's work, and a notion of {\em capacity} and {\em metric dimension} for classes of functions that map into a bounded metric space. The report can be retrieved by anonymous ftp from the UCSC Tech report library. An example follows: unix> ftp midgard.ucsc.edu # (or ftp 128.114.134.15) Connected ... Name (...): anonymous Password: yourname at cs.anyuniversity.edu (i.e. your email address) (Please use your email address so we can correspond with you.) Guest login ok, access restrictions apply. ftp> cd pub/tr ftp> binary ftp> get ucsc-crl-91-02.ps.Z 200 PORT command successful. 150 Opening BINARY mode data connection for ucsc-crl-91-02.ps.Z (576429 bytes). 226 Transfer complete. local: ucsc-crl-91-02.ps.Z remote: ucsc-crl-91-02.ps.Z 576429 bytes received in 10 seconds (70 Kbytes/s) ftp> quit unix> uncompress ucsc-crl-91-02.ps.Z unix> lpr -P(your_local_postscript_printer) ucsc-crl-91-02.ps (Note: you will need a printer with a large memory.) (also: some other UCSC tech reports are available as well and more will be added soon. ftp the file INDEX to see what's there.) If you have any difficulties with the above, please send e-mail to jean at cis.ucsc.edu. DO NOT "reply" to this message, please. -David From N.E.Sharkey at cs.exeter.ac.uk Fri Jan 25 09:54:38 1991 From: N.E.Sharkey at cs.exeter.ac.uk (Noel Sharkey) Date: Fri, 25 Jan 91 14:54:38 GMT Subject: IJCNN-91-SEATTLE In-Reply-To: Gary Cottrell's message of Tue, 22 Jan 91 10:51:45 PST <9101221851.AA18040@desi.ucsd.edu> Message-ID: <16315.9101251454@entropy.cs.exeter.ac.uk> I agree with gary about CNLP (and cognition). Natural language does get ignored at many of the connectionist (or neural net) meetings. It may be that up until fairly recently there wasn't much research in that area. But there is certainly quite a lot now. - (You can mail lyn at my address below for a potted history and references up to 1990). There has been an explosion of work since 1988 and much of it is related to cognitive areas (e.g. whether or not connectionist research offers a new theory of representation). CNLP also offers much of interest to general neuro-computing such as the nature of connectionist compositionality and the encoding and recognition of recursive structures. CNLP research can provide input into some of the big questions about the relationship between neural net and mind (i use the more neutral term neural net rather than brain), though of course i am broadening the field here. Obviously, i would like to see this area represented more at the major conferences. Do others feel that this field is being backgrounded? and if so why do you think it is? Perhaps it is felt that this is an inappropriate area for connectionist research - that it is too high level, or the it should be left to the symbol grinders. who knows? From barto at envy.cs.umass.edu Sat Jan 26 09:55:38 1991 From: barto at envy.cs.umass.edu (Andy Barto) Date: Sat, 26 Jan 91 09:55:38 EST Subject: reference for nonsymmetric interconnection networks Message-ID: <9101261455.AA16863@envy.cs.umass.edu> From VAINA at buenga.bu.edu Sat Jan 26 21:20:00 1991 From: VAINA at buenga.bu.edu (VAINA@buenga.bu.edu) Date: Sat, 26 Jan 91 21:20 EST Subject: The COMPUTING BRAIN LECTURE AT BU-ENGINEERING Message-ID: From: BUENGA::CORTEX 26-JAN-1991 18:58 To: @CORTEX-NEW,IN%CORTEX-IN-DISTRIBUTION CC: CORTEX Subj: COMPUTING BRAIN LECTURE SERIES - John Hopfield *************************************************************************** THE COMPUTING BRAIN LECTURE SERIES *************************************************************************** " THE DYNAMICS OF COMPUTING " Professor John Hopfield California Institute of Technology Wednesday, February 27, 1991 at 5 pm Old Engineering Building - Room 150 110 Cummington Street, Boston, Ma Tea at 4 pm in Room 129 (same address as above) Lecture open to all For further information contact: Professor Lucia M. Vaina 353-2455 or vaina at buenga.bu.edu *************************************************************************** From hertz at nordita.dk Mon Jan 28 06:04:05 1991 From: hertz at nordita.dk (hertz@nordita.dk) Date: Mon, 28 Jan 91 11:04:05 GMT Subject: preprint available Message-ID: <9101281104.AA01456@thor.dk> The following technical report has been placed in the neuroprose archives at Ohio State University: Dynamics of Generalization in Linear Perceptrons Anders Krogh John Hertz Niels Bohr Institut Nordita Abstract: We study the evolution of the generalization ability of a simple linear perceptron with N inputs which learns to imitate a ``teacher perceptron''. The system is trained on p = \alpha N binary example inputs and the generalization ability measured by testing for agreement with the teacher on all 2^N possible binary input patterns. The dynamics may be solved analytically and exhibits a phase transition from imperfect to perfect generalization at \alpha = 1. Except at this point the generalization ability approaches its asymptotic value exponentially, with critical slowing down near the transition; the relaxation time is \propto (1-\sqrt{\alpha})^{-2}. Right at the critical point, the approach to perfect generalization follows a power law \propto t^{-1/2}. In the presence of noise, the generalization ability is degraded by an amount \propto (\sqrt{\alpha}-1)^{-1} just above \alpha = 1. This paper will appear in the NIPS-90 proceedings. To retrieve it by anonymous ftp, do the following: unix> ftp cheops.cis.ohio-state.edu # (or ftp 128.146.8.62) Name (cheops.cis.ohio-state.edu:): anonymous Password (cheops.cis.ohio-state.edu:anonymous): ftp> cd pub/neuroprose ftp> binary ftp> get krogh.generalization.ps.Z ftp> quit unix> uncompress krogh.generalization.ps unix> lpr -P(your_local_postscript_printer) krogh.generalization.ps An old-fashioned paper preprint version is also available -- send requests to hertz at nordita.dk or John Hertz Nordita Blegdamsvej 17 DK-2100 Copenhagen Denmark From pjh at compsci.stirling.ac.uk Mon Jan 28 11:07:12 1991 From: pjh at compsci.stirling.ac.uk (Peter J.B. Hancock) Date: 28 Jan 91 16:07:12 GMT (Mon) Subject: MSc in NEURAL COMPUTATION Message-ID: <9101281607.AA01507@uk.ac.stir.cs.lira> M.Sc. in NEURAL COMPUTATION: A one-year full time course at the University of Stirling, Scotland, offered by the Centre for Cognitive and Computational Neuroscience (CCCN), and the Departments of Psychology, Computing Science, and Applied Mathematics. Aims and context: The course is designed for students entering the field from any of a variety of disciplines, e.g. Computing, Psychology, Biology, Engineering, Mathematics, Physics. It aims to combine extensive practical experience with a concern for basic principles. The study of neural computation in general is combined with an in-depth analysis of vision. The first few weeks of the course form an integrated crash course in the basic techniques and ideas. During the autumn semester lectures, seminars, and specified practical exercises predominate. In the spring and summer work based on each student's own interests and abilities predominates. This culminates in a research project that can be submitted anytime between July 1 and September 1. Where work on the M. Sc. has been of a sufficiently high standard it can be converted into the first year of a Ph. D. program. Courses: Autumn: 1. Principles of neural computation. 2. Principles of vision. 3. Cognitive Neuroscience. 4. Computational and Mathematical techniques. Spring and summer: 1. Advanced practical courses, including e.g. properties, design and use of neurocomputational systems, image processing, visual psychophysics. 2. Advanced topics in neural computation, vision, and cognitive neuroscience. 3. Research project. CCCN: The CCCN is a broadly-based interdisciplinary research centre. It has a well established reputation for work on vision, neural nets, and neuropsychology. The atmosphere is informal, friendly, and enthusiastic. Research meetings are held once or twice a week during semester. Students, research staff, and teaching staff work closely together. The centre has excellent lab and office space overlooking lakes and mountains. The university is located in the most beautiful landscaped campus in Europe. It has excellent sporting facilities. Some of the most striking regions of the Scottish highlands are within easy reach. Eligibility: Applicants should have a first degree, e.g. B.A., B.Sc., in any of a variety of disciplines, e.g. Computing, Psychology, Biology, Mathematics Engineering, Physics. For further information and application forms contact: School Office, School of Human Sciences, Stirling University, Stirling FK9 4LA, SCOTLAND Specific enquiries to: Dr W A Phillips, CCCN, Psychology, Stirling University, Scotland e-mail: WAP at UK.AC.STIRLING.CS No deadline for applications is specified. From jose at learning.siemens.com Mon Jan 28 08:46:06 1991 From: jose at learning.siemens.com (Steve Hanson) Date: Mon, 28 Jan 91 08:46:06 EST Subject: IJCNN-91-SEATTLE Message-ID: <9101281346.AA28682@learning.siemens.com.siemens.com> NIPS*91 is certainly interested in Cognitive Science and Natural Language research. And I would like to point out that NIPS*90 had an 2*fold increase in submissions in that area from the year before. Steve Hanson NIPS*91 Program Chair From andercha at grieg.CS.ColoState.EDU Mon Jan 28 13:46:28 1991 From: andercha at grieg.CS.ColoState.EDU (charles anderson) Date: Mon, 28 Jan 91 11:46:28 MST Subject: Call for Papers for 1991 Machine Learning Workshop Message-ID: <9101281846.AA04012@grieg.CS.ColoState.Edu> CALL FOR PAPERS 1991 MACHINE LEARNING WORKSHOP Northwestern University June 27-29, 1991 CONSTRUCTIVE INDUCTION Selection of an appropriate representation is critical to the success of most learning systems. In difficult learning problems (e.g., protein folding, word pronunciation, relation learning), considerable human effort is often required to identify the basic terms of the representation language. Constructive induction offers a partial solution to this problem by automatically introducing new terms into the representation as needed. Automatically constructing new terms is difficult because the environment or teacher usually provides only indirect feedback, thus raising the issue of credit assignment. However, as learning systems face tasks of greater autonomy and complexity, effective methods for constructive induction are becoming increasingly important. The objective of this workshop is to provide a forum for the interchange of ideas among researchers actively working on constructive induction issues. It is intended to identify commonalities and differences among various existing and emerging approaches such as knowledge-based term construction, relation learning, theory revision in analytic systems, learning of hidden-units in multi-layer neural networks, rule-creation in classifier systems, inverse resolution, and qualitative-law discovery. Submissions are encouraged in the following topic areas: o Empirical approaches and the use of inductive biases o Use of domain knowledge in the construction and evaluation of new terms o Construction of or from relational predicates o Theory revision in analytic-learning systems o Unsupervised learning and credit assignment in constructive induction o Interpreting hidden units as constructed features o Constructive induction in human learning o Techniques for handling noise and uncertainty o Experimental studies of constructive induction systems o Theoretical proofs, frameworks, and comparative analyses o Comparison of techniques from empirical learning, analytical learning, classifier systems, and neural networks Send six copies of paper submissions (3000 word maximum) to Christopher Matheus, GTE Laboratories, 40 Sylvan Road, MS-45, Waltham MA 02254 (matheus at gte.com). Submissions must be received by February 1, 1991. Include a cover page with authors' names, addresses, phone numbers, electronic mail addresses, paper title, and a 300 (maximum) word abstract. Do not indicate or allude to authorship anywhere within the paper. Acceptance notification will be mailed by March 31, 1991. Accepted papers will be allotted four two-column pages for publication in the Proceedings of the 1991 Machine Learning Workshop. Organizing Committee: Program Committee: Christopher Matheus, GTE Laboratories Chuck Anderson, Colorado State George Drastal, Siemens Corp. Research Gunar Liepins, Oak Ridge National Lab. Larry Rendell, University of Illinois Douglas Medin, University of Michigan Paul Utgoff, University of Massachusetts From N.E.Sharkey at cs.exeter.ac.uk Mon Jan 28 13:37:58 1991 From: N.E.Sharkey at cs.exeter.ac.uk (Noel Sharkey) Date: Mon, 28 Jan 91 18:37:58 GMT Subject: references to CNLP Message-ID: <259.9101281837@entropy.cs.exeter.ac.uk> I said that people could obtain a copy of references to CNLP by writing to lyn at my address. But i forgot to enclose the address. I have had several messages now pointing out difficulties. My apologies to all. noel sharkey The addresses are: JANET: lyn at uk.ac.exeter.cs UUCP: lyn at expya.uucp BITNET: lyn at cs.exeter.ac.uk@UKACRL From andercha at grieg.CS.ColoState.EDU Mon Jan 28 17:06:28 1991 From: andercha at grieg.CS.ColoState.EDU (charles anderson) Date: Mon, 28 Jan 91 15:06:28 MST Subject: CFP Constructive Induction Workshop, Due March 1st Message-ID: <9101282206.AA04326@grieg.CS.ColoState.Edu> CALL FOR PAPERS 1991 MACHINE LEARNING WORKSHOP Northwestern University June 27-29, 1991 CONSTRUCTIVE INDUCTION Selection of an appropriate representation is critical to the success of most learning systems. In difficult learning problems (e.g., protein folding, word pronunciation, relation learning), considerable human effort is often required to identify the basic terms of the representation language. Constructive induction offers a partial solution to this problem by automatically introducing new terms into the representation as needed. Automatically constructing new terms is difficult because the environment or teacher usually provides only indirect feedback, thus raising the issue of credit assignment. However, as learning systems face tasks of greater autonomy and complexity, effective methods for constructive induction are becoming increasingly important. The objective of this workshop is to provide a forum for the interchange of ideas among researchers actively working on constructive induction issues. It is intended to identify commonalities and differences among various existing and emerging approaches such as knowledge-based term construction, relation learning, theory revision in analytic systems, learning of hidden-units in multi-layer neural networks, rule-creation in classifier systems, inverse resolution, and qualitative-law discovery. Submissions are encouraged in the following topic areas: o Empirical approaches and the use of inductive biases o Use of domain knowledge in the construction and evaluation of new terms o Construction of or from relational predicates o Theory revision in analytic-learning systems o Unsupervised learning and credit assignment in constructive induction o Interpreting hidden units as constructed features o Constructive induction in human learning o Techniques for handling noise and uncertainty o Experimental studies of constructive induction systems o Theoretical proofs, frameworks, and comparative analyses o Comparison of techniques from empirical learning, analytical learning, classifier systems, and neural networks Send six copies of paper submissions (4000 word maximum) to Christopher Matheus, GTE Laboratories, 40 Sylvan Road, MS-45, Waltham MA 02254 (matheus at gte.com). Submissions must be received by March 1, 1991. Include a cover page with authors' names, addresses, phone numbers, electronic mail addresses, paper title, and a 300 (maximum) word abstract. Do not indicate or allude to authorship anywhere within the paper. Acceptance notification will be mailed by April 30, 1991. Accepted papers will be allotted four two-column pages for publication in the Proceedings of the 1991 Machine Learning Workshop. Organizing Committee: Program Committee: Christopher Matheus, GTE Laboratories Chuck Anderson, Colorado State George Drastal, Siemens Corp. Research Gunar Liepins, Oak Ridge National Lab. Larry Rendell, University of Illinois Douglas Medin, University of Michigan Paul Utgoff, University of Massachusetts From smieja at gmdzi.uucp Mon Jan 28 10:01:13 1991 From: smieja at gmdzi.uucp (Frank Smieja) Date: Mon, 28 Jan 91 14:01:13 -0100 Subject: Vision (What's wrong with Marr's model) Message-ID: <9101281301.AA26852@gmdzi.gmd.de> I received the following suggestions for alternative (or more modern) views on the vision problem, as opposed to the Marr viewpoint. Thanks very much to those who replied. Young, D.S. (1990). Quantitative ecological optics. In Ph. Jorrand & V. Sgurev (Eds.), Artificial Intelligence IV: Methodology, Systems, Applications (pp. 423-431). Amsterdam: North-Holland ----------- A. Sloman `On designing a visual system: Towards a Gibsonian computational model of vision' Journal of Experimental and Theoretical AI 1,4, 1989 It is concerned with fairly high level design requirements for a visual system that needs to be able to cope with real-time constraints, optical information of variable quality, a multitude of different uses for vision and, and many different links between the vision sub-system and various other sub-systems. Aaron Sloman, School of Cognitive and Computing Sciences, Univ of Sussex, Brighton, BN1 9QH, England EMAIL aarons at cogs.sussex.ac.uk or: aarons%uk.ac.sussex.cogs at nsfnet-relay.ac.uk --------- D. Weinshal and S. Edelman, "Computational Vision: A Critical Review", MIT-AI Technical Report #??? (sorry!), 1989, and @article{Hild87, author = "E. C. Hildreth and C. Koch", year = "1987", journal = "Annual Reviews of Neuroscience", volume = "10", pages = "477-533", title = "The Analysis of Visual Motion: From Computation Theory to Neuronal Mechanisms" } --------- I have written two papers that call into doubt Marr & Nishihara's proposals about shape recognition being accomplished by matching object-centered, viewpoint-independent shape representations. They are: Tarr, M. J. & Pinker, S. (1989) Mental rotation and orientation-dependence in shape recognition. @i[Cognitive Psychology, 21], 233-282. Tarr, M. J. & Pinker, S. (1990) When does human object recognition use a viewer-centered recognition frame? @i[Psychological Science, 1], 253-256. --Steve Pinker ---------- From haussler at saturn.ucsc.edu Tue Jan 29 20:10:26 1991 From: haussler at saturn.ucsc.edu (David Haussler) Date: Tue, 29 Jan 91 17:10:26 -0800 Subject: problems ftping large UCSC tech report fixed (we hope) Message-ID: <9101300110.AA11994@saturn.ucsc.edu> Several people had problems printing the tech report Decision Theoretic Generalizations of the PAC Model for Neural Net and Other Learning Applications UCSC-CRL-91-02 which I announced a few days ago. We have now split it into 2 parts so that it can be printed on most printers. revised ftp instructions follow: unix> ftp midgard.ucsc.edu # (or ftp 128.114.134.15) Connected ... Name (...): anonymous Password: yourname at cs.anyuniversity.edu (i.e. your email address) (Please use your email address so we can correspond with you.) Guest login ok, access restrictions apply. ftp> cd pub/tr ftp> binary ftp> get ucsc-crl-91-02.part1.ps.Z ... 310226 bytes received in 4.4 seconds (70 Kbytes/s) ftp> get ucsc-crl-91-02.part2.ps.Z ... 277165 bytes received in 4.4 seconds (70 Kbytes/s) ftp> quit unix> uncompress ucsc-crl-91-02.part1.ps.Z unix> uncompress ucsc-crl-91-02.part2.ps.Z unix> lpr -P(your_local_postscript_printer) ucsc-crl-91-02.part1.ps unix> lpr -P(your_local_postscript_printer) ucsc-crl-91-02.part2.ps (Some other UCSC tech reports are also available. We fixed some problems that we were having with these earlier as well, notably 90-16. ftp the file INDEX to see what's there.) If you have any difficulties with the above, please send e-mail to jean at cis.ucsc.edu. DO NOT "reply" to this message, please. -David From pjh at compsci.stirling.ac.uk Tue Jan 29 10:31:50 1991 From: pjh at compsci.stirling.ac.uk (Peter J.B. Hancock) Date: 29 Jan 91 15:31:50 GMT (Tue) Subject: re MSc in Neural Computation Message-ID: <9101291531.AA05715@uk.ac.stir.cs.lira> Due to vagaries in the UK email service and the fact that 'cs' is the country code for Czechoslovakia, the email address given on the recent announcement of an MSc in neural computation at Stirling was incorrect. It should be: wap at uk.ac.stirling.compsci However, most users will probably at least have to reverse this: wap at compsci.stirling.ac.uk Apologies for any confusion and the wasted bandwidth. Peter Hancock From reggia at cs.UMD.EDU Thu Jan 31 10:56:20 1991 From: reggia at cs.UMD.EDU (James A. Reggia) Date: Thu, 31 Jan 91 10:56:20 -0500 Subject: CALL FOR PAPERS: CONNECTIONIST MODELS IN BIOMEDICINE Message-ID: <9101311556.AA12211@mimsy.UMD.EDU> CALL FOR PAPERS: The 15th Symposium on Computer Applications in Medical Care will include a Program Area Track on Connectionism, Simulation and Modeling. Submission of papers is welcomed. Papers are solicited which report on original research, system development or survey the state of the art in an aspect of this wide- ranging field. Papers in previous years have addressed such topics as modelling invertebrate nervous systems, modelling disorders of higher cortical functions, development of high-level languages for building connectionist models, and systems for medical diagnosis, among other topics. Deadline for receipt of manuscripts is April 1, 1991. The conference will be held November 17-20, 1991 in Washington, DC. For submittal forms please write: Paul D. Clayton, PhD SCAMC Program Chair, 1991 AMIA 11140 Rockville Pike Box 324 Rockville, MD 20852 or contact Gail Mutnik at mutnik at lhc.nlm.nih.gov by email. If you have questions about whether your paper would be appropriate for this conference please contact me at: Stan Tuhrim SSTMS at CUNYVM.CUNY.EDU From ff at sun8.lri.fr Thu Jan 31 12:45:46 1991 From: ff at sun8.lri.fr (ff@sun8.lri.fr) Date: Thu, 31 Jan 91 18:45:46 +0100 Subject: change in receiver name Message-ID: <9101311745.AA06011@sun3a.lri.fr> Please, could you change the subscription destination to: ----------------------------------- conx at FRLRI61.BITNET@CUNYVM.CUNY.EDU ----------------------------------- (instead of: ff at FRLRI61.BITNET@CUNYVM.CUNY.EDU ) best regards Francoise Fogelman