From Connectionists-Request at cs.cmu.edu Thu Jun 1 10:13:45 1989 From: Connectionists-Request at cs.cmu.edu (Connectionists-Request@cs.cmu.edu) Date: Thu, 01 Jun 89 10:13:45 EDT Subject: EURASIP Workshop on Neural Networks Message-ID: <17603.612713625@B.GP.CS.CMU.EDU> CALL FOR PAPERS EURASIP WORKSHOP ON NEURAL NETWORKS Sesimbra, Portugal February 15-17, 1990 The workshop will be held at the Hotel do Mar in Sesimbra, Portugal. It will take place in 1990, from February 15 morning to 17 noon, and will be sponsored by EURASIP, the European Association for Signal Processing. It will be open to participants from all countries, both from inside and outside of Europe. Contributions from all fields related to the neural network area are welcome. A (non-exclusive) list of topics is given below. Care is being taken to ensure that the workshop will have a high level of quality. Proposed contributions will be evaluated by an international technical committee. A proceedings volume will be published, and will be handed to participants at the beginning of the workshop. The number of participants will be limited to 50. Full contributions will take the form of oral presentations, and will correspond to papers in the proceedings. Some short contributions will also be accepted, for presentation of ongoing work, projects (ESPRIT, BRAIN, DARPA,...), etc. They will be presented in poster format, and will not originate any written publication. A small number of non-contributing participants may also be accepted. The official language of the workshop will be English. TOPICS: - signal processing (speech, image,...) - pattern recognition - algorithms (training procedures, new structures, speedups,...) - generalization - implementation - specific applications where NN have been proved better than other approaches - industrial projects and realizations SUBMISSION PROCEDURES: Submissions, both for long and for short contributions, will consist of (strictly) 2-page summaries. Three copies should be sent directly to the Technical Chairman, at the address given below. The calendar for contributions is as follows: Full contributions Short contributions Deadline for submission June 15, 1989 Oct 1, 1989 Notif. of acceptance Sept 1, 1989 Nov 15, 1989 Camera-ready paper Nov 1, 1989 ORGANIZING COMMITTEE General Chairman: Luis B. Almeida, INESC, Apartado 10105, P-1017 Lisboa, Codex, Portugal Phone: +351-1-544607; Fax: +351-1-525843; E-mail: {any backbone, uunet}!mcvax!inesc!lba Technical Chairman: Christian J. Wellekens, Philips Research Laboratory Brussels, Av. Van Becelaere 2, Box 8, B-1170 Brussels, Belgium Phone: +32-2-6742275; Fax: +32-2-6742299; E-mail: wlk at prlb2.uucp Technical committee: John Bridle (Royal Signal and Radar Establishment, Malvern, UK), Herve Bourlard (Intern. Computer Science Institute, Berkeley, USA), Frank Fallside (University of Cambridge, Cambridge, UK), Francoise Fogelman (Ecole de H. Etudes en Informatique, Paris, France), Jeanny Herault (Institut Nat. Polytech. de Grenoble, Grenoble, France), Larry Jackel (AT\&T Bell Labs, Holmdel, NJ, USA), Renato de Mori (McGill University, Montreal, Canada), H. Muehlenbein (GMD, Sankt Augustin, FRG). REGISTRATION, FINANCE, LOCAL ARRANGEMENTS: Joao Bilhim, INESC, Apartado 10105, P-1017 Lisboa, Codex, Portugal Phone: +351-1-545150; Fax: +351-1-525843. WORKSHOP SPONSOR EURASIP - European Association for Signal Processing CO-SPONSORS: INESC - Instituto de Engenharia de Sistemas e Computadores, Lisbon, Portugal IEEE, Portugal Section THE LOCATION: Sesimbra is a fishermens village, located in a nice region about 30 km south of Lisbon. Special transportation from/to Lisbon will be arranged. The workshop will end on a Saturday at lunch time; therefore, the participants will have the option of either flying back home in the afternoon, or staying for sightseeing for the remainder of the weekend in Sesimbra and/or Lisbon. An optional program for accompanying persons is being organized. From tds at ai.mit.edu Thu Jun 1 12:54:10 1989 From: tds at ai.mit.edu (tds@ai.mit.edu) Date: Thu, 1 Jun 89 12:54:10 edt Subject: supervised learning In-Reply-To: Alain Grumbach's message of Tue, 30 May 89 17:34:56 +0200 <8905301534.AA17056@ulysse.enst.fr> Message-ID: <8906011654.AA02920@mauriac.ai.mit.edu> I agree that there is some artificiality in the distinction between supervised and unsupervised learning. Traditionally, the distinction seems to be made based on whether the learning algorithm has available to it the desired outputs of the network. If the algorithm can be described by an energy function (such as minimal output error for Backpropagation), then supervised learning seems to require an energy function which explicitly includes the desired outputs of the network. Unsupervised learning (in some cases) can be described by energy functions which specify properties or statistics of the outputs (or sometimes the weights). However, it seems easy to imagine a continuum of energy functions between these two types. For example, a BP net which requires outputs to be close to the desired points within some tolerance, or an unsupervised algorithm designed so that the desired properties of the outputs can be satisfied only by specific output values which are actually the desired outputs. And what about an algorithm whose energy function is the mean squared distance to the desired weights themselves? (Learning in this case is very easy and fast, but not particularly interesting!) This is complicated by the fact that very often the same solution (final set of weights) can be obtained from very different algorithms with very different energy functions (imagine training a BP network to converge to a Kohonen network). So there seems to be an artificial distinction between supervised nets (gradient descent on an energy function defined by mean-squared output error) and unsupervised nets (everything else). But there does seem to be some nice intuitive idea behind this. Perhaps it is based on the difference between giving someone the right answer to a math problem and teaching them how to solve it themselves. In other words, in real life we seem to make the supervised/unsupervised distinction quite naturally. Does anyone have any ideas why this distinction is so pervasive? Terry Sanger tds at wheaties.ai.mit.edu From spencer at iris.ucdavis.edu Thu Jun 1 11:18:31 1989 From: spencer at iris.ucdavis.edu (Richard Spencer) Date: Thu, 1 Jun 89 08:18:31 pdt Subject: please remove me from the mailing list Message-ID: <8906011518.AA16749@iris> I appologize for sending this to the regular address, but I have lost the appropriate address for this kind of correspondence. In any event, I asked to be removed a month ago and the volume of mail has decreased, but it hasn't stopped completely. I don't understand that, but I would appreciate it if you could prevent it. Thank you for the effort Richard Spencer From tbaker at cse.ogc.edu Thu Jun 1 20:24:27 1989 From: tbaker at cse.ogc.edu (Thomas Baker) Date: Thu, 1 Jun 89 17:24:27 -0700 Subject: Image Databases Message-ID: <8906020024.AA01888@ogccse.OGC.EDU> I am a student doing research in image pattern recognition applications of connectionist networks. I would like to get a copy of some image databases. One database that I have seen research on is the U.S. Postal Service handwritten zip code characters. There is also a more general database of pictures, most research that I have seen uses the same pictures (i.e. Lena). Where do I get a copy of this data? The method that I would prefer is to order a tape from the group that officially distributes the data. I also have ftp access, or I can supply my own tape if somebody has the data locally. Thanks in advance for any help you can give me. Thomas Baker UUCP: ..ucbvax!tektronix!ogccse!tbaker Oregon Graduate Center CSNET: tbaker at cse.ogc.edu Beaverton, Oregon 97005 From AGB11 at PHOENIX.CAMBRIDGE.AC.UK Fri Jun 2 15:43:11 1989 From: AGB11 at PHOENIX.CAMBRIDGE.AC.UK (AGB11@PHOENIX.CAMBRIDGE.AC.UK) Date: Fri, 02 Jun 89 15:43:11 BST Subject: Supervised and Unsupervised Learning Message-ID: Here are some thoughts on the distinction between supervised and unsupervised learning. First, despite the apparent exhaustiveness of the terminology, these two types of learning are not all that there are (see below). Supervised learning means that the classes of the training instances are known and available to the learning system. Or when not applied to a classification problem, it means that particular functional values are available for the training instances (although perhaps corrupted by noise). Unsupervised learning usually refers to the case where the class labels or function values are not provided with the training instances. I agree with Terry Sanger that there can be all sorts of problems falling between these extremes, but there are still other kinds of learning tasks. I've never much liked the term unsupervised learning because (aside from the fact that together with supervised learning it misleadingly implies that there are no other kinds of learning) it seems sometimes to be understood as meaning a method that can do the same thing a supervised method can but doesn't need the supervision. My view (and I welcome comments on this) is that what is usually called unsupervised learning is a kind of supervised learning with a fixed, builtin teacher. This teacher is embodied in some principle, such as principal component analysis, clustering with a specific criterion function on clusterings, infomax, etc. So, according to this view, unsupervised learning is a more specific type of process than is supervised learning. Methods are usually not built to accomodate the possibility of some input specifying what criterion to use in the learning process. Then there are other sorts of learning tasks where the information given by the learning system's environment is not as specific as a specification of desired responses. For example, in a reinforcement learning task there are desired responses but the system is not told what they are: it has to search for them (in addition to the search in parameter space for weights to remember the results of previous searches). For example, Widrow, Gupta, and Maitra (Systems Man and Cybernetics, vol 5, 1973, pp. 455-465) discuss "learning with a critic". Rewards and punishments are logically different from signed errors or desired responses. Lots of other people (including me) have discussed this type of learning task. In thinking about supervised and unsupervised learning tasks as these terms are used in everyday language, I think it is greatly misleading to assume that the technical meanings of these terms adequately characterize the kind of learning we see in animals. Usually, it seems to me, the kind of supervision we see really is a process whereby a sequence of problem-solving tasks are presented, often with graded difficulty, and for each the learning system faces a complex learning control task. Of course, to solve these tasks, various parts of the sytem will probably be facing unsupervised, supervised, reinforcement, and probably other kinds of tasks that are all worked on together. Similarly, unsupervised learning used in the vernacular seems to mean a process where all these things go on, except the system itself in creating the sequence of problem-solving tasks (cf. Mitchell's LEX learning system). Finally, I think it is important to distinguish carefully between learning tasks and learning algorithms or procedures (although I haven't been particularly careful about this above). Tasks are characterized by the kinds of information the system is allowed to get, how much it has at the start, what the objective is, etc. A specific learning algorithm is more-or-less capable of achieving various objectives in various kinds of tasks. Usually a real application encompasses lots of different learning tasks depending on where you draw the line between the learning system and the rest. A. Barto agb11 at phx.cam.ac.uk From tp at ai.mit.edu Fri Jun 2 15:32:32 1989 From: tp at ai.mit.edu (Tomaso Poggio) Date: Fri, 2 Jun 89 15:32:32 EDT Subject: Image Databases In-Reply-To: Thomas Baker's message of Thu, 1 Jun 89 17:24:27 -0700 <8906020024.AA01888@ogccse.OGC.EDU> Message-ID: <8906021932.AA02272@rice-chex.ai.mit.edu> Could you let me know what you learn from your request? I am also interested in obtaining similar kinds of data... From hinton at ai.toronto.edu Fri Jun 2 15:45:52 1989 From: hinton at ai.toronto.edu (Geoffrey Hinton) Date: Fri, 2 Jun 89 15:45:52 EDT Subject: supervised learning In-Reply-To: Your message of Tue, 30 May 89 11:34:56 -0400. Message-ID: <89Jun2.154559edt.11075@ephemeral.ai.toronto.edu> I think the current useage of the terms "supervised" and "unsupervised" learning is pretty much OK. To try to change the meanings of these terms now would cause even more confusion. There is a natural and important distinction between data sets (wherever they come from) that consist of Input-output pairs where the task is to predict the output given the input, and data-sets that simply consist of an ensemble of vectors where the task (if its specified at all) is typically to find natural clusters, or a compact code, or a code with independent components. Naturally there are tricky cases. One of these is when we create a "multi-completion" task by trying to predict each part of the input from all the other parts. We are really turning an unsupervised task into a whole set of supervised tasks. Another problem arises when we have some LOCAL objective function (internal to the network) that acts as a teacher. If we backpropagate derivatives from this LOCAL function, are we doing supervised or unsupervised learning? It might be reasonable to say that we are doing learning that is locally supervised but globally unsupervised (i.e. the local supervision information is not derived from more global supervision information that is supplied with the data). Its worth noting that the statistics literature makes a very similar basic distinction. It uses the term "discriminant" analysis for supervised learning and "cluster analysis" for unsupervised learning (or the subset of unsupervised learning that they know how to do.) Geoff From honavar at cs.wisc.edu Sat Jun 3 18:21:22 1989 From: honavar at cs.wisc.edu (Vasant Honavar) Date: Sat, 3 Jun 89 17:21:22 -0500 Subject: Supervised learning Message-ID: <8906032221.AA00949@goat.cs.wisc.edu> A number of "supervised" learning tasks can be formulated as learning tasks in which the "feedback" comes from a different sensory modality - e.g., the feedback for a language learning may come from a visual scene. Learning in the two modalities may be initiated through a process of "bootstrapping". Feedback can be viewed as coming from simply another input stream. Instead of working with one input stream (as is typically the case with most backprop-like schemes), the system learns by forming associations between two or more input streams. In this case, it is hard to argue that the learning is any more "supervised" than in any of the so-called "unsupervised" methods. This is conceptually quite similar to the self-supervised methods (Hinton, 1987; "Connectionist Learning Procedures"). It therefor appears useful to view the various learning schemes as part of a continuum: Schemes requiring very specific feedback lie (e.g., the desired output) on one extreme; those requiring no feedback (all of learning is simply storing some abstractions of the input) lie on the other extreme; and a host of schemes using various rather diffuse and non-specific forms of feedback (and possibly feedback that is hidden in the form of local objective functions, information- theoretic "constraints", the locus of interactions between units, etc.) lie in the middle. Vasant Honavar From gmdzi!joerg%zsv at uunet.UU.NET Mon Jun 5 07:36:31 1989 From: gmdzi!joerg%zsv at uunet.UU.NET (Joerg Kindermann) Date: Mon, 5 Jun 89 13:36:31 +0200 Subject: speech recognition data Message-ID: <8906051136.AA04744@zsv.gmd.de> I would like to get any pointers to *mass* data which can be used for speech recognition tasks (talker independent, both isolated words and fluent speech). Please send e-mail, if there is enough interest, I'll post a summary of this request. Thanks in advance Joerg. address: Dr. Joerg Kindermann Gesellschaft fuer Mathematik und Datenverarbeitung (GMD) Schloss Birlinghoven Postfach 1240 D-5205 St. Augustin 1 WEST GERMANY e-mail: joerg at gmdzi.uucp From der at elrond.Stanford.EDU Mon Jun 5 12:20:48 1989 From: der at elrond.Stanford.EDU (Dave Rumelhart) Date: Mon, 5 Jun 89 09:20:48 PDT Subject: Supervised learning Message-ID: It has seemed to me that the supervised/unsupervised distinction is insufficiently fine-grained and is wrongly associated with learning methods. (For example, backprop is often thought of as supervised whereas, say, Hebbian rules are thought of as unsupervised.) We ought to distinguish between kinds of learning tasks on the one hand and learning rules on the other. I have distinguished four learning tasks: (1) PATTERN ASSOCIATION in which the goal is to learn a series of I/O pairs. In this case the networks learns to be the function which maps input elements to their corresponding output elements. This is usually what people have in mind with supervised learning. (2) AUTOASSOCIATION in which the goal is to store a pattern so that, essentially, any part of the pattern can be used to retrieve the whole pattern. This is the connectionist implementation of content-addressible memory. This is difficult to classify in the supervised/unsupervised dimension. On the one hand it involves a single input item and hence should be unsupervised, on the other hand, the network is told exactly what to store, so it should be viewed as unsupervised. (3) REGULARITY-DETECTION in which the goal is to discover statistical regularities (such as clusters or featural decompositions etc.) in the input patterns. This would seem to be the prototype unsupervised case. (4) REINFORCEMENT LEARNING in which the goal is to learn a set of actions which either maximize some postive reinforcement variable or minimize some negative reinforcement variable or both. This to is difficult to classify on the supervised/unsupervised dimension. It would appear that it is supervised because the environment is guiding its behavior. On the other hand it is unsupervised in as much as its free to make whatever response it wishes so long as it maximizes reinforcement. Perhaps partially supervised would due for this case. More important than this classification, it is important to realize that these categories are (nearly) orthogonol to the learning rules employed. Thus, we have (1) CORRELATIONAL (Hebbian) LEARNING which can be employed for pattern association tasks, autoassociation tasks, regularity detection tasks and (perhaps) reinforcement learning tasks. Similarly, (2) ERROR-CORRECTION LEARNING (backprop, Widrow-Hoff, Perceptron, etc.) can be employed for each of the classes. Pattern association is the obvious application, but it is obvious how it can be employed for autoassociation tasks. Error-correction methods can be used to build an autoencoder and thereby be used to extract features or principle components of the input patterns and thereby act as a regularity detector. Backprop, for example, can also be used in reinfoprcement learning situations. (3) COMPETITIVE LEARNING mechanisms can also be used for at least regularity detection and for autoencoding and (probably) for possibly for pattern association. (4) BOLTZMANN LEARNING is just as versitile as error correction learning. The point is simply that a cross-classification of learning task by learning rule is required to classify the kinds of connectionist learning applications we see. der From mehra at ptolemy.arc.nasa.gov Mon Jun 5 13:37:32 1989 From: mehra at ptolemy.arc.nasa.gov (Pankaj Mehra) Date: Mon, 5 Jun 89 10:37:32 PDT Subject: Supervised learning Message-ID: <8906051737.AA25680@ptolemy.arc.nasa.gov> The difference between "supervised" learning and reinforcement learning is largely due to the nature of feedback. To borrow a term from Ron Williams, the feedback is "prescriptive" in the former type and "evaluative" in the latter. Yet another distinction, first noted by Barto, Sutton, and Anderson, in their 1983 paper, is due to the synchronicity and delay in feedback. If feedback is delayed, the learner needs "memory" of recent decisions, and a temporal credit assignment mechanism (Sutton, 1988) to distribute the feedback among memorized decisions. Asynchronicity in feedback can (roughly) be defined as the property of a training environment so that it cannot be determined precisely which output of the network will be followed by reinforcement or correction. IMHO, this is an important difference between knowledge-based and connectionist learning systems. The AI model of learning (Dietterich, 1981) can handle only synchronous delays in feedback. The reason why ANSs can handle asynchronous delays in feedback is because their architecture is inherently asynchronous. - Pankaj {mehra at cs.uiuc.edu} References: (Sutton,88) Machine Learning, vol. 3, pp. 9-44 (Dietterich,Buchanan,81) The Role of Critic in Learning Systems, Stanford TR STAN-CS-81-891 (Barto et al.,83) IEEE trans. Sys. Man Cyb., vol. SMC-13, pp. 834-846 From neural!yann at att.att.com Mon Jun 5 10:47:33 1989 From: neural!yann at att.att.com (neural!yann@att.att.com) Date: Mon, 05 Jun 89 10:47:33 -0400 Subject: supervised learning In-Reply-To: Your message of Fri, 02 Jun 89 15:45:52 -0400. Message-ID: <8906051445.AA27851@neural.UUCP> I entirely agree with Andy Barto when he says that unsupervised learning is a special type of supervised learning where the objective function is hidden or implicit. From the ALGORITHMIC point of view, the important thing is that we just need to look for good supervised learning procedures. Interesting unsupervised procedures can usually be derived from supervised ones. Look at how back-prop can be stripped down to perform principal component analysis (I am not saying that ALL unsupervisd procedures can be derived from supervised procedures). Of course from the TASK point of view, it looks like there is a qualitative difference between supervised and unsupervised learning, but they are just different ways of using the same general principles. I don't think the difference is relevant if you are interested in the general principles. Yann From lyn at CS.EXETER.AC.UK Wed Jun 7 13:36:25 1989 From: lyn at CS.EXETER.AC.UK (Lyn Shackleton) Date: Wed, 7 Jun 89 13:36:25 BST Subject: Connection Science Message-ID: <2628.8906071236@exsc.cs.exeter.ac.uk> ANNOUNCEMENT Issue 1. of the new journal CONNECTION SCIENCE has just gone to press and Issue 2. will follow shortly. The editors are very pleased with the response they have received and would welcome more high quality submissions or theoretical notes. VOLUME 1 NUMBER 1 CONTENTS Michael C Mozer & Paul Smolensky 'Using Relevance to Reduce Network Size Automatically' James Hendler 'The Design and Implementation of Symbolic Marker-Passing Systems' Eduardo R Caianello, Patrik E Eklund & Aldo G S Ventre 'Implementations of the C-Calculus' Charles P Dolan & Paul Smolensky 'Tensor Product Production System: A Modular Architecture and Representation' Christopher J Thornton 'Learning Mechanisms which Construct Neighbourhood Representations' Ronald J Williams & David Zipser 'Experimental Analysis of the Real-Time Recurrent Learning Algorithm' Editor: Dr NOEL E SHARKEY, Centre for Connection Science, Dept of Computer Science, University of Exeter, UK Associate Editors: Andy CLARK (University of Sussex, Brighton, UK) Gary COTTRELL (University of California, San Diego, USA) James A HENDLER (University of Maryland, USA) Ronan REILLY (St Patrick's College, Dublin, Ireland) Richard SUTTON (GTE Laboratories, Waltham, MA, USA) FORTHCOMING IN VOLUMES 1 & 2 Special Issue on Natural Language, edited by Ronan Reilly & Noel Sharkey Special Issue on Hybrid Symbolic/Connectionist Systems, edited by James Hendler For further details please contact. lyn shackleton (assistant editor) Centre for Connection Science JANET: lyn at uk.ac.exeter.cs Dept. Computer Science University of Exeter UUCP: !ukc!expya!lyn Exeter EX4 4PT Devon BITNET: lyn at cs.exeter.ac.uk.UKACRL U.K. Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa02160; 8 Jun 89 10:56:34 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa11156; 8 Jun 89 10:50:51 EDT Received: from PRINCETON.EDU by CS.CMU.EDU; 8 Jun 89 10:48:22 EDT Received: from clarity.Princeton.EDU by Princeton.EDU (5.58+++/2.17) id AA15362; Thu, 8 Jun 89 09:09:45 EDT Received: by clarity.Princeton.EDU (4.0/1.81) id AA08666; Thu, 8 Jun 89 09:14:22 EDT Date: Thu, 8 Jun 89 09:14:22 EDT From: harnad at Princeton.EDU Message-Id: <8906081314.AA08666 at clarity.Princeton.EDU> To: connectionists at CS.CMU.EDU Subject: Categorization and Supervision CATEGORIZATION AND SUPERVISION The question of the definition and nature of supervised vs. unsupervised learning touches on some analogous points in the theory of categorization. Perhaps some of the underlying logic is more obvious in the case of categorization: There are two kinds of categorization tasks, one of which I've dubbed "imposed" categorization and the other "ad lib" categorization (otherwise known as similarity judgment). In both kinds of categorization task a set of inputs is sorted into a set of categories, but in imposed categorization there is feedback about whether the sorting is correct or incorrect (and when it is incorrect, there is usually also information indicating which category would have been correct), whereas in ad lib categorization there is no feedback (and often no nonarbitrary criterion for "correctness" either): The subject is simply asked to sort the inputs into categories as he sees fit. (Sometimes even less is asked for in ad lib categorization: The subject just rates pairs of inputs on how similar he judges them, and then the "categorization" is inferred by some suitable multidimensional statistical analysis such as scaling or cluster analysis.) The logical structure and informational demands of imposed and ad lib categorization are very different. Although both are influenced by learning, it is clear that imposed categorization (except if it is inborn) depends critically on learning through feedback from the external consequences of MIScategorization. Inputs are provisionally sorted and labeled, the consequences of the sorting and labeling somehow matter to the organism, and the feedback from incorrect sorting and labeling gives rise to a learning process that converges eventually on correct sorting and labeling. The categories are "imposed" by the environmental consequences of miscategorization. The category "label" can of course be any operant response, so imposed categorization is the paradigm for learned operant behaviors as well as many evolved sensorimotor adaptations (i.e. Skinner's "selection by consequences," although Skinner of course provides no mechanism for the learning, whereas connectionism seems to have some candidates). Ad lib categorization, on the other hand, does not depend directly on learning (although it is no doubt influenced by the outcome of prior imposed category learning). Logically, all it depends on is passive exposure to the inputs; by definition there are no consequences arising from miscategorization (if there are, then we are back into supervised learning). One last point: Learning an imposed categorization is usually dependent on finding the stimulus features that will reliably sort the imputs into the correct categories. An imposed categorization problem is "hard" to the degree to which this is not a trivial task. (Put another way, it depends on how "underdetermined" the categorization problem is.) A trivial imposed categorization problem is one in which the ad lib categorization happens to be a solution to the imposed categorization as well, i.e., the perceived similarity structure of the input set is already such that it "relaxes" into the right sorting through mere exposure, without the need for feedback. I think this last point may be at the heart of some of the misunderstandings about supervised vs. unsupervised learning: Logically speaking, there can be no "correct" or "incorrect" categorization in ad lib categorization; the "correctness" criterion is just in the mind of the experimenter. But if the hard work (i.e., finding and weighting the features that will reliably sort the inputs "correctly") has already been done by prior imposed category learning (or the evolution of our sensorimotor systems) then a given categorization problem may spuriously give the appearance of having been solved by "unsupervised" learning alone. In reality, however, the "solution" will have been well-prepared by prior learning and evolution, since, apart from the mind of the experimenter, "correctness" is DEFINED by the consequences of miscategorization, i.e., by supervision. Reference: Harnad, S. (Ed.) (1987) Categorical Perception: The Groundwork of Cognition. NY: Cambridge University Press. Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa09828; 8 Jun 89 18:43:21 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa15405; 8 Jun 89 15:44:14 EDT Received: from EE.ECN.PURDUE.EDU by CS.CMU.EDU; 8 Jun 89 15:42:42 EDT Received: by ee.ecn.purdue.edu (5.61/1.18jrs) id AA11580; Thu, 8 Jun 89 14:42:19 -0500 Message-Id: <8906081942.AA11580 at ee.ecn.purdue.edu> To: connectionists at CS.CMU.EDU Subject: TR available Date: Tue, 06 Jun 89 17:05:00 EST From: Manoel Fernando Tenorio Sender: tenorio at ee.ecn.purdue.edu Bcc:neuron-request at hplabs.hp.com -------- The Tech Report below will be available by June, 15. Please do not reply to this posting. Send all you requests to jld at ee.ecn.purdue.edu Self Organizing Neural Network for Optimum Supervised Learning Manoel Fernando Tenorio Wei-Tsih Lee School of Electrical Engineering School of Electrical Engineering Purdue University Purdue University W. Lafayette, IN. 47907 W. Lafayette, IN. 47907 tenorio at ee.ecn.purdue.edu lwt at ed.ecn.purdue.edu Summary Current neural network algorithms can be classified by the following characteristics: the architecture of the network, the error criteria used, the neuron transfer function, and the algorithm used during learning. For example: in the case of back propagation, one would classify the algorithm as a fixed architecture (feedforward in most cases), using a MSE criteria, and a sigmoid function on a weighted sum of the input, with the Generalized Delta Rule performing a gradient descent in the weight space. This characterization is important in order to assess the power of such algorithms from a modeling viewpoint. The expressive power of a network is intimately related with these four features. In this paper, we will discuss a neural network algorithm with noticeably different characteristics from current networks. The Self Organizing Neural Network (SONN) [TeLe88] is an algorithm that through a search process creates the network necessary and optimum in the sense of performance and complexity. SONN can be classified as follows. The network architecture is constructed through a search using Simulated Annealing (SA),and it is optimum in that sense. The error criteria used is a modification of the Minimum Description Length Criteria called the Structure Estimation Criteria (SEC); it takes into account both the performance of the algorithm and the complexity of the structure generated. The neuron transfer function is individually chosen from a pool of functions, and the weights are adjusted during the neuron creation. This function pool can be selected with a priori knowledge of the problem, or simply use a class of non-linearities shown to be general enough for a wide variety of problems. Although the algorithm is stochastic in nature (SA), we show that its performance is extremely high both in comparative and absolute terms. In [TeLe88], we have used SONN as an algorithm to identify and predict chaotic series, particularly the Mackey-Glass equation [LaFa87, Mood88] was used. For comparison, the experiments of using Back Propagation for this problem were replicated under the same computational environment. The results indicated that for about 10% of the computational effort, the SONN delivered a 2.11 times better model (normalized RMSE). Some inherited aspects of the algorithm are even more interesting: there were 3.75 times less weights, 15 times less connections, 6.51 times less epochs over the data set, and only 1/5 of the data was fed to the algorithm. Furthermore, the algorithm generates a symbolic representation of the network which can be used to substitute it, or be used for the analysis of the problem. ****************************************************************************** We have further developed the algorithm, and although not part of the report above, it will be part of a paper submitted to NIPS'89. There, some major improvements on the algorithm are reported. The same chaotic series problem can now run with 26.4 less epochs over the data set that BP, and have generated the same model in about 18.5 seconds of computer time. (This is down from 2 CPU hours in a Gould NP1 Powernode 9080). Performance on a Sun 3-60 was sightly over 1 minute. These performance figures include the use of an 8 times larger function pool; the final performance now independs of the size of the pool. Other aspects of the algorithm are also important considering. Because of its stochastic nature, no two runs of the algorithm should be the same. This can become a hindrance if a suboptimal solution is desired, since at every run the set of suboptimal models can be different. A report on modifications of the original SONN to run on an A* search are presented. Since the algorithm generates partial structures at each iteration, the learning process is only optimized for the structure presently generated. If such substructure is used as a part of a larger structure, then no provision is made to readjust its weights making the final model slightly stiff. A provision for melting the structure (parametric readjustment) is also discussed. Finally, the combination of symbolic processing with this numerical method can lead to construction of AI-NN based methods for supervised and unsupervised learning. The ability of SONN to take symbolic constraints and produce symbolic information can make such a system possible. Implications of this design are also explored. [LaFa87] - Alans Lapedes and Robert Farber, How Neural Networks Work, TR LA-UR-88-418, Los Alamos, 1987. [Mood88] - J. Moody, Fast Learning in Multi-Resolution Hierarchies, Advances in Neural Information Processing Systems, D. Touresky, Ed., Morgan Kaufmann, 1989 (NIPS88). [TeLe88] - M. F. Tenorio and W-T Lee, Self Organizing Neural Networks for the Identification Problem, Advances in Neural Information Processing Systems, D. Touresky, Ed., Morgan Kaufmann, 1989 (NIPS88). Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa11898; 8 Jun 89 22:28:55 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa18227; 8 Jun 89 18:49:26 EDT Received: from EPHEMERAL.AI.TORONTO.EDU by CS.CMU.EDU; 8 Jun 89 18:47:03 EDT Received: from localhost (stdin) by ephemeral.ai.toronto.edu with SMTP id 11256; Thu, 8 Jun 89 18:46:41 EDT To: harnad at PRINCETON.EDU cc: connectionists at CS.CMU.EDU Subject: Re: Categorization and Supervision In-reply-to: Your message of Thu, 08 Jun 89 09:14:22 -0400. Date: Thu, 8 Jun 89 18:46:27 EDT From: Geoffrey Hinton Message-Id: <89Jun8.184641edt.11256 at ephemeral.ai.toronto.edu> Steve Harnad's message about supervised versus unsupervised learning makes an apparently plausible point that is DEEPLY wrong. He says "Logically speaking, there can be no "correct" or "incorrect" categorization in ad lib (his term for "unsupervised") categorization; the "correctness" criterion is just in the mind of the experimenter. But if the hard work (i.e., finding and weighting the features that will reliably sort the inputs "correctly") has already been done by prior imposed category learning (or the evolution of our sensorimotor systems) then a given categorization problem may spuriously give the appearance of having been solved by "unsupervised" learning alone. In reality, however, the "solution" will have been well-prepared by prior learning and evolution, since, apart from the mind of the experimenter, "correctness" is DEFINED by the consequences of miscategorization, i.e., by supervision." The mistake in this line of reasoning is as follows: Using the Kolmogorov notion of complexity, we can distinguish between good and bad models of some data without any additional teacher. A good model is one that has low kolmogorov complexity and fits the data closely (there is always a data-fit vs model-complexity trade-off). Clustering data into categories is just one particularly tractable way of modelling data. Naturally, ideas based on Kolmogorov complexity are easiest to apply if we start with a restricted class of possible models (so there is still plenty of room for evolution to be helpful). But restricting the class of models is not the same as implicitly saying which particular models (i.e. specific categorizations) are correct. For example, we could insist on modeling some data as having arisen from a mixture of gaussians (one per cluster), but this doesnt tell us which data to put in which cluster, or how many clusters to use. For that we need to trade-off the complexity of the model (number of clusters) against the data-fit. Cheeseman (and others) have shown that this approach can be made to work nicely in practice. Geoff Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa02050; 9 Jun 89 4:54:31 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa22494; 9 Jun 89 1:24:46 EDT Received: from PRINCETON.EDU by CS.CMU.EDU; 9 Jun 89 01:23:14 EDT Received: from clarity.Princeton.EDU by Princeton.EDU (5.58+++/2.17) id AA19296; Fri, 9 Jun 89 01:23:15 EDT Received: by clarity.Princeton.EDU (4.0/1.81) id AA17091; Fri, 9 Jun 89 01:28:02 EDT Date: Fri, 9 Jun 89 01:28:02 EDT From: harnad at Princeton.EDU Message-Id: <8906090528.AA17091 at clarity.Princeton.EDU> To: connectionists at CS.CMU.EDU Subject: Re: Categorization and Supervision Geoff Hinton wrote: >> Steve Harnad's message about supervised versus unsupervised learning makes an >> apparently plausible point that is DEEPLY wrong. [Harnad] says >> "Logically speaking, there can be no "correct" or "incorrect" categorization in ad lib (his term for "unsupervised") categorization..." >> The mistake in this line of reasoning is as follows: Using the Kolmogorov >> notion of complexity, we can distinguish between good and bad models of some >> data without any additional teacher. A good model is one that has low >> kolmogorov complexity and fits the data closely... I am afraid Geoff has not understood my point. I was not speaking about algorithms or models but about human categorization performance and its constraints. There may well be a "model" or internal criterion or constraint according to which input can be preferentially categorized in a particular way, but that still has nothing to do with the correctness or incorrectness of the categorization, which, as a logical matter, can only be determined by some external consequence of categorizing INcorrectly. That's what categorizing correctly MEANS. Otherwise "categorization" and "correctness" are just figures of speech (and solipsistic ones, at that). My point about imposed vs. ad lib categorization (which is not only plausible, but, till grasped and refuted on its own terms, stands, uncorrected) was based purely on external performance considerations, not model-specific internal ones. I think that the supervised/unsupervised learning distinction has given rise to misunderstandings precisely because it equivocates between external and internal considerations. I focused on the external question of what the "correctness" of a categorization really consists in so as to highlight some of the logical, methodological and informational features of categorization, and indeed of learning in general. It is a purely logical point that, even if arrived at by purely internal, unsupervised means, the "correctness" of a "correct" human categorization must be based on some external performance constraint, and hence at least a potential "supervisor." Otherwise the "correctness" is merely in the mind of the theorist, or the interpreter (or the astrologist, if the magical number happens to be 12). Now, that simple point having been made, I might add that whereas I do not find it implausible that SOME categorization problems (for which the requisite potential supervision from the external consequences of miscategorization must, I continue to insist, exist in principle) might nevertheless be solved through internal constraints alone, with no need for any actual recourse to the external supervision, I find it highly implausible that ALL, MOST, or even MANY categorization problems should be soluble that way -- and certainly not the ones I called the "hard" (underdetermined) categorization problems. Connectionism should not be TOO ambitious. It's a proud enough hope to aspire to be THE method that reliably finds invariant features in generalized nontrivial induction WITH supervision -- without going on to claim to be able to do it all with your eyes closed and your hands tied behind your back... Stevan Harnad Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa07681; 9 Jun 89 13:45:13 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa03720; 9 Jun 89 13:39:42 EDT Received: from GORT.CS.BUFFALO.EDU by CS.CMU.EDU; 9 Jun 89 13:37:27 EDT Received: from sybil.cs.Buffalo.EDU by gort.cs.Buffalo.EDU (5.59/1.1) id AA10660; Fri, 9 Jun 89 13:37:19 EDT Received: by sybil.cs.Buffalo.EDU (4.12/1.1) id AA06218; Fri, 9 Jun 89 13:38:32 edt Date: Fri, 9 Jun 89 13:38:32 edt From: Arun Jagota Message-Id: <8906091738.AA06218 at sybil.cs.Buffalo.EDU> To: connectionists at CS.CMU.EDU Subject: Re: Supervised/Unsupervised Learning Steve Harnad wrote : >It is a >purely logical point that, even if arrived at by purely internal, >unsupervised means, the "correctness" of a "correct" human categorization >must be based on some external performance constraint, and hence at >least a potential "supervisor." Otherwise the "correctness" is merely >in the mind of the theorist, or the interpreter (or the astrologist, if >the magical number happens to be 12). There are times when an organism might wish to categorize events (in the input space) for internal reasons only. The external correctness of the categorization may not be important (there may even be none), as long as there is an internal consistency of classification and recognition. >Now, that simple point having been made, I might add that whereas I do >not find it implausible that SOME categorization problems (for which >the requisite potential supervision from the external consequences of >miscategorization must, I continue to insist, exist in principle) might >nevertheless be solved through internal constraints alone, with no >need for any actual recourse to the external supervision, I find it >highly implausible that ALL, MOST, or even MANY categorization problems >should be soluble that way There are situations for which I don't necessarily think of categorization as a problem that has to be solved, rather a process that has to be performed. I think there are instances when the exact nature of the categories is not as important as the process itself (as an aid to event recognition/content addressibility). Consider a dictionary: We could clasify words a) lexically sorted b) By word-size c) By a hash function (sum of all ASCII character codes mod constant) etc The choice of the categorization to use is determined more by what we want to use it for (retrieval etc) than by any issues of its external correctness. I don't wish to overstate the case for such an unsupervised learning, but I think it deserves to be studied as an independent task in it's own right. Arun Jagota (jagota at cs.buffalo.edu) Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa13460; 9 Jun 89 19:50:58 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa05685; 9 Jun 89 15:02:51 EDT Received: from HELIOS.NORTHEASTERN.EDU by CS.CMU.EDU; 9 Jun 89 15:00:28 EDT Received: from corwin.ccs.northeastern.edu by helios.northeastern.edu id aa14361; 9 Jun 89 15:00 EDT Received: by corwin.CCS.Northeastern.EDU (5.51/SMI-3.2+CCS-main-2.6) id AA10984; Fri, 9 Jun 89 14:58:44 ADT Date: Fri, 9 Jun 89 14:58:44 ADT From: steve gallant Message-Id: <8906091758.AA10984 at corwin.CCS.Northeastern.EDU> To: connectionists at CS.CMU.EDU Subject: Re: Supervised/Unsupervised and Autoassociative learning Here's my suggestion for a taxonomy of learning problems: I. Supervised Learning: correct activations given for output cells for each training example A. Hard Learning Problems: network structure is fixed and activations are not given for intermediate cells B. Easy Learning Problems: correct activations given for intermediate cells OR network structure allowed to be modified OR single-cell problem II. Unsupervised Learning: correct activations not given for output cells There are further subdivisions for learning ALGORITHMS that can cut across the above PROBLEM classes. For example: 1. One-Shot Learning Algorithms: each training example is looked at at most one time 2. Iterative Algorithms: training examples may be looked at several times Autoassociative Algorithms are a special case of IB above because each output cell can be trained independently of the other cells, reducing the problem to single-cell supervised learning. However these models have different dynamics, in that output values are copied to input cells for the next iteration. Most learning algs for autoassociative networks are one-shot algorithms (eg. linear autoassociator, BSB, Hopfield nets). Of course iterative algorithms can be used to increase the capacity of autoassociative models (eg. perceptron learning, pocket alg.). We could also add intermediate cells to an autoassociative network to allow arbitrary sets of patterns to be stored. Sorry if you got this message twice, but I don't think the first try made it. Steve Gallant Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa03549; 9 Jun 89 22:18:27 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa08388; 9 Jun 89 18:09:32 EDT Received: from PRINCETON.EDU by CS.CMU.EDU; 9 Jun 89 18:06:53 EDT Received: from clarity.Princeton.EDU by Princeton.EDU (5.58+++/2.17) id AA15388; Fri, 9 Jun 89 18:06:49 EDT Received: by clarity.Princeton.EDU (4.0/1.81) id AA24806; Fri, 9 Jun 89 18:11:32 EDT Date: Fri, 9 Jun 89 18:11:32 EDT From: harnad at Princeton.EDU Message-Id: <8906092211.AA24806 at clarity.Princeton.EDU> To: connectionists at CS.CMU.EDU Subject: Re: Unsupervised Category Learning Arun Jagota wrote: >> an organism might wish to categorize events (in the input space) for >> internal reasons only. The external correctness of the categorization >> may not be important (there may even be none), as long as there is an >> internal consistency of classification and recognition... I think there >> are instances when the exact nature of the categories is not as >> important as the process itself (as an aid to event recognition/content >> addressibility). Consider a dictionary: We could classify words a) >> lexically sorted b) By word-size c) By a hash function... The choice of >> the categorization to use is determined more by what we want to use it >> for (retrieval etc) than by any issues of its external correctness. I think there is a confusion of phenomena here, including (1) organisms' adaptive behavior in their environments, (2) intelligent machines we use for various purposes, and (3) something like finger painting or playing with tarot cards. I have no particular interest in doing or in modeling (3) (categorizing events for "internal reasons only"), and I don't think it's a well-defined problem. (1) is clearly "supervised" by the external consequences for survival and reproduction arising from how the organism sorts and responds to classes of objects, events and states of affairs in the world. (2) could in principle be governed by machine-internal constraints only, but the "usefulness" of the categories (e.g., word classifications) to US is again clearly dependent on external consequences (to US), exactly as in (1). Perhaps an example will give a better idea of the distinction I'm emphasizing: Suppose you had N "objects." Consider the many ways you could sort and label them: By size, by shape, by color, by function, by age, as natural or synthetic, living or nonliving, etc. In each case, the name of the same "object" changes, as does the membership of the category to which it is assigned. With sufficient imagination an enormous number of possible categorizations could be generated for the same set of objects. What makes some of those categorizations "correct" (or, equally important, incorrect) and others arbitrary? The answer is surely related to why it is that we bother to categorize at all: Sorting (and especially mis-sorting) objects some ways (e.g., as edibles vs inedibles) has important consequences for us, whereas sorting them other ways (e.g., bigger/smaller than a breadbox, moon in Aquarius) does not. The consequences do not depend on "internal" considerations but on external ones, so I continue to doubt that unsupervised internal constraints can account for much that is of interest in human category learning. Stevan Harnad Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa05838; 10 Jun 89 4:18:33 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa12044; 9 Jun 89 23:42:04 EDT Received: from UICSRD.CSRD.UIUC.EDU by CS.CMU.EDU; 9 Jun 89 23:10:20 EDT Received: from s16.csrd.uiuc.edu by uicsrd.csrd.uiuc.edu with SMTP (5.61+/IDA-1.2.8) id AA16942; Fri, 9 Jun 89 22:10:11 -0500 Received: by s16.csrd.uiuc.edu (3.2/9.2) id AA08968; Fri, 9 Jun 89 22:10:07 CDT Date: Fri, 9 Jun 89 22:10:07 CDT From: George Cybenko Message-Id: <8906100310.AA08968 at s16.csrd.uiuc.edu> To: connectionists at CS.CMU.EDU Re: Unsupervised learning What do people mean by "data" in unsupervised learning? Surely it is more than just bit strings....if you get "data" in the form of bit strings don't you have to know what those bit strings mean? Does the bit string represent an image, a vector of 32 bit floating point numbers in VAX or MC68000 arithmetic, an integer, or an ASCII representation of some characters? It seems to me that the interpretation of the bit string in this way determines a metric that is imposed by the semantics of the problem from which the data came. That metric qualifies as a form of "supervision". Moreover, any discussion of Kolmogorov complexity vs. modeling errors requires some notion of error, hence some notion of a metric imposed on the data. Put another way, I don't think that there can be a useful "unsupervised" learning procedure that accepts only bit strings as input with no interpretations of what those bit strings mean. I suspect that there might be a canonical format into which all problems can be transformed that would usually give meaningful classifications but then it would be argued that the transformation is a type of supervision. In fact, this question of how to interpret and represent input data is a key criticism of connectionist modeling as given by many neuro- biologists, from my understanding of that debate. George Cybenko that debate. Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa05853; 10 Jun 89 4:27:32 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa12632; 10 Jun 89 0:04:04 EDT Received: from THINK.COM by CS.CMU.EDU; 10 Jun 89 00:02:41 EDT Return-Path: Received: from kulla.think.com by Think.COM; Sat, 10 Jun 89 00:02:35 EDT Received: by kulla.think.com; Sat, 10 Jun 89 00:01:33 EDT Date: Sat, 10 Jun 89 00:01:33 EDT From: singer at Think.COM Message-Id: <8906100401.AA04047 at kulla.think.com> To: connectionists at CS.CMU.EDU Subject: Sparse vs. Dense networks In general sparsely connected networks run faster than densely connected ones. Ignoring this advantage, are there any theoretical benefits to sparsely connected nets? Are schemes utilitzing local connectivity patterns, perhaps including the use of additional layers, buying the researcher some advantage besides more efficient use of limited amounts of computer time? Naively it would seem that the more interconnections the better (even if, after training, many have near-zero weights), is this true? Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa12627; 10 Jun 89 19:40:08 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa20829; 10 Jun 89 19:35:56 EDT Received: from GOAT.CS.WISC.EDU by CS.CMU.EDU; 10 Jun 89 19:34:30 EDT Date: Sat, 10 Jun 89 18:34:12 -0500 From: Vasant Honavar Message-Id: <8906102334.AA02423 at goat.cs.wisc.edu> Received: by goat.cs.wisc.edu; Sat, 10 Jun 89 18:34:12 -0500 To: singer at Think.COM Subject: Re: Sparse vs. Dense networks Cc: connectionists at CS.CMU.EDU, honavar at cs.wisc.edu The tradeoff between connectivity and performance, can in general, be quite complex. There is reason to think that smaller networks (with fewer nodes and links) may be better than larger ones for getting improved generalization as shown by the work of Hinton and others. The use of some topological constraints (such as local receptive fields - in the case of vision) on network connectivity can yield improved performance in learning (after discounting the saving in computer time for simulations) (Honavar and Uhr, 1988; Moody and Darken, 198?; Qian and Sejnowski, 1988). Local interactions and systematic initiation and termination of plasticity, combined with a Hebbian-like learning rule, can, through a process of self-organization, yield network structures that resemble that of orientation columns in the primary visual cortex (Linsker, 1988). Modularity is another means of restricting interactions between different parts of a network. When tasks are relatively independent, they are best handled by seperate or sparsely interacting modules. This is shown by the work of Ruekl and Kosslyn (198?) - Seperate modules learning object identification and object location perform better than a monolithic network learning both. Many tasks that connectionist models attempt to solve are NP-complete; combinatorially explosive solutions are not feasible. Other factors being equal, architectures that exploit the natural constraints of the domain yield better performance (speed, cost) than those that don't. Network architecture (size, topology, etc) interact intimately with the learning processes (weight modification; generation of new links (Honavar, 1988) - and possibly, nodes; regulation of plasticity) as well as the structure of the domain (e.g., vision, speech) at different levels. Connectivity appropriate for a particular problem domain cannot be determined independent of these factors. Vasant Honavar (honavar at cs.wisc.edu) Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa21277; 11 Jun 89 19:11:08 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa27443; 11 Jun 89 17:25:58 EDT Received: from ATHENA.MIT.EDU by CS.CMU.EDU; 11 Jun 89 17:23:40 EDT Received: by ATHENA.MIT.EDU (5.45/4.7) id AA27878; Sun, 11 Jun 89 17:25:12 EDT Message-Id: <8906112125.AA27878 at ATHENA.MIT.EDU> Date: Sun, 11 Jun 89 17:15:39 edt From: Kyle Cave Site: MIT Center for Cognitive Science To: connectionists at CS.CMU.EDU Subject: modularity I'd like to add two notes concerning modularity to Honavar's letter. First, the paper by Rueckl, Cave, & Kosslyn (1989) that he referred to can be found in the most recent issue of the Journal of Cognitive Neuroscience. Second, anyone interested in how modular problems can be approached in a connectionist framework will be interested in recent work by Robbie Jacobs, who is constructing systems that learn to devote separate subnets to independent problems. Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa02004; 12 Jun 89 8:48:00 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa02852; 12 Jun 89 8:41:06 EDT Received: from GATEWAY.MITRE.ORG by CS.CMU.EDU; 12 Jun 89 08:38:49 EDT Received: by gateway.mitre.org (5.54/SMI-2.2) id AA18039; Mon, 12 Jun 89 08:39:09 EDT Return-Path: Received: by marzipan.mitre.org (4.0/SMI-2.2) id AA01478; Mon, 12 Jun 89 08:36:28 EDT Date: Mon, 12 Jun 89 08:36:28 EDT From: alexis%yummy at gateway.mitre.org Message-Id: <8906121236.AA01478 at marzipan.mitre.org> To: singer at Think.COM Cc: connectionists at CS.CMU.EDU In-Reply-To: singer at Think.COM's message of Sat, 10 Jun 89 00:01:33 EDT <8906100401.AA04047 at kulla.think.com> Subject: Sparse vs. Dense networks Reply-To: alexis%yummy at gateway.mitre.org "Sparsely connected networks" (or "tessellated connections" as we tend to think of them) can also be used to hardwire known constraints. In speech or vision or something like the Penzias problem the researcher knows before hand that it makes sence to most if not all of the computation locally, possibly in something like a pyramid architecture. This restriction forces the network to use clues that are considered "reasonable." As an example, a few years ago we spent a fair amount of time using a NN to recognize characters independant of rotation. At first the "D" had the largest letter, so the net just counted pixels (a valid, but as far as we were concerned "wrong," discriminant). We originally worked around this by modifying the training database, but latter we found we could get the same results by using tessellation. Limiting the receptive fields forced the net to primarily use gradients, which proved to be a much more robust method in general. The moral of this story is that you can force a network to do certain classes of algorithms by restricting the connections. Since there is often a limit on the training data available these restrictions are even more important. Combine all this with shared connections (ala the Blue Book), and share tesselations, and tesselations of tesselations, and "skip-level" arcs into shared tesselations, ... and you can really control and add to what you and your net can do. alexis. Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa06612; 12 Jun 89 14:17:18 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa04131; 12 Jun 89 9:32:19 EDT Received: from RELAY.CS.NET by CS.CMU.EDU; 12 Jun 89 09:30:25 EDT Received: from relay2.cs.net by RELAY.CS.NET id ab08738; 12 Jun 89 9:24 EDT Received: from cs.brandeis.edu by RELAY.CS.NET id ab18119; 12 Jun 89 9:13 EDT Received: by cs.brandeis.edu (14.2/6.0.GT) id AA03801; Mon, 12 Jun 89 09:07:57 edt Date: Mon, 12 Jun 89 09:07:57 edt From: Ron Sun Posted-Date: Mon, 12 Jun 89 09:07:57 edt To: connectionists.2 at cs.brandeis.edu The following two tech reports are available from rsun%cs.brandeis.edu at relay.cs.net or R. Sun Brandeis U. CS Waltham, MA 02254 ############################################################# A Discrete Neural Network Model for Conceptual Representation and Reasoning Ron Sun Computer Science Dept. Brandeis University Waltham, MA 02254 Current connectionist models are oversimplified in terms of the internal mechanisms of individual neurons and the communication between them. Although connectionist models offer significant advantages in certain aspects, this oversimplification leads to the inefficiency of these models in addressing issues in explicit symbolic processing, which is proven to be essential to human intelligence. What we are aiming at is a connectionist architecture which is capa- ble of simple, flexible representations of high level knowledge structures and efficient performance of reasoning based on the data. We first propose a discrete neural net- work model which contains state variables for each neuron in which a set of discrete states is explicitly specified instead of a continuous activation function. A technique is developed for representing concepts in this network, which utilizes the connections to define the concepts and represents the concepts in both verbal and compiled forms. The main advantage is that this scheme can handle variable bindings efficiently. A reasoning scheme is developed in the discrete neural network model, which utilizes the inherent parallelism in a neural network model, performing all possible inference steps in parallel, implementable in a fine-grained massively parallel computer. (to appear in Proc. CogSci Conf. 1989) ############################################################### Model local neural networks in the lobster stomatogastric ganglion Ron Sun Eve Marder David Waltz Brandeis University Waltham, MA 02254 ABSTRACT We describe a simulation study of the pyloric network of the lobster stomatogastric ganglion. We demonstrate that a few simple activation functions are sufficient to describe the oscillatory behavior of the network. Our aim is to determine the essential mechanisms necessary to specify the operation of biological neural networks so that we can incorporate them into connectionist models. Our model includes rhythmically active bursting neurons and long time-constant synaptic relations. In the process of doing this work, various models and algorithms were compared. We have derived some connectionist learning algorithms. They have proved useful in terms of ease and accuracy in model generation. (to appear in IJCNN-89) ************************************************************** Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa09540; 12 Jun 89 17:29:12 EDT Received: from ri.cmu.edu by Q.CS.CMU.EDU id aa06000; 12 Jun 89 11:32:32 EDT Received: from HELIOS.NORTHEASTERN.EDU by RI.CMU.EDU; 12 Jun 89 11:30:32 EDT Received: from corwin.ccs.northeastern.edu by helios.northeastern.edu id aa23199; 8 Jun 89 12:12 EDT Received: by corwin.CCS.Northeastern.EDU (5.51/SMI-3.2+CCS-main-2.6) id AA01649; Thu, 8 Jun 89 12:10:34 ADT Date: Thu, 8 Jun 89 12:10:34 ADT From: steve gallant Message-Id: <8906081510.AA01649 at corwin.CCS.Northeastern.EDU> To: connectionists at RI.CMU.EDU Subject: supervised vs. unsupervised learning and autoassociators Here's my suggestion for a taxonomy of learning problems: I. Supervised Learning: correct activations given for output cells for each training example A. Hard Learning Problems: network structure is fixed and activations are not given for intermediate cells B. Easy Learning Problems: correct activations given for intermediate cells OR network structure allowed to be modified OR single-cell problem II. Unsupervised Learning: correct activations not given for output cells There are further subdivisions for learning ALGORITHMS that can cut across the above PROBLEM classes. For example: 1. One-Shot Learning Algorithms: each training example is looked at at most one time 2. Iterative Algorithms: training examples may be looked at several times Autoassociative Algorithms are a special case of IB above because each output cell can be trained independently of the other cells, reducing the problem to single-cell supervised learning. However these models have different dynamics, in that output values are copied to input cells for the next iteration. Most learning algs for autoassociative networks are one-shot algorithms (eg. linear autoassociator, BSB, Hopfield nets). Of course iterative algorithms can be used to increase the capacity of autoassociative models (eg. perceptron learning, pocket alg.). We could also add intermediate cells to an autoassociative network to allow arbitrary sets of patterns to be stored. Steve Gallant Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa13248; 13 Jun 89 0:10:26 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa09991; 12 Jun 89 15:51:06 EDT Received: from AI.CS.WISC.EDU by CS.CMU.EDU; 12 Jun 89 15:48:14 EDT Date: Mon, 12 Jun 89 14:48:02 -0500 From: Leonard Uhr Message-Id: <8906121948.AA11341 at ai.cs.wisc.edu> Received: by ai.cs.wisc.edu; Mon, 12 Jun 89 14:48:02 -0500 To: singer at Think.COM Subject: desirability of sparse, local, converging connectivity Cc: connectionists at CS.CMU.EDU I think there is a strong set of arguments for sparse local connectivity, with multiple converging layers to give global functions. In a word: Sparse connectivity keeps the number of links down to O(kN), rather than O(N**2). Consider brains: 10**12 neurons each with only roughly 3*10**3 synaptic links. 10**24 links is far too much to handle either serially (time) or in parallel (wires). Obviously the links should be those with non-zero weight, and that usually means local (tho not always). Irrelevant links with random initial weights almost certainly obscure the functional links and breed confusion. Less-than-complete connectivity means that the convergence necessary to compute global interactions needs additional layers O(logN). - Len Uhr Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa15637; 13 Jun 89 5:22:36 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa11703; 12 Jun 89 17:16:46 EDT Received: from MARS.NJIT.EDU by CS.CMU.EDU; 12 Jun 89 17:15:15 EDT Received: by mars.njit.edu (5.57/Ultrix2.4-C) id AA14240; Mon, 12 Jun 89 17:14:31 EDT Date: Mon, 12 Jun 89 17:14:31 EDT From: nirwan ansari fac ee Message-Id: <8906122114.AA14240 at mars.njit.edu> To: Connectionists at CS.CMU.EDU Subject: 1989 INNS meeting, Sep 5-9, 1989. Return-Receipt-To: ang at mars.njit.edu Last March, I submitted a paper to the 1989 INNS meeting, Sep 5-9, 1989. Just found out that this meeting has been cancelled. Yet, I haven't been notified of the fate of my paper. I called Schman Associates and INNS office - no one knew where my paper ended up. Anybody out there knows what were happening to the papers submitted to this meeting? I still got the receipt of my express mail... Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa15939; 13 Jun 89 6:00:25 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa13235; 12 Jun 89 18:46:14 EDT Received: from EPHEMERAL.AI.TORONTO.EDU by CS.CMU.EDU; 12 Jun 89 18:43:56 EDT Received: from localhost (stdin) by ephemeral.ai.toronto.edu with SMTP id 10882; Mon, 12 Jun 89 18:42:36 EDT To: harnad at PRINCETON.EDU cc: connectionists at CS.CMU.EDU Subject: Re: Categorization and Supervision In-reply-to: Your message of Fri, 09 Jun 89 01:28:02 -0400. Date: Mon, 12 Jun 89 18:41:56 EDT From: Geoffrey Hinton Message-Id: <89Jun12.184236edt.10882 at ephemeral.ai.toronto.edu> I am afraid Steve has not understood my point. He simply repeats his conviction again: "It is a purely logical point that, even if arrived at by purely internal, unsupervised means, the "correctness" of a "correct" human categorization must be based on some external performance constraint, and hence at least a potential "supervisor." Otherwise the "correctness" is merely in the mind of the theorist, or the interpreter (or the astrologist, if the magical number happens to be 12)." The whole debate is about whether there is a sense in which a categorization can be "correct" even though no external supervision is supplied. Steve thinks that any such categorization would be "merely in the mind of the theorist" (and hence, I guess he thinks, arbitrary). I think that this is wrong. If one model of the data is overwhelmingly simpler than any other, then its not just in the mind of the theorist. Its correct. The nice thing about the Kolmogorov-Chaitin view of complexity is that (in the limit) it doesnt need to mention the mind of the observer (i.e. in the limit, one model can be simpler than another WHATEVER the programming language in which we measure simplicity). In another message on the same subject, Steve says: "Perhaps an example will give a better idea of the distinction I'm emphasizing: Suppose you had N "objects." Consider the many ways you could sort and label them: By size, by shape, by color, by function, by age, as natural or synthetic, living or nonliving, etc. In each case, the name of the same "object" changes, as does the membership of the category to which it is assigned. With sufficient imagination an enormous number of possible categorizations could be generated for the same set of objects. What makes some of those categorizations "correct" (or, equally important, incorrect) and others arbitrary? The answer is surely related to why it is that we bother to categorize at all: Sorting (and especially mis-sorting) objects some ways (e.g., as edibles vs inedibles) has important consequences for us, whereas sorting them other ways (e.g., bigger/smaller than a breadbox, moon in Aquarius) does not. The consequences do not depend on "internal" considerations but on external ones, so I continue to doubt that unsupervised internal constraints can account for much that is of interest in human category learning." Again, this is just a repetition of his viewpoint. If you consider Peter Cheeseman's work on categorization (which found a new class of stars without supervision), it becomes clear that unsupervised categorization is NOT arbitrary. Geoff PS: I think I have now stated my beliefs clearly, and I do not intend to clutter up the network with any more meta-level junk. I think this debate will be resolved by technical progress in getting unsupervised learning to work nicely, not by a priori assertions about what is necessary and what is impossible. Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa16749; 13 Jun 89 7:28:02 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa14729; 12 Jun 89 20:10:35 EDT Received: from PRINCETON.EDU by CS.CMU.EDU; 12 Jun 89 20:08:14 EDT Received: from clarity.Princeton.EDU by Princeton.EDU (5.58+++/2.17) id AA07505; Mon, 12 Jun 89 20:08:02 EDT Received: by clarity.Princeton.EDU (4.0/1.81) id AA09043; Mon, 12 Jun 89 20:12:47 EDT Date: Mon, 12 Jun 89 20:12:47 EDT From: harnad at Princeton.EDU Message-Id: <8906130012.AA09043 at clarity.Princeton.EDU> To: connectionists at CS.CMU.EDU Subject: Re: Categorization & Supervision Geoff writes: >> I am afraid Steve has not understood my point... about whether there >> is a sense in which a categorization can be "correct" even though no >> external supervision is supplied... If one model of the data is >> overwhelmingly simpler than any other, then it's not just in the mind of >> the theorist. It's correct. Let me quickly cite two senses (that I've already noted) in which such a categorization can indeed be "correct," just as Geoff writes in the foregoing passage, so as to confirm that the misunderstanding's not on this side of the border: (1) Under the above conditions it would certainly be (tautologically) "correct" that the simplest categorization was indeed the simplest categorization, and that would be true entirely independently of any external supervisory criterion. (2) It COULD also be "correct" (i) accidentally, or because of prior (ii) innate or (iii) learned preparation, or because the categorization problem happened to be (iv) easy (all four of which I've already mentioned in prior postings) that the categorization arrived at by the internal simplicity criterion ALSO happens to correspond to a categorization that DOES match an external supervisory criterion. (Edibles and inedibles could conceivably sort this way, though it's unlikely.) I take it that (1) in and of itself is of interest only to the complexity theorist. (2) Is a possibility; the burden of proof is on anyone who wishes to claim that many, most or all of our external categorizations in the world can indeed be captured this way (the "some" I already noted in my first posting). I have already given (in a response to someone else's posting on this question) one a priori reason not to expect simplicity or any other a priori symmetry to succeed very often: We typically can and do categorize the very SAME N inputs in many, many DIFFERENT ways, depending on external contingencies; so it is hard to imagine how one and the same internal criterion (or several) could second guess ALL of these alternative categorizations. The winning features are certainly lurking SOMEWHERE in the input, otherwise successful categorization would not be possible; but finding them is hardly likely to be an a priori or internal matter -- and certainly not in what I called the "hard" (underdetermined) cases. That's what learning is all about. And without supervision it's just thrashing in the dark (except in the easy cases that wear their category structure on their ears -- or the already "prepared" cases, but those involve theft rather than honest toil for a learning mechanism). Let me close by clarifying some possible sources of misunderstanding in this discussion of the relation between supervised/unsupervised learning algorithms and imposed/ad-lib categorization performance tasks: (A) A pure ad lib sorting task is to give a subject N objects and ask him to sort them any way he wishes. A variant on this is to ask him to (A1) sort them into k categories any way he wishes. (B) A pure imposed sorting task is to give a subject N objects and k categories with k category names. The subject is asked to sort them correctly and is given feedback on every trial; whenever there is an error, the name of the correct category is provided. The task is trivial if the subject can sort correctly from the beginning, or as soon as he knows how many categories there are and what their names are (i.e., if task B is the same as task A1) or soon thereafter. The task is nontrivial if the the N objects are sufficiently interconfusable, and the winning features sufficiently nonobvious, as to require many supervised trials in order to learn which features will reliably sort the objects correctly. (The solution should also not be a matter of rote memory for every object-label pair: i.e., N should be very large, and categorization should reach 100% before having sampled them all). A variant on B would be (B1) to provide feedback only on whether or not the categorization is correct, but without indicating the correct category name when something has been miscategorized. Note that for dichotomous categories (k = 2), B and B1 are the same (and that locally, all categorizations are dichotomous: "Is this an `X' or isn't it?"). In any case, both B and B1 would be cases of supervised learning. Note also that the choice of k, and of which categorization (and hence which potential features) are "correct," is entirely up to the experimenter in imposed categorization -- just as it is entirely up to external contingencies in the world, and the consequences of miscategorization for the organism, in natural categorization. Cases in which A happened to match these external contingencies would be special ones indeed. Stevan Harnad Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa06509; 13 Jun 89 13:14:03 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa21735; 13 Jun 89 9:18:34 EDT Received: from PRINCETON.EDU by CS.CMU.EDU; 13 Jun 89 09:15:26 EDT Received: from suspicion.Princeton.EDU by Princeton.EDU (5.58+++/2.17) id AA26730; Tue, 13 Jun 89 09:15:27 EDT Received: by suspicion.Princeton.EDU (4.0/1.81) id AA00904; Tue, 13 Jun 89 09:20:13 EDT Date: Tue, 13 Jun 89 09:20:13 EDT From: "Stephen J. Hanson" Message-Id: <8906131320.AA00904 at suspicion.Princeton.EDU> To: connectionists at CS.CMU.EDU Subject: sup vs unsup one other connection with this issue is a profitable distinction that has been made in the animal learning literature for some decades now. It has to do with the notion that there is a set of operations or procedures and there is a concomitant set processes. Classical or Pavlovian conditioning is an "open loop"" procedure in control lingo, and is clearly a supervised procedure (analogous to delta rule/back-prop/boltzmann) while Skinner's Operant or Instrumental conditioning--"Reinforcement Learning" is a "closed loop" procedure, the organism's action determine consequences. Clearly this a weakening of the supervision and of the information provided by the teacher. What is interesting is one can find process theories that run the gamut from complete distinction between classical and operant conditioning (less so these days) to equivalence. Consequently, from a process point of view there may be NO distinction between the procedures..this can cause some confusion at the procedure level depending on the process theory one subscribes to.. Now one other interesting point in this context raised a moment ago (at least I saw the mail a moment ago) by Cybenko...There is really nothing analogous in the animal learning area with "unsupervised" learning --procedures might be "sensitization", pseudo-conditioning, pre-exposure... all which have relatively complex process accounts. As Cybenko pointed out, "exposure" or Unsupervised procedures really do impose some implicit metric on the system, one which interacts with the way the network computes activation. In a similar sense, animals, in fact, come prewired or "prepared" for classical and operant conditioning with certain classes of stimuli and certain classes of responses and not other classes. These kinds of predispositions mitigate the distinction between supervised and unsupervised learning, of course there is a set unsupervised operations which one can impose on the system. Steve Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa10206; 13 Jun 89 17:45:16 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa24897; 13 Jun 89 10:31:58 EDT Received: from PRINCETON.EDU by CS.CMU.EDU; 13 Jun 89 10:28:59 EDT Received: from clarity.Princeton.EDU by Princeton.EDU (5.58+++/2.17) id AA28936; Tue, 13 Jun 89 10:29:03 EDT Received: by clarity.Princeton.EDU (4.0/1.81) id AA10919; Tue, 13 Jun 89 10:33:48 EDT Date: Tue, 13 Jun 89 10:33:48 EDT From: harnad at Princeton.EDU Message-Id: <8906131433.AA10919 at clarity.Princeton.EDU> To: connectionists at CS.CMU.EDU Subject: Re: Categorization & Supervision In a variant of a comment that appears to have been posted twice, Geoff Hinton added: > Cheeseman's work on categorization (which found a new class of stars > without supervision)... [illustrates] that unsupervised categorization > is NOT arbitrary... I think this debate will be resolved by technical > progress in getting unsupervised learning to work nicely, not by a > priori assertions about what is necessary and what is impossible. This seems an apt point to ponder again the words of the historian J. H. Hexter on the subject of negative a prioris. He wrote: In an academic generation a little overaddicted to "politesse," it may be worth saying that violent destruction is not necessarily worthless and futile. Even though it leaves doubt about the right road for London, it helps if someone rips up, however violently, a `To London' sign on the Dover cliffs pointing south... But I'm certainly prepared to agree that time itself can arbitrate whether or not it has been well spent checking if the unsupervised discovery of a new class of stars also happens to lead us to the nuts-and-bolts categories imposed by the contingencies of nonverbal and verbal life on terra firma... Stevan Harnad Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa10313; 13 Jun 89 17:54:43 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa21431; 13 Jun 89 9:12:22 EDT Received: from FLASH.BELLCORE.COM by CS.CMU.EDU; 13 Jun 89 09:10:33 EDT Received: by flash.bellcore.com (5.58/1.1) id AA15789; Tue, 13 Jun 89 09:09:19 EDT Date: Tue, 13 Jun 89 09:09:19 EDT From: Stephen J Hanson Message-Id: <8906131309.AA15789 at flash.bellcore.com> To: connectionists at CS.CMU.EDU Subject: test please ignore Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa11670; 13 Jun 89 20:44:06 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa27730; 13 Jun 89 12:31:06 EDT Received: from ARPA.ATT.COM by CS.CMU.EDU; 13 Jun 89 12:29:09 EDT From: neural!yann at att.att.com Message-Id: <8906131533.AA05289 at neural.UUCP> Received: from localhost by lesun.UUCP (4.0/4.7) id AA00895; Tue, 13 Jun 89 11:35:26 EDT To: att!cs.cmu.edu!connectionists at att.att.com Cc: att!ai.toronto.edu!hinton at att.att.com Subject: Kolmogorov-Chaitin complexity (was: Categorization and Supervision) In-Reply-To: Your message of Mon, 12 Jun 89 18:41:56 -0400. Reply-To: neural!yann at neural.att.com Date: Tue, 13 Jun 89 11:35:23 -0400 >From: Yann le Cun Like Geoff I "do not intend to clutter up the network with any more meta-level junk", so let's talk about more technical issues. Geoff Hinton says: " If one model of the data is overwhelmingly simpler than any other, then its not just in the mind of the theorist. Its correct. The nice thing about the Kolmogorov-Chaitin view of complexity is that (in the limit) it doesnt need to mention the mind of the observer (i.e. in the limit, one model can be simpler than another WHATEVER the programming language in which we measure simplicity)." The bad thing about KC complexity is that it DOES mention the mind of the observer (or rather, the language he uses) for all finitely complex objects. Infinitely complex objects are infinitely complex for all measures of complexity (in Chaitin's framework they correspond to non-computable functions). But the complexity of finite objects depends on the language used to describe the object. If Ca(x) is the complexity of object x expressed in language a, and Cb(x) the complexity of the same object expressed in an other language b, then there is a constant c such that for any x : Ca(x) < Cb(x) + c the constant c can be interpreted as the complexity of a language translator from b to a. The trouble is: c can be very large, so that the comparison of the complexity of two real (computable) objects depends on the language used to describe the objects. We can have Ca(x) < Ca(y) and Cb(x) > Cb(y). Using this thing as a Universal Criterion for Unsupervised Learning looks quite hopeless. - Yann Le Cun Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa17748; 14 Jun 89 11:19:05 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa03400; 13 Jun 89 19:20:19 EDT Received: from IBM.COM by CS.CMU.EDU; 13 Jun 89 19:17:52 EDT Date: 13 Jun 89 18:33:31 EDT From: Dimitri Kanevsky To: connectionists at CS.CMU.EDU Message-Id: <061389.183332.dimitri at ibm.com> Subject: Categorization & Supervision I have been followng the discussion of supervised learning between G. Hinton and S. Harnad and it is not at all clear to me why Hinton would expect the correspondence between simplicity and correctness to be anything but accidental. Dimitri Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa23564; 14 Jun 89 18:55:59 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa05110; 13 Jun 89 21:55:17 EDT Received: from VMA.CC.CMU.EDU by CS.CMU.EDU; 13 Jun 89 21:53:23 EDT Received: from CMUCCVMA by VMA.CC.CMU.EDU ; Tue, 13 Jun 89 21:52:36 EDT Received: from JHUVMS.BITNET (INS_ATGE) by CMUCCVMA (Mailer X1.25) with BSMTP id 0258; Tue, 13 Jun 89 21:52:35 EDT Date: Tue, 13 Jun 89 21:51 EST From: INS_ATGE%JHUVMS.BITNET at VMA.CC.CMU.EDU MMDF-Warning: Parse error in original version of preceding line at Q.CS.CMU.EDU Subject: Two Questions To: connectionists at CS.CMU.EDU X-Original-To: connectionists at cs.cmu.edu, INS_ATGE I am currently writing a parallel backpropogation program on The Connection Machine, with the immediate task of identifying insonified objects from active sonar data. I was wondering if anyone could give me a reference for a detailed paper on conjugate gradient network teaching technqiues. From what I have picked up from casual conversation, it appears that this method can often lead to faster learning. I would also appreciate information dealing with NN analysis of sonar data (citations in literature besides Sejnowski and Gorman, personal communication, just to get an idea of how the program is stacking up against earlier work). -Thomas G. Edwards ins_atge at jhuvms.BITNET ins_atge at jhunix.hcf.jhu.edu tedwards at cmsun.nrl.navy.mil Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa04553; 15 Jun 89 0:36:56 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa14557; 14 Jun 89 13:29:12 EDT Received: from PRINCETON.EDU by CS.CMU.EDU; 14 Jun 89 08:26:23 EDT Received: from clarity.Princeton.EDU by Princeton.EDU (5.58+++/2.17) id AA07121; Wed, 14 Jun 89 08:26:25 EDT Received: by clarity.Princeton.EDU (4.0/1.81) id AA16907; Wed, 14 Jun 89 08:31:09 EDT Date: Wed, 14 Jun 89 08:31:09 EDT From: harnad at Princeton.EDU Message-Id: <8906141231.AA16907 at clarity.Princeton.EDU> To: connectionists at CS.CMU.EDU Subject: Proper Place of Connectionism... ON THE PROPER PLACE OF CONNECTIONISM IN MODELLING OUR BEHAVIORAL CAPACITIES (Abstract of paper presented at First Annual Meeting of the American Psychological Society, Alexandria VA, June 11 1989) Stevan Harnad Psychology Department Princeton University Princeton NJ 08544 Connectionism is a family of statistical techniques for extracting complex higher-order correlations from data. It can also be interpreted and implemented as a neural network of interconnected units with weighted positive and negative interconnections. Many claims and counterclaims have been made about connectionism: Some have said it will supplant artificial intelligence (symbol manipulation) and explain how we learn and how our brain works. Others have said it is just a limited family of statistical pattern recognition techniques and will not be able to account for most of our behavior and cognition. I will try to sketch how connectionist processes could play a crucial but partial role in modeling our behavioral capacities in learning and representing invariances in the input, thereby mediating the "grounding" of symbolic representations in analog sensory representations. The behavioral capacity I will focus on is categorization: Our ability to sort and label inputs correctly on the basis of feedback from the consequences of miscategorization. Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa22843; 15 Jun 89 7:40:47 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa01101; 14 Jun 89 19:39:50 EDT Received: from ICSIB.BERKELEY.EDU by CS.CMU.EDU; 14 Jun 89 18:55:26 EDT Received: from icsib6. (icsib6.Berkeley.EDU) by icsib.Berkeley.EDU (4.0/SMI-4.0) id AA21096; Wed, 14 Jun 89 15:59:22 PDT Received: by icsib6. (4.0/SMI-4.0) id AA11131; Wed, 14 Jun 89 15:56:27 PDT Date: Wed, 14 Jun 89 15:56:27 PDT From: Andreas Stolcke Message-Id: <8906142256.AA11131 at icsib6.> To: connectionists at CS.CMU.EDU Subject: Tech. Report available The following Technical Report is available from ICSI: ______________________________________________________________________ TR-89-032 Andreas Stolcke 5/01/89 19 pages,$1.75 "A Connectionist Model of Unification" A general approach to encode and unify recursively nested feature structures in connectionist networks is described. The unification algorithm implemented by the net is based on iterative coarsening of equivalence classes of graph nodes. This method allows the reformulation of unification as a constraint satisfaction problem and enables the connectionist implementation to take full advantage of the potential parallelism inherent in unification, resuting in sublinear time complexity. Moreover, the method is able to process any number of feature structures in parallel, searching for possible unifications and making decisions among mutually exclusive unifications where necessary. ______________________________________________________________________ International Computer Science Institute Technical Reports There is a small charge to cover postage and handling for each report. This charge is waived for ICSI sponsors and for institutions having an exchange agreement with the Institute. Please use the form at the back of the list for your order. Make any necessary additions or corrections to the address label on the form, and return it to the International Computer Science Institute. NOTE: Qualifying institutions may choose to participate in a technical report exchange program and receive ICSI TRs at no charge. To arrange an exchange agreement write, call or send e-mail message to: Librarian International Computer Science Institute 1947 Center Street, Suite 600 Berkeley, CA 94704 info at icsi.berkeley.EDU (415) 643-9153. ______________________________________________________________________ Technical Report Order Form International Computer Science Institute 1947 Center Street, Suite 600 Berkeley, CA 94704 Please enclose your check to cover postage and handling. Prepayment is required for all materials. Charges will be waived for ICSI sponsors and for institutions that have exchange agreements with the Institute. Make checks payable to "ICSI". __________________________________________________________ | | | | | | TR Number | Quantity | Postage & Handling| Total | |______________|___________|___________________|___________| | | | | | | |______________|___________|___________________|_______|___| | | | | | | |______________|___________|___________________|_______|___| | | | | | | |______________|___________|___________________|_______|___| | | | | | | |______________|___________|___________________|_______|___| | | | | | | |______________|___________|___________________|_______|___| | | | | | | |______________|___________|___________________|_______|___| | | | | | | |______________|___________|___________________|_______|___| | | | | | | |______________|___________|___________________|_______|___| | | | | | | |______________|___________|___________________|_______|___| | | | | | | |______________|___________|___________________|_______|___| | | | | | | |______________|___________|___________________|_______|___| | | | | | | |______________|___________|___________________|_______|___| | | | | | | |______________|___________|___________________|_______|___| | | | | | | |______________|___________|___________________|_______|___| | | | | | Total Amount | | | |___________________|_______|___| NAME AND ADDRESS __ __________________________ |__| Please note change of address __ as shown. __________________________ |__| Please remove my name from this __ mailing list. __________________________ |__| Please add my name to this mailing list. __________________________ __________________________ Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa14169; 16 Jun 89 12:09:45 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa27023; 16 Jun 89 12:04:26 EDT Received: from VMA.CC.CMU.EDU by CS.CMU.EDU; 16 Jun 89 06:34:46 EDT Received: from CMUCCVMA by VMA.CC.CMU.EDU ; Fri, 16 Jun 89 06:34:14 EDT Received: from UKACRL.BITNET by CMUCCVMA (Mailer X1.25) with BSMTP id 2217; Fri, 16 Jun 89 06:34:13 EDT Received: from RL.IB by UKACRL.BITNET (Mailer X1.25) with BSMTP id 7570; Fri, 16 Jun 89 10:51:44 BST Received: Via: UK.AC.EX.CS; 16 JUN 89 10:51:38 BST Received: from exsc by expya.cs.exeter.ac.uk; Fri, 16 Jun 89 10:58:04-0000 From: Lyn Shackleton Date: Fri, 16 Jun 89 10:50:29 BST Message-Id: <702.8906160950 at exsc.cs.exeter.ac.uk> To: connectionists at CS.CMU.EDU Subject: journal reviewers ******* CONNECTION SCIENCE ****** Editor: Noel E. Sharkey Because fo the number of specialist submissions, the journal is currently expanding its review panel. This is an interdisciplinary journal with an emphasis on replicability of results. If you wish to volunteer please send details of your review area to the address below. Or write for further details. lyn shackleton (assistant editor) Centre for Connection Science JANET: lyn at uk.ac.exeter.cs Dept. Computer Science University of Exeter UUCP: !ukc!expya!lyn Exeter EX4 4PT Devon BITNET: lyn at cs.exeter.ac.uk.UKACRL U.K. Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa09476; 16 Jun 89 18:22:40 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa28639; 16 Jun 89 13:20:32 EDT Received: from PRINCETON.EDU by CS.CMU.EDU; 16 Jun 89 13:18:10 EDT Received: from suspicion.Princeton.EDU by Princeton.EDU (5.58+++/2.17) id AA00231; Fri, 16 Jun 89 11:30:25 EDT Received: by suspicion.Princeton.EDU (4.0/1.81) id AA02388; Fri, 16 Jun 89 11:35:14 EDT Date: Fri, 16 Jun 89 11:35:14 EDT From: jose at confidence.Princeton.EDU Message-Id: <8906161535.AA02388 at suspicion.Princeton.EDU> To: connectionists at CS.CMU.EDU Subject: lost papers... through a series of complex, political and unnecessarily confusing and obfuscating moves by various parties..this field's attempt to have one coherent, reasonable quality meeting has been foiled. Now we have (at least) 3 of varying coherence and quality. IJCNN which is meeting very soon this month, INNS will meet sometime in January, and NIPS will occur as always in November. It is too bad that greed and self-interest seems to take the place of the sort of nurturing and careful, thoughtful, long term commitment to a field that has such great potential. It would be a shame to see the polarization within the field allow people's work and thought to be lost or ignored. Steve Hanson Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa08295; 16 Jun 89 21:24:38 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa03304; 16 Jun 89 17:21:10 EDT Received: from ANDREW.CMU.EDU by CS.CMU.EDU; 16 Jun 89 17:16:05 EDT Received: by andrew.cmu.edu (5.54/3.15) id for connectionists at cs.cmu.edu; Fri, 16 Jun 89 17:15:56 EDT Received: via switchmail; Fri, 16 Jun 89 17:15:53 -0400 (EDT) Received: from chicory.psy.cmu.edu via qmail ID ; Fri, 16 Jun 89 17:13:50 -0400 (EDT) Received: from chicory.psy.cmu.edu via qmail ID ; Fri, 16 Jun 89 17:13:39 -0400 (EDT) Received: from BatMail.robin.v2.8.CUILIB.3.45.SNAP.NOT.LINKED.chicory.psy.cmu.edu.sun3.35 via MS.5.6.chicory.psy.cmu.edu.sun3_35; Fri, 16 Jun 89 17:13:33 -0400 (EDT) Message-Id: Date: Fri, 16 Jun 89 17:13:33 -0400 (EDT) From: "James L. McClelland" To: connectionists at CS.CMU.EDU Subject: Let 1,000 flowers bloom In-Reply-To: <8906161535.AA02388 at suspicion.Princeton.EDU> References: <8906161535.AA02388 at suspicion.Princeton.EDU> In response to Steve Hanson's last message: I myself do not mind the fact that there are three conectionist meetings, as well as others where connectionist work can be presented. Sure, there are factions and there is politics; this has lead to some fragmentation. But, there is an upside. Different perspectives all have a chance to be represented, and different goals have a chance to be served. The NIPS meeting, for example, is a small, high-quality meeting dedicated to highly quantitative, computational-theory type work with a biological flavor; I think it would be a shame if these characteristics were lost in an attempt to achieve some grand synthesis. The other conferences have their specific strengths as well. Meanwhile, non-connectionist conferences like Cognitive Science and AAAI attract some of the good connectionist work applied to language and higher-level aspects of cognition. People seem to be gravitating toward attending the one or two of these that best suit their own needs and interests within the broad range of connectionist research. Seems to me things are pretty close to the way they ought to be. It would be nice if the process of arriving at this state could have been smoother, and maybe in the end it won't turn out that there need to be quite so many meetings, but things could be a heck of a lot worse. -- Jay McClelland Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa00328; 17 Jun 89 3:02:39 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa07666; 16 Jun 89 22:50:01 EDT Received: from EPHEMERAL.AI.TORONTO.EDU by CS.CMU.EDU; 16 Jun 89 22:48:23 EDT Received: from localhost (stdin) by ephemeral.ai.toronto.edu with SMTP id 10958; Fri, 16 Jun 89 15:33:20 EDT To: connectionists at CS.CMU.EDU Subject: CRG-TR-89-3 Date: Fri, 16 Jun 89 15:33:15 EDT From: Carol Plathan Message-Id: <89Jun16.153320edt.10958 at ephemeral.ai.toronto.edu> The Technical Report CRG-TR-89-3 by Hinton and Shallice (May 1989) can be obtained by sending me your full mailing address. An abstract of this Report follows: LESIONING A CONNECTIONIST NETWORK: INVESTIGATIONS OF ACQUIRED DYSLEXIA ----------------------------------------------------------------------- Geoffrey E. Hinton Tim Shallice Department of Computer Science MRC Applied Psychology Unit University of Toronto Cambridge, UK ABSTRACT: -------- A connectionist network which had been trained to map orthographic representation into semantic ones was systematically 'lesioned'. Wherever it was lesioned it produced more Visual, Semantic, and Mixed visual and semantic errors than would be expected by chance. With more severe lesions it showed relatively spared categorical discrimination when item identification was not possible. Both phenomena are qualitatively similar to those observed in neurological patients. The error pattern is that characteristically found in deep dyslexia. The spared categorical discrimination is observed in semantic access dyslexia and also in a form of pure alexia. It is concluded that the lesioning of connectionist networks may produce phenomena which mimic non-transparent aspects of the behaviour of neurological patients. Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa04289; 17 Jun 89 18:05:49 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa13079; 17 Jun 89 17:30:52 EDT Received: from IBM.COM by CS.CMU.EDU; 17 Jun 89 17:28:36 EDT Date: 17 Jun 89 17:05:59 EDT From: Scott Kirkpatrick To: connectionists at CS.CMU.EDU cc: Steve at confidence.princeton.edu Message-Id: <061789.170559.kirk at ibm.com> Subject: NIPS 89 status and schedule Authors have been known to call me, Rich Lippman (the program committee chair) or Kathie Hibbard (who runs the local office in Denver), asking when the NIPS program decisions will be made, announced, etc... I'll give our schedule in order to restrict the phone calls to those which can help us to catch mistakes. Steve Hanson (NIPS publicity, and responsible for our eye-catching blue poster) -- please put a version of this on all the other bulletin boards. Deadline for abstracts and summaries was May 30, 1989. We have now received over 460 contributions (almost 50% more than last year!). They are now logged in, and cards acknowledging receipt will be mailed next week to authors. Authors who have not received an acknowledgement by June 30 should write to Kathie Hibbard at the NIPS office; it's possible we got your address wrong in our database, and this will help us catch these things. Refereeing will take July. Collecting the results and defining a final program will be done in August. We plan to mail letters informing authors of the outcome during the first week of September. At that time, we will send all registration material, information about prices, and a complete program. If you haven't heard from us in late September, again please write, to help us straighten things out. Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa05039; 17 Jun 89 20:45:24 EDT Received: from csvax.caltech.edu by Q.CS.CMU.EDU id aa14984; 17 Jun 89 20:41:52 EDT Received: from aurel.caltech.edu by csvax.caltech.edu (5.59/1.2) id AA09482; Sat, 17 Jun 89 16:36:47 PDT Received: from smaug.caltech.edu. by aurel.caltech.edu (4.0/SMI-4.0) id AA00438; Sat, 17 Jun 89 16:39:32 PDT Received: by smaug.caltech.edu. (4.0/SMI-4.0) id AA28797; Sat, 17 Jun 89 14:39:57 PDT Date: Sat, 17 Jun 89 14:39:57 PDT From: Jim Bower Message-Id: <8906172139.AA28797 at smaug.caltech.edu.> To: connectionists at Q.CS.CMU.EDU Subject: 3 meetings Concerning Steve Hanson's comments on meetings. I think that it is only fair to note that not all the recent history of neural-net meetings have been characterized by "greed and self-interest". In order for greed to be an issue there must be an opportunity for organizers to make money. In order for self-interest to be an issue there must be a biasable mechanism for self promotion. Opportunities for commercialization and the often outrageous associated claims apply to both cases. In this regard, in my view, it is unfair to mention all three national neural network meetings in the same sentence. One of the three meetings was founded and continues to be organized to provide a "nurturing, careful, and thoughtful" forum committed to the long term support and growth of a field with "considerable potential". Most of you know which meeting that is, but if a clue is necessary, it is the only one that takes place in a city not known as a national focus for greed and self-interest. Jim Bower Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa03661; 18 Jun 89 15:29:07 EDT Received: from vma.cc.cmu.edu by Q.CS.CMU.EDU id aa02704; 18 Jun 89 15:25:31 EDT Received: from CMUCCVMA by VMA.CC.CMU.EDU ; Sun, 18 Jun 89 15:25:51 EDT Received: from UKACRL.BITNET by CMUCCVMA (Mailer X1.25) with BSMTP id 7292; Sun, 18 Jun 89 15:25:49 EDT Received: from RL.IB by UKACRL.BITNET (Mailer X1.25) with BSMTP id 7878; Sun, 18 Jun 89 20:23:50 BST Received: Via: UK.AC.EX.CS; 18 JUN 89 20:23:43 BST To: cs.exeter-connect at CS.EXETER.AC.UK Received: Received: from q.cs.cmu.edu by NSFnet-Relay.AC.UK via NSFnet with SMTP id aa09349; 18 Jun 89 3:58 BS Received: from csvax.caltech.edu by Q.CS.CMU.EDU id aa14984; 17 Jun 89 20:41:52 ED Received: from aurel.caltech.edu by csvax.caltech.edu (5.59/1.2) id AA09482; Sat, 17 Jun 89 16:36:47 PD Received: from smaug.caltech.edu. by aurel.caltech.edu (4.0/SMI-4.0) id AA00438; Sat, 17 Jun 89 16:39:32 PD Received: by smaug.caltech.edu. (4.0/SMI-4.0) id AA28797; Sat, 17 Jun 89 14:39:57 PD Date: Sat, 17 Jun 89 14:39:57 PDT From: Jim Bower Message-Id: <8906172139.AA28797 at smaug.caltech.edu.> To: connectionists at Q.CS.CMU.EDU Subject: 3 meetings Original-Sender: Connectionists-Request at edu.cmu.cs.q Sender: connect-bb-request at CS.EXETER.AC.UK Concerning Steve Hanson's comments on meetings. I think that it is only fair to note that not all the recent history of neural-net meetings have been characterized by "greed and self-interest". In order for greed to be an issue there must be an opportunity for organizers to make money. In order for self-interest to be an issue there must be a biasable mechanism for self promotion. Opportunities for commercialization and the often outrageous associated claims apply to both cases. In this regard, in my view, it is unfair to mention all three national neural network meetings in the same sentence. One of the three meetings was founded and continues to be organized to provide a "nurturing, careful, and thoughtful" forum committed to the long term support and growth of a field with "considerable potential". Most of you know which meeting that is, but if a clue is necessary, it is the only one that takes place in a city not known as a national focus for greed and self-interest. Jim Bower Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa06023; 18 Jun 89 18:08:05 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa02751; 18 Jun 89 15:28:06 EDT Received: from VMA.CC.CMU.EDU by CS.CMU.EDU; 18 Jun 89 15:24:17 EDT Received: from CMUCCVMA by VMA.CC.CMU.EDU ; Sun, 18 Jun 89 15:24:45 EDT Received: from UKACRL.BITNET by CMUCCVMA (Mailer X1.25) with BSMTP id 7252; Sun, 18 Jun 89 15:24:40 EDT Received: from RL.IB by UKACRL.BITNET (Mailer X1.25) with BSMTP id 7874; Sun, 18 Jun 89 20:22:47 BST Received: Via: UK.AC.EX.CS; 18 JUN 89 20:22:42 BST To: cs.exeter-connect at CS.EXETER.AC.UK Received: Received: from q.cs.cmu.edu by NSFnet-Relay.AC.UK via NSFnet with SMTP id aa01674; 17 Jun 89 9:11 BS Received: Received: from VMA.CC.CMU.EDU by CS.CMU.EDU; 16 Jun 89 06:34:46 EDT Received: from CMUCCVMA by VMA.CC.CMU.EDU ; Fri, 16 Jun 89 06:34:14 EDT Received: from UKACRL.BITNET by CMUCCVMA (Mailer X1.25) with BSMTP id 2217; Fri, 16 Jun 89 06:34:13 ED Received: Received: Original-Via: UK.AC.EX.CS; 16 JUN 89 10:51:38 BST Received: from exsc by expya.cs.exeter.ac.uk; Fri, 16 Jun 89 10:58:04-0000 From: Lyn Shackleton Date: Fri, 16 Jun 89 10:50:29 BST Message-Id: <702.8906160950 at exsc.cs.exeter.ac.uk> To: connectionists at CS.CMU.EDU Subject: journal reviewers Original-Sender: Connectionists-Request at edu.cmu.cs.q Sender: connect-bb-request at CS.EXETER.AC.UK ******* CONNECTION SCIENCE ****** Editor: Noel E. Sharkey Because fo the number of specialist submissions, the journal is currently expanding its review panel. This is an interdisciplinary journal with an emphasis on replicability of results. If you wish to volunteer please send details of your review area to the address below. Or write for further details. lyn shackleton (assistant editor) Centre for Connection Science JANET: lyn at uk.ac.exeter.cs Dept. Computer Science University of Exeter UUCP: !ukc!expya!lyn Exeter EX4 4PT Devon BITNET: lyn at cs.exeter.ac.uk.UKACRL U.K. Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa06079; 18 Jun 89 18:12:51 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa02706; 18 Jun 89 15:25:48 EDT Received: from VMA.CC.CMU.EDU by CS.CMU.EDU; 18 Jun 89 15:22:25 EDT Received: from CMUCCVMA by VMA.CC.CMU.EDU ; Sun, 18 Jun 89 15:22:45 EDT Received: from UKACRL.BITNET by CMUCCVMA (Mailer X1.25) with BSMTP id 7140; Sun, 18 Jun 89 15:22:43 EDT Received: from RL.IB by UKACRL.BITNET (Mailer X1.25) with BSMTP id 7856; Sun, 18 Jun 89 20:20:44 BST Received: Via: UK.AC.EX.CS; 18 JUN 89 20:20:36 BST To: cs.exeter-connect at CS.EXETER.AC.UK Received: Received: from q.cs.cmu.edu by NSFnet-Relay.AC.UK via NSFnet with SMTP id aa01325; 17 Jun 89 8:37 BS Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa07666; 16 Jun 89 22:50:01 EDT Received: from EPHEMERAL.AI.TORONTO.EDU by CS.CMU.EDU; 16 Jun 89 22:48:23 EDT Received: To: connectionists at CS.CMU.EDU Subject: CRG-TR-89-3 Date: Fri, 16 Jun 89 15:33:15 EDT From: Carol Plathan Message-Id: <89Jun16.153320edt.10958 at ephemeral.ai.toronto.edu> Original-Sender: Connectionists-Request at edu.cmu.cs.q Sender: connect-bb-request at CS.EXETER.AC.UK The Technical Report CRG-TR-89-3 by Hinton and Shallice (May 1989) can be obtained by sending me your full mailing address. An abstract of this Report follows: LESIONING A CONNECTIONIST NETWORK: INVESTIGATIONS OF ACQUIRED DYSLEXIA ----------------------------------------------------------------------- Geoffrey E. Hinton Tim Shallice Department of Computer Science MRC Applied Psychology Unit University of Toronto Cambridge, UK ABSTRACT: -------- A connectionist network which had been trained to map orthographic representation into semantic ones was systematically 'lesioned'. Wherever it was lesioned it produced more Visual, Semantic, and Mixed visual and semantic errors than would be expected by chance. With more severe lesions it showed relatively spared categorical discrimination when item identification was not possible. Both phenomena are qualitatively similar to those observed in neurological patients. The error pattern is that characteristically found in deep dyslexia. The spared categorical discrimination is observed in semantic access dyslexia and also in a form of pure alexia. It is concluded that the lesioning of connectionist networks may produce phenomena which mimic non-transparent aspects of the behaviour of neurological patients. Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa08767; 19 Jun 89 0:41:09 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa02712; 18 Jun 89 15:26:22 EDT Received: from VMA.CC.CMU.EDU by CS.CMU.EDU; 18 Jun 89 15:22:58 EDT Received: from CMUCCVMA by VMA.CC.CMU.EDU ; Sun, 18 Jun 89 15:23:18 EDT Received: from UKACRL.BITNET by CMUCCVMA (Mailer X1.25) with BSMTP id 7170; Sun, 18 Jun 89 15:23:17 EDT Received: from RL.IB by UKACRL.BITNET (Mailer X1.25) with BSMTP id 7866; Sun, 18 Jun 89 20:21:07 BST Received: Via: UK.AC.EX.CS; 18 JUN 89 20:21:02 BST To: cs.exeter-connect at CS.EXETER.AC.UK Received: Received: from q.cs.cmu.edu by NSFnet-Relay.AC.UK via NSFnet with SMTP id aa01599; 17 Jun 89 9:03 BS Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa28639; 16 Jun 89 13:20:32 EDT Received: from PRINCETON.EDU by CS.CMU.EDU; 16 Jun 89 13:18:10 EDT Received: from suspicion.Princeton.EDU by Princeton.EDU (5.58+++/2.17) id AA00231; Fri, 16 Jun 89 11:30:25 ED Received: by suspicion.Princeton.EDU (4.0/1.81) id AA02388; Fri, 16 Jun 89 11:35:14 ED Date: Fri, 16 Jun 89 11:35:14 EDT From: jose at CONFIDENCE.PRINCETON.EDU Message-Id: <8906161535.AA02388 at suspicion.Princeton.EDU> To: connectionists at CS.CMU.EDU Subject: lost papers... Original-Sender: Connectionists-Request at edu.cmu.cs.q Sender: connect-bb-request at CS.EXETER.AC.UK through a series of complex, political and unnecessarily confusing and obfuscating moves by various parties..this field's attempt to have one coherent, reasonable quality meeting has been foiled. Now we have (at least) 3 of varying coherence and quality. IJCNN which is meeting very soon this month, INNS will meet sometime in January, and NIPS will occur as always in November. It is too bad that greed and self-interest seems to take the place of the sort of nurturing and careful, thoughtful, long term commitment to a field that has such great potential. It would be a shame to see the polarization within the field allow people's work and thought to be lost or ignored. Steve Hanson Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa10897; 19 Jun 89 5:48:18 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa14967; 19 Jun 89 5:40:18 EDT Received: from VMA.CC.CMU.EDU by CS.CMU.EDU; 19 Jun 89 05:38:17 EDT Received: from CMUCCVMA by VMA.CC.CMU.EDU ; Mon, 19 Jun 89 05:38:43 EDT Received: from UKACRL.BITNET by CMUCCVMA (Mailer X1.25) with BSMTP id 0828; Mon, 19 Jun 89 05:38:42 EDT Received: from RL.IB by UKACRL.BITNET (Mailer X1.25) with BSMTP id 6641; Mon, 19 Jun 89 10:36:41 BST Received: Via: UK.AC.EX.CS; 19 JUN 89 10:36:23 BST Received: From: Noel Sharkey Date: Mon, 19 Jun 89 10:35:56 BST Message-Id: <4309.8906190935 at entropy.cs.exeter.ac.uk> To: carol at AI.TORONTO.EDU Cc: connectionists at CS.CMU.EDU In-Reply-To: Carol Plathan's message of Fri, 16 Jun 89 15:33:15 EDT <89Jun16.153320edt.10958 at ephemeral.ai.toronto.edu Subject: CRG-TR-89-3 please send me a copy of Report CRG-TR-89-3 by Hinton and Shallice (May 1989). LESIONING A CONNECTIONIST NETWORK: INVESTIGATIONS OF ACQUIRED DYSLEXIA noel sharkey Centre for Connection Science JANET: noel at uk.ac.exeter.cs Dept. Computer Science University of Exeter UUCP: !ukc!expya!noel Exeter EX4 4PT Devon BITNET: noel at cs.exeter.ac.uk@UKACRL U.K. Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa01375; 19 Jun 89 10:01:27 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa15712; 19 Jun 89 6:05:21 EDT Received: from VMA.CC.CMU.EDU by CS.CMU.EDU; 19 Jun 89 06:03:46 EDT Received: from CMUCCVMA by VMA.CC.CMU.EDU ; Mon, 19 Jun 89 06:04:13 EDT Received: from UKACRL.BITNET by CMUCCVMA (Mailer X1.25) with BSMTP id 0907; Mon, 19 Jun 89 06:04:12 EDT Received: from RL.IB by UKACRL.BITNET (Mailer X1.25) with BSMTP id 7010; Mon, 19 Jun 89 10:50:54 BST Received: Via: UK.AC.EX.CS; 19 JUN 89 10:50:38 BST Received: From: Noel Sharkey Date: Mon, 19 Jun 89 10:50:37 BST Message-Id: <4312.8906190950 at entropy.cs.exeter.ac.uk> To: jose at CONFIDENCE.PRINCETON.EDU Cc: connectionists at CS.CMU.EDU In-Reply-To: jose at edu.princeton.confidence's message of Fri, 16 Jun 89 11:35:14 EDT <8906161535.AA02388 at suspicion.Princeton.EDU Subject: lost papers... Hanson's comment seems a bit bitter ... i wonder what is really behind it. In a field growing as rapidly as connectionism, I would have thought that we need a lot more annual meetings. And there will of course be more when we in Europe get our act together. I think that the field is getting far to large to be unified. Papers come at me from all directions - the structure of the hippocampus to reasoning in natural language understanding and low-level visual perception. Surely it is inevidable that the "field" will fractionate into many specialised sub-fields as is happening at the cognitive science meeting etc. as Jay pointed out. Imagine having one annual psychology meeting, or one annual physics meeting. noel sharkey Centre for Connection Science JANET: noel at uk.ac.exeter.cs Dept. Computer Science University of Exeter UUCP: !ukc!expya!noel Exeter EX4 4PT Devon BITNET: noel at cs.exeter.ac.uk@UKACRL U.K. Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa02032; 19 Jun 89 10:53:39 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa17474; 19 Jun 89 8:49:38 EDT Received: from PRINCETON.EDU by CS.CMU.EDU; 19 Jun 89 08:47:31 EDT Received: from suspicion.Princeton.EDU by Princeton.EDU (5.58+++/2.17) id AA12779; Mon, 19 Jun 89 08:47:20 EDT Received: by suspicion.Princeton.EDU (4.0/1.81) id AA00471; Mon, 19 Jun 89 08:52:13 EDT Date: Mon, 19 Jun 89 08:52:13 EDT From: "Stephen J. Hanson" Message-Id: <8906191252.AA00471 at suspicion.Princeton.EDU> To: jose at confidence.Princeton.EDU, noel%CS.EXETER.AC.UK at pucc.Princeton.EDU Subject: Re: lost papers... Cc: connectionists at CS.CMU.EDU Flowers and all.. (flame on) Actually, I agree with Jay and Noel, diversity is the spice of life. In fact, one of the key aspects of this field is the fact that one can find 8-9 diciplines in the room represented. I think we forget sometimes how remarkable this actually is. I also think it is important to remember as we rush to sell our version of the story, or make a septobijjillion dollars selling neural net black boxes or teaching neural nets to the great unwashed, that we don't shoot ourselves in our collective feet --after all who is the competition here? remember some sort of proto-AI killed off this field less than 40 years ago..all I was suggesting was that it would be nice if there was sort of agreement about organization, politics and coherence about progress in the field--god knows--not about the subject matter. I realize this is unlikely and somewhat naive, and more (conferences, journals, etc..) is usually better in any field.. its just that all these flowers look a bit carnivorous. (flame off) and enough.. back to some substance please. jose Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa20520; 20 Jun 89 14:35:03 EDT Received: from ri.cmu.edu by Q.CS.CMU.EDU id aa14829; 20 Jun 89 14:08:20 EDT Received: from TRACTATUS.BELLCORE.COM by RI.CMU.EDU; 20 Jun 89 14:05:10 EDT Received: by tractatus.bellcore.com (5.61/1.34) id AA13847; Mon, 19 Jun 89 08:51:16 -0400 Date: Mon, 19 Jun 89 08:51:16 -0400 From: Stephen J Hanson Message-Id: <8906191251.AA13847 at tractatus.bellcore.com> To: DJERS at TUCC.TUCC.EDU, TheoryNet at ibm.com, ailist at kl.sri.com, arpanet-bboards at mc.lcs.mit.edu, biotech%umdc.BITNET at cunyvm.cuny.edu, chipman at NPRDC.NAVY.MIL, conferences at hplabs.hp.com, connectionists at RI.CMU.EDU, dynsys-l%unc.BITNET at cunyvm.cuny.edu, epsynet%uhupvm1.BITNET at cunyvm.cuny.edu, gazzaniga at tractatus.bellcore.com, hirst at ROCKY2.ROCKEFELLER.EDU, info-futures at bu-cs.bu.edu, kaiser%yorkvm1.BITNET at cunyvm.cuny.edu, keeler at mcc.com, mike%bucasb.bu.edu at bu-it.bu.edu, msgs at tractatus.bellcore.com, msgs at confidence.princeton.edu, neuron at ti-csl.csc.ti.com, parsym at sumex-aim.stanford.edu, physics at mc.lcs.mit.edu, self-org at mc.lcs.mit.edu, soft-eng at MINTAKA.LCS.MIT.EDU, taylor at hplabsz.hpl.hp.com, vision-list at ADS.COM Subject: NIPS Schedule *********NIPS UPDATE********** Deadline for abstracts and summaries was May 30, 1989. We have now received over 460 contributions (almost 50% more than last year!). They are now logged in, and cards acknowledging receipt will be mailed next week to authors. Authors who have not received an acknowledgement by June 30 should write to Kathie Hibbard at the NIPS office; it's possible we got your address wrong in our database, and this will help us catch these things. Refereeing will take July. Collecting the results and defining a final program will be done in August. We plan to mail letters informing authors of the outcome during the first week of September. At that time, we will send all registration material, information about prices, and a complete program. If you haven't heard from us in late September, again please write, to help us straighten things out. **********NIPS UPDATE*********** Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa27436; 20 Jun 89 19:58:42 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa16421; 20 Jun 89 15:22:48 EDT Received: from ELROND.STANFORD.EDU by CS.CMU.EDU; 20 Jun 89 15:20:05 EDT Received: by elrond.Stanford.EDU (3.2/4.7); Tue, 20 Jun 89 12:03:08 PDT Date: Tue, 20 Jun 89 12:03:08 PDT From: Dave Rumelhart To: jose at confidence.Princeton.EDU, noel%CS.EXETER.AC.UK at pucc.Princeton.EDU Subject: Re: lost papers... Cc: connectionists at CS.CMU.EDU Not that I want to prolong this discussion, but as a member of the INNS board, I should like to clarify one point concerning the the IJCNN meetings and the INNS. In fact, the movment has been toward cooperation between the large IEEE sponsored meeting and the large INNS sponsored meeting, formerly known as the ICNN meeting. There has been an agreement whereby IEEE and INNS will co-sponsor two meetings per year -- roughly a summer meeting and a winter meeting. These jointly sponsored meetings have been dubbed IJCNN or International JOINT Conference on Neural Networks with the JOINT to signify the joint sponsorship. The movement of the INNS annual meeting from September 1989 to January 1990 has been by way of cooperating, not by way of competing. The current plan to continue to jointly sponsor two meetings per year as long as there is sufficient interest. In addition INNS and IEEE will probably help sponsor occasional European and Japanese meetings from time to time. That there will, in addition, of course, be a number of smaller meetings and workshops sponsored my other groups is, in my opinion, healthy and natural. There are many people with many interests working on things they call (or are willing to call) neural networks. It is normal that special interest groups should form within such a interdisciplinary field. I hope these comments are of some use to those bewildered by the array of meetings. I, in fact, believe that the establishment of the joint meeting plan between INNS and IEEE was a major accomplishment and certainly a move in the right direction for the field. David E. Rumelhart der at psych.stanford.edu Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa01011; 21 Jun 89 3:51:42 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa24477; 21 Jun 89 3:48:22 EDT Received: from UUNET.UU.NET by CS.CMU.EDU; 21 Jun 89 03:46:49 EDT Received: from munnari.UUCP by uunet.uu.net (5.61/1.14) with UUCP id AA14955; Wed, 21 Jun 89 03:46:42 -0400 From: munnari!cluster.cs.su.OZ.AU!ray at uunet.uu.net Received: from munnari.oz.au (munnari.oz) by murtoa.cs.mu.oz (5.5) id AA01986; Wed, 21 Jun 89 16:13:32 EST (from ray at cluster.cs.su.OZ.AU for uunet!connectionists at CS.CMU.EDU) Received: from cluster.cs.su.oz (via basser) by munnari.oz.au with SunIII (5.61+IDA+MU) id AA20510; Wed, 21 Jun 89 13:21:46 +1000 (from ray at cluster.cs.su.OZ.AU for connectionists at CS.CMU.EDU@murtoa.cs.mu.OZ.AU) Date: Wed, 21 Jun 89 13:20:51 +1000 Message-Id: <8906210321.20510 at munnari.oz.au> To: connectionists at CS.CMU.EDU Subject: location of IJCNN conferences Cc: munnari!ray at uunet.uu.net > From Connectionists-Request at q.cs.cmu.edu@murtoa.cs.mu.oz > From: der%elrond.Stanford.EDU at murtoa.cs.mu.oz (Dave Rumelhart) > Date: Tue, 20 Jun 89 12:03:08 PDT > To: jose at ... noel%... > Subject: Re: lost papers... > Cc: connectionists at CS.CMU.EDU > > ... The current plan to continue to jointly sponsor two meetings per > year as long as there is sufficient interest. In addition INNS and > IEEE will probably help sponsor occasional European and Japanese > meetings from time to time. This is one aspect of the current arrangement that bothers me. The INNS is an international society, but its *premier* conference is held only in the USA. Given the high percentage of US INNS members, and the amount of NN research done in the USA, its fitting that the bulk of NN conferences be held there. But the INNS' premier conference should be held occasionally outside the USA. The IJCAI and AAAI have a good arrangement. The IJCAI conferences are held every second year, with every second conference held in North America. The AAAI holds its own conference in the USA in the three out of four years that the IJCAI is not in North America. I think the exact periodicities for the AAAI and IJCAI may not suit the IEEE and INNS. It seems that people want more frequent conferences. But I think the general idea is sound. Perhaps every second occurrence of the January conference could be held outside the USA? (Leaving three out of every four IJCNN conferences in the USA.) As I understand it, the current arrangement will be reviewed in a year or two. I was glad to see the INNS and IEEE get together and coordinate the conferences. I just think some fine tuning is required. Raymond Lister Basser Department of Computer Science University of Sydney AUSTRALIA Internet: ray at basser.cs.su.oz.au@uunet.uu.net Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa05561; 21 Jun 89 7:39:33 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa26050; 21 Jun 89 5:08:45 EDT Received: from UUNET.UU.NET by CS.CMU.EDU; 21 Jun 89 05:06:33 EDT Received: from munnari.UUCP by uunet.uu.net (5.61/1.14) with UUCP id AA01454; Wed, 21 Jun 89 05:06:24 -0400 From: munnari!cluster.cs.su.OZ.AU!ray at uunet.uu.net Received: from munnari.oz.au (munnari.oz) by murtoa.cs.mu.oz (5.5) id AA06593; Wed, 21 Jun 89 18:38:56 EST (from ray at cluster.cs.su.OZ.AU for uunet!connectionists at CS.CMU.EDU) Received: from cluster.cs.su.oz (via basser) by munnari.oz.au with SunIII (5.61+IDA+MU) id AA24824; Wed, 21 Jun 89 18:38:52 +1000 (from ray at cluster.cs.su.OZ.AU for connectionists at CS.CMU.EDU@murtoa.cs.mu.OZ.AU) Date: Wed, 21 Jun 89 18:37:48 +1000 Message-Id: <8906210838.24824 at munnari.oz.au> To: singer at THINK.COM Subject: Re: Sparse vs. Dense networks Cc: munnari!ray at uunet.uu.net, connectionists at CS.CMU.EDU Hopfield and Tank's approach to the N city traveling salesman problem (TSP) ('"Neural" Computation of Decisions in Optimization Problems' Biol. Cybern. 52, pp 141-152 (1985)) used a N**2 matrix of neurons. Each neuron is connected to kN other neurons. Bailey and Hammerstrom pointed out that this high level of interconnect raises the area requirement to N**3. ('Why VLSI Implementations of Associative VLCNs Require Connection Multiplexing', IEEE International Conference on Neural Networks, San Diego (1988), pp II-173 to II-180 - anyone interested in implementation issues, but without a background in VLSI design, should read this paper). An N**2 area requirement is pushing it. N**3 is just about impractical, for any but very small TSPs. Adding metal layers for interconnect doesn't beat the problem (unless the number of metal layers is a function of N, which is impractical). I have devised an approach that uses an N**2 array of neurons, like H&T, but which requires only log N interconnect per neuron (giving an overall area requirement of (N**2)*(log N). My approach does not use multiplexing. It works by restricting the matrix to legal solutions. Despite this restriction, the approach generates quite good TSP solutions. Not only does the approach reduce the level of interconnect to practical levels, it also suggests that the capacity of analog approaches to move within the volume of the solution hypercube is not as important as previously thought. If you would like to know more about my approach, send your complete postal address, and I'll send you a paper. Raymond Lister Basser Department of Computer Science University of Sydney NSW 2006 AUSTRALIA Internet: ray%basser.cs.su.oz.au at uunet.uu.net CSNET: ray%basser.cs.su.oz at csnet-relay UUCP: {uunet,hplabs,pyramid,mcvax,ukc,nttlab}!munnari!basser.cs.su.oz!ray JANET: munnari!basser.cs.su.oz!ray at ukc Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa08791; 21 Jun 89 17:04:10 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa00924; 21 Jun 89 7:48:08 EDT Received: from TI.COM by CS.CMU.EDU; 21 Jun 89 07:45:45 EDT Received: by ti.com id AA01436; Tue, 20 Jun 89 21:32:43 CDT Message-Id: <8906210232.AA01436 at ti.com> Received: by tilde id AA29832; Tue, 20 Jun 89 21:30:34 CDT Date: Tue, 20 Jun 89 21:30:34 CDT From: lugowski at ngstl1.csc.ti.com To: connectionists at CS.CMU.EDU Subject: objection! As an original member of connectionists at cs.cmu (50+ folks), I have considered unsubscribing because of the recent runaway discussions a la Self-Org or AIList. Thus, I will keep my note short and ask for no replies, although I anticipate other strong opinions on the same subject. Please consider the note that follows to be my personal opinion. -- Marek Lugowski, TI AI Lab/IU CS Dept. I object to an abstract posted to the list recently, exerpted below, as the definition of connectionism given there is grossly misleading. As *one* prototype that does not fit the misleadingly drawn category, I suggest that my work is entirely connectionist; has been perceived by connectionists to be connectionist as early as 1986, yet in *no way* fits the definition cited below. I object knowing that the author presented at the Emergent Computation conference May 22-26 and had plenty opportunity to disabuse himself in Los Alamos of such distortions. The fact that he apprently chose not to do so is what I object to the strongest, only then to the distortion itself. Lugowski, Marek. "Computational Metabolism: Towards Biological Geometries for Computing", pp. 341 - 368, in _Artificial Life_, 2nd printing, Christopher Langton, ed., Addison-Wesley, Reading, MA: 1989. ----------------------- Objectionable citation: (Abstract of paper presented... June 11 1989) Connectionism is a family of statistical techniques for extracting complex higher-order correlations from data. ----------------------- Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa08797; 21 Jun 89 17:04:15 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa01359; 21 Jun 89 7:56:32 EDT Received: from TRACTATUS.BELLCORE.COM by CS.CMU.EDU; 21 Jun 89 07:54:30 EDT Received: by tractatus.bellcore.com (5.61/1.34) id AA13801; Mon, 19 Jun 89 08:20:12 -0400 Date: Mon, 19 Jun 89 08:20:12 -0400 From: Stephen J Hanson Message-Id: <8906191220.AA13801 at tractatus.bellcore.com> To: connectionists at CS.CMU.EDU Subject: test please ignore.. Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa09103; 21 Jun 89 17:37:15 EDT Received: from ri.cmu.edu by Q.CS.CMU.EDU id aa02801; 21 Jun 89 9:48:41 EDT Received: from TRACTATUS.BELLCORE.COM by RI.CMU.EDU; 21 Jun 89 09:46:13 EDT Received: by tractatus.bellcore.com (5.61/1.34) id AA16491; Wed, 21 Jun 89 09:45:36 -0400 Date: Wed, 21 Jun 89 09:45:36 -0400 From: Stephen J Hanson Message-Id: <8906211345.AA16491 at tractatus.bellcore.com> To: Connectionists at RI.CMU.EDU Subject: NIPS ------------------NIPS UPDATE------------------ Deadline for abstracts and summaries was May 30, 1989. We have now received over 460 contributions (almost 50% more than last year!). They are now logged in, and cards acknowledging receipt will be mailed next week to authors. Authors who have not received an acknowledgement by June 30 should write to Kathie Hibbard at the NIPS office; it's possible we got your address wrong in our database, and this will help us catch these things. Refereeing will take July. Collecting the results and defining a final program will be done in August. We plan to mail letters informing authors of the outcome during the first week of September. At that time, we will send all registration material, information about prices, and a complete program. If you haven't heard from us in late September, again please write, to help us straighten things out. ------------------NIPS UPDATE------------------ Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa00127; 22 Jun 89 16:48:56 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa18103; 22 Jun 89 11:59:23 EDT Received: from BOULDER.COLORADO.EDU by CS.CMU.EDU; 22 Jun 89 11:57:15 EDT Return-Path: Received: by boulder.Colorado.EDU (cu-hub.022489) Received: by tigger.colorado.edu (cu.generic.021288) Date: Thu, 22 Jun 89 09:56:59 MDT From: Dennis Sanger Message-Id: <8906221556.AA24065 at tigger> To: connectionists at cs.cmu.edu Subject: TR available: Contribution Analysis University of Colorado at Boulder Technical Report CU-CS-435-89 is now available: Contribution Analysis: A Technique for Assigning Responsibilities to Hidden Units in Connectionist Networks Dennis Sanger AT&T Bell Laboratories and the University of Colorado at Boulder ABSTRACT: Contributions, the products of hidden unit activations and weights, are presented as a valuable tool for investigating the inner workings of neural nets. Using a scaled-down version of NETtalk, a fully automated method for summarizing in a compact form both local and distributed hidden-unit responsibilities is demonstrated. Contributions are shown to be more useful for ascertaining hidden-unit responsibilities than either weights or hidden-unit activations. Among the results yielded by contribution analysis: for the example net, redundant output units are handled by identical patterns of hidden units, and the amount of responsibility a hidden unit takes on is inversely proportional to the number of hidden units. Please send requests to conn_tech_report at boulder.colorado.edu. Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa00855; 22 Jun 89 17:27:17 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa19129; 22 Jun 89 12:36:08 EDT Received: from BOULDER.COLORADO.EDU by CS.CMU.EDU; 22 Jun 89 12:33:22 EDT Return-Path: Received: by boulder.Colorado.EDU (cu-hub.022489) Received: by tigger.colorado.edu (cu.generic.021288) Date: Thu, 22 Jun 89 10:33:10 MDT From: Phillip E. Gardner Message-Id: <8906221633.AA25139 at tigger> To: connectionists at cs.cmu.edu, pdp at boulder.Colorado.EDU Subject: learning spatial data I'm interested in teaching a network to learn its way around a building. I want to use a technique similar to the one outlined by Dr. Widrow at the last NIPPS conference where a network learned how to backup a truck. If you could send me some references that might help, including references to what Dr. Widrow talked about, I would be most thankful. Sincerely, Phil Gardner gardner at boulder.colorado.edu Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa03744; 22 Jun 89 22:23:10 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa25755; 22 Jun 89 22:18:22 EDT Received: from UCSD.EDU by CS.CMU.EDU; 22 Jun 89 22:15:45 EDT Received: from sdbio2.ucsd.edu by ucsd.edu; id AA27631 sendmail 5.60/UCSD-1.0 Thu, 22 Jun 89 19:14:05 PDT for connectionists at cs.cmu.edu Received: by sdbio2.UCSD.EDU (3.2/UCSDGENERIC2) id AA23341 for delivery to terry at helmholtz.sdsc.edu; Thu, 22 Jun 89 19:16:14 PDT Date: Thu, 22 Jun 89 19:16:14 PDT From: terry%sdbio2 at ucsd.edu (Terry Sejnowski) Message-Id: <8906230216.AA23341 at sdbio2.UCSD.EDU> To: connectionists at cs.cmu.edu Subject: Neural Computation, Vol. 1, No. 2 Cc: terry at helmholtz.sdsc.edu NEURAL COMPUTATION -- Issue #2 -- July 1, 1989 Views: Recurrent backpropagation and the dynamical approach to adaptive neural computation. F. J. Pineda New models for motor control. J. S. Altman and J. Kien Seeing chips: Analog VLSI circuits for computer vision. C. Koch A proposal for more powerful learning algorithms. E. B. Baum Letters: A possible neural mechanism for computing shape from shading. A. Pentland Optimization in model matching and perceptual organization. E. Mjolsness, G. Gindi and P. Anandan Distributed parallel processing in the vestibulo-oculomotor system. T. J. Anastasio and D. A. Robinson A neural model for generation of some behaviors in the fictive scratch reflex. R. Shadmehr A robot that walks: Emergent behaviors from a carefully evolved network. R. A. Brooks Learning state space trajectories in recurrent neural networks. B. A. Pearlmutter. A learning algorithm for continually running fully recurrent neural networks. R. J. Williams and D. Zipser. Fast learning in networks of locally-tuned processing units. J. Moody and C. J. Darken. ----- SUBSCRIPTIONS: ______ $35. Student ______ $45. Individual ______ $90. Institution Add $9. for postage outside USA and Canada surface mail or $17. for air mail. MIT Press Journals, 55 Hayward Street, Cambridge, MA 02142. (617) 253-2889. ----- Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa24809; 23 Jun 89 11:39:13 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa13269; 23 Jun 89 11:34:30 EDT Received: from UUNET.UU.NET by CS.CMU.EDU; 23 Jun 89 11:31:42 EDT Received: from unido.UUCP by uunet.uu.net (5.61/1.14) with UUCP id AA22532; Fri, 23 Jun 89 11:31:36 -0400 Received: from gmdzi.UUCP (gmdzi) (1961) by unido.irb.informatik.uni-dortmund.de for uunet id AP04399; Fri, 23 Jun 89 15:19:41 +0100 Received: by gmdzi.UUCP id AA19582; Fri, 23 Jun 89 16:20:34 -0200 Received: by zsv.gmd.de id AA03551; Fri, 23 Jun 89 16:20:55 +0200 Date: Fri, 23 Jun 89 16:20:55 +0200 From: unido!gmdzi!zsv!joerg at uunet.UU.NET (Joerg Kindermann) Message-Id: <8906231420.AA03551 at zsv.gmd.de> To: connectionists at cs.cmu.edu Cc: gmdzi!zsv!joerg at uunet.UU.NET Subject: wanted: guest researcher If you are a postgraduate student of scientist with a strong background in neural networks, we are interested to get in touch with you: We are a small team (5 scientists plus students) doing research in neural networks here at the GMD. Currently we are applying to get funding for several positions of guest researchers. But: we need strong arguments (i.e. good people who are interested in a stay) to actually get the money. Our research interests are both theoretical and application oriented. The main focus is on temporal computation (time series analysis) by neural networks. We are using multi-layer recurrent networks and gradient learning algorithms (backpropagation, reinforcement). Applications are speech recognition, analysis of medical data (ECG, ...), and navigation tasks for autonomous vehicles (2-D simulation only). A second research direction is the optimization of neural networks by means of genetic algorithms. We are using both SUN3s and a parallel Computer (64 cpu transputer-based). So, if you are interested, please write a letter, indicating your background in neural networks and preferred dates for your stay. Dr. Joerg Kindermann Gesellschaft fuer Mathematik und Datenverarbeitung mbH (GMD) Postfach 1240 email: joerg at gmdzi.uucp D-5205 St. Augustin 1, FRG phone: (+49 02241) 142437 Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa16084; 25 Jun 89 12:30:57 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa27315; 25 Jun 89 12:25:53 EDT Received: from THINK.COM by CS.CMU.EDU; 25 Jun 89 12:24:12 EDT Return-Path: Received: from leander.think.com by Think.COM; Sun, 25 Jun 89 12:24:30 EDT Received: by leander.think.com; Sun, 25 Jun 89 12:22:43 EDT Date: Sun, 25 Jun 89 12:22:43 EDT From: singer at Think.COM Message-Id: <8906251622.AA07513 at leander.think.com> To: unido!gmdzi!zsv!joerg at uunet.uu.net Cc: connectionists at cs.cmu.edu, gmdzi!zsv!joerg at uunet.uu.net In-Reply-To: Joerg Kindermann's message of Fri, 23 Jun 89 16:20:55 +0200 <8906231420.AA03551 at zsv.gmd.de> Subject: wanted: guest researcher Date: Fri, 23 Jun 89 16:20:55 +0200 From: unido!gmdzi!zsv!joerg at uunet.uu.net (Joerg Kindermann) If you are a postgraduate student of scientist with a strong background in neural networks, we are interested to get in touch with you: [...] Dr. Joerg Kindermann Gesellschaft fuer Mathematik und Datenverarbeitung mbH (GMD) Postfach 1240 email: joerg at gmdzi.uucp D-5205 St. Augustin 1, FRG phone: (+49 02241) 142437 Though I am not a postgraduate student (i.e. I do not have a PhD), your invitation made me very interested. Especially when I saw your address. Is your organization the same one in which work relating the methods of statistical mechanics to "Darwinian" optimization paradigms has been done? Unfortunately I do not have the particular papers and names at my disposal right now, but the Gesellschaft sounds familiar. My own background includes a Bachelor of Science in Neural Science and a bachelor of Arts in Philosophy. I made an oral presentation at the first NIPS (Neural Information Processing) Conference in Denver, CO in 1987 on hybrid neural net/biological systems. I spent a year beginning my PhD studies at Johns Hopkins with Terry Sejnowski, but had to stop temoprarily because Dr Sejnowski moved to California. I am currently employed by Thinking Machines Corporation workingon their 64K processor Connection Machine doing neural network research, genetic algorithm research, combinatorial optimization work, and statistical work. I also have a working knowledge of German from having worked in Frankfurt for a summer. I would be extremely interested in further discussing this with you, if my qua lifications seem appropriate. Alexander Singer Thinkning Machines Corp. 245 First St. Cambridge, MA 02142 USA Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa01703; 25 Jun 89 17:39:39 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa27905; 25 Jun 89 12:36:58 EDT Received: from [128.188.1.13] by CS.CMU.EDU; 25 Jun 89 12:33:55 EDT Received: by net2.m2c.org (5.57/sendmail.28-May-87) id AA00561; Sun, 25 Jun 89 12:28:13 EDT Received: by m2c.m2c.org (5.57/sendmail.28-May-87) id AA05897; Sun, 25 Jun 89 12:30:16 EDT Received: by wpi (4.12/4.7) id AA09483; Sun, 25 Jun 89 12:32:44 edt Date: Sun, 25 Jun 89 12:32:44 edt From: weili at wpi.wpi.edu (Wei Li) Message-Id: <8906251632.AA09483 at wpi> To: connectionists at CS.CMU.EDU Subject: information on fundings wanted Hi, if any one could send me some notes on fundings from NSF, AFOR, NIH and DARPA, including areas of interested projects, amount of available money, dead line for accepting proposals, and phone numbers of people to contact to, I will appriate it very much. This information was given in IJCNN 89 Washington D.C. neural network conference. My e-mail address is weili at wpi.wpi.edu ---- Wei Li Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa09527; 28 Jun 89 9:22:00 EDT Received: from ri.cmu.edu by Q.CS.CMU.EDU id aa08554; 28 Jun 89 8:08:24 EDT Received: from VMA.CC.CMU.EDU by RI.CMU.EDU; 28 Jun 89 08:04:57 EDT Received: from CMUCCVMA by VMA.CC.CMU.EDU ; Wed, 28 Jun 89 08:03:12 EDT Received: from EB0UB011.BITNET (D4PBPHB2) by CMUCCVMA (Mailer X1.25) with BSMTP id 2266; Wed, 28 Jun 89 08:03:11 EDT Date: Wed, 28 Jun 89 12:58:05 HOE To: connectionists at c.cs.cmu.edu From: D4PBPHB2%EB0UB011.BITNET at VMA.CC.CMU.EDU Comment: CROSSNET mail via MAILER at CMUCCVMA Date: 28 June 1989, 12:57:19 HOE From: D4PBPHB2 at EB0UB011 To: connectionists at c.cs.cmu.edu Add/Subscribe Perfecto Herrera Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa01812; 29 Jun 89 6:05:54 EDT Received: from ri.cmu.edu by Q.CS.CMU.EDU id aa21472; 29 Jun 89 6:00:41 EDT Received: from UUNET.UU.NET by RI.CMU.EDU; 29 Jun 89 05:58:31 EDT Received: from munnari.UUCP by uunet.uu.net (5.61/1.14) with UUCP id AA02442; Thu, 29 Jun 89 05:58:21 -0400 Received: from munnari.oz.au (munnari.oz) by murtoa.cs.mu.oz (5.5) id AA07438; Thu, 29 Jun 89 18:42:41 EST (from guy at flinders.cs.flinders.oz for uunet!connectionists at RI.CMU.EDU) Received: from flinders.cs.flinders.oz (via murtoa) by munnari.oz.au with SunIII (5.61+IDA+MU) id AA08527; Thu, 29 Jun 89 18:33:36 +1000 (from guy at flinders.cs.flinders.oz for connectionists at RI.CMU.EDU@murtoa.cs.mu.OZ.AU) Message-Id: <8906290833.8527 at munnari.oz.au> Received: by flinders.cs.flinders.oz.au(4.0/SMI-3.2) id AA03504; Thu, 29 Jun 89 17:56:22 CST Date: Thu, 29 Jun 89 17:56:22 CST From: munnari!cs.flinders.oz.au!guy at uunet.UU.NET (Guy Smith) To: connectionists at RI.CMU.EDU Subject: Tech Report available Cc: munnari!guy at uunet.UU.NET The Tech Report "Back Propagation with Discrete Weights and Activations" describes a modification of BP which generates a net with discrete (but not integral) weights and activations. The modification is simple: weights and activations are restricted to discrete values. The weights/activations calculated by BP are rounded to one of the neighbouring discrete values. For simple discrete problems, the learning performance of the net was not much affected until the granularity of the legal weight/activation values was as coarse as ten values per integer (ie 0.0, 0.1, 0.2, ...). To request a copy, mail to "guy at cs.flinders.oz..." or write to Guy Smith, Computer Science Department, Flinders University, Adelaide 5042, AUSTRALIA. Guy Smith. Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa03460; 29 Jun 89 9:22:27 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa00460; 29 Jun 89 9:17:33 EDT Received: from DST.BOLTZ.CS.CMU.EDU by CS.CMU.EDU; 29 Jun 89 08:24:18 EDT Received: from DST.BOLTZ.CS.CMU.EDU by DST.BOLTZ.CS.CMU.EDU; 29 Jun 89 08:23:30 EDT To: connectionists at cs.cmu.edu Reply-To: Dave.Touretzky at cs.cmu.edu cc: copetas at cs.cmu.edu Subject: "Rules and Maps" tech report Date: Thu, 29 Jun 89 08:23:22 EDT Message-ID: <3871.615126202 at DST.BOLTZ.CS.CMU.EDU> From: Dave.Touretzky at B.GP.CS.CMU.EDU Rules and Maps in Connectionist Symbol Processing Technical Report CMU-CS-89-158 David S. Touretzky School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213-3890 ABSTRACT: This report contains two papers to be presented at the Eleventh Annual Conference of the Cognitive Science Society. The first describes a simulation of chunking in a connectionist network. The network applies context-sensitive rewrite rules to strings of symbols as they flow through its input buffer. Chunking is implemented as a form of self-supervised learning using backpropagation. Over time, the network improves its efficiency by replacing simple rule sequences with more complex chunks. The second paper describes the first implementation of Lakoff's new theory of cognitive phonology. His approach is based on a multilevel representation of utterances to which all rules apply in parallel. Cognitive phonology is free of the rule ordering constraints that make classical generative theories computationally awkward. The connectionist implementation utilizes a novel ``many maps'' architecture that may explain certain constraints on phonological rules not adequately accounted for by more abstract models. ================ To order copies of this tech report, please send email to Catherine Copetas (copetas at cs.cmu.edu), or write the School of Computer Science at the address above. If you previously requested a copy of CMU-CS-89-147 (Connectionism and Compositional Semantics), you will receive a copy of this new report automatically. In fact, it should arrive in your mailbox today. Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa10761; 29 Jun 89 15:12:55 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa02123; 29 Jun 89 10:41:34 EDT Received: from VMA.CC.CMU.EDU by CS.CMU.EDU; 29 Jun 89 10:40:08 EDT Received: from CMUCCVMA by VMA.CC.CMU.EDU ; Thu, 29 Jun 89 10:39:37 EDT Received: from UKACRL.BITNET by CMUCCVMA (Mailer X1.25) with BSMTP id 0225; Thu, 29 Jun 89 10:39:36 EDT Received: from RL.IB by UKACRL.BITNET (Mailer X1.25) with BSMTP id 7397; Thu, 29 Jun 89 15:37:20 BST Received: Via: UK.AC.EX.CS; 29 JUN 89 15:37:17 BST Received: from entropy by expya.cs.exeter.ac.uk; Thu, 29 Jun 89 15:29:29-0000 From: Noel Sharkey Date: Thu, 29 Jun 89 15:28:14 BST Message-Id: <6270.8906291428 at entropy.cs.exeter.ac.uk> To: connectionists at CS.CMU.EDU Subject: post graduate studentships RESEARCH STUDENTSHIPS IN COMPUTER SCIENCE University of Exeter The Department of Computer Science invites applications for Science and Engineering Research Council (SERC) PhD. quotas for October, 1989. Any area of computer science within the Department's research interests (new generation architectures and languages, methodology, interfaces, logics) will be considered. However, for a least one of the quotas, preference will be given to candidates with an interest in CONNECTIONIST or NEURAL NETWORK research, particularly in relation to applications within the domain of natural language processing or simulations of human memory. Candidates should possess a first degree of at least 2(i) standard in order to be eligible for the award of SERC research studentship. Closing date for applications is 14th July, 1989. For further information about the Department's research interests, eligibility consideration and application procedure, please contact: .nf Dr Noel Sharkey, Reader in Computer Science JANET: noel at uk.ac.exeter.cs Dept. Computer Science University of Exeter Exeter EX4 4PT U.K. (Telephone: 0392 264061) .fi Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa15407; 29 Jun 89 22:07:18 EDT Received: from ri.cmu.edu by Q.CS.CMU.EDU id aa10205; 29 Jun 89 17:36:17 EDT Received: from CS.UTEXAS.EDU by RI.CMU.EDU; 29 Jun 89 17:34:27 EDT Date: Thu, 29 Jun 89 16:25:24 CDT From: yu at cs.utexas.edu (Yeong-Ho Yu) Posted-Date: Thu, 29 Jun 89 16:25:24 CDT Message-Id: <8906292125.AA27045 at cs.utexas.edu> Received: by cs.utexas.edu (5.59/36.2) id AA27045; Thu, 29 Jun 89 16:25:24 CDT To: guy!s.flinders.oz.au!guy at uunet.UU.NET Cc: connectionists at RI.CMU.EDU, munnari!guy at uunet.UU.NET In-Reply-To: Guy Smith's message of Thu, 29 Jun 89 17:56:22 CST <8906290833.8527 at munnari.oz.au> Subject: Tech Report available I'd like to get a copy of your tech report "Back Propagation with Discrete Weights and Activations". My address is Yeong-Ho Yu AI Lab The University of Texas at Austin Austin, TX 78712 (yu at cs.utexas.edu) Thanks in advance. Yeong ---------- Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa23828; 30 Jun 89 12:43:18 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa15581; 30 Jun 89 9:30:39 EDT Received: from VMA.CC.CMU.EDU by CS.CMU.EDU; 30 Jun 89 09:28:32 EDT Received: from CMUCCVMA by VMA.CC.CMU.EDU ; Fri, 30 Jun 89 09:28:04 EDT Received: from WEIZMANN.WEIZMANN.AC.IL by CMUCCVMA (Mailer X1.25) with BSMTP id 7131; Fri, 30 Jun 89 09:28:03 EDT Received: by WEIZMANN (Mailer R2.03B) id 3415; Fri, 30 Jun 89 10:00:56 +0300 Date: Fri, 30 Jun 89 09:52:40 +0300 From: "Harel Shouval, Tal Grossman" Subject: Preprint available To: connectionists at cs.cmu.edu The following preprint describes a theoretical and experimental work on optical neural network that is based on a negative weights nn model. Please send your requests by email to: feshouva at weizmann (bitnet), or write to: Harel Shouval, Electronics Dept., Weizmann Inst. Rehovot 76100, ISRAEL. --------------------- An All-Optical Hopfield Network: Theory and Experiment ------------------------------------------------------- Harel Shouval, Itzhak Shariv, Tal Grossman, Asher A. Friesem and Eytan Domany. Dept. of Electronics, Weizmann Institute of Science, Rehovot 76100 Israel. --- ABSTRACT --- Realization of an all-optical Hopfield-type neural network is made possible by eliminating the need for subtracting light intensities. This can be done without significntly degrading the network's preformance, if only inhibitory connections (i.e. $J_{ij}<0$) are used. We present theoretical analysis of such a network, and its experimental implementation, that uses a liquid crystal light valve for the neurons and an array of sub-holograms for the interconnections. ----------------------------------------------------------------------- From Connectionists-Request at cs.cmu.edu Thu Jun 1 10:13:45 1989 From: Connectionists-Request at cs.cmu.edu (Connectionists-Request@cs.cmu.edu) Date: Thu, 01 Jun 89 10:13:45 EDT Subject: EURASIP Workshop on Neural Networks Message-ID: <17603.612713625@B.GP.CS.CMU.EDU> CALL FOR PAPERS EURASIP WORKSHOP ON NEURAL NETWORKS Sesimbra, Portugal February 15-17, 1990 The workshop will be held at the Hotel do Mar in Sesimbra, Portugal. It will take place in 1990, from February 15 morning to 17 noon, and will be sponsored by EURASIP, the European Association for Signal Processing. It will be open to participants from all countries, both from inside and outside of Europe. Contributions from all fields related to the neural network area are welcome. A (non-exclusive) list of topics is given below. Care is being taken to ensure that the workshop will have a high level of quality. Proposed contributions will be evaluated by an international technical committee. A proceedings volume will be published, and will be handed to participants at the beginning of the workshop. The number of participants will be limited to 50. Full contributions will take the form of oral presentations, and will correspond to papers in the proceedings. Some short contributions will also be accepted, for presentation of ongoing work, projects (ESPRIT, BRAIN, DARPA,...), etc. They will be presented in poster format, and will not originate any written publication. A small number of non-contributing participants may also be accepted. The official language of the workshop will be English. TOPICS: - signal processing (speech, image,...) - pattern recognition - algorithms (training procedures, new structures, speedups,...) - generalization - implementation - specific applications where NN have been proved better than other approaches - industrial projects and realizations SUBMISSION PROCEDURES: Submissions, both for long and for short contributions, will consist of (strictly) 2-page summaries. Three copies should be sent directly to the Technical Chairman, at the address given below. The calendar for contributions is as follows: Full contributions Short contributions Deadline for submission June 15, 1989 Oct 1, 1989 Notif. of acceptance Sept 1, 1989 Nov 15, 1989 Camera-ready paper Nov 1, 1989 ORGANIZING COMMITTEE General Chairman: Luis B. Almeida, INESC, Apartado 10105, P-1017 Lisboa, Codex, Portugal Phone: +351-1-544607; Fax: +351-1-525843; E-mail: {any backbone, uunet}!mcvax!inesc!lba Technical Chairman: Christian J. Wellekens, Philips Research Laboratory Brussels, Av. Van Becelaere 2, Box 8, B-1170 Brussels, Belgium Phone: +32-2-6742275; Fax: +32-2-6742299; E-mail: wlk at prlb2.uucp Technical committee: John Bridle (Royal Signal and Radar Establishment, Malvern, UK), Herve Bourlard (Intern. Computer Science Institute, Berkeley, USA), Frank Fallside (University of Cambridge, Cambridge, UK), Francoise Fogelman (Ecole de H. Etudes en Informatique, Paris, France), Jeanny Herault (Institut Nat. Polytech. de Grenoble, Grenoble, France), Larry Jackel (AT\&T Bell Labs, Holmdel, NJ, USA), Renato de Mori (McGill University, Montreal, Canada), H. Muehlenbein (GMD, Sankt Augustin, FRG). REGISTRATION, FINANCE, LOCAL ARRANGEMENTS: Joao Bilhim, INESC, Apartado 10105, P-1017 Lisboa, Codex, Portugal Phone: +351-1-545150; Fax: +351-1-525843. WORKSHOP SPONSOR EURASIP - European Association for Signal Processing CO-SPONSORS: INESC - Instituto de Engenharia de Sistemas e Computadores, Lisbon, Portugal IEEE, Portugal Section THE LOCATION: Sesimbra is a fishermens village, located in a nice region about 30 km south of Lisbon. Special transportation from/to Lisbon will be arranged. The workshop will end on a Saturday at lunch time; therefore, the participants will have the option of either flying back home in the afternoon, or staying for sightseeing for the remainder of the weekend in Sesimbra and/or Lisbon. An optional program for accompanying persons is being organized. From tds at ai.mit.edu Thu Jun 1 12:54:10 1989 From: tds at ai.mit.edu (tds@ai.mit.edu) Date: Thu, 1 Jun 89 12:54:10 edt Subject: supervised learning In-Reply-To: Alain Grumbach's message of Tue, 30 May 89 17:34:56 +0200 <8905301534.AA17056@ulysse.enst.fr> Message-ID: <8906011654.AA02920@mauriac.ai.mit.edu> I agree that there is some artificiality in the distinction between supervised and unsupervised learning. Traditionally, the distinction seems to be made based on whether the learning algorithm has available to it the desired outputs of the network. If the algorithm can be described by an energy function (such as minimal output error for Backpropagation), then supervised learning seems to require an energy function which explicitly includes the desired outputs of the network. Unsupervised learning (in some cases) can be described by energy functions which specify properties or statistics of the outputs (or sometimes the weights). However, it seems easy to imagine a continuum of energy functions between these two types. For example, a BP net which requires outputs to be close to the desired points within some tolerance, or an unsupervised algorithm designed so that the desired properties of the outputs can be satisfied only by specific output values which are actually the desired outputs. And what about an algorithm whose energy function is the mean squared distance to the desired weights themselves? (Learning in this case is very easy and fast, but not particularly interesting!) This is complicated by the fact that very often the same solution (final set of weights) can be obtained from very different algorithms with very different energy functions (imagine training a BP network to converge to a Kohonen network). So there seems to be an artificial distinction between supervised nets (gradient descent on an energy function defined by mean-squared output error) and unsupervised nets (everything else). But there does seem to be some nice intuitive idea behind this. Perhaps it is based on the difference between giving someone the right answer to a math problem and teaching them how to solve it themselves. In other words, in real life we seem to make the supervised/unsupervised distinction quite naturally. Does anyone have any ideas why this distinction is so pervasive? Terry Sanger tds at wheaties.ai.mit.edu From spencer at iris.ucdavis.edu Thu Jun 1 11:18:31 1989 From: spencer at iris.ucdavis.edu (Richard Spencer) Date: Thu, 1 Jun 89 08:18:31 pdt Subject: please remove me from the mailing list Message-ID: <8906011518.AA16749@iris> I appologize for sending this to the regular address, but I have lost the appropriate address for this kind of correspondence. In any event, I asked to be removed a month ago and the volume of mail has decreased, but it hasn't stopped completely. I don't understand that, but I would appreciate it if you could prevent it. Thank you for the effort Richard Spencer From tbaker at cse.ogc.edu Thu Jun 1 20:24:27 1989 From: tbaker at cse.ogc.edu (Thomas Baker) Date: Thu, 1 Jun 89 17:24:27 -0700 Subject: Image Databases Message-ID: <8906020024.AA01888@ogccse.OGC.EDU> I am a student doing research in image pattern recognition applications of connectionist networks. I would like to get a copy of some image databases. One database that I have seen research on is the U.S. Postal Service handwritten zip code characters. There is also a more general database of pictures, most research that I have seen uses the same pictures (i.e. Lena). Where do I get a copy of this data? The method that I would prefer is to order a tape from the group that officially distributes the data. I also have ftp access, or I can supply my own tape if somebody has the data locally. Thanks in advance for any help you can give me. Thomas Baker UUCP: ..ucbvax!tektronix!ogccse!tbaker Oregon Graduate Center CSNET: tbaker at cse.ogc.edu Beaverton, Oregon 97005 From AGB11 at PHOENIX.CAMBRIDGE.AC.UK Fri Jun 2 15:43:11 1989 From: AGB11 at PHOENIX.CAMBRIDGE.AC.UK (AGB11@PHOENIX.CAMBRIDGE.AC.UK) Date: Fri, 02 Jun 89 15:43:11 BST Subject: Supervised and Unsupervised Learning Message-ID: Here are some thoughts on the distinction between supervised and unsupervised learning. First, despite the apparent exhaustiveness of the terminology, these two types of learning are not all that there are (see below). Supervised learning means that the classes of the training instances are known and available to the learning system. Or when not applied to a classification problem, it means that particular functional values are available for the training instances (although perhaps corrupted by noise). Unsupervised learning usually refers to the case where the class labels or function values are not provided with the training instances. I agree with Terry Sanger that there can be all sorts of problems falling between these extremes, but there are still other kinds of learning tasks. I've never much liked the term unsupervised learning because (aside from the fact that together with supervised learning it misleadingly implies that there are no other kinds of learning) it seems sometimes to be understood as meaning a method that can do the same thing a supervised method can but doesn't need the supervision. My view (and I welcome comments on this) is that what is usually called unsupervised learning is a kind of supervised learning with a fixed, builtin teacher. This teacher is embodied in some principle, such as principal component analysis, clustering with a specific criterion function on clusterings, infomax, etc. So, according to this view, unsupervised learning is a more specific type of process than is supervised learning. Methods are usually not built to accomodate the possibility of some input specifying what criterion to use in the learning process. Then there are other sorts of learning tasks where the information given by the learning system's environment is not as specific as a specification of desired responses. For example, in a reinforcement learning task there are desired responses but the system is not told what they are: it has to search for them (in addition to the search in parameter space for weights to remember the results of previous searches). For example, Widrow, Gupta, and Maitra (Systems Man and Cybernetics, vol 5, 1973, pp. 455-465) discuss "learning with a critic". Rewards and punishments are logically different from signed errors or desired responses. Lots of other people (including me) have discussed this type of learning task. In thinking about supervised and unsupervised learning tasks as these terms are used in everyday language, I think it is greatly misleading to assume that the technical meanings of these terms adequately characterize the kind of learning we see in animals. Usually, it seems to me, the kind of supervision we see really is a process whereby a sequence of problem-solving tasks are presented, often with graded difficulty, and for each the learning system faces a complex learning control task. Of course, to solve these tasks, various parts of the sytem will probably be facing unsupervised, supervised, reinforcement, and probably other kinds of tasks that are all worked on together. Similarly, unsupervised learning used in the vernacular seems to mean a process where all these things go on, except the system itself in creating the sequence of problem-solving tasks (cf. Mitchell's LEX learning system). Finally, I think it is important to distinguish carefully between learning tasks and learning algorithms or procedures (although I haven't been particularly careful about this above). Tasks are characterized by the kinds of information the system is allowed to get, how much it has at the start, what the objective is, etc. A specific learning algorithm is more-or-less capable of achieving various objectives in various kinds of tasks. Usually a real application encompasses lots of different learning tasks depending on where you draw the line between the learning system and the rest. A. Barto agb11 at phx.cam.ac.uk From tp at ai.mit.edu Fri Jun 2 15:32:32 1989 From: tp at ai.mit.edu (Tomaso Poggio) Date: Fri, 2 Jun 89 15:32:32 EDT Subject: Image Databases In-Reply-To: Thomas Baker's message of Thu, 1 Jun 89 17:24:27 -0700 <8906020024.AA01888@ogccse.OGC.EDU> Message-ID: <8906021932.AA02272@rice-chex.ai.mit.edu> Could you let me know what you learn from your request? I am also interested in obtaining similar kinds of data... From hinton at ai.toronto.edu Fri Jun 2 15:45:52 1989 From: hinton at ai.toronto.edu (Geoffrey Hinton) Date: Fri, 2 Jun 89 15:45:52 EDT Subject: supervised learning In-Reply-To: Your message of Tue, 30 May 89 11:34:56 -0400. Message-ID: <89Jun2.154559edt.11075@ephemeral.ai.toronto.edu> I think the current useage of the terms "supervised" and "unsupervised" learning is pretty much OK. To try to change the meanings of these terms now would cause even more confusion. There is a natural and important distinction between data sets (wherever they come from) that consist of Input-output pairs where the task is to predict the output given the input, and data-sets that simply consist of an ensemble of vectors where the task (if its specified at all) is typically to find natural clusters, or a compact code, or a code with independent components. Naturally there are tricky cases. One of these is when we create a "multi-completion" task by trying to predict each part of the input from all the other parts. We are really turning an unsupervised task into a whole set of supervised tasks. Another problem arises when we have some LOCAL objective function (internal to the network) that acts as a teacher. If we backpropagate derivatives from this LOCAL function, are we doing supervised or unsupervised learning? It might be reasonable to say that we are doing learning that is locally supervised but globally unsupervised (i.e. the local supervision information is not derived from more global supervision information that is supplied with the data). Its worth noting that the statistics literature makes a very similar basic distinction. It uses the term "discriminant" analysis for supervised learning and "cluster analysis" for unsupervised learning (or the subset of unsupervised learning that they know how to do.) Geoff From honavar at cs.wisc.edu Sat Jun 3 18:21:22 1989 From: honavar at cs.wisc.edu (Vasant Honavar) Date: Sat, 3 Jun 89 17:21:22 -0500 Subject: Supervised learning Message-ID: <8906032221.AA00949@goat.cs.wisc.edu> A number of "supervised" learning tasks can be formulated as learning tasks in which the "feedback" comes from a different sensory modality - e.g., the feedback for a language learning may come from a visual scene. Learning in the two modalities may be initiated through a process of "bootstrapping". Feedback can be viewed as coming from simply another input stream. Instead of working with one input stream (as is typically the case with most backprop-like schemes), the system learns by forming associations between two or more input streams. In this case, it is hard to argue that the learning is any more "supervised" than in any of the so-called "unsupervised" methods. This is conceptually quite similar to the self-supervised methods (Hinton, 1987; "Connectionist Learning Procedures"). It therefor appears useful to view the various learning schemes as part of a continuum: Schemes requiring very specific feedback lie (e.g., the desired output) on one extreme; those requiring no feedback (all of learning is simply storing some abstractions of the input) lie on the other extreme; and a host of schemes using various rather diffuse and non-specific forms of feedback (and possibly feedback that is hidden in the form of local objective functions, information- theoretic "constraints", the locus of interactions between units, etc.) lie in the middle. Vasant Honavar From gmdzi!joerg%zsv at uunet.UU.NET Mon Jun 5 07:36:31 1989 From: gmdzi!joerg%zsv at uunet.UU.NET (Joerg Kindermann) Date: Mon, 5 Jun 89 13:36:31 +0200 Subject: speech recognition data Message-ID: <8906051136.AA04744@zsv.gmd.de> I would like to get any pointers to *mass* data which can be used for speech recognition tasks (talker independent, both isolated words and fluent speech). Please send e-mail, if there is enough interest, I'll post a summary of this request. Thanks in advance Joerg. address: Dr. Joerg Kindermann Gesellschaft fuer Mathematik und Datenverarbeitung (GMD) Schloss Birlinghoven Postfach 1240 D-5205 St. Augustin 1 WEST GERMANY e-mail: joerg at gmdzi.uucp From der at elrond.Stanford.EDU Mon Jun 5 12:20:48 1989 From: der at elrond.Stanford.EDU (Dave Rumelhart) Date: Mon, 5 Jun 89 09:20:48 PDT Subject: Supervised learning Message-ID: It has seemed to me that the supervised/unsupervised distinction is insufficiently fine-grained and is wrongly associated with learning methods. (For example, backprop is often thought of as supervised whereas, say, Hebbian rules are thought of as unsupervised.) We ought to distinguish between kinds of learning tasks on the one hand and learning rules on the other. I have distinguished four learning tasks: (1) PATTERN ASSOCIATION in which the goal is to learn a series of I/O pairs. In this case the networks learns to be the function which maps input elements to their corresponding output elements. This is usually what people have in mind with supervised learning. (2) AUTOASSOCIATION in which the goal is to store a pattern so that, essentially, any part of the pattern can be used to retrieve the whole pattern. This is the connectionist implementation of content-addressible memory. This is difficult to classify in the supervised/unsupervised dimension. On the one hand it involves a single input item and hence should be unsupervised, on the other hand, the network is told exactly what to store, so it should be viewed as unsupervised. (3) REGULARITY-DETECTION in which the goal is to discover statistical regularities (such as clusters or featural decompositions etc.) in the input patterns. This would seem to be the prototype unsupervised case. (4) REINFORCEMENT LEARNING in which the goal is to learn a set of actions which either maximize some postive reinforcement variable or minimize some negative reinforcement variable or both. This to is difficult to classify on the supervised/unsupervised dimension. It would appear that it is supervised because the environment is guiding its behavior. On the other hand it is unsupervised in as much as its free to make whatever response it wishes so long as it maximizes reinforcement. Perhaps partially supervised would due for this case. More important than this classification, it is important to realize that these categories are (nearly) orthogonol to the learning rules employed. Thus, we have (1) CORRELATIONAL (Hebbian) LEARNING which can be employed for pattern association tasks, autoassociation tasks, regularity detection tasks and (perhaps) reinforcement learning tasks. Similarly, (2) ERROR-CORRECTION LEARNING (backprop, Widrow-Hoff, Perceptron, etc.) can be employed for each of the classes. Pattern association is the obvious application, but it is obvious how it can be employed for autoassociation tasks. Error-correction methods can be used to build an autoencoder and thereby be used to extract features or principle components of the input patterns and thereby act as a regularity detector. Backprop, for example, can also be used in reinfoprcement learning situations. (3) COMPETITIVE LEARNING mechanisms can also be used for at least regularity detection and for autoencoding and (probably) for possibly for pattern association. (4) BOLTZMANN LEARNING is just as versitile as error correction learning. The point is simply that a cross-classification of learning task by learning rule is required to classify the kinds of connectionist learning applications we see. der From mehra at ptolemy.arc.nasa.gov Mon Jun 5 13:37:32 1989 From: mehra at ptolemy.arc.nasa.gov (Pankaj Mehra) Date: Mon, 5 Jun 89 10:37:32 PDT Subject: Supervised learning Message-ID: <8906051737.AA25680@ptolemy.arc.nasa.gov> The difference between "supervised" learning and reinforcement learning is largely due to the nature of feedback. To borrow a term from Ron Williams, the feedback is "prescriptive" in the former type and "evaluative" in the latter. Yet another distinction, first noted by Barto, Sutton, and Anderson, in their 1983 paper, is due to the synchronicity and delay in feedback. If feedback is delayed, the learner needs "memory" of recent decisions, and a temporal credit assignment mechanism (Sutton, 1988) to distribute the feedback among memorized decisions. Asynchronicity in feedback can (roughly) be defined as the property of a training environment so that it cannot be determined precisely which output of the network will be followed by reinforcement or correction. IMHO, this is an important difference between knowledge-based and connectionist learning systems. The AI model of learning (Dietterich, 1981) can handle only synchronous delays in feedback. The reason why ANSs can handle asynchronous delays in feedback is because their architecture is inherently asynchronous. - Pankaj {mehra at cs.uiuc.edu} References: (Sutton,88) Machine Learning, vol. 3, pp. 9-44 (Dietterich,Buchanan,81) The Role of Critic in Learning Systems, Stanford TR STAN-CS-81-891 (Barto et al.,83) IEEE trans. Sys. Man Cyb., vol. SMC-13, pp. 834-846 From neural!yann at att.att.com Mon Jun 5 10:47:33 1989 From: neural!yann at att.att.com (neural!yann@att.att.com) Date: Mon, 05 Jun 89 10:47:33 -0400 Subject: supervised learning In-Reply-To: Your message of Fri, 02 Jun 89 15:45:52 -0400. Message-ID: <8906051445.AA27851@neural.UUCP> I entirely agree with Andy Barto when he says that unsupervised learning is a special type of supervised learning where the objective function is hidden or implicit. From the ALGORITHMIC point of view, the important thing is that we just need to look for good supervised learning procedures. Interesting unsupervised procedures can usually be derived from supervised ones. Look at how back-prop can be stripped down to perform principal component analysis (I am not saying that ALL unsupervisd procedures can be derived from supervised procedures). Of course from the TASK point of view, it looks like there is a qualitative difference between supervised and unsupervised learning, but they are just different ways of using the same general principles. I don't think the difference is relevant if you are interested in the general principles. Yann From lyn at CS.EXETER.AC.UK Wed Jun 7 13:36:25 1989 From: lyn at CS.EXETER.AC.UK (Lyn Shackleton) Date: Wed, 7 Jun 89 13:36:25 BST Subject: Connection Science Message-ID: <2628.8906071236@exsc.cs.exeter.ac.uk> ANNOUNCEMENT Issue 1. of the new journal CONNECTION SCIENCE has just gone to press and Issue 2. will follow shortly. The editors are very pleased with the response they have received and would welcome more high quality submissions or theoretical notes. VOLUME 1 NUMBER 1 CONTENTS Michael C Mozer & Paul Smolensky 'Using Relevance to Reduce Network Size Automatically' James Hendler 'The Design and Implementation of Symbolic Marker-Passing Systems' Eduardo R Caianello, Patrik E Eklund & Aldo G S Ventre 'Implementations of the C-Calculus' Charles P Dolan & Paul Smolensky 'Tensor Product Production System: A Modular Architecture and Representation' Christopher J Thornton 'Learning Mechanisms which Construct Neighbourhood Representations' Ronald J Williams & David Zipser 'Experimental Analysis of the Real-Time Recurrent Learning Algorithm' Editor: Dr NOEL E SHARKEY, Centre for Connection Science, Dept of Computer Science, University of Exeter, UK Associate Editors: Andy CLARK (University of Sussex, Brighton, UK) Gary COTTRELL (University of California, San Diego, USA) James A HENDLER (University of Maryland, USA) Ronan REILLY (St Patrick's College, Dublin, Ireland) Richard SUTTON (GTE Laboratories, Waltham, MA, USA) FORTHCOMING IN VOLUMES 1 & 2 Special Issue on Natural Language, edited by Ronan Reilly & Noel Sharkey Special Issue on Hybrid Symbolic/Connectionist Systems, edited by James Hendler For further details please contact. lyn shackleton (assistant editor) Centre for Connection Science JANET: lyn at uk.ac.exeter.cs Dept. Computer Science University of Exeter UUCP: !ukc!expya!lyn Exeter EX4 4PT Devon BITNET: lyn at cs.exeter.ac.uk.UKACRL U.K. Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa02160; 8 Jun 89 10:56:34 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa11156; 8 Jun 89 10:50:51 EDT Received: from PRINCETON.EDU by CS.CMU.EDU; 8 Jun 89 10:48:22 EDT Received: from clarity.Princeton.EDU by Princeton.EDU (5.58+++/2.17) id AA15362; Thu, 8 Jun 89 09:09:45 EDT Received: by clarity.Princeton.EDU (4.0/1.81) id AA08666; Thu, 8 Jun 89 09:14:22 EDT Date: Thu, 8 Jun 89 09:14:22 EDT From: harnad at Princeton.EDU Message-Id: <8906081314.AA08666 at clarity.Princeton.EDU> To: connectionists at CS.CMU.EDU Subject: Categorization and Supervision CATEGORIZATION AND SUPERVISION The question of the definition and nature of supervised vs. unsupervised learning touches on some analogous points in the theory of categorization. Perhaps some of the underlying logic is more obvious in the case of categorization: There are two kinds of categorization tasks, one of which I've dubbed "imposed" categorization and the other "ad lib" categorization (otherwise known as similarity judgment). In both kinds of categorization task a set of inputs is sorted into a set of categories, but in imposed categorization there is feedback about whether the sorting is correct or incorrect (and when it is incorrect, there is usually also information indicating which category would have been correct), whereas in ad lib categorization there is no feedback (and often no nonarbitrary criterion for "correctness" either): The subject is simply asked to sort the inputs into categories as he sees fit. (Sometimes even less is asked for in ad lib categorization: The subject just rates pairs of inputs on how similar he judges them, and then the "categorization" is inferred by some suitable multidimensional statistical analysis such as scaling or cluster analysis.) The logical structure and informational demands of imposed and ad lib categorization are very different. Although both are influenced by learning, it is clear that imposed categorization (except if it is inborn) depends critically on learning through feedback from the external consequences of MIScategorization. Inputs are provisionally sorted and labeled, the consequences of the sorting and labeling somehow matter to the organism, and the feedback from incorrect sorting and labeling gives rise to a learning process that converges eventually on correct sorting and labeling. The categories are "imposed" by the environmental consequences of miscategorization. The category "label" can of course be any operant response, so imposed categorization is the paradigm for learned operant behaviors as well as many evolved sensorimotor adaptations (i.e. Skinner's "selection by consequences," although Skinner of course provides no mechanism for the learning, whereas connectionism seems to have some candidates). Ad lib categorization, on the other hand, does not depend directly on learning (although it is no doubt influenced by the outcome of prior imposed category learning). Logically, all it depends on is passive exposure to the inputs; by definition there are no consequences arising from miscategorization (if there are, then we are back into supervised learning). One last point: Learning an imposed categorization is usually dependent on finding the stimulus features that will reliably sort the imputs into the correct categories. An imposed categorization problem is "hard" to the degree to which this is not a trivial task. (Put another way, it depends on how "underdetermined" the categorization problem is.) A trivial imposed categorization problem is one in which the ad lib categorization happens to be a solution to the imposed categorization as well, i.e., the perceived similarity structure of the input set is already such that it "relaxes" into the right sorting through mere exposure, without the need for feedback. I think this last point may be at the heart of some of the misunderstandings about supervised vs. unsupervised learning: Logically speaking, there can be no "correct" or "incorrect" categorization in ad lib categorization; the "correctness" criterion is just in the mind of the experimenter. But if the hard work (i.e., finding and weighting the features that will reliably sort the inputs "correctly") has already been done by prior imposed category learning (or the evolution of our sensorimotor systems) then a given categorization problem may spuriously give the appearance of having been solved by "unsupervised" learning alone. In reality, however, the "solution" will have been well-prepared by prior learning and evolution, since, apart from the mind of the experimenter, "correctness" is DEFINED by the consequences of miscategorization, i.e., by supervision. Reference: Harnad, S. (Ed.) (1987) Categorical Perception: The Groundwork of Cognition. NY: Cambridge University Press. Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa09828; 8 Jun 89 18:43:21 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa15405; 8 Jun 89 15:44:14 EDT Received: from EE.ECN.PURDUE.EDU by CS.CMU.EDU; 8 Jun 89 15:42:42 EDT Received: by ee.ecn.purdue.edu (5.61/1.18jrs) id AA11580; Thu, 8 Jun 89 14:42:19 -0500 Message-Id: <8906081942.AA11580 at ee.ecn.purdue.edu> To: connectionists at CS.CMU.EDU Subject: TR available Date: Tue, 06 Jun 89 17:05:00 EST From: Manoel Fernando Tenorio Sender: tenorio at ee.ecn.purdue.edu Bcc:neuron-request at hplabs.hp.com -------- The Tech Report below will be available by June, 15. Please do not reply to this posting. Send all you requests to jld at ee.ecn.purdue.edu Self Organizing Neural Network for Optimum Supervised Learning Manoel Fernando Tenorio Wei-Tsih Lee School of Electrical Engineering School of Electrical Engineering Purdue University Purdue University W. Lafayette, IN. 47907 W. Lafayette, IN. 47907 tenorio at ee.ecn.purdue.edu lwt at ed.ecn.purdue.edu Summary Current neural network algorithms can be classified by the following characteristics: the architecture of the network, the error criteria used, the neuron transfer function, and the algorithm used during learning. For example: in the case of back propagation, one would classify the algorithm as a fixed architecture (feedforward in most cases), using a MSE criteria, and a sigmoid function on a weighted sum of the input, with the Generalized Delta Rule performing a gradient descent in the weight space. This characterization is important in order to assess the power of such algorithms from a modeling viewpoint. The expressive power of a network is intimately related with these four features. In this paper, we will discuss a neural network algorithm with noticeably different characteristics from current networks. The Self Organizing Neural Network (SONN) [TeLe88] is an algorithm that through a search process creates the network necessary and optimum in the sense of performance and complexity. SONN can be classified as follows. The network architecture is constructed through a search using Simulated Annealing (SA),and it is optimum in that sense. The error criteria used is a modification of the Minimum Description Length Criteria called the Structure Estimation Criteria (SEC); it takes into account both the performance of the algorithm and the complexity of the structure generated. The neuron transfer function is individually chosen from a pool of functions, and the weights are adjusted during the neuron creation. This function pool can be selected with a priori knowledge of the problem, or simply use a class of non-linearities shown to be general enough for a wide variety of problems. Although the algorithm is stochastic in nature (SA), we show that its performance is extremely high both in comparative and absolute terms. In [TeLe88], we have used SONN as an algorithm to identify and predict chaotic series, particularly the Mackey-Glass equation [LaFa87, Mood88] was used. For comparison, the experiments of using Back Propagation for this problem were replicated under the same computational environment. The results indicated that for about 10% of the computational effort, the SONN delivered a 2.11 times better model (normalized RMSE). Some inherited aspects of the algorithm are even more interesting: there were 3.75 times less weights, 15 times less connections, 6.51 times less epochs over the data set, and only 1/5 of the data was fed to the algorithm. Furthermore, the algorithm generates a symbolic representation of the network which can be used to substitute it, or be used for the analysis of the problem. ****************************************************************************** We have further developed the algorithm, and although not part of the report above, it will be part of a paper submitted to NIPS'89. There, some major improvements on the algorithm are reported. The same chaotic series problem can now run with 26.4 less epochs over the data set that BP, and have generated the same model in about 18.5 seconds of computer time. (This is down from 2 CPU hours in a Gould NP1 Powernode 9080). Performance on a Sun 3-60 was sightly over 1 minute. These performance figures include the use of an 8 times larger function pool; the final performance now independs of the size of the pool. Other aspects of the algorithm are also important considering. Because of its stochastic nature, no two runs of the algorithm should be the same. This can become a hindrance if a suboptimal solution is desired, since at every run the set of suboptimal models can be different. A report on modifications of the original SONN to run on an A* search are presented. Since the algorithm generates partial structures at each iteration, the learning process is only optimized for the structure presently generated. If such substructure is used as a part of a larger structure, then no provision is made to readjust its weights making the final model slightly stiff. A provision for melting the structure (parametric readjustment) is also discussed. Finally, the combination of symbolic processing with this numerical method can lead to construction of AI-NN based methods for supervised and unsupervised learning. The ability of SONN to take symbolic constraints and produce symbolic information can make such a system possible. Implications of this design are also explored. [LaFa87] - Alans Lapedes and Robert Farber, How Neural Networks Work, TR LA-UR-88-418, Los Alamos, 1987. [Mood88] - J. Moody, Fast Learning in Multi-Resolution Hierarchies, Advances in Neural Information Processing Systems, D. Touresky, Ed., Morgan Kaufmann, 1989 (NIPS88). [TeLe88] - M. F. Tenorio and W-T Lee, Self Organizing Neural Networks for the Identification Problem, Advances in Neural Information Processing Systems, D. Touresky, Ed., Morgan Kaufmann, 1989 (NIPS88). Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa11898; 8 Jun 89 22:28:55 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa18227; 8 Jun 89 18:49:26 EDT Received: from EPHEMERAL.AI.TORONTO.EDU by CS.CMU.EDU; 8 Jun 89 18:47:03 EDT Received: from localhost (stdin) by ephemeral.ai.toronto.edu with SMTP id 11256; Thu, 8 Jun 89 18:46:41 EDT To: harnad at PRINCETON.EDU cc: connectionists at CS.CMU.EDU Subject: Re: Categorization and Supervision In-reply-to: Your message of Thu, 08 Jun 89 09:14:22 -0400. Date: Thu, 8 Jun 89 18:46:27 EDT From: Geoffrey Hinton Message-Id: <89Jun8.184641edt.11256 at ephemeral.ai.toronto.edu> Steve Harnad's message about supervised versus unsupervised learning makes an apparently plausible point that is DEEPLY wrong. He says "Logically speaking, there can be no "correct" or "incorrect" categorization in ad lib (his term for "unsupervised") categorization; the "correctness" criterion is just in the mind of the experimenter. But if the hard work (i.e., finding and weighting the features that will reliably sort the inputs "correctly") has already been done by prior imposed category learning (or the evolution of our sensorimotor systems) then a given categorization problem may spuriously give the appearance of having been solved by "unsupervised" learning alone. In reality, however, the "solution" will have been well-prepared by prior learning and evolution, since, apart from the mind of the experimenter, "correctness" is DEFINED by the consequences of miscategorization, i.e., by supervision." The mistake in this line of reasoning is as follows: Using the Kolmogorov notion of complexity, we can distinguish between good and bad models of some data without any additional teacher. A good model is one that has low kolmogorov complexity and fits the data closely (there is always a data-fit vs model-complexity trade-off). Clustering data into categories is just one particularly tractable way of modelling data. Naturally, ideas based on Kolmogorov complexity are easiest to apply if we start with a restricted class of possible models (so there is still plenty of room for evolution to be helpful). But restricting the class of models is not the same as implicitly saying which particular models (i.e. specific categorizations) are correct. For example, we could insist on modeling some data as having arisen from a mixture of gaussians (one per cluster), but this doesnt tell us which data to put in which cluster, or how many clusters to use. For that we need to trade-off the complexity of the model (number of clusters) against the data-fit. Cheeseman (and others) have shown that this approach can be made to work nicely in practice. Geoff Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa02050; 9 Jun 89 4:54:31 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa22494; 9 Jun 89 1:24:46 EDT Received: from PRINCETON.EDU by CS.CMU.EDU; 9 Jun 89 01:23:14 EDT Received: from clarity.Princeton.EDU by Princeton.EDU (5.58+++/2.17) id AA19296; Fri, 9 Jun 89 01:23:15 EDT Received: by clarity.Princeton.EDU (4.0/1.81) id AA17091; Fri, 9 Jun 89 01:28:02 EDT Date: Fri, 9 Jun 89 01:28:02 EDT From: harnad at Princeton.EDU Message-Id: <8906090528.AA17091 at clarity.Princeton.EDU> To: connectionists at CS.CMU.EDU Subject: Re: Categorization and Supervision Geoff Hinton wrote: >> Steve Harnad's message about supervised versus unsupervised learning makes an >> apparently plausible point that is DEEPLY wrong. [Harnad] says >> "Logically speaking, there can be no "correct" or "incorrect" categorization in ad lib (his term for "unsupervised") categorization..." >> The mistake in this line of reasoning is as follows: Using the Kolmogorov >> notion of complexity, we can distinguish between good and bad models of some >> data without any additional teacher. A good model is one that has low >> kolmogorov complexity and fits the data closely... I am afraid Geoff has not understood my point. I was not speaking about algorithms or models but about human categorization performance and its constraints. There may well be a "model" or internal criterion or constraint according to which input can be preferentially categorized in a particular way, but that still has nothing to do with the correctness or incorrectness of the categorization, which, as a logical matter, can only be determined by some external consequence of categorizing INcorrectly. That's what categorizing correctly MEANS. Otherwise "categorization" and "correctness" are just figures of speech (and solipsistic ones, at that). My point about imposed vs. ad lib categorization (which is not only plausible, but, till grasped and refuted on its own terms, stands, uncorrected) was based purely on external performance considerations, not model-specific internal ones. I think that the supervised/unsupervised learning distinction has given rise to misunderstandings precisely because it equivocates between external and internal considerations. I focused on the external question of what the "correctness" of a categorization really consists in so as to highlight some of the logical, methodological and informational features of categorization, and indeed of learning in general. It is a purely logical point that, even if arrived at by purely internal, unsupervised means, the "correctness" of a "correct" human categorization must be based on some external performance constraint, and hence at least a potential "supervisor." Otherwise the "correctness" is merely in the mind of the theorist, or the interpreter (or the astrologist, if the magical number happens to be 12). Now, that simple point having been made, I might add that whereas I do not find it implausible that SOME categorization problems (for which the requisite potential supervision from the external consequences of miscategorization must, I continue to insist, exist in principle) might nevertheless be solved through internal constraints alone, with no need for any actual recourse to the external supervision, I find it highly implausible that ALL, MOST, or even MANY categorization problems should be soluble that way -- and certainly not the ones I called the "hard" (underdetermined) categorization problems. Connectionism should not be TOO ambitious. It's a proud enough hope to aspire to be THE method that reliably finds invariant features in generalized nontrivial induction WITH supervision -- without going on to claim to be able to do it all with your eyes closed and your hands tied behind your back... Stevan Harnad Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa07681; 9 Jun 89 13:45:13 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa03720; 9 Jun 89 13:39:42 EDT Received: from GORT.CS.BUFFALO.EDU by CS.CMU.EDU; 9 Jun 89 13:37:27 EDT Received: from sybil.cs.Buffalo.EDU by gort.cs.Buffalo.EDU (5.59/1.1) id AA10660; Fri, 9 Jun 89 13:37:19 EDT Received: by sybil.cs.Buffalo.EDU (4.12/1.1) id AA06218; Fri, 9 Jun 89 13:38:32 edt Date: Fri, 9 Jun 89 13:38:32 edt From: Arun Jagota Message-Id: <8906091738.AA06218 at sybil.cs.Buffalo.EDU> To: connectionists at CS.CMU.EDU Subject: Re: Supervised/Unsupervised Learning Steve Harnad wrote : >It is a >purely logical point that, even if arrived at by purely internal, >unsupervised means, the "correctness" of a "correct" human categorization >must be based on some external performance constraint, and hence at >least a potential "supervisor." Otherwise the "correctness" is merely >in the mind of the theorist, or the interpreter (or the astrologist, if >the magical number happens to be 12). There are times when an organism might wish to categorize events (in the input space) for internal reasons only. The external correctness of the categorization may not be important (there may even be none), as long as there is an internal consistency of classification and recognition. >Now, that simple point having been made, I might add that whereas I do >not find it implausible that SOME categorization problems (for which >the requisite potential supervision from the external consequences of >miscategorization must, I continue to insist, exist in principle) might >nevertheless be solved through internal constraints alone, with no >need for any actual recourse to the external supervision, I find it >highly implausible that ALL, MOST, or even MANY categorization problems >should be soluble that way There are situations for which I don't necessarily think of categorization as a problem that has to be solved, rather a process that has to be performed. I think there are instances when the exact nature of the categories is not as important as the process itself (as an aid to event recognition/content addressibility). Consider a dictionary: We could clasify words a) lexically sorted b) By word-size c) By a hash function (sum of all ASCII character codes mod constant) etc The choice of the categorization to use is determined more by what we want to use it for (retrieval etc) than by any issues of its external correctness. I don't wish to overstate the case for such an unsupervised learning, but I think it deserves to be studied as an independent task in it's own right. Arun Jagota (jagota at cs.buffalo.edu) Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa13460; 9 Jun 89 19:50:58 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa05685; 9 Jun 89 15:02:51 EDT Received: from HELIOS.NORTHEASTERN.EDU by CS.CMU.EDU; 9 Jun 89 15:00:28 EDT Received: from corwin.ccs.northeastern.edu by helios.northeastern.edu id aa14361; 9 Jun 89 15:00 EDT Received: by corwin.CCS.Northeastern.EDU (5.51/SMI-3.2+CCS-main-2.6) id AA10984; Fri, 9 Jun 89 14:58:44 ADT Date: Fri, 9 Jun 89 14:58:44 ADT From: steve gallant Message-Id: <8906091758.AA10984 at corwin.CCS.Northeastern.EDU> To: connectionists at CS.CMU.EDU Subject: Re: Supervised/Unsupervised and Autoassociative learning Here's my suggestion for a taxonomy of learning problems: I. Supervised Learning: correct activations given for output cells for each training example A. Hard Learning Problems: network structure is fixed and activations are not given for intermediate cells B. Easy Learning Problems: correct activations given for intermediate cells OR network structure allowed to be modified OR single-cell problem II. Unsupervised Learning: correct activations not given for output cells There are further subdivisions for learning ALGORITHMS that can cut across the above PROBLEM classes. For example: 1. One-Shot Learning Algorithms: each training example is looked at at most one time 2. Iterative Algorithms: training examples may be looked at several times Autoassociative Algorithms are a special case of IB above because each output cell can be trained independently of the other cells, reducing the problem to single-cell supervised learning. However these models have different dynamics, in that output values are copied to input cells for the next iteration. Most learning algs for autoassociative networks are one-shot algorithms (eg. linear autoassociator, BSB, Hopfield nets). Of course iterative algorithms can be used to increase the capacity of autoassociative models (eg. perceptron learning, pocket alg.). We could also add intermediate cells to an autoassociative network to allow arbitrary sets of patterns to be stored. Sorry if you got this message twice, but I don't think the first try made it. Steve Gallant Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa03549; 9 Jun 89 22:18:27 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa08388; 9 Jun 89 18:09:32 EDT Received: from PRINCETON.EDU by CS.CMU.EDU; 9 Jun 89 18:06:53 EDT Received: from clarity.Princeton.EDU by Princeton.EDU (5.58+++/2.17) id AA15388; Fri, 9 Jun 89 18:06:49 EDT Received: by clarity.Princeton.EDU (4.0/1.81) id AA24806; Fri, 9 Jun 89 18:11:32 EDT Date: Fri, 9 Jun 89 18:11:32 EDT From: harnad at Princeton.EDU Message-Id: <8906092211.AA24806 at clarity.Princeton.EDU> To: connectionists at CS.CMU.EDU Subject: Re: Unsupervised Category Learning Arun Jagota wrote: >> an organism might wish to categorize events (in the input space) for >> internal reasons only. The external correctness of the categorization >> may not be important (there may even be none), as long as there is an >> internal consistency of classification and recognition... I think there >> are instances when the exact nature of the categories is not as >> important as the process itself (as an aid to event recognition/content >> addressibility). Consider a dictionary: We could classify words a) >> lexically sorted b) By word-size c) By a hash function... The choice of >> the categorization to use is determined more by what we want to use it >> for (retrieval etc) than by any issues of its external correctness. I think there is a confusion of phenomena here, including (1) organisms' adaptive behavior in their environments, (2) intelligent machines we use for various purposes, and (3) something like finger painting or playing with tarot cards. I have no particular interest in doing or in modeling (3) (categorizing events for "internal reasons only"), and I don't think it's a well-defined problem. (1) is clearly "supervised" by the external consequences for survival and reproduction arising from how the organism sorts and responds to classes of objects, events and states of affairs in the world. (2) could in principle be governed by machine-internal constraints only, but the "usefulness" of the categories (e.g., word classifications) to US is again clearly dependent on external consequences (to US), exactly as in (1). Perhaps an example will give a better idea of the distinction I'm emphasizing: Suppose you had N "objects." Consider the many ways you could sort and label them: By size, by shape, by color, by function, by age, as natural or synthetic, living or nonliving, etc. In each case, the name of the same "object" changes, as does the membership of the category to which it is assigned. With sufficient imagination an enormous number of possible categorizations could be generated for the same set of objects. What makes some of those categorizations "correct" (or, equally important, incorrect) and others arbitrary? The answer is surely related to why it is that we bother to categorize at all: Sorting (and especially mis-sorting) objects some ways (e.g., as edibles vs inedibles) has important consequences for us, whereas sorting them other ways (e.g., bigger/smaller than a breadbox, moon in Aquarius) does not. The consequences do not depend on "internal" considerations but on external ones, so I continue to doubt that unsupervised internal constraints can account for much that is of interest in human category learning. Stevan Harnad Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa05838; 10 Jun 89 4:18:33 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa12044; 9 Jun 89 23:42:04 EDT Received: from UICSRD.CSRD.UIUC.EDU by CS.CMU.EDU; 9 Jun 89 23:10:20 EDT Received: from s16.csrd.uiuc.edu by uicsrd.csrd.uiuc.edu with SMTP (5.61+/IDA-1.2.8) id AA16942; Fri, 9 Jun 89 22:10:11 -0500 Received: by s16.csrd.uiuc.edu (3.2/9.2) id AA08968; Fri, 9 Jun 89 22:10:07 CDT Date: Fri, 9 Jun 89 22:10:07 CDT From: George Cybenko Message-Id: <8906100310.AA08968 at s16.csrd.uiuc.edu> To: connectionists at CS.CMU.EDU Re: Unsupervised learning What do people mean by "data" in unsupervised learning? Surely it is more than just bit strings....if you get "data" in the form of bit strings don't you have to know what those bit strings mean? Does the bit string represent an image, a vector of 32 bit floating point numbers in VAX or MC68000 arithmetic, an integer, or an ASCII representation of some characters? It seems to me that the interpretation of the bit string in this way determines a metric that is imposed by the semantics of the problem from which the data came. That metric qualifies as a form of "supervision". Moreover, any discussion of Kolmogorov complexity vs. modeling errors requires some notion of error, hence some notion of a metric imposed on the data. Put another way, I don't think that there can be a useful "unsupervised" learning procedure that accepts only bit strings as input with no interpretations of what those bit strings mean. I suspect that there might be a canonical format into which all problems can be transformed that would usually give meaningful classifications but then it would be argued that the transformation is a type of supervision. In fact, this question of how to interpret and represent input data is a key criticism of connectionist modeling as given by many neuro- biologists, from my understanding of that debate. George Cybenko that debate. Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa05853; 10 Jun 89 4:27:32 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa12632; 10 Jun 89 0:04:04 EDT Received: from THINK.COM by CS.CMU.EDU; 10 Jun 89 00:02:41 EDT Return-Path: Received: from kulla.think.com by Think.COM; Sat, 10 Jun 89 00:02:35 EDT Received: by kulla.think.com; Sat, 10 Jun 89 00:01:33 EDT Date: Sat, 10 Jun 89 00:01:33 EDT From: singer at Think.COM Message-Id: <8906100401.AA04047 at kulla.think.com> To: connectionists at CS.CMU.EDU Subject: Sparse vs. Dense networks In general sparsely connected networks run faster than densely connected ones. Ignoring this advantage, are there any theoretical benefits to sparsely connected nets? Are schemes utilitzing local connectivity patterns, perhaps including the use of additional layers, buying the researcher some advantage besides more efficient use of limited amounts of computer time? Naively it would seem that the more interconnections the better (even if, after training, many have near-zero weights), is this true? Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa12627; 10 Jun 89 19:40:08 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa20829; 10 Jun 89 19:35:56 EDT Received: from GOAT.CS.WISC.EDU by CS.CMU.EDU; 10 Jun 89 19:34:30 EDT Date: Sat, 10 Jun 89 18:34:12 -0500 From: Vasant Honavar Message-Id: <8906102334.AA02423 at goat.cs.wisc.edu> Received: by goat.cs.wisc.edu; Sat, 10 Jun 89 18:34:12 -0500 To: singer at Think.COM Subject: Re: Sparse vs. Dense networks Cc: connectionists at CS.CMU.EDU, honavar at cs.wisc.edu The tradeoff between connectivity and performance, can in general, be quite complex. There is reason to think that smaller networks (with fewer nodes and links) may be better than larger ones for getting improved generalization as shown by the work of Hinton and others. The use of some topological constraints (such as local receptive fields - in the case of vision) on network connectivity can yield improved performance in learning (after discounting the saving in computer time for simulations) (Honavar and Uhr, 1988; Moody and Darken, 198?; Qian and Sejnowski, 1988). Local interactions and systematic initiation and termination of plasticity, combined with a Hebbian-like learning rule, can, through a process of self-organization, yield network structures that resemble that of orientation columns in the primary visual cortex (Linsker, 1988). Modularity is another means of restricting interactions between different parts of a network. When tasks are relatively independent, they are best handled by seperate or sparsely interacting modules. This is shown by the work of Ruekl and Kosslyn (198?) - Seperate modules learning object identification and object location perform better than a monolithic network learning both. Many tasks that connectionist models attempt to solve are NP-complete; combinatorially explosive solutions are not feasible. Other factors being equal, architectures that exploit the natural constraints of the domain yield better performance (speed, cost) than those that don't. Network architecture (size, topology, etc) interact intimately with the learning processes (weight modification; generation of new links (Honavar, 1988) - and possibly, nodes; regulation of plasticity) as well as the structure of the domain (e.g., vision, speech) at different levels. Connectivity appropriate for a particular problem domain cannot be determined independent of these factors. Vasant Honavar (honavar at cs.wisc.edu) Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa21277; 11 Jun 89 19:11:08 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa27443; 11 Jun 89 17:25:58 EDT Received: from ATHENA.MIT.EDU by CS.CMU.EDU; 11 Jun 89 17:23:40 EDT Received: by ATHENA.MIT.EDU (5.45/4.7) id AA27878; Sun, 11 Jun 89 17:25:12 EDT Message-Id: <8906112125.AA27878 at ATHENA.MIT.EDU> Date: Sun, 11 Jun 89 17:15:39 edt From: Kyle Cave Site: MIT Center for Cognitive Science To: connectionists at CS.CMU.EDU Subject: modularity I'd like to add two notes concerning modularity to Honavar's letter. First, the paper by Rueckl, Cave, & Kosslyn (1989) that he referred to can be found in the most recent issue of the Journal of Cognitive Neuroscience. Second, anyone interested in how modular problems can be approached in a connectionist framework will be interested in recent work by Robbie Jacobs, who is constructing systems that learn to devote separate subnets to independent problems. Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa02004; 12 Jun 89 8:48:00 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa02852; 12 Jun 89 8:41:06 EDT Received: from GATEWAY.MITRE.ORG by CS.CMU.EDU; 12 Jun 89 08:38:49 EDT Received: by gateway.mitre.org (5.54/SMI-2.2) id AA18039; Mon, 12 Jun 89 08:39:09 EDT Return-Path: Received: by marzipan.mitre.org (4.0/SMI-2.2) id AA01478; Mon, 12 Jun 89 08:36:28 EDT Date: Mon, 12 Jun 89 08:36:28 EDT From: alexis%yummy at gateway.mitre.org Message-Id: <8906121236.AA01478 at marzipan.mitre.org> To: singer at Think.COM Cc: connectionists at CS.CMU.EDU In-Reply-To: singer at Think.COM's message of Sat, 10 Jun 89 00:01:33 EDT <8906100401.AA04047 at kulla.think.com> Subject: Sparse vs. Dense networks Reply-To: alexis%yummy at gateway.mitre.org "Sparsely connected networks" (or "tessellated connections" as we tend to think of them) can also be used to hardwire known constraints. In speech or vision or something like the Penzias problem the researcher knows before hand that it makes sence to most if not all of the computation locally, possibly in something like a pyramid architecture. This restriction forces the network to use clues that are considered "reasonable." As an example, a few years ago we spent a fair amount of time using a NN to recognize characters independant of rotation. At first the "D" had the largest letter, so the net just counted pixels (a valid, but as far as we were concerned "wrong," discriminant). We originally worked around this by modifying the training database, but latter we found we could get the same results by using tessellation. Limiting the receptive fields forced the net to primarily use gradients, which proved to be a much more robust method in general. The moral of this story is that you can force a network to do certain classes of algorithms by restricting the connections. Since there is often a limit on the training data available these restrictions are even more important. Combine all this with shared connections (ala the Blue Book), and share tesselations, and tesselations of tesselations, and "skip-level" arcs into shared tesselations, ... and you can really control and add to what you and your net can do. alexis. Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa06612; 12 Jun 89 14:17:18 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa04131; 12 Jun 89 9:32:19 EDT Received: from RELAY.CS.NET by CS.CMU.EDU; 12 Jun 89 09:30:25 EDT Received: from relay2.cs.net by RELAY.CS.NET id ab08738; 12 Jun 89 9:24 EDT Received: from cs.brandeis.edu by RELAY.CS.NET id ab18119; 12 Jun 89 9:13 EDT Received: by cs.brandeis.edu (14.2/6.0.GT) id AA03801; Mon, 12 Jun 89 09:07:57 edt Date: Mon, 12 Jun 89 09:07:57 edt From: Ron Sun Posted-Date: Mon, 12 Jun 89 09:07:57 edt To: connectionists.2 at cs.brandeis.edu The following two tech reports are available from rsun%cs.brandeis.edu at relay.cs.net or R. Sun Brandeis U. CS Waltham, MA 02254 ############################################################# A Discrete Neural Network Model for Conceptual Representation and Reasoning Ron Sun Computer Science Dept. Brandeis University Waltham, MA 02254 Current connectionist models are oversimplified in terms of the internal mechanisms of individual neurons and the communication between them. Although connectionist models offer significant advantages in certain aspects, this oversimplification leads to the inefficiency of these models in addressing issues in explicit symbolic processing, which is proven to be essential to human intelligence. What we are aiming at is a connectionist architecture which is capa- ble of simple, flexible representations of high level knowledge structures and efficient performance of reasoning based on the data. We first propose a discrete neural net- work model which contains state variables for each neuron in which a set of discrete states is explicitly specified instead of a continuous activation function. A technique is developed for representing concepts in this network, which utilizes the connections to define the concepts and represents the concepts in both verbal and compiled forms. The main advantage is that this scheme can handle variable bindings efficiently. A reasoning scheme is developed in the discrete neural network model, which utilizes the inherent parallelism in a neural network model, performing all possible inference steps in parallel, implementable in a fine-grained massively parallel computer. (to appear in Proc. CogSci Conf. 1989) ############################################################### Model local neural networks in the lobster stomatogastric ganglion Ron Sun Eve Marder David Waltz Brandeis University Waltham, MA 02254 ABSTRACT We describe a simulation study of the pyloric network of the lobster stomatogastric ganglion. We demonstrate that a few simple activation functions are sufficient to describe the oscillatory behavior of the network. Our aim is to determine the essential mechanisms necessary to specify the operation of biological neural networks so that we can incorporate them into connectionist models. Our model includes rhythmically active bursting neurons and long time-constant synaptic relations. In the process of doing this work, various models and algorithms were compared. We have derived some connectionist learning algorithms. They have proved useful in terms of ease and accuracy in model generation. (to appear in IJCNN-89) ************************************************************** Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa09540; 12 Jun 89 17:29:12 EDT Received: from ri.cmu.edu by Q.CS.CMU.EDU id aa06000; 12 Jun 89 11:32:32 EDT Received: from HELIOS.NORTHEASTERN.EDU by RI.CMU.EDU; 12 Jun 89 11:30:32 EDT Received: from corwin.ccs.northeastern.edu by helios.northeastern.edu id aa23199; 8 Jun 89 12:12 EDT Received: by corwin.CCS.Northeastern.EDU (5.51/SMI-3.2+CCS-main-2.6) id AA01649; Thu, 8 Jun 89 12:10:34 ADT Date: Thu, 8 Jun 89 12:10:34 ADT From: steve gallant Message-Id: <8906081510.AA01649 at corwin.CCS.Northeastern.EDU> To: connectionists at RI.CMU.EDU Subject: supervised vs. unsupervised learning and autoassociators Here's my suggestion for a taxonomy of learning problems: I. Supervised Learning: correct activations given for output cells for each training example A. Hard Learning Problems: network structure is fixed and activations are not given for intermediate cells B. Easy Learning Problems: correct activations given for intermediate cells OR network structure allowed to be modified OR single-cell problem II. Unsupervised Learning: correct activations not given for output cells There are further subdivisions for learning ALGORITHMS that can cut across the above PROBLEM classes. For example: 1. One-Shot Learning Algorithms: each training example is looked at at most one time 2. Iterative Algorithms: training examples may be looked at several times Autoassociative Algorithms are a special case of IB above because each output cell can be trained independently of the other cells, reducing the problem to single-cell supervised learning. However these models have different dynamics, in that output values are copied to input cells for the next iteration. Most learning algs for autoassociative networks are one-shot algorithms (eg. linear autoassociator, BSB, Hopfield nets). Of course iterative algorithms can be used to increase the capacity of autoassociative models (eg. perceptron learning, pocket alg.). We could also add intermediate cells to an autoassociative network to allow arbitrary sets of patterns to be stored. Steve Gallant Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa13248; 13 Jun 89 0:10:26 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa09991; 12 Jun 89 15:51:06 EDT Received: from AI.CS.WISC.EDU by CS.CMU.EDU; 12 Jun 89 15:48:14 EDT Date: Mon, 12 Jun 89 14:48:02 -0500 From: Leonard Uhr Message-Id: <8906121948.AA11341 at ai.cs.wisc.edu> Received: by ai.cs.wisc.edu; Mon, 12 Jun 89 14:48:02 -0500 To: singer at Think.COM Subject: desirability of sparse, local, converging connectivity Cc: connectionists at CS.CMU.EDU I think there is a strong set of arguments for sparse local connectivity, with multiple converging layers to give global functions. In a word: Sparse connectivity keeps the number of links down to O(kN), rather than O(N**2). Consider brains: 10**12 neurons each with only roughly 3*10**3 synaptic links. 10**24 links is far too much to handle either serially (time) or in parallel (wires). Obviously the links should be those with non-zero weight, and that usually means local (tho not always). Irrelevant links with random initial weights almost certainly obscure the functional links and breed confusion. Less-than-complete connectivity means that the convergence necessary to compute global interactions needs additional layers O(logN). - Len Uhr Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa15637; 13 Jun 89 5:22:36 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa11703; 12 Jun 89 17:16:46 EDT Received: from MARS.NJIT.EDU by CS.CMU.EDU; 12 Jun 89 17:15:15 EDT Received: by mars.njit.edu (5.57/Ultrix2.4-C) id AA14240; Mon, 12 Jun 89 17:14:31 EDT Date: Mon, 12 Jun 89 17:14:31 EDT From: nirwan ansari fac ee Message-Id: <8906122114.AA14240 at mars.njit.edu> To: Connectionists at CS.CMU.EDU Subject: 1989 INNS meeting, Sep 5-9, 1989. Return-Receipt-To: ang at mars.njit.edu Last March, I submitted a paper to the 1989 INNS meeting, Sep 5-9, 1989. Just found out that this meeting has been cancelled. Yet, I haven't been notified of the fate of my paper. I called Schman Associates and INNS office - no one knew where my paper ended up. Anybody out there knows what were happening to the papers submitted to this meeting? I still got the receipt of my express mail... Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa15939; 13 Jun 89 6:00:25 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa13235; 12 Jun 89 18:46:14 EDT Received: from EPHEMERAL.AI.TORONTO.EDU by CS.CMU.EDU; 12 Jun 89 18:43:56 EDT Received: from localhost (stdin) by ephemeral.ai.toronto.edu with SMTP id 10882; Mon, 12 Jun 89 18:42:36 EDT To: harnad at PRINCETON.EDU cc: connectionists at CS.CMU.EDU Subject: Re: Categorization and Supervision In-reply-to: Your message of Fri, 09 Jun 89 01:28:02 -0400. Date: Mon, 12 Jun 89 18:41:56 EDT From: Geoffrey Hinton Message-Id: <89Jun12.184236edt.10882 at ephemeral.ai.toronto.edu> I am afraid Steve has not understood my point. He simply repeats his conviction again: "It is a purely logical point that, even if arrived at by purely internal, unsupervised means, the "correctness" of a "correct" human categorization must be based on some external performance constraint, and hence at least a potential "supervisor." Otherwise the "correctness" is merely in the mind of the theorist, or the interpreter (or the astrologist, if the magical number happens to be 12)." The whole debate is about whether there is a sense in which a categorization can be "correct" even though no external supervision is supplied. Steve thinks that any such categorization would be "merely in the mind of the theorist" (and hence, I guess he thinks, arbitrary). I think that this is wrong. If one model of the data is overwhelmingly simpler than any other, then its not just in the mind of the theorist. Its correct. The nice thing about the Kolmogorov-Chaitin view of complexity is that (in the limit) it doesnt need to mention the mind of the observer (i.e. in the limit, one model can be simpler than another WHATEVER the programming language in which we measure simplicity). In another message on the same subject, Steve says: "Perhaps an example will give a better idea of the distinction I'm emphasizing: Suppose you had N "objects." Consider the many ways you could sort and label them: By size, by shape, by color, by function, by age, as natural or synthetic, living or nonliving, etc. In each case, the name of the same "object" changes, as does the membership of the category to which it is assigned. With sufficient imagination an enormous number of possible categorizations could be generated for the same set of objects. What makes some of those categorizations "correct" (or, equally important, incorrect) and others arbitrary? The answer is surely related to why it is that we bother to categorize at all: Sorting (and especially mis-sorting) objects some ways (e.g., as edibles vs inedibles) has important consequences for us, whereas sorting them other ways (e.g., bigger/smaller than a breadbox, moon in Aquarius) does not. The consequences do not depend on "internal" considerations but on external ones, so I continue to doubt that unsupervised internal constraints can account for much that is of interest in human category learning." Again, this is just a repetition of his viewpoint. If you consider Peter Cheeseman's work on categorization (which found a new class of stars without supervision), it becomes clear that unsupervised categorization is NOT arbitrary. Geoff PS: I think I have now stated my beliefs clearly, and I do not intend to clutter up the network with any more meta-level junk. I think this debate will be resolved by technical progress in getting unsupervised learning to work nicely, not by a priori assertions about what is necessary and what is impossible. Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa16749; 13 Jun 89 7:28:02 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa14729; 12 Jun 89 20:10:35 EDT Received: from PRINCETON.EDU by CS.CMU.EDU; 12 Jun 89 20:08:14 EDT Received: from clarity.Princeton.EDU by Princeton.EDU (5.58+++/2.17) id AA07505; Mon, 12 Jun 89 20:08:02 EDT Received: by clarity.Princeton.EDU (4.0/1.81) id AA09043; Mon, 12 Jun 89 20:12:47 EDT Date: Mon, 12 Jun 89 20:12:47 EDT From: harnad at Princeton.EDU Message-Id: <8906130012.AA09043 at clarity.Princeton.EDU> To: connectionists at CS.CMU.EDU Subject: Re: Categorization & Supervision Geoff writes: >> I am afraid Steve has not understood my point... about whether there >> is a sense in which a categorization can be "correct" even though no >> external supervision is supplied... If one model of the data is >> overwhelmingly simpler than any other, then it's not just in the mind of >> the theorist. It's correct. Let me quickly cite two senses (that I've already noted) in which such a categorization can indeed be "correct," just as Geoff writes in the foregoing passage, so as to confirm that the misunderstanding's not on this side of the border: (1) Under the above conditions it would certainly be (tautologically) "correct" that the simplest categorization was indeed the simplest categorization, and that would be true entirely independently of any external supervisory criterion. (2) It COULD also be "correct" (i) accidentally, or because of prior (ii) innate or (iii) learned preparation, or because the categorization problem happened to be (iv) easy (all four of which I've already mentioned in prior postings) that the categorization arrived at by the internal simplicity criterion ALSO happens to correspond to a categorization that DOES match an external supervisory criterion. (Edibles and inedibles could conceivably sort this way, though it's unlikely.) I take it that (1) in and of itself is of interest only to the complexity theorist. (2) Is a possibility; the burden of proof is on anyone who wishes to claim that many, most or all of our external categorizations in the world can indeed be captured this way (the "some" I already noted in my first posting). I have already given (in a response to someone else's posting on this question) one a priori reason not to expect simplicity or any other a priori symmetry to succeed very often: We typically can and do categorize the very SAME N inputs in many, many DIFFERENT ways, depending on external contingencies; so it is hard to imagine how one and the same internal criterion (or several) could second guess ALL of these alternative categorizations. The winning features are certainly lurking SOMEWHERE in the input, otherwise successful categorization would not be possible; but finding them is hardly likely to be an a priori or internal matter -- and certainly not in what I called the "hard" (underdetermined) cases. That's what learning is all about. And without supervision it's just thrashing in the dark (except in the easy cases that wear their category structure on their ears -- or the already "prepared" cases, but those involve theft rather than honest toil for a learning mechanism). Let me close by clarifying some possible sources of misunderstanding in this discussion of the relation between supervised/unsupervised learning algorithms and imposed/ad-lib categorization performance tasks: (A) A pure ad lib sorting task is to give a subject N objects and ask him to sort them any way he wishes. A variant on this is to ask him to (A1) sort them into k categories any way he wishes. (B) A pure imposed sorting task is to give a subject N objects and k categories with k category names. The subject is asked to sort them correctly and is given feedback on every trial; whenever there is an error, the name of the correct category is provided. The task is trivial if the subject can sort correctly from the beginning, or as soon as he knows how many categories there are and what their names are (i.e., if task B is the same as task A1) or soon thereafter. The task is nontrivial if the the N objects are sufficiently interconfusable, and the winning features sufficiently nonobvious, as to require many supervised trials in order to learn which features will reliably sort the objects correctly. (The solution should also not be a matter of rote memory for every object-label pair: i.e., N should be very large, and categorization should reach 100% before having sampled them all). A variant on B would be (B1) to provide feedback only on whether or not the categorization is correct, but without indicating the correct category name when something has been miscategorized. Note that for dichotomous categories (k = 2), B and B1 are the same (and that locally, all categorizations are dichotomous: "Is this an `X' or isn't it?"). In any case, both B and B1 would be cases of supervised learning. Note also that the choice of k, and of which categorization (and hence which potential features) are "correct," is entirely up to the experimenter in imposed categorization -- just as it is entirely up to external contingencies in the world, and the consequences of miscategorization for the organism, in natural categorization. Cases in which A happened to match these external contingencies would be special ones indeed. Stevan Harnad Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa06509; 13 Jun 89 13:14:03 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa21735; 13 Jun 89 9:18:34 EDT Received: from PRINCETON.EDU by CS.CMU.EDU; 13 Jun 89 09:15:26 EDT Received: from suspicion.Princeton.EDU by Princeton.EDU (5.58+++/2.17) id AA26730; Tue, 13 Jun 89 09:15:27 EDT Received: by suspicion.Princeton.EDU (4.0/1.81) id AA00904; Tue, 13 Jun 89 09:20:13 EDT Date: Tue, 13 Jun 89 09:20:13 EDT From: "Stephen J. Hanson" Message-Id: <8906131320.AA00904 at suspicion.Princeton.EDU> To: connectionists at CS.CMU.EDU Subject: sup vs unsup one other connection with this issue is a profitable distinction that has been made in the animal learning literature for some decades now. It has to do with the notion that there is a set of operations or procedures and there is a concomitant set processes. Classical or Pavlovian conditioning is an "open loop"" procedure in control lingo, and is clearly a supervised procedure (analogous to delta rule/back-prop/boltzmann) while Skinner's Operant or Instrumental conditioning--"Reinforcement Learning" is a "closed loop" procedure, the organism's action determine consequences. Clearly this a weakening of the supervision and of the information provided by the teacher. What is interesting is one can find process theories that run the gamut from complete distinction between classical and operant conditioning (less so these days) to equivalence. Consequently, from a process point of view there may be NO distinction between the procedures..this can cause some confusion at the procedure level depending on the process theory one subscribes to.. Now one other interesting point in this context raised a moment ago (at least I saw the mail a moment ago) by Cybenko...There is really nothing analogous in the animal learning area with "unsupervised" learning --procedures might be "sensitization", pseudo-conditioning, pre-exposure... all which have relatively complex process accounts. As Cybenko pointed out, "exposure" or Unsupervised procedures really do impose some implicit metric on the system, one which interacts with the way the network computes activation. In a similar sense, animals, in fact, come prewired or "prepared" for classical and operant conditioning with certain classes of stimuli and certain classes of responses and not other classes. These kinds of predispositions mitigate the distinction between supervised and unsupervised learning, of course there is a set unsupervised operations which one can impose on the system. Steve Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa10206; 13 Jun 89 17:45:16 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa24897; 13 Jun 89 10:31:58 EDT Received: from PRINCETON.EDU by CS.CMU.EDU; 13 Jun 89 10:28:59 EDT Received: from clarity.Princeton.EDU by Princeton.EDU (5.58+++/2.17) id AA28936; Tue, 13 Jun 89 10:29:03 EDT Received: by clarity.Princeton.EDU (4.0/1.81) id AA10919; Tue, 13 Jun 89 10:33:48 EDT Date: Tue, 13 Jun 89 10:33:48 EDT From: harnad at Princeton.EDU Message-Id: <8906131433.AA10919 at clarity.Princeton.EDU> To: connectionists at CS.CMU.EDU Subject: Re: Categorization & Supervision In a variant of a comment that appears to have been posted twice, Geoff Hinton added: > Cheeseman's work on categorization (which found a new class of stars > without supervision)... [illustrates] that unsupervised categorization > is NOT arbitrary... I think this debate will be resolved by technical > progress in getting unsupervised learning to work nicely, not by a > priori assertions about what is necessary and what is impossible. This seems an apt point to ponder again the words of the historian J. H. Hexter on the subject of negative a prioris. He wrote: In an academic generation a little overaddicted to "politesse," it may be worth saying that violent destruction is not necessarily worthless and futile. Even though it leaves doubt about the right road for London, it helps if someone rips up, however violently, a `To London' sign on the Dover cliffs pointing south... But I'm certainly prepared to agree that time itself can arbitrate whether or not it has been well spent checking if the unsupervised discovery of a new class of stars also happens to lead us to the nuts-and-bolts categories imposed by the contingencies of nonverbal and verbal life on terra firma... Stevan Harnad Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa10313; 13 Jun 89 17:54:43 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa21431; 13 Jun 89 9:12:22 EDT Received: from FLASH.BELLCORE.COM by CS.CMU.EDU; 13 Jun 89 09:10:33 EDT Received: by flash.bellcore.com (5.58/1.1) id AA15789; Tue, 13 Jun 89 09:09:19 EDT Date: Tue, 13 Jun 89 09:09:19 EDT From: Stephen J Hanson Message-Id: <8906131309.AA15789 at flash.bellcore.com> To: connectionists at CS.CMU.EDU Subject: test please ignore Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa11670; 13 Jun 89 20:44:06 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa27730; 13 Jun 89 12:31:06 EDT Received: from ARPA.ATT.COM by CS.CMU.EDU; 13 Jun 89 12:29:09 EDT From: neural!yann at att.att.com Message-Id: <8906131533.AA05289 at neural.UUCP> Received: from localhost by lesun.UUCP (4.0/4.7) id AA00895; Tue, 13 Jun 89 11:35:26 EDT To: att!cs.cmu.edu!connectionists at att.att.com Cc: att!ai.toronto.edu!hinton at att.att.com Subject: Kolmogorov-Chaitin complexity (was: Categorization and Supervision) In-Reply-To: Your message of Mon, 12 Jun 89 18:41:56 -0400. Reply-To: neural!yann at neural.att.com Date: Tue, 13 Jun 89 11:35:23 -0400 >From: Yann le Cun Like Geoff I "do not intend to clutter up the network with any more meta-level junk", so let's talk about more technical issues. Geoff Hinton says: " If one model of the data is overwhelmingly simpler than any other, then its not just in the mind of the theorist. Its correct. The nice thing about the Kolmogorov-Chaitin view of complexity is that (in the limit) it doesnt need to mention the mind of the observer (i.e. in the limit, one model can be simpler than another WHATEVER the programming language in which we measure simplicity)." The bad thing about KC complexity is that it DOES mention the mind of the observer (or rather, the language he uses) for all finitely complex objects. Infinitely complex objects are infinitely complex for all measures of complexity (in Chaitin's framework they correspond to non-computable functions). But the complexity of finite objects depends on the language used to describe the object. If Ca(x) is the complexity of object x expressed in language a, and Cb(x) the complexity of the same object expressed in an other language b, then there is a constant c such that for any x : Ca(x) < Cb(x) + c the constant c can be interpreted as the complexity of a language translator from b to a. The trouble is: c can be very large, so that the comparison of the complexity of two real (computable) objects depends on the language used to describe the objects. We can have Ca(x) < Ca(y) and Cb(x) > Cb(y). Using this thing as a Universal Criterion for Unsupervised Learning looks quite hopeless. - Yann Le Cun Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa17748; 14 Jun 89 11:19:05 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa03400; 13 Jun 89 19:20:19 EDT Received: from IBM.COM by CS.CMU.EDU; 13 Jun 89 19:17:52 EDT Date: 13 Jun 89 18:33:31 EDT From: Dimitri Kanevsky To: connectionists at CS.CMU.EDU Message-Id: <061389.183332.dimitri at ibm.com> Subject: Categorization & Supervision I have been followng the discussion of supervised learning between G. Hinton and S. Harnad and it is not at all clear to me why Hinton would expect the correspondence between simplicity and correctness to be anything but accidental. Dimitri Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa23564; 14 Jun 89 18:55:59 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa05110; 13 Jun 89 21:55:17 EDT Received: from VMA.CC.CMU.EDU by CS.CMU.EDU; 13 Jun 89 21:53:23 EDT Received: from CMUCCVMA by VMA.CC.CMU.EDU ; Tue, 13 Jun 89 21:52:36 EDT Received: from JHUVMS.BITNET (INS_ATGE) by CMUCCVMA (Mailer X1.25) with BSMTP id 0258; Tue, 13 Jun 89 21:52:35 EDT Date: Tue, 13 Jun 89 21:51 EST From: INS_ATGE%JHUVMS.BITNET at VMA.CC.CMU.EDU MMDF-Warning: Parse error in original version of preceding line at Q.CS.CMU.EDU Subject: Two Questions To: connectionists at CS.CMU.EDU X-Original-To: connectionists at cs.cmu.edu, INS_ATGE I am currently writing a parallel backpropogation program on The Connection Machine, with the immediate task of identifying insonified objects from active sonar data. I was wondering if anyone could give me a reference for a detailed paper on conjugate gradient network teaching technqiues. From what I have picked up from casual conversation, it appears that this method can often lead to faster learning. I would also appreciate information dealing with NN analysis of sonar data (citations in literature besides Sejnowski and Gorman, personal communication, just to get an idea of how the program is stacking up against earlier work). -Thomas G. Edwards ins_atge at jhuvms.BITNET ins_atge at jhunix.hcf.jhu.edu tedwards at cmsun.nrl.navy.mil Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa04553; 15 Jun 89 0:36:56 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa14557; 14 Jun 89 13:29:12 EDT Received: from PRINCETON.EDU by CS.CMU.EDU; 14 Jun 89 08:26:23 EDT Received: from clarity.Princeton.EDU by Princeton.EDU (5.58+++/2.17) id AA07121; Wed, 14 Jun 89 08:26:25 EDT Received: by clarity.Princeton.EDU (4.0/1.81) id AA16907; Wed, 14 Jun 89 08:31:09 EDT Date: Wed, 14 Jun 89 08:31:09 EDT From: harnad at Princeton.EDU Message-Id: <8906141231.AA16907 at clarity.Princeton.EDU> To: connectionists at CS.CMU.EDU Subject: Proper Place of Connectionism... ON THE PROPER PLACE OF CONNECTIONISM IN MODELLING OUR BEHAVIORAL CAPACITIES (Abstract of paper presented at First Annual Meeting of the American Psychological Society, Alexandria VA, June 11 1989) Stevan Harnad Psychology Department Princeton University Princeton NJ 08544 Connectionism is a family of statistical techniques for extracting complex higher-order correlations from data. It can also be interpreted and implemented as a neural network of interconnected units with weighted positive and negative interconnections. Many claims and counterclaims have been made about connectionism: Some have said it will supplant artificial intelligence (symbol manipulation) and explain how we learn and how our brain works. Others have said it is just a limited family of statistical pattern recognition techniques and will not be able to account for most of our behavior and cognition. I will try to sketch how connectionist processes could play a crucial but partial role in modeling our behavioral capacities in learning and representing invariances in the input, thereby mediating the "grounding" of symbolic representations in analog sensory representations. The behavioral capacity I will focus on is categorization: Our ability to sort and label inputs correctly on the basis of feedback from the consequences of miscategorization. Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa22843; 15 Jun 89 7:40:47 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa01101; 14 Jun 89 19:39:50 EDT Received: from ICSIB.BERKELEY.EDU by CS.CMU.EDU; 14 Jun 89 18:55:26 EDT Received: from icsib6. (icsib6.Berkeley.EDU) by icsib.Berkeley.EDU (4.0/SMI-4.0) id AA21096; Wed, 14 Jun 89 15:59:22 PDT Received: by icsib6. (4.0/SMI-4.0) id AA11131; Wed, 14 Jun 89 15:56:27 PDT Date: Wed, 14 Jun 89 15:56:27 PDT From: Andreas Stolcke Message-Id: <8906142256.AA11131 at icsib6.> To: connectionists at CS.CMU.EDU Subject: Tech. Report available The following Technical Report is available from ICSI: ______________________________________________________________________ TR-89-032 Andreas Stolcke 5/01/89 19 pages,$1.75 "A Connectionist Model of Unification" A general approach to encode and unify recursively nested feature structures in connectionist networks is described. The unification algorithm implemented by the net is based on iterative coarsening of equivalence classes of graph nodes. This method allows the reformulation of unification as a constraint satisfaction problem and enables the connectionist implementation to take full advantage of the potential parallelism inherent in unification, resuting in sublinear time complexity. Moreover, the method is able to process any number of feature structures in parallel, searching for possible unifications and making decisions among mutually exclusive unifications where necessary. ______________________________________________________________________ International Computer Science Institute Technical Reports There is a small charge to cover postage and handling for each report. This charge is waived for ICSI sponsors and for institutions having an exchange agreement with the Institute. Please use the form at the back of the list for your order. Make any necessary additions or corrections to the address label on the form, and return it to the International Computer Science Institute. NOTE: Qualifying institutions may choose to participate in a technical report exchange program and receive ICSI TRs at no charge. To arrange an exchange agreement write, call or send e-mail message to: Librarian International Computer Science Institute 1947 Center Street, Suite 600 Berkeley, CA 94704 info at icsi.berkeley.EDU (415) 643-9153. ______________________________________________________________________ Technical Report Order Form International Computer Science Institute 1947 Center Street, Suite 600 Berkeley, CA 94704 Please enclose your check to cover postage and handling. Prepayment is required for all materials. Charges will be waived for ICSI sponsors and for institutions that have exchange agreements with the Institute. Make checks payable to "ICSI". __________________________________________________________ | | | | | | TR Number | Quantity | Postage & Handling| Total | |______________|___________|___________________|___________| | | | | | | |______________|___________|___________________|_______|___| | | | | | | |______________|___________|___________________|_______|___| | | | | | | |______________|___________|___________________|_______|___| | | | | | | |______________|___________|___________________|_______|___| | | | | | | |______________|___________|___________________|_______|___| | | | | | | |______________|___________|___________________|_______|___| | | | | | | |______________|___________|___________________|_______|___| | | | | | | |______________|___________|___________________|_______|___| | | | | | | |______________|___________|___________________|_______|___| | | | | | | |______________|___________|___________________|_______|___| | | | | | | |______________|___________|___________________|_______|___| | | | | | | |______________|___________|___________________|_______|___| | | | | | | |______________|___________|___________________|_______|___| | | | | | | |______________|___________|___________________|_______|___| | | | | | Total Amount | | | |___________________|_______|___| NAME AND ADDRESS __ __________________________ |__| Please note change of address __ as shown. __________________________ |__| Please remove my name from this __ mailing list. __________________________ |__| Please add my name to this mailing list. __________________________ __________________________ Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa14169; 16 Jun 89 12:09:45 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa27023; 16 Jun 89 12:04:26 EDT Received: from VMA.CC.CMU.EDU by CS.CMU.EDU; 16 Jun 89 06:34:46 EDT Received: from CMUCCVMA by VMA.CC.CMU.EDU ; Fri, 16 Jun 89 06:34:14 EDT Received: from UKACRL.BITNET by CMUCCVMA (Mailer X1.25) with BSMTP id 2217; Fri, 16 Jun 89 06:34:13 EDT Received: from RL.IB by UKACRL.BITNET (Mailer X1.25) with BSMTP id 7570; Fri, 16 Jun 89 10:51:44 BST Received: Via: UK.AC.EX.CS; 16 JUN 89 10:51:38 BST Received: from exsc by expya.cs.exeter.ac.uk; Fri, 16 Jun 89 10:58:04-0000 From: Lyn Shackleton Date: Fri, 16 Jun 89 10:50:29 BST Message-Id: <702.8906160950 at exsc.cs.exeter.ac.uk> To: connectionists at CS.CMU.EDU Subject: journal reviewers ******* CONNECTION SCIENCE ****** Editor: Noel E. Sharkey Because fo the number of specialist submissions, the journal is currently expanding its review panel. This is an interdisciplinary journal with an emphasis on replicability of results. If you wish to volunteer please send details of your review area to the address below. Or write for further details. lyn shackleton (assistant editor) Centre for Connection Science JANET: lyn at uk.ac.exeter.cs Dept. Computer Science University of Exeter UUCP: !ukc!expya!lyn Exeter EX4 4PT Devon BITNET: lyn at cs.exeter.ac.uk.UKACRL U.K. Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa09476; 16 Jun 89 18:22:40 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa28639; 16 Jun 89 13:20:32 EDT Received: from PRINCETON.EDU by CS.CMU.EDU; 16 Jun 89 13:18:10 EDT Received: from suspicion.Princeton.EDU by Princeton.EDU (5.58+++/2.17) id AA00231; Fri, 16 Jun 89 11:30:25 EDT Received: by suspicion.Princeton.EDU (4.0/1.81) id AA02388; Fri, 16 Jun 89 11:35:14 EDT Date: Fri, 16 Jun 89 11:35:14 EDT From: jose at confidence.Princeton.EDU Message-Id: <8906161535.AA02388 at suspicion.Princeton.EDU> To: connectionists at CS.CMU.EDU Subject: lost papers... through a series of complex, political and unnecessarily confusing and obfuscating moves by various parties..this field's attempt to have one coherent, reasonable quality meeting has been foiled. Now we have (at least) 3 of varying coherence and quality. IJCNN which is meeting very soon this month, INNS will meet sometime in January, and NIPS will occur as always in November. It is too bad that greed and self-interest seems to take the place of the sort of nurturing and careful, thoughtful, long term commitment to a field that has such great potential. It would be a shame to see the polarization within the field allow people's work and thought to be lost or ignored. Steve Hanson Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa08295; 16 Jun 89 21:24:38 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa03304; 16 Jun 89 17:21:10 EDT Received: from ANDREW.CMU.EDU by CS.CMU.EDU; 16 Jun 89 17:16:05 EDT Received: by andrew.cmu.edu (5.54/3.15) id for connectionists at cs.cmu.edu; Fri, 16 Jun 89 17:15:56 EDT Received: via switchmail; Fri, 16 Jun 89 17:15:53 -0400 (EDT) Received: from chicory.psy.cmu.edu via qmail ID ; Fri, 16 Jun 89 17:13:50 -0400 (EDT) Received: from chicory.psy.cmu.edu via qmail ID ; Fri, 16 Jun 89 17:13:39 -0400 (EDT) Received: from BatMail.robin.v2.8.CUILIB.3.45.SNAP.NOT.LINKED.chicory.psy.cmu.edu.sun3.35 via MS.5.6.chicory.psy.cmu.edu.sun3_35; Fri, 16 Jun 89 17:13:33 -0400 (EDT) Message-Id: Date: Fri, 16 Jun 89 17:13:33 -0400 (EDT) From: "James L. McClelland" To: connectionists at CS.CMU.EDU Subject: Let 1,000 flowers bloom In-Reply-To: <8906161535.AA02388 at suspicion.Princeton.EDU> References: <8906161535.AA02388 at suspicion.Princeton.EDU> In response to Steve Hanson's last message: I myself do not mind the fact that there are three conectionist meetings, as well as others where connectionist work can be presented. Sure, there are factions and there is politics; this has lead to some fragmentation. But, there is an upside. Different perspectives all have a chance to be represented, and different goals have a chance to be served. The NIPS meeting, for example, is a small, high-quality meeting dedicated to highly quantitative, computational-theory type work with a biological flavor; I think it would be a shame if these characteristics were lost in an attempt to achieve some grand synthesis. The other conferences have their specific strengths as well. Meanwhile, non-connectionist conferences like Cognitive Science and AAAI attract some of the good connectionist work applied to language and higher-level aspects of cognition. People seem to be gravitating toward attending the one or two of these that best suit their own needs and interests within the broad range of connectionist research. Seems to me things are pretty close to the way they ought to be. It would be nice if the process of arriving at this state could have been smoother, and maybe in the end it won't turn out that there need to be quite so many meetings, but things could be a heck of a lot worse. -- Jay McClelland Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa00328; 17 Jun 89 3:02:39 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa07666; 16 Jun 89 22:50:01 EDT Received: from EPHEMERAL.AI.TORONTO.EDU by CS.CMU.EDU; 16 Jun 89 22:48:23 EDT Received: from localhost (stdin) by ephemeral.ai.toronto.edu with SMTP id 10958; Fri, 16 Jun 89 15:33:20 EDT To: connectionists at CS.CMU.EDU Subject: CRG-TR-89-3 Date: Fri, 16 Jun 89 15:33:15 EDT From: Carol Plathan Message-Id: <89Jun16.153320edt.10958 at ephemeral.ai.toronto.edu> The Technical Report CRG-TR-89-3 by Hinton and Shallice (May 1989) can be obtained by sending me your full mailing address. An abstract of this Report follows: LESIONING A CONNECTIONIST NETWORK: INVESTIGATIONS OF ACQUIRED DYSLEXIA ----------------------------------------------------------------------- Geoffrey E. Hinton Tim Shallice Department of Computer Science MRC Applied Psychology Unit University of Toronto Cambridge, UK ABSTRACT: -------- A connectionist network which had been trained to map orthographic representation into semantic ones was systematically 'lesioned'. Wherever it was lesioned it produced more Visual, Semantic, and Mixed visual and semantic errors than would be expected by chance. With more severe lesions it showed relatively spared categorical discrimination when item identification was not possible. Both phenomena are qualitatively similar to those observed in neurological patients. The error pattern is that characteristically found in deep dyslexia. The spared categorical discrimination is observed in semantic access dyslexia and also in a form of pure alexia. It is concluded that the lesioning of connectionist networks may produce phenomena which mimic non-transparent aspects of the behaviour of neurological patients. Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa04289; 17 Jun 89 18:05:49 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa13079; 17 Jun 89 17:30:52 EDT Received: from IBM.COM by CS.CMU.EDU; 17 Jun 89 17:28:36 EDT Date: 17 Jun 89 17:05:59 EDT From: Scott Kirkpatrick To: connectionists at CS.CMU.EDU cc: Steve at confidence.princeton.edu Message-Id: <061789.170559.kirk at ibm.com> Subject: NIPS 89 status and schedule Authors have been known to call me, Rich Lippman (the program committee chair) or Kathie Hibbard (who runs the local office in Denver), asking when the NIPS program decisions will be made, announced, etc... I'll give our schedule in order to restrict the phone calls to those which can help us to catch mistakes. Steve Hanson (NIPS publicity, and responsible for our eye-catching blue poster) -- please put a version of this on all the other bulletin boards. Deadline for abstracts and summaries was May 30, 1989. We have now received over 460 contributions (almost 50% more than last year!). They are now logged in, and cards acknowledging receipt will be mailed next week to authors. Authors who have not received an acknowledgement by June 30 should write to Kathie Hibbard at the NIPS office; it's possible we got your address wrong in our database, and this will help us catch these things. Refereeing will take July. Collecting the results and defining a final program will be done in August. We plan to mail letters informing authors of the outcome during the first week of September. At that time, we will send all registration material, information about prices, and a complete program. If you haven't heard from us in late September, again please write, to help us straighten things out. Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa05039; 17 Jun 89 20:45:24 EDT Received: from csvax.caltech.edu by Q.CS.CMU.EDU id aa14984; 17 Jun 89 20:41:52 EDT Received: from aurel.caltech.edu by csvax.caltech.edu (5.59/1.2) id AA09482; Sat, 17 Jun 89 16:36:47 PDT Received: from smaug.caltech.edu. by aurel.caltech.edu (4.0/SMI-4.0) id AA00438; Sat, 17 Jun 89 16:39:32 PDT Received: by smaug.caltech.edu. (4.0/SMI-4.0) id AA28797; Sat, 17 Jun 89 14:39:57 PDT Date: Sat, 17 Jun 89 14:39:57 PDT From: Jim Bower Message-Id: <8906172139.AA28797 at smaug.caltech.edu.> To: connectionists at Q.CS.CMU.EDU Subject: 3 meetings Concerning Steve Hanson's comments on meetings. I think that it is only fair to note that not all the recent history of neural-net meetings have been characterized by "greed and self-interest". In order for greed to be an issue there must be an opportunity for organizers to make money. In order for self-interest to be an issue there must be a biasable mechanism for self promotion. Opportunities for commercialization and the often outrageous associated claims apply to both cases. In this regard, in my view, it is unfair to mention all three national neural network meetings in the same sentence. One of the three meetings was founded and continues to be organized to provide a "nurturing, careful, and thoughtful" forum committed to the long term support and growth of a field with "considerable potential". Most of you know which meeting that is, but if a clue is necessary, it is the only one that takes place in a city not known as a national focus for greed and self-interest. Jim Bower Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa03661; 18 Jun 89 15:29:07 EDT Received: from vma.cc.cmu.edu by Q.CS.CMU.EDU id aa02704; 18 Jun 89 15:25:31 EDT Received: from CMUCCVMA by VMA.CC.CMU.EDU ; Sun, 18 Jun 89 15:25:51 EDT Received: from UKACRL.BITNET by CMUCCVMA (Mailer X1.25) with BSMTP id 7292; Sun, 18 Jun 89 15:25:49 EDT Received: from RL.IB by UKACRL.BITNET (Mailer X1.25) with BSMTP id 7878; Sun, 18 Jun 89 20:23:50 BST Received: Via: UK.AC.EX.CS; 18 JUN 89 20:23:43 BST To: cs.exeter-connect at CS.EXETER.AC.UK Received: Received: from q.cs.cmu.edu by NSFnet-Relay.AC.UK via NSFnet with SMTP id aa09349; 18 Jun 89 3:58 BS Received: from csvax.caltech.edu by Q.CS.CMU.EDU id aa14984; 17 Jun 89 20:41:52 ED Received: from aurel.caltech.edu by csvax.caltech.edu (5.59/1.2) id AA09482; Sat, 17 Jun 89 16:36:47 PD Received: from smaug.caltech.edu. by aurel.caltech.edu (4.0/SMI-4.0) id AA00438; Sat, 17 Jun 89 16:39:32 PD Received: by smaug.caltech.edu. (4.0/SMI-4.0) id AA28797; Sat, 17 Jun 89 14:39:57 PD Date: Sat, 17 Jun 89 14:39:57 PDT From: Jim Bower Message-Id: <8906172139.AA28797 at smaug.caltech.edu.> To: connectionists at Q.CS.CMU.EDU Subject: 3 meetings Original-Sender: Connectionists-Request at edu.cmu.cs.q Sender: connect-bb-request at CS.EXETER.AC.UK Concerning Steve Hanson's comments on meetings. I think that it is only fair to note that not all the recent history of neural-net meetings have been characterized by "greed and self-interest". In order for greed to be an issue there must be an opportunity for organizers to make money. In order for self-interest to be an issue there must be a biasable mechanism for self promotion. Opportunities for commercialization and the often outrageous associated claims apply to both cases. In this regard, in my view, it is unfair to mention all three national neural network meetings in the same sentence. One of the three meetings was founded and continues to be organized to provide a "nurturing, careful, and thoughtful" forum committed to the long term support and growth of a field with "considerable potential". Most of you know which meeting that is, but if a clue is necessary, it is the only one that takes place in a city not known as a national focus for greed and self-interest. Jim Bower Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa06023; 18 Jun 89 18:08:05 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa02751; 18 Jun 89 15:28:06 EDT Received: from VMA.CC.CMU.EDU by CS.CMU.EDU; 18 Jun 89 15:24:17 EDT Received: from CMUCCVMA by VMA.CC.CMU.EDU ; Sun, 18 Jun 89 15:24:45 EDT Received: from UKACRL.BITNET by CMUCCVMA (Mailer X1.25) with BSMTP id 7252; Sun, 18 Jun 89 15:24:40 EDT Received: from RL.IB by UKACRL.BITNET (Mailer X1.25) with BSMTP id 7874; Sun, 18 Jun 89 20:22:47 BST Received: Via: UK.AC.EX.CS; 18 JUN 89 20:22:42 BST To: cs.exeter-connect at CS.EXETER.AC.UK Received: Received: from q.cs.cmu.edu by NSFnet-Relay.AC.UK via NSFnet with SMTP id aa01674; 17 Jun 89 9:11 BS Received: Received: from VMA.CC.CMU.EDU by CS.CMU.EDU; 16 Jun 89 06:34:46 EDT Received: from CMUCCVMA by VMA.CC.CMU.EDU ; Fri, 16 Jun 89 06:34:14 EDT Received: from UKACRL.BITNET by CMUCCVMA (Mailer X1.25) with BSMTP id 2217; Fri, 16 Jun 89 06:34:13 ED Received: Received: Original-Via: UK.AC.EX.CS; 16 JUN 89 10:51:38 BST Received: from exsc by expya.cs.exeter.ac.uk; Fri, 16 Jun 89 10:58:04-0000 From: Lyn Shackleton Date: Fri, 16 Jun 89 10:50:29 BST Message-Id: <702.8906160950 at exsc.cs.exeter.ac.uk> To: connectionists at CS.CMU.EDU Subject: journal reviewers Original-Sender: Connectionists-Request at edu.cmu.cs.q Sender: connect-bb-request at CS.EXETER.AC.UK ******* CONNECTION SCIENCE ****** Editor: Noel E. Sharkey Because fo the number of specialist submissions, the journal is currently expanding its review panel. This is an interdisciplinary journal with an emphasis on replicability of results. If you wish to volunteer please send details of your review area to the address below. Or write for further details. lyn shackleton (assistant editor) Centre for Connection Science JANET: lyn at uk.ac.exeter.cs Dept. Computer Science University of Exeter UUCP: !ukc!expya!lyn Exeter EX4 4PT Devon BITNET: lyn at cs.exeter.ac.uk.UKACRL U.K. Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa06079; 18 Jun 89 18:12:51 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa02706; 18 Jun 89 15:25:48 EDT Received: from VMA.CC.CMU.EDU by CS.CMU.EDU; 18 Jun 89 15:22:25 EDT Received: from CMUCCVMA by VMA.CC.CMU.EDU ; Sun, 18 Jun 89 15:22:45 EDT Received: from UKACRL.BITNET by CMUCCVMA (Mailer X1.25) with BSMTP id 7140; Sun, 18 Jun 89 15:22:43 EDT Received: from RL.IB by UKACRL.BITNET (Mailer X1.25) with BSMTP id 7856; Sun, 18 Jun 89 20:20:44 BST Received: Via: UK.AC.EX.CS; 18 JUN 89 20:20:36 BST To: cs.exeter-connect at CS.EXETER.AC.UK Received: Received: from q.cs.cmu.edu by NSFnet-Relay.AC.UK via NSFnet with SMTP id aa01325; 17 Jun 89 8:37 BS Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa07666; 16 Jun 89 22:50:01 EDT Received: from EPHEMERAL.AI.TORONTO.EDU by CS.CMU.EDU; 16 Jun 89 22:48:23 EDT Received: To: connectionists at CS.CMU.EDU Subject: CRG-TR-89-3 Date: Fri, 16 Jun 89 15:33:15 EDT From: Carol Plathan Message-Id: <89Jun16.153320edt.10958 at ephemeral.ai.toronto.edu> Original-Sender: Connectionists-Request at edu.cmu.cs.q Sender: connect-bb-request at CS.EXETER.AC.UK The Technical Report CRG-TR-89-3 by Hinton and Shallice (May 1989) can be obtained by sending me your full mailing address. An abstract of this Report follows: LESIONING A CONNECTIONIST NETWORK: INVESTIGATIONS OF ACQUIRED DYSLEXIA ----------------------------------------------------------------------- Geoffrey E. Hinton Tim Shallice Department of Computer Science MRC Applied Psychology Unit University of Toronto Cambridge, UK ABSTRACT: -------- A connectionist network which had been trained to map orthographic representation into semantic ones was systematically 'lesioned'. Wherever it was lesioned it produced more Visual, Semantic, and Mixed visual and semantic errors than would be expected by chance. With more severe lesions it showed relatively spared categorical discrimination when item identification was not possible. Both phenomena are qualitatively similar to those observed in neurological patients. The error pattern is that characteristically found in deep dyslexia. The spared categorical discrimination is observed in semantic access dyslexia and also in a form of pure alexia. It is concluded that the lesioning of connectionist networks may produce phenomena which mimic non-transparent aspects of the behaviour of neurological patients. Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa08767; 19 Jun 89 0:41:09 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa02712; 18 Jun 89 15:26:22 EDT Received: from VMA.CC.CMU.EDU by CS.CMU.EDU; 18 Jun 89 15:22:58 EDT Received: from CMUCCVMA by VMA.CC.CMU.EDU ; Sun, 18 Jun 89 15:23:18 EDT Received: from UKACRL.BITNET by CMUCCVMA (Mailer X1.25) with BSMTP id 7170; Sun, 18 Jun 89 15:23:17 EDT Received: from RL.IB by UKACRL.BITNET (Mailer X1.25) with BSMTP id 7866; Sun, 18 Jun 89 20:21:07 BST Received: Via: UK.AC.EX.CS; 18 JUN 89 20:21:02 BST To: cs.exeter-connect at CS.EXETER.AC.UK Received: Received: from q.cs.cmu.edu by NSFnet-Relay.AC.UK via NSFnet with SMTP id aa01599; 17 Jun 89 9:03 BS Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa28639; 16 Jun 89 13:20:32 EDT Received: from PRINCETON.EDU by CS.CMU.EDU; 16 Jun 89 13:18:10 EDT Received: from suspicion.Princeton.EDU by Princeton.EDU (5.58+++/2.17) id AA00231; Fri, 16 Jun 89 11:30:25 ED Received: by suspicion.Princeton.EDU (4.0/1.81) id AA02388; Fri, 16 Jun 89 11:35:14 ED Date: Fri, 16 Jun 89 11:35:14 EDT From: jose at CONFIDENCE.PRINCETON.EDU Message-Id: <8906161535.AA02388 at suspicion.Princeton.EDU> To: connectionists at CS.CMU.EDU Subject: lost papers... Original-Sender: Connectionists-Request at edu.cmu.cs.q Sender: connect-bb-request at CS.EXETER.AC.UK through a series of complex, political and unnecessarily confusing and obfuscating moves by various parties..this field's attempt to have one coherent, reasonable quality meeting has been foiled. Now we have (at least) 3 of varying coherence and quality. IJCNN which is meeting very soon this month, INNS will meet sometime in January, and NIPS will occur as always in November. It is too bad that greed and self-interest seems to take the place of the sort of nurturing and careful, thoughtful, long term commitment to a field that has such great potential. It would be a shame to see the polarization within the field allow people's work and thought to be lost or ignored. Steve Hanson Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa10897; 19 Jun 89 5:48:18 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa14967; 19 Jun 89 5:40:18 EDT Received: from VMA.CC.CMU.EDU by CS.CMU.EDU; 19 Jun 89 05:38:17 EDT Received: from CMUCCVMA by VMA.CC.CMU.EDU ; Mon, 19 Jun 89 05:38:43 EDT Received: from UKACRL.BITNET by CMUCCVMA (Mailer X1.25) with BSMTP id 0828; Mon, 19 Jun 89 05:38:42 EDT Received: from RL.IB by UKACRL.BITNET (Mailer X1.25) with BSMTP id 6641; Mon, 19 Jun 89 10:36:41 BST Received: Via: UK.AC.EX.CS; 19 JUN 89 10:36:23 BST Received: From: Noel Sharkey Date: Mon, 19 Jun 89 10:35:56 BST Message-Id: <4309.8906190935 at entropy.cs.exeter.ac.uk> To: carol at AI.TORONTO.EDU Cc: connectionists at CS.CMU.EDU In-Reply-To: Carol Plathan's message of Fri, 16 Jun 89 15:33:15 EDT <89Jun16.153320edt.10958 at ephemeral.ai.toronto.edu Subject: CRG-TR-89-3 please send me a copy of Report CRG-TR-89-3 by Hinton and Shallice (May 1989). LESIONING A CONNECTIONIST NETWORK: INVESTIGATIONS OF ACQUIRED DYSLEXIA noel sharkey Centre for Connection Science JANET: noel at uk.ac.exeter.cs Dept. Computer Science University of Exeter UUCP: !ukc!expya!noel Exeter EX4 4PT Devon BITNET: noel at cs.exeter.ac.uk@UKACRL U.K. Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa01375; 19 Jun 89 10:01:27 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa15712; 19 Jun 89 6:05:21 EDT Received: from VMA.CC.CMU.EDU by CS.CMU.EDU; 19 Jun 89 06:03:46 EDT Received: from CMUCCVMA by VMA.CC.CMU.EDU ; Mon, 19 Jun 89 06:04:13 EDT Received: from UKACRL.BITNET by CMUCCVMA (Mailer X1.25) with BSMTP id 0907; Mon, 19 Jun 89 06:04:12 EDT Received: from RL.IB by UKACRL.BITNET (Mailer X1.25) with BSMTP id 7010; Mon, 19 Jun 89 10:50:54 BST Received: Via: UK.AC.EX.CS; 19 JUN 89 10:50:38 BST Received: From: Noel Sharkey Date: Mon, 19 Jun 89 10:50:37 BST Message-Id: <4312.8906190950 at entropy.cs.exeter.ac.uk> To: jose at CONFIDENCE.PRINCETON.EDU Cc: connectionists at CS.CMU.EDU In-Reply-To: jose at edu.princeton.confidence's message of Fri, 16 Jun 89 11:35:14 EDT <8906161535.AA02388 at suspicion.Princeton.EDU Subject: lost papers... Hanson's comment seems a bit bitter ... i wonder what is really behind it. In a field growing as rapidly as connectionism, I would have thought that we need a lot more annual meetings. And there will of course be more when we in Europe get our act together. I think that the field is getting far to large to be unified. Papers come at me from all directions - the structure of the hippocampus to reasoning in natural language understanding and low-level visual perception. Surely it is inevidable that the "field" will fractionate into many specialised sub-fields as is happening at the cognitive science meeting etc. as Jay pointed out. Imagine having one annual psychology meeting, or one annual physics meeting. noel sharkey Centre for Connection Science JANET: noel at uk.ac.exeter.cs Dept. Computer Science University of Exeter UUCP: !ukc!expya!noel Exeter EX4 4PT Devon BITNET: noel at cs.exeter.ac.uk@UKACRL U.K. Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa02032; 19 Jun 89 10:53:39 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa17474; 19 Jun 89 8:49:38 EDT Received: from PRINCETON.EDU by CS.CMU.EDU; 19 Jun 89 08:47:31 EDT Received: from suspicion.Princeton.EDU by Princeton.EDU (5.58+++/2.17) id AA12779; Mon, 19 Jun 89 08:47:20 EDT Received: by suspicion.Princeton.EDU (4.0/1.81) id AA00471; Mon, 19 Jun 89 08:52:13 EDT Date: Mon, 19 Jun 89 08:52:13 EDT From: "Stephen J. Hanson" Message-Id: <8906191252.AA00471 at suspicion.Princeton.EDU> To: jose at confidence.Princeton.EDU, noel%CS.EXETER.AC.UK at pucc.Princeton.EDU Subject: Re: lost papers... Cc: connectionists at CS.CMU.EDU Flowers and all.. (flame on) Actually, I agree with Jay and Noel, diversity is the spice of life. In fact, one of the key aspects of this field is the fact that one can find 8-9 diciplines in the room represented. I think we forget sometimes how remarkable this actually is. I also think it is important to remember as we rush to sell our version of the story, or make a septobijjillion dollars selling neural net black boxes or teaching neural nets to the great unwashed, that we don't shoot ourselves in our collective feet --after all who is the competition here? remember some sort of proto-AI killed off this field less than 40 years ago..all I was suggesting was that it would be nice if there was sort of agreement about organization, politics and coherence about progress in the field--god knows--not about the subject matter. I realize this is unlikely and somewhat naive, and more (conferences, journals, etc..) is usually better in any field.. its just that all these flowers look a bit carnivorous. (flame off) and enough.. back to some substance please. jose Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa20520; 20 Jun 89 14:35:03 EDT Received: from ri.cmu.edu by Q.CS.CMU.EDU id aa14829; 20 Jun 89 14:08:20 EDT Received: from TRACTATUS.BELLCORE.COM by RI.CMU.EDU; 20 Jun 89 14:05:10 EDT Received: by tractatus.bellcore.com (5.61/1.34) id AA13847; Mon, 19 Jun 89 08:51:16 -0400 Date: Mon, 19 Jun 89 08:51:16 -0400 From: Stephen J Hanson Message-Id: <8906191251.AA13847 at tractatus.bellcore.com> To: DJERS at TUCC.TUCC.EDU, TheoryNet at ibm.com, ailist at kl.sri.com, arpanet-bboards at mc.lcs.mit.edu, biotech%umdc.BITNET at cunyvm.cuny.edu, chipman at NPRDC.NAVY.MIL, conferences at hplabs.hp.com, connectionists at RI.CMU.EDU, dynsys-l%unc.BITNET at cunyvm.cuny.edu, epsynet%uhupvm1.BITNET at cunyvm.cuny.edu, gazzaniga at tractatus.bellcore.com, hirst at ROCKY2.ROCKEFELLER.EDU, info-futures at bu-cs.bu.edu, kaiser%yorkvm1.BITNET at cunyvm.cuny.edu, keeler at mcc.com, mike%bucasb.bu.edu at bu-it.bu.edu, msgs at tractatus.bellcore.com, msgs at confidence.princeton.edu, neuron at ti-csl.csc.ti.com, parsym at sumex-aim.stanford.edu, physics at mc.lcs.mit.edu, self-org at mc.lcs.mit.edu, soft-eng at MINTAKA.LCS.MIT.EDU, taylor at hplabsz.hpl.hp.com, vision-list at ADS.COM Subject: NIPS Schedule *********NIPS UPDATE********** Deadline for abstracts and summaries was May 30, 1989. We have now received over 460 contributions (almost 50% more than last year!). They are now logged in, and cards acknowledging receipt will be mailed next week to authors. Authors who have not received an acknowledgement by June 30 should write to Kathie Hibbard at the NIPS office; it's possible we got your address wrong in our database, and this will help us catch these things. Refereeing will take July. Collecting the results and defining a final program will be done in August. We plan to mail letters informing authors of the outcome during the first week of September. At that time, we will send all registration material, information about prices, and a complete program. If you haven't heard from us in late September, again please write, to help us straighten things out. **********NIPS UPDATE*********** Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa27436; 20 Jun 89 19:58:42 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa16421; 20 Jun 89 15:22:48 EDT Received: from ELROND.STANFORD.EDU by CS.CMU.EDU; 20 Jun 89 15:20:05 EDT Received: by elrond.Stanford.EDU (3.2/4.7); Tue, 20 Jun 89 12:03:08 PDT Date: Tue, 20 Jun 89 12:03:08 PDT From: Dave Rumelhart To: jose at confidence.Princeton.EDU, noel%CS.EXETER.AC.UK at pucc.Princeton.EDU Subject: Re: lost papers... Cc: connectionists at CS.CMU.EDU Not that I want to prolong this discussion, but as a member of the INNS board, I should like to clarify one point concerning the the IJCNN meetings and the INNS. In fact, the movment has been toward cooperation between the large IEEE sponsored meeting and the large INNS sponsored meeting, formerly known as the ICNN meeting. There has been an agreement whereby IEEE and INNS will co-sponsor two meetings per year -- roughly a summer meeting and a winter meeting. These jointly sponsored meetings have been dubbed IJCNN or International JOINT Conference on Neural Networks with the JOINT to signify the joint sponsorship. The movement of the INNS annual meeting from September 1989 to January 1990 has been by way of cooperating, not by way of competing. The current plan to continue to jointly sponsor two meetings per year as long as there is sufficient interest. In addition INNS and IEEE will probably help sponsor occasional European and Japanese meetings from time to time. That there will, in addition, of course, be a number of smaller meetings and workshops sponsored my other groups is, in my opinion, healthy and natural. There are many people with many interests working on things they call (or are willing to call) neural networks. It is normal that special interest groups should form within such a interdisciplinary field. I hope these comments are of some use to those bewildered by the array of meetings. I, in fact, believe that the establishment of the joint meeting plan between INNS and IEEE was a major accomplishment and certainly a move in the right direction for the field. David E. Rumelhart der at psych.stanford.edu Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa01011; 21 Jun 89 3:51:42 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa24477; 21 Jun 89 3:48:22 EDT Received: from UUNET.UU.NET by CS.CMU.EDU; 21 Jun 89 03:46:49 EDT Received: from munnari.UUCP by uunet.uu.net (5.61/1.14) with UUCP id AA14955; Wed, 21 Jun 89 03:46:42 -0400 From: munnari!cluster.cs.su.OZ.AU!ray at uunet.uu.net Received: from munnari.oz.au (munnari.oz) by murtoa.cs.mu.oz (5.5) id AA01986; Wed, 21 Jun 89 16:13:32 EST (from ray at cluster.cs.su.OZ.AU for uunet!connectionists at CS.CMU.EDU) Received: from cluster.cs.su.oz (via basser) by munnari.oz.au with SunIII (5.61+IDA+MU) id AA20510; Wed, 21 Jun 89 13:21:46 +1000 (from ray at cluster.cs.su.OZ.AU for connectionists at CS.CMU.EDU@murtoa.cs.mu.OZ.AU) Date: Wed, 21 Jun 89 13:20:51 +1000 Message-Id: <8906210321.20510 at munnari.oz.au> To: connectionists at CS.CMU.EDU Subject: location of IJCNN conferences Cc: munnari!ray at uunet.uu.net > From Connectionists-Request at q.cs.cmu.edu@murtoa.cs.mu.oz > From: der%elrond.Stanford.EDU at murtoa.cs.mu.oz (Dave Rumelhart) > Date: Tue, 20 Jun 89 12:03:08 PDT > To: jose at ... noel%... > Subject: Re: lost papers... > Cc: connectionists at CS.CMU.EDU > > ... The current plan to continue to jointly sponsor two meetings per > year as long as there is sufficient interest. In addition INNS and > IEEE will probably help sponsor occasional European and Japanese > meetings from time to time. This is one aspect of the current arrangement that bothers me. The INNS is an international society, but its *premier* conference is held only in the USA. Given the high percentage of US INNS members, and the amount of NN research done in the USA, its fitting that the bulk of NN conferences be held there. But the INNS' premier conference should be held occasionally outside the USA. The IJCAI and AAAI have a good arrangement. The IJCAI conferences are held every second year, with every second conference held in North America. The AAAI holds its own conference in the USA in the three out of four years that the IJCAI is not in North America. I think the exact periodicities for the AAAI and IJCAI may not suit the IEEE and INNS. It seems that people want more frequent conferences. But I think the general idea is sound. Perhaps every second occurrence of the January conference could be held outside the USA? (Leaving three out of every four IJCNN conferences in the USA.) As I understand it, the current arrangement will be reviewed in a year or two. I was glad to see the INNS and IEEE get together and coordinate the conferences. I just think some fine tuning is required. Raymond Lister Basser Department of Computer Science University of Sydney AUSTRALIA Internet: ray at basser.cs.su.oz.au@uunet.uu.net Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa05561; 21 Jun 89 7:39:33 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa26050; 21 Jun 89 5:08:45 EDT Received: from UUNET.UU.NET by CS.CMU.EDU; 21 Jun 89 05:06:33 EDT Received: from munnari.UUCP by uunet.uu.net (5.61/1.14) with UUCP id AA01454; Wed, 21 Jun 89 05:06:24 -0400 From: munnari!cluster.cs.su.OZ.AU!ray at uunet.uu.net Received: from munnari.oz.au (munnari.oz) by murtoa.cs.mu.oz (5.5) id AA06593; Wed, 21 Jun 89 18:38:56 EST (from ray at cluster.cs.su.OZ.AU for uunet!connectionists at CS.CMU.EDU) Received: from cluster.cs.su.oz (via basser) by munnari.oz.au with SunIII (5.61+IDA+MU) id AA24824; Wed, 21 Jun 89 18:38:52 +1000 (from ray at cluster.cs.su.OZ.AU for connectionists at CS.CMU.EDU@murtoa.cs.mu.OZ.AU) Date: Wed, 21 Jun 89 18:37:48 +1000 Message-Id: <8906210838.24824 at munnari.oz.au> To: singer at THINK.COM Subject: Re: Sparse vs. Dense networks Cc: munnari!ray at uunet.uu.net, connectionists at CS.CMU.EDU Hopfield and Tank's approach to the N city traveling salesman problem (TSP) ('"Neural" Computation of Decisions in Optimization Problems' Biol. Cybern. 52, pp 141-152 (1985)) used a N**2 matrix of neurons. Each neuron is connected to kN other neurons. Bailey and Hammerstrom pointed out that this high level of interconnect raises the area requirement to N**3. ('Why VLSI Implementations of Associative VLCNs Require Connection Multiplexing', IEEE International Conference on Neural Networks, San Diego (1988), pp II-173 to II-180 - anyone interested in implementation issues, but without a background in VLSI design, should read this paper). An N**2 area requirement is pushing it. N**3 is just about impractical, for any but very small TSPs. Adding metal layers for interconnect doesn't beat the problem (unless the number of metal layers is a function of N, which is impractical). I have devised an approach that uses an N**2 array of neurons, like H&T, but which requires only log N interconnect per neuron (giving an overall area requirement of (N**2)*(log N). My approach does not use multiplexing. It works by restricting the matrix to legal solutions. Despite this restriction, the approach generates quite good TSP solutions. Not only does the approach reduce the level of interconnect to practical levels, it also suggests that the capacity of analog approaches to move within the volume of the solution hypercube is not as important as previously thought. If you would like to know more about my approach, send your complete postal address, and I'll send you a paper. Raymond Lister Basser Department of Computer Science University of Sydney NSW 2006 AUSTRALIA Internet: ray%basser.cs.su.oz.au at uunet.uu.net CSNET: ray%basser.cs.su.oz at csnet-relay UUCP: {uunet,hplabs,pyramid,mcvax,ukc,nttlab}!munnari!basser.cs.su.oz!ray JANET: munnari!basser.cs.su.oz!ray at ukc Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa08791; 21 Jun 89 17:04:10 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa00924; 21 Jun 89 7:48:08 EDT Received: from TI.COM by CS.CMU.EDU; 21 Jun 89 07:45:45 EDT Received: by ti.com id AA01436; Tue, 20 Jun 89 21:32:43 CDT Message-Id: <8906210232.AA01436 at ti.com> Received: by tilde id AA29832; Tue, 20 Jun 89 21:30:34 CDT Date: Tue, 20 Jun 89 21:30:34 CDT From: lugowski at ngstl1.csc.ti.com To: connectionists at CS.CMU.EDU Subject: objection! As an original member of connectionists at cs.cmu (50+ folks), I have considered unsubscribing because of the recent runaway discussions a la Self-Org or AIList. Thus, I will keep my note short and ask for no replies, although I anticipate other strong opinions on the same subject. Please consider the note that follows to be my personal opinion. -- Marek Lugowski, TI AI Lab/IU CS Dept. I object to an abstract posted to the list recently, exerpted below, as the definition of connectionism given there is grossly misleading. As *one* prototype that does not fit the misleadingly drawn category, I suggest that my work is entirely connectionist; has been perceived by connectionists to be connectionist as early as 1986, yet in *no way* fits the definition cited below. I object knowing that the author presented at the Emergent Computation conference May 22-26 and had plenty opportunity to disabuse himself in Los Alamos of such distortions. The fact that he apprently chose not to do so is what I object to the strongest, only then to the distortion itself. Lugowski, Marek. "Computational Metabolism: Towards Biological Geometries for Computing", pp. 341 - 368, in _Artificial Life_, 2nd printing, Christopher Langton, ed., Addison-Wesley, Reading, MA: 1989. ----------------------- Objectionable citation: (Abstract of paper presented... June 11 1989) Connectionism is a family of statistical techniques for extracting complex higher-order correlations from data. ----------------------- Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa08797; 21 Jun 89 17:04:15 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa01359; 21 Jun 89 7:56:32 EDT Received: from TRACTATUS.BELLCORE.COM by CS.CMU.EDU; 21 Jun 89 07:54:30 EDT Received: by tractatus.bellcore.com (5.61/1.34) id AA13801; Mon, 19 Jun 89 08:20:12 -0400 Date: Mon, 19 Jun 89 08:20:12 -0400 From: Stephen J Hanson Message-Id: <8906191220.AA13801 at tractatus.bellcore.com> To: connectionists at CS.CMU.EDU Subject: test please ignore.. Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa09103; 21 Jun 89 17:37:15 EDT Received: from ri.cmu.edu by Q.CS.CMU.EDU id aa02801; 21 Jun 89 9:48:41 EDT Received: from TRACTATUS.BELLCORE.COM by RI.CMU.EDU; 21 Jun 89 09:46:13 EDT Received: by tractatus.bellcore.com (5.61/1.34) id AA16491; Wed, 21 Jun 89 09:45:36 -0400 Date: Wed, 21 Jun 89 09:45:36 -0400 From: Stephen J Hanson Message-Id: <8906211345.AA16491 at tractatus.bellcore.com> To: Connectionists at RI.CMU.EDU Subject: NIPS ------------------NIPS UPDATE------------------ Deadline for abstracts and summaries was May 30, 1989. We have now received over 460 contributions (almost 50% more than last year!). They are now logged in, and cards acknowledging receipt will be mailed next week to authors. Authors who have not received an acknowledgement by June 30 should write to Kathie Hibbard at the NIPS office; it's possible we got your address wrong in our database, and this will help us catch these things. Refereeing will take July. Collecting the results and defining a final program will be done in August. We plan to mail letters informing authors of the outcome during the first week of September. At that time, we will send all registration material, information about prices, and a complete program. If you haven't heard from us in late September, again please write, to help us straighten things out. ------------------NIPS UPDATE------------------ Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa00127; 22 Jun 89 16:48:56 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa18103; 22 Jun 89 11:59:23 EDT Received: from BOULDER.COLORADO.EDU by CS.CMU.EDU; 22 Jun 89 11:57:15 EDT Return-Path: Received: by boulder.Colorado.EDU (cu-hub.022489) Received: by tigger.colorado.edu (cu.generic.021288) Date: Thu, 22 Jun 89 09:56:59 MDT From: Dennis Sanger Message-Id: <8906221556.AA24065 at tigger> To: connectionists at cs.cmu.edu Subject: TR available: Contribution Analysis University of Colorado at Boulder Technical Report CU-CS-435-89 is now available: Contribution Analysis: A Technique for Assigning Responsibilities to Hidden Units in Connectionist Networks Dennis Sanger AT&T Bell Laboratories and the University of Colorado at Boulder ABSTRACT: Contributions, the products of hidden unit activations and weights, are presented as a valuable tool for investigating the inner workings of neural nets. Using a scaled-down version of NETtalk, a fully automated method for summarizing in a compact form both local and distributed hidden-unit responsibilities is demonstrated. Contributions are shown to be more useful for ascertaining hidden-unit responsibilities than either weights or hidden-unit activations. Among the results yielded by contribution analysis: for the example net, redundant output units are handled by identical patterns of hidden units, and the amount of responsibility a hidden unit takes on is inversely proportional to the number of hidden units. Please send requests to conn_tech_report at boulder.colorado.edu. Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa00855; 22 Jun 89 17:27:17 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa19129; 22 Jun 89 12:36:08 EDT Received: from BOULDER.COLORADO.EDU by CS.CMU.EDU; 22 Jun 89 12:33:22 EDT Return-Path: Received: by boulder.Colorado.EDU (cu-hub.022489) Received: by tigger.colorado.edu (cu.generic.021288) Date: Thu, 22 Jun 89 10:33:10 MDT From: Phillip E. Gardner Message-Id: <8906221633.AA25139 at tigger> To: connectionists at cs.cmu.edu, pdp at boulder.Colorado.EDU Subject: learning spatial data I'm interested in teaching a network to learn its way around a building. I want to use a technique similar to the one outlined by Dr. Widrow at the last NIPPS conference where a network learned how to backup a truck. If you could send me some references that might help, including references to what Dr. Widrow talked about, I would be most thankful. Sincerely, Phil Gardner gardner at boulder.colorado.edu Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa03744; 22 Jun 89 22:23:10 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa25755; 22 Jun 89 22:18:22 EDT Received: from UCSD.EDU by CS.CMU.EDU; 22 Jun 89 22:15:45 EDT Received: from sdbio2.ucsd.edu by ucsd.edu; id AA27631 sendmail 5.60/UCSD-1.0 Thu, 22 Jun 89 19:14:05 PDT for connectionists at cs.cmu.edu Received: by sdbio2.UCSD.EDU (3.2/UCSDGENERIC2) id AA23341 for delivery to terry at helmholtz.sdsc.edu; Thu, 22 Jun 89 19:16:14 PDT Date: Thu, 22 Jun 89 19:16:14 PDT From: terry%sdbio2 at ucsd.edu (Terry Sejnowski) Message-Id: <8906230216.AA23341 at sdbio2.UCSD.EDU> To: connectionists at cs.cmu.edu Subject: Neural Computation, Vol. 1, No. 2 Cc: terry at helmholtz.sdsc.edu NEURAL COMPUTATION -- Issue #2 -- July 1, 1989 Views: Recurrent backpropagation and the dynamical approach to adaptive neural computation. F. J. Pineda New models for motor control. J. S. Altman and J. Kien Seeing chips: Analog VLSI circuits for computer vision. C. Koch A proposal for more powerful learning algorithms. E. B. Baum Letters: A possible neural mechanism for computing shape from shading. A. Pentland Optimization in model matching and perceptual organization. E. Mjolsness, G. Gindi and P. Anandan Distributed parallel processing in the vestibulo-oculomotor system. T. J. Anastasio and D. A. Robinson A neural model for generation of some behaviors in the fictive scratch reflex. R. Shadmehr A robot that walks: Emergent behaviors from a carefully evolved network. R. A. Brooks Learning state space trajectories in recurrent neural networks. B. A. Pearlmutter. A learning algorithm for continually running fully recurrent neural networks. R. J. Williams and D. Zipser. Fast learning in networks of locally-tuned processing units. J. Moody and C. J. Darken. ----- SUBSCRIPTIONS: ______ $35. Student ______ $45. Individual ______ $90. Institution Add $9. for postage outside USA and Canada surface mail or $17. for air mail. MIT Press Journals, 55 Hayward Street, Cambridge, MA 02142. (617) 253-2889. ----- Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa24809; 23 Jun 89 11:39:13 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa13269; 23 Jun 89 11:34:30 EDT Received: from UUNET.UU.NET by CS.CMU.EDU; 23 Jun 89 11:31:42 EDT Received: from unido.UUCP by uunet.uu.net (5.61/1.14) with UUCP id AA22532; Fri, 23 Jun 89 11:31:36 -0400 Received: from gmdzi.UUCP (gmdzi) (1961) by unido.irb.informatik.uni-dortmund.de for uunet id AP04399; Fri, 23 Jun 89 15:19:41 +0100 Received: by gmdzi.UUCP id AA19582; Fri, 23 Jun 89 16:20:34 -0200 Received: by zsv.gmd.de id AA03551; Fri, 23 Jun 89 16:20:55 +0200 Date: Fri, 23 Jun 89 16:20:55 +0200 From: unido!gmdzi!zsv!joerg at uunet.UU.NET (Joerg Kindermann) Message-Id: <8906231420.AA03551 at zsv.gmd.de> To: connectionists at cs.cmu.edu Cc: gmdzi!zsv!joerg at uunet.UU.NET Subject: wanted: guest researcher If you are a postgraduate student of scientist with a strong background in neural networks, we are interested to get in touch with you: We are a small team (5 scientists plus students) doing research in neural networks here at the GMD. Currently we are applying to get funding for several positions of guest researchers. But: we need strong arguments (i.e. good people who are interested in a stay) to actually get the money. Our research interests are both theoretical and application oriented. The main focus is on temporal computation (time series analysis) by neural networks. We are using multi-layer recurrent networks and gradient learning algorithms (backpropagation, reinforcement). Applications are speech recognition, analysis of medical data (ECG, ...), and navigation tasks for autonomous vehicles (2-D simulation only). A second research direction is the optimization of neural networks by means of genetic algorithms. We are using both SUN3s and a parallel Computer (64 cpu transputer-based). So, if you are interested, please write a letter, indicating your background in neural networks and preferred dates for your stay. Dr. Joerg Kindermann Gesellschaft fuer Mathematik und Datenverarbeitung mbH (GMD) Postfach 1240 email: joerg at gmdzi.uucp D-5205 St. Augustin 1, FRG phone: (+49 02241) 142437 Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa16084; 25 Jun 89 12:30:57 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa27315; 25 Jun 89 12:25:53 EDT Received: from THINK.COM by CS.CMU.EDU; 25 Jun 89 12:24:12 EDT Return-Path: Received: from leander.think.com by Think.COM; Sun, 25 Jun 89 12:24:30 EDT Received: by leander.think.com; Sun, 25 Jun 89 12:22:43 EDT Date: Sun, 25 Jun 89 12:22:43 EDT From: singer at Think.COM Message-Id: <8906251622.AA07513 at leander.think.com> To: unido!gmdzi!zsv!joerg at uunet.uu.net Cc: connectionists at cs.cmu.edu, gmdzi!zsv!joerg at uunet.uu.net In-Reply-To: Joerg Kindermann's message of Fri, 23 Jun 89 16:20:55 +0200 <8906231420.AA03551 at zsv.gmd.de> Subject: wanted: guest researcher Date: Fri, 23 Jun 89 16:20:55 +0200 From: unido!gmdzi!zsv!joerg at uunet.uu.net (Joerg Kindermann) If you are a postgraduate student of scientist with a strong background in neural networks, we are interested to get in touch with you: [...] Dr. Joerg Kindermann Gesellschaft fuer Mathematik und Datenverarbeitung mbH (GMD) Postfach 1240 email: joerg at gmdzi.uucp D-5205 St. Augustin 1, FRG phone: (+49 02241) 142437 Though I am not a postgraduate student (i.e. I do not have a PhD), your invitation made me very interested. Especially when I saw your address. Is your organization the same one in which work relating the methods of statistical mechanics to "Darwinian" optimization paradigms has been done? Unfortunately I do not have the particular papers and names at my disposal right now, but the Gesellschaft sounds familiar. My own background includes a Bachelor of Science in Neural Science and a bachelor of Arts in Philosophy. I made an oral presentation at the first NIPS (Neural Information Processing) Conference in Denver, CO in 1987 on hybrid neural net/biological systems. I spent a year beginning my PhD studies at Johns Hopkins with Terry Sejnowski, but had to stop temoprarily because Dr Sejnowski moved to California. I am currently employed by Thinking Machines Corporation workingon their 64K processor Connection Machine doing neural network research, genetic algorithm research, combinatorial optimization work, and statistical work. I also have a working knowledge of German from having worked in Frankfurt for a summer. I would be extremely interested in further discussing this with you, if my qua lifications seem appropriate. Alexander Singer Thinkning Machines Corp. 245 First St. Cambridge, MA 02142 USA Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa01703; 25 Jun 89 17:39:39 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa27905; 25 Jun 89 12:36:58 EDT Received: from [128.188.1.13] by CS.CMU.EDU; 25 Jun 89 12:33:55 EDT Received: by net2.m2c.org (5.57/sendmail.28-May-87) id AA00561; Sun, 25 Jun 89 12:28:13 EDT Received: by m2c.m2c.org (5.57/sendmail.28-May-87) id AA05897; Sun, 25 Jun 89 12:30:16 EDT Received: by wpi (4.12/4.7) id AA09483; Sun, 25 Jun 89 12:32:44 edt Date: Sun, 25 Jun 89 12:32:44 edt From: weili at wpi.wpi.edu (Wei Li) Message-Id: <8906251632.AA09483 at wpi> To: connectionists at CS.CMU.EDU Subject: information on fundings wanted Hi, if any one could send me some notes on fundings from NSF, AFOR, NIH and DARPA, including areas of interested projects, amount of available money, dead line for accepting proposals, and phone numbers of people to contact to, I will appriate it very much. This information was given in IJCNN 89 Washington D.C. neural network conference. My e-mail address is weili at wpi.wpi.edu ---- Wei Li Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa09527; 28 Jun 89 9:22:00 EDT Received: from ri.cmu.edu by Q.CS.CMU.EDU id aa08554; 28 Jun 89 8:08:24 EDT Received: from VMA.CC.CMU.EDU by RI.CMU.EDU; 28 Jun 89 08:04:57 EDT Received: from CMUCCVMA by VMA.CC.CMU.EDU ; Wed, 28 Jun 89 08:03:12 EDT Received: from EB0UB011.BITNET (D4PBPHB2) by CMUCCVMA (Mailer X1.25) with BSMTP id 2266; Wed, 28 Jun 89 08:03:11 EDT Date: Wed, 28 Jun 89 12:58:05 HOE To: connectionists at c.cs.cmu.edu From: D4PBPHB2%EB0UB011.BITNET at VMA.CC.CMU.EDU Comment: CROSSNET mail via MAILER at CMUCCVMA Date: 28 June 1989, 12:57:19 HOE From: D4PBPHB2 at EB0UB011 To: connectionists at c.cs.cmu.edu Add/Subscribe Perfecto Herrera Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa01812; 29 Jun 89 6:05:54 EDT Received: from ri.cmu.edu by Q.CS.CMU.EDU id aa21472; 29 Jun 89 6:00:41 EDT Received: from UUNET.UU.NET by RI.CMU.EDU; 29 Jun 89 05:58:31 EDT Received: from munnari.UUCP by uunet.uu.net (5.61/1.14) with UUCP id AA02442; Thu, 29 Jun 89 05:58:21 -0400 Received: from munnari.oz.au (munnari.oz) by murtoa.cs.mu.oz (5.5) id AA07438; Thu, 29 Jun 89 18:42:41 EST (from guy at flinders.cs.flinders.oz for uunet!connectionists at RI.CMU.EDU) Received: from flinders.cs.flinders.oz (via murtoa) by munnari.oz.au with SunIII (5.61+IDA+MU) id AA08527; Thu, 29 Jun 89 18:33:36 +1000 (from guy at flinders.cs.flinders.oz for connectionists at RI.CMU.EDU@murtoa.cs.mu.OZ.AU) Message-Id: <8906290833.8527 at munnari.oz.au> Received: by flinders.cs.flinders.oz.au(4.0/SMI-3.2) id AA03504; Thu, 29 Jun 89 17:56:22 CST Date: Thu, 29 Jun 89 17:56:22 CST From: munnari!cs.flinders.oz.au!guy at uunet.UU.NET (Guy Smith) To: connectionists at RI.CMU.EDU Subject: Tech Report available Cc: munnari!guy at uunet.UU.NET The Tech Report "Back Propagation with Discrete Weights and Activations" describes a modification of BP which generates a net with discrete (but not integral) weights and activations. The modification is simple: weights and activations are restricted to discrete values. The weights/activations calculated by BP are rounded to one of the neighbouring discrete values. For simple discrete problems, the learning performance of the net was not much affected until the granularity of the legal weight/activation values was as coarse as ten values per integer (ie 0.0, 0.1, 0.2, ...). To request a copy, mail to "guy at cs.flinders.oz..." or write to Guy Smith, Computer Science Department, Flinders University, Adelaide 5042, AUSTRALIA. Guy Smith. Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa03460; 29 Jun 89 9:22:27 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa00460; 29 Jun 89 9:17:33 EDT Received: from DST.BOLTZ.CS.CMU.EDU by CS.CMU.EDU; 29 Jun 89 08:24:18 EDT Received: from DST.BOLTZ.CS.CMU.EDU by DST.BOLTZ.CS.CMU.EDU; 29 Jun 89 08:23:30 EDT To: connectionists at cs.cmu.edu Reply-To: Dave.Touretzky at cs.cmu.edu cc: copetas at cs.cmu.edu Subject: "Rules and Maps" tech report Date: Thu, 29 Jun 89 08:23:22 EDT Message-ID: <3871.615126202 at DST.BOLTZ.CS.CMU.EDU> From: Dave.Touretzky at B.GP.CS.CMU.EDU Rules and Maps in Connectionist Symbol Processing Technical Report CMU-CS-89-158 David S. Touretzky School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213-3890 ABSTRACT: This report contains two papers to be presented at the Eleventh Annual Conference of the Cognitive Science Society. The first describes a simulation of chunking in a connectionist network. The network applies context-sensitive rewrite rules to strings of symbols as they flow through its input buffer. Chunking is implemented as a form of self-supervised learning using backpropagation. Over time, the network improves its efficiency by replacing simple rule sequences with more complex chunks. The second paper describes the first implementation of Lakoff's new theory of cognitive phonology. His approach is based on a multilevel representation of utterances to which all rules apply in parallel. Cognitive phonology is free of the rule ordering constraints that make classical generative theories computationally awkward. The connectionist implementation utilizes a novel ``many maps'' architecture that may explain certain constraints on phonological rules not adequately accounted for by more abstract models. ================ To order copies of this tech report, please send email to Catherine Copetas (copetas at cs.cmu.edu), or write the School of Computer Science at the address above. If you previously requested a copy of CMU-CS-89-147 (Connectionism and Compositional Semantics), you will receive a copy of this new report automatically. In fact, it should arrive in your mailbox today. Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa10761; 29 Jun 89 15:12:55 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa02123; 29 Jun 89 10:41:34 EDT Received: from VMA.CC.CMU.EDU by CS.CMU.EDU; 29 Jun 89 10:40:08 EDT Received: from CMUCCVMA by VMA.CC.CMU.EDU ; Thu, 29 Jun 89 10:39:37 EDT Received: from UKACRL.BITNET by CMUCCVMA (Mailer X1.25) with BSMTP id 0225; Thu, 29 Jun 89 10:39:36 EDT Received: from RL.IB by UKACRL.BITNET (Mailer X1.25) with BSMTP id 7397; Thu, 29 Jun 89 15:37:20 BST Received: Via: UK.AC.EX.CS; 29 JUN 89 15:37:17 BST Received: from entropy by expya.cs.exeter.ac.uk; Thu, 29 Jun 89 15:29:29-0000 From: Noel Sharkey Date: Thu, 29 Jun 89 15:28:14 BST Message-Id: <6270.8906291428 at entropy.cs.exeter.ac.uk> To: connectionists at CS.CMU.EDU Subject: post graduate studentships RESEARCH STUDENTSHIPS IN COMPUTER SCIENCE University of Exeter The Department of Computer Science invites applications for Science and Engineering Research Council (SERC) PhD. quotas for October, 1989. Any area of computer science within the Department's research interests (new generation architectures and languages, methodology, interfaces, logics) will be considered. However, for a least one of the quotas, preference will be given to candidates with an interest in CONNECTIONIST or NEURAL NETWORK research, particularly in relation to applications within the domain of natural language processing or simulations of human memory. Candidates should possess a first degree of at least 2(i) standard in order to be eligible for the award of SERC research studentship. Closing date for applications is 14th July, 1989. For further information about the Department's research interests, eligibility consideration and application procedure, please contact: .nf Dr Noel Sharkey, Reader in Computer Science JANET: noel at uk.ac.exeter.cs Dept. Computer Science University of Exeter Exeter EX4 4PT U.K. (Telephone: 0392 264061) .fi Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa15407; 29 Jun 89 22:07:18 EDT Received: from ri.cmu.edu by Q.CS.CMU.EDU id aa10205; 29 Jun 89 17:36:17 EDT Received: from CS.UTEXAS.EDU by RI.CMU.EDU; 29 Jun 89 17:34:27 EDT Date: Thu, 29 Jun 89 16:25:24 CDT From: yu at cs.utexas.edu (Yeong-Ho Yu) Posted-Date: Thu, 29 Jun 89 16:25:24 CDT Message-Id: <8906292125.AA27045 at cs.utexas.edu> Received: by cs.utexas.edu (5.59/36.2) id AA27045; Thu, 29 Jun 89 16:25:24 CDT To: guy!s.flinders.oz.au!guy at uunet.UU.NET Cc: connectionists at RI.CMU.EDU, munnari!guy at uunet.UU.NET In-Reply-To: Guy Smith's message of Thu, 29 Jun 89 17:56:22 CST <8906290833.8527 at munnari.oz.au> Subject: Tech Report available I'd like to get a copy of your tech report "Back Propagation with Discrete Weights and Activations". My address is Yeong-Ho Yu AI Lab The University of Texas at Austin Austin, TX 78712 (yu at cs.utexas.edu) Thanks in advance. Yeong ---------- Received: from q.cs.cmu.edu by B.GP.CS.CMU.EDU id aa23828; 30 Jun 89 12:43:18 EDT Received: from cs.cmu.edu by Q.CS.CMU.EDU id aa15581; 30 Jun 89 9:30:39 EDT Received: from VMA.CC.CMU.EDU by CS.CMU.EDU; 30 Jun 89 09:28:32 EDT Received: from CMUCCVMA by VMA.CC.CMU.EDU ; Fri, 30 Jun 89 09:28:04 EDT Received: from WEIZMANN.WEIZMANN.AC.IL by CMUCCVMA (Mailer X1.25) with BSMTP id 7131; Fri, 30 Jun 89 09:28:03 EDT Received: by WEIZMANN (Mailer R2.03B) id 3415; Fri, 30 Jun 89 10:00:56 +0300 Date: Fri, 30 Jun 89 09:52:40 +0300 From: "Harel Shouval, Tal Grossman" Subject: Preprint available To: connectionists at cs.cmu.edu The following preprint describes a theoretical and experimental work on optical neural network that is based on a negative weights nn model. Please send your requests by email to: feshouva at weizmann (bitnet), or write to: Harel Shouval, Electronics Dept., Weizmann Inst. Rehovot 76100, ISRAEL. --------------------- An All-Optical Hopfield Network: Theory and Experiment ------------------------------------------------------- Harel Shouval, Itzhak Shariv, Tal Grossman, Asher A. Friesem and Eytan Domany. Dept. of Electronics, Weizmann Institute of Science, Rehovot 76100 Israel. --- ABSTRACT --- Realization of an all-optical Hopfield-type neural network is made possible by eliminating the need for subtracting light intensities. This can be done without significntly degrading the network's preformance, if only inhibitory connections (i.e. $J_{ij}<0$) are used. We present theoretical analysis of such a network, and its experimental implementation, that uses a liquid crystal light valve for the neurons and an array of sub-holograms for the interconnections. -----------------------------------------------------------------------