From janet at psy.uq.oz.au Thu Feb 1 02:46:37 1996 From: janet at psy.uq.oz.au (Janet Wiles) Date: Thu, 1 Feb 1996 17:46:37 +1000 (EST) Subject: Cognitive Modelling Workshop (Virtual and Physical) Message-ID: CALL FOR PAPERS We are pleased to announce a COGNITIVE MODELLING WORKSHOP in conjunction with the Australian Conference on Neural Networks (ACNN'96) and the electronic cognitive science journal Noetica This announcement can be accessed on the WWW at URL: http://psy.uq.edu.au/CogPsych/acnn96/cfp.html -------------------------------------------------------------------- Noetica/ACNN'96 Cognitive Modelling Workshop: Call For Papers VIRTUAL WORKSHOP February 1 to March 26, 1996 PHYSICAL WORKSHOP Canberra Australia April 9, 1996 Memory, time, change and structure in ANNs: Distilling cognitive models into their functional components Organisers: Janet Wiles and J. Devin McAuley Departments of Computer Science and Psychology University of Queensland QLD 4072 Australia janet at psy.uq.edu.au devin at psy.uq.edu.au Call For Papers Web Page http://psy.uq.edu.au/CogPsych/acnn96/cfp.html Workshop Web Page http://psy.uq.edu.au/CogPsych/acnn96/workshop.html Aim The Workshop aim is to identify the functional roles of artificial neural network (ANN) components and to understand how they combine to explain cognitive phenomena. Existing ANN models will be distilled into their functional components through case study analysis, targeting three traditional strengths of ANNs - mechanisms for memory, time and change; and one area of weakness - mechanisms for structure (see below for details of these four areas). Workshop Format The Workshop will be held in two parts: a virtual workshop via the World Wide Web followed by the physical workshop at ACNN'96. February - March 1996 (Part 1 - Virtual Workshop): All members of the cognitive modelling and ANN research communities are invited to submit case studies of neural-network-based cognitive models (their own or established models from the literature) and commentaries on the workshop issues and case studies. Submissions judged appropriate for the workshop will be posted to the Workshop Web Page as they arrive, and will be collated into a Special Issue of the electronic journal Noetica. Multiple case studies of an ANN model may be accepted if they address different cognitive phenomena. It is OK to participate in the virtual workshop without attending the physical workshop. April 9, 1996 (Part 2 - Physical Workshop): A physical workshop will be held as part of the 1996 Australian Conference on Neural Networks (ACNN'96) in Canberra Australia. At the workshop, the collection of case studies and commentaries will be available in hard copy form. The physical workshop will be 90 minutes long, beginning with an introduction (review of the issues); then presentation of submitted and invited Case Studies; and closing with a discussion of what's missing from the list of available mechanisms. A summary of the issues raised in the discussion will be compiled afterwards, and made available via the workshop web page. Further details on presentations will be announced closer to the date of the workshop. Rationale For many ANN models of cognitive phenomena, interesting behaviour appears to arise from the model as a total package, and it is often a challenge to understand how aspects of the behaviour are supported by components of the ANN. The goal of this workshop is to further such understanding: Specifically, to identify the functional roles of ANN components, the link from the component to the overall behaviour and how they combine to explain cognitive phenomena. The four target areas (memory, time, change and structure) are not disjoint, but rather, provide overlapping viewpoints from which to examine models. In essence, we believe these target areas are where to look for the ``sources of power'' in a model. We use the term "distillation" to refer to the process of identifying the functional components of a model with respect to the four target areas. The task of distillation requires exploring the details of a model in order to clarify its source of power, stripping away other aspects. It focuses on the computational properties of the model formalism, providing a method for: (1) understanding the computational components of a newly presented model, and how they give rise to its behavior, (2) discerning novel computational components that may prove useful in model development, and (3) comparing ANN models that target similar cognitive tasks. The workshop is specifically intended for cognitive modellers who use ANNs, but we anticipate that it will be of interest to the wider ANN community. The case study format grounds the analysis of the functional components of ANNs in the cognitive modelling literature, and focuses on phenomena that do admit a computational explanation. The workshop is intended as much as a learning experience as a communicative one. ---------------------------------------------------------------------------- Submission Details Each Case Study should be based on a published ANN simulation in an area of cognitive science. It should address all four target areas of memory, time, change and structure, with the order of sections and choice of sub-headings up to the individual. Some models may have little to say about one or more of these areas - make this explicit. Include simulation details relevant to understanding the functional components of the model but complete replication details are not necessary. Issues beyond the scope of the functional components and the behaviour they support should not be included in the case study itself, but may be appropriate as commentary. Maximum length is 2500 words including references. Where possible, papers should be submitted in html format (but ascii and postscript will also be accepted). The URL for each submission or the source document can be emailed to the organisers at janet at psy.uq.edu.au or devin at psy.uq.edu.au between Feb 1 and March 26, 1996. Case Study Format See Case Study #1 , "Elman's SRN and the discovery of lexical classes" as a guide: http://psych.psy.uq.oz.au/CogPsych/acnn96/case1.html Target paper: Give the full reference to the original paper and relevant simulation. Introduction: In this section introduce the task addressed by the model and the ANN used. Distinguish between the cognitive task of the model and its instantiation in the input/output task of the network. Is there a gap between the the cognitive task and the input/output task of the ANN? For some studies this mismatch may be intentional, as the cognitive task can be viewed as a by-product of another process (e.g., discovery of lexical classes via the prediction task in Elman's SRN). In others, the mismatch between cognitive task and input/output task may be less benign, obscuring the contribution of the model towards understanding the phenomenon. For the ANN task, consider the following questions: What are the inputs and outputs of the model? How is the task information encoded in the input representation? How is the model's response encoded by the output representation? How does the network address the cognitive task? Memory: Identify the information to be stored in memory, then describe the mechanisms. There have been a range of mechanisms proposed for storing and retrieving memories in neural networks: such as implicit long-term coding of memories distributed in the weights; memory as an attractor; short-term memory as transient decay of activations; limit-cycle encoding of memories with synchrony as a method of retrieval. Consider the what and how of memory storage and retrieval: What information needs memory? How is it stored and retrieved? Time: Describe how time is treated in the data, processing and parameters of the network. For example, does the network consider time as an absolute measure in which events in the input are time-stamped with reference to an external clock, as a sequence in which only the order of events is specified, or as a relative measure in which durations are ratios of one another? Methods for processing temporal information with neural networks have included: using a fixed or sliding time window which maps time into space; learning of time delays in the network weights; sequential processing of time slices; and encoding time as the phase angle of an oscillator. What measures of time are used by the network? How are they represented and what are the underlying mechanisms? Change: In this section, consider the types of changes that occur in the neural network, parameters, data, etc, over a range of timescales: * evolutionary change such as a genetic algorithm operating over network parameters; * generational change such as networks training the next generation of networks; * development and aging such as adding or removing units; * learning such as changing weights based on training data; * transient behavior such as activation equation dynamics What types of changes occur in the selected model and how are they implemented in the network's mechanisms? (Note that few of these aspects are expected to apply to any one model, with many case studies focusing on change as learning.) Structure: In this section, consider how structured information in the environment is represented as structured information in the network (e.g., an implicit grammar in training data can be encoded in the hidden-unit space of a recurrent net; and higher-order bindings can be stored using tensors or phase synchrony). What structure is coded directly into the architecture of the network? Is the network partitioned into modules to directly encode structure? What generalization is the network capable of? Is the generalization due to direct coding of structure or does it learn it from the training data? There has been an ongoing debate in the ANN literature on the generalization abilities of networks. Are there important aspects of structure that cannot be represented, learned or generalized by the network? Where possible, identify structure in the environment that may be expected to be reflected in the ANN model but is not. Discussion and Conclusions: In the final section, discuss how the functional components reviewed in the previous sections combine to explain the cognitive phenomena targeted by the case study. Commentary Format The commentary section of the workshop is provided as an outlet for interpretation, elaboration, and substantive criticism of case studies. It is included as part of the workshop format to complement the case studies, which are intended to be compact and focussed on the workshop aim of distilling ANN models into their functional components. Each commentary should discuss one or more case studies and have a maximum length of 1000 words including references. ---------------------------------------------------------------------------- From pjs at aig.jpl.nasa.gov Thu Feb 1 12:29:16 1996 From: pjs at aig.jpl.nasa.gov (Padhraic J. Smyth) Date: Thu, 1 Feb 1996 09:29:16 -0800 (PST) Subject: CFP for special issue of Machine Learning Journal Message-ID: <199602011729.JAA08415@amorgos.jpl.nasa.gov> CALL FOR PAPERS SPECIAL ISSUE OF THE MACHINE LEARNING JOURNAL ON LEARNING WITH PROBABILISTIC REPRESENTATIONS Guest editors: Pat Langley (ISLE/Stanford University) Gregory Provan (Rockwell Science Center/ISLE) Padhraic Smyth (JPL/University of California, Irvine) In recent years, probabilistic formalisms for representing knowledge and inference techniques for using such knowledge have come to play an important role in artificial intelligence. The further development of algorithms for inducing such probabilistic knowledge from experience has resulted in novel approaches to machine learning. To increase awareness of such probabilistic methods, including their relation to each other and to other induction techniques, Machine Learning will publish a special issue on this topic. We encourage submission of papers that address all aspects of learning with probabilistic representations, including but not limited to: Bayesian networks, probabilistic concept hierarchies, naive Bayesian classifiers, mixture models, (hidden) Markov models, and stochastic context-free grammars. We consider any work on learning over representations with explicit probabilistic semantics to fall within the scope of this issue. Submissions should describe clearly the learning task, the representation of data and learned knowledge, the performance element that uses this knowledge, and the induction algorithm itself. Moreover, we encourage authors to decompose their characterization of learning into the processes of (i) selecting a model (or family of models): what are the properties of the model representation ? (ii) selecting a method for evaluating the quality of a fitted model: given a particular parametrization of the model what is the performance criterion by which one can judge its quality ? and (iii) the algorithmic specification of how to search over parameter and model space. An ideal paper will specify these three items clearly and relatively independently. Papers should also evaluate the proposed methods using techniques acknowledged in the machine learning literature, including but not limited to: experimental studies of algorithm behavior on natural and synthetic data (but not the latter alone), theoretical analyses of algorithm behavior, ability to model psychological phenomena, and evidence of successful application in real-world contexts. We especially encourage comparisons that clarify relations among different probabilistic methods or to nonprobabilistic techniques. Papers should meet the standard submission requirements given in the Machine Learning instructions to authors, including having length between 8,000 and 12,000 words. Hardcopies of each submission should be mailed to: Karen Cullen (5 copies) Pat Langley (1 copy) Kluwer Academic Publishers Institute for the Study 101 Philip Drive of Learning and Expertise Assinippi Park 2164 Staunton Court Norwell, MA 02061 Palo Alto, CA 94306 by the submission deadline, July 1, 1996. The review process will take into account the usual criteria, including clarity of presentation, originality of the contribution, and quality of evaluation. We encourage potential authors to contact Pat Langley (langley at cs.stanford.edu), Gregory Provan (provan at jupiter.risc.rockwell.com), or Padhraic Smyth (pjs at aig.jpl.nasa.gov) prior to submission if they have questions. From john at dcs.rhbnc.ac.uk Fri Feb 2 11:45:57 1996 From: john at dcs.rhbnc.ac.uk (John Shawe-Taylor) Date: Fri, 02 Feb 96 16:45:57 +0000 Subject: Technical Report Series in Neural and Computational Learning Message-ID: <199602021645.QAA32350@platon.cs.rhbnc.ac.uk> The European Community ESPRIT Working Group in Neural and Computational Learning Theory (NeuroCOLT) has produced a set of new Technical Reports available from the remote ftp site described below. They cover topics in real valued complexity theory, computational learning theory, and analysis of the computational power of continuous neural networks. Abstracts are included for the titles. *** Please note that the location of the files was changed at the beginning of ** the year, so that any copies you have of the previous instructions should be * discarded. The new location and instructions are given at the end of the list. ---------------------------------------- NeuroCOLT Technical Report NC-TR-96-030: ---------------------------------------- Exponentially many local minima for single neurons by Peter Auer, University of California, Santa Cruz, USA, Mark Herbster, University of California, Santa Cruz, USA, Manfred K. Warmuth, University of California, Santa Cruz, USA Abstract: We show that for a single neuron with the logistic function as the transfer function the number of local minima of the error function based on the square loss can grow exponentially in the dimension. ---------------------------------------- NeuroCOLT Technical Report NC-TR-96-031: ---------------------------------------- An Efficient Implementation of Sigmoidal Neural Nets in Temporal Coding with Noisy Spiking Neurons by Wolfgang Maass, Institute for Theoretical Computer Science, Technische Universitaet Graz, Austria Abstract: We show that networks of rather realistic models for biological neurons can in principle simulate arbitrary feedforward sigmoidal neural nets in a way which has previously not been considered. This new approach is based on temporal coding by single spikes (respectively by the timing of synchronous firing in pools of neurons), rather than on the traditional interpretation of analog variables in terms of firing rates. The resulting new simulation is substantially faster and apparently more consistent with experimental results about fast information processing in cortical neural systems. As a consequence we can show that networks of noisy spiking neurons are "universal approximators" in the sense that they can approximate with regard to temporal coding {\it any} given continuous function of several variables. Our new proposal for the possible organization of computations in biological neural systems has some interesting consequences for the type of learning rules that would be needed to explain the self-organization of such neural circuits. Finally, our fast and noise-robust implementation of sigmoidal neural nets via temporal coding points to possible new ways of implementing sigmoidal neural nets with pulse stream VLSI. ---------------------------------------- NeuroCOLT Technical Report NC-TR-96-032: ---------------------------------------- A Framework for Stuctural Risk Minimisation by John Shawe-Taylor, Royal Holloway, University of London, UK Peter Bartlett, RSISE, Australian National University, Australia Robert Williamson, Australian National University, Australia Martin Anthony, London School of Economics, UK Abstract: The paper introduces a framework for studying structural risk minimisation. The model views structural risk minimisation in a PAC context. It then generalises to the case when the hierarchy of classes is chosen in response to the data, hence explaining the impressive performance of the maximal margin hyperplane algorithm of Vapnik. ---------------------------------------- NeuroCOLT Technical Report NC-TR-96-033: ---------------------------------------- Learning to Compress Ergodic Sources by Jonathan Baxter, Royal Holloway, University of London and London School of Economics, UK John Shawe-Taylor, Royal Holloway, University of London, UK Abstract: We present an adaptive coding technique which is shown to achieve the optimal coding in the limit as the size of the text grows, while the data structures associated with the code only grow linearly with the text. The approach relies on Huffman codes which are generated relative to the context in which a particular character occurs. The Huffman codes themselves are inferred from the data that has already been seen. A key part of the paper involves showing that the loss per character incurred by the learning process tends to zero as the size of the text tends to infinity. ---------------------------------------- NeuroCOLT Technical Report NC-TR-96-034: ---------------------------------------- Theory and Applications of Agnostic PAC-Learning with Small Decision Trees by Peter Auer, University of California, Santa Cruz, USA, Mark Herbster, University of California, Santa Cruz, USA, Robert C. Holte, University of Ottawa, Canada, Wolfgang Maass, Technische Universitaet Graz, Austria Abstract: We exhibit a theoretically founded algorithm $\Tii$ for agnostic PAC-learning of decision trees of at most 2 levels, whose computation time is almost linear in the size of the training set. We evaluate the performance of this learning algorithm $\Tii$ on 15 common ``real-world'' datasets, and show that for most of these datasets $\Tii$ provides simple decision trees with little or no loss in predictive power (compared with C4.5). In fact, for datasets with continuous attributes its error rate tends to be lower than that of C4.5. To the best of our knowledge this is the first time that a PAC-learning algorithm is shown to be applicable to ``real-world'' classification problems. Since one can {\em prove} that $\Tii$ is an agnostic PAC-learning algorithm, $\Tii$ is {\em guaranteed} to produce close to optimal 2-level decision trees from sufficiently large training sets for {\em any} (!) distribution of data. In this regard $\Tii$ differs strongly from all other learning algorithms that are considered in applied machine learning, for which no {\em guarantee} can be given about their performance on {\em new } datasets. We also demonstrate that this algorithm $\Tii$ can be used as a diagnostic tool for the investigation of the expressive limits of 2-level decision trees. Finally, T2, in combination with new bounds on the VC-dimension of decision trees of bounded depth that we derive, provides us now for the first time with the tools necessary for comparing learning curves of decision trees for ``real-world'' datasets with the theoretical estimates of PAC-learning theory. ---------------------------------------- NeuroCOLT Technical Report NC-TR-96-035: ---------------------------------------- A recurrent network that performs a context-sensitive prediction task by Mark Steijvers, Indiana University, Peter Gr\"{u}nwald, CWI, Dept. of Algorithmics, The Netherlands Abstract: We address the problem of processing a context-sensitive language with a recurrent neural network (RN). So far, the language processing capabilities of RNs have only been investigated for regular and context-free languages. We present an extremely simple RN with only one parameter for its two hidden nodes that can perform a prediction task on sequences of symbols from the language $\{ (ba^{k})^n \mid k \geq 0, n > 0 \}$, a language that is context-sensitive but not context-free. The input to the RN consists of any string of the language, one symbol at a time. The network should then, at all times, predict the symbol that should follow. This means that the network must be able to count the number of $a$'s in the first subsequence and to retain this number for future use. Our network can solve the task for $k=1$ up to $k=120$. The network represents the count of $a$'s in the subsequence by having different limit cycles for every different number of $a$'s counted. The limit cycles are related in such a way that the representation of network states in which an $a$ should be predicted are linearly separable from those in which a $b$ should be predicted. Our work shows that connectionism in general can handle more complex formal languages than was previously known. ---------------------------------------- NeuroCOLT Technical Report NC-TR-96-036: ---------------------------------------- Tight worst-case loss bounds for predicting with expert advice by David Haussler, University of California, Santa Cruz, USA, Jyrki Kivinen, University of Helsinki, Finland, Manfred K. Warmuth, University of California, Santa Cruz, USA Abstract: We consider on-line algorithms for predicting binary or continuous-valued outcomes, when the algorithm has available the predictions made by $N$ experts. For a sequence of trials, we compute total losses for both the algorithm and the experts under a loss function. At the end of the trial sequence, we compare the total loss of the algorithm to the total loss of the best expert, \ie, the expert with the least loss on the particular trial sequence. For a large class of loss functions, with binary outcomes the total loss of the algorithm proposed by Vovk exceeds the total loss of the best expert at most by the amount $c\ln N$, where $c$ is a constant determined by the loss function. This upper bound does not depend on any assumptions on how the experts' predictions or the outcomes are generated, and the trial sequence can be arbitrarily long. We give a straightforward method for finding the correct value $c$ and show by a lower bound that for this value of $c$, the upper bound is asymptotically tight. The lower bound is based on a probabilistic adversary argument. The class of loss functions for which the $c\ln N$ upper bound holds includes the square loss, the logarithmic loss, and the Hellinger loss. We also consider another class of loss functions, including the absolute loss, for which we have an $\Omega\left(\sqrt{\ell\log N}\right)$ lower bound, where $\ell$ is the number of trials. We show that for the square and logarithmic loss functions, Vovk's algorithm achieves the same worst-case upper bounds with continuous-valued outcomes as with binary outcomes. For the absolute loss, we show how bounds earlier achieved for binary outcomes can be achieved with continuous-valued outcomes using a slightly more complicated algorithm. ---------------------------------------- NeuroCOLT Technical Report NC-TR-96-037: ---------------------------------------- Exponentiated Gradient Versus Gradient Descent for Linear Predictors by Jyrki Kivinen, University of Helsinki, Finland, Manfred K. Warmuth, University of California, Santa Cruz, USA Abstract: We consider two algorithm for on-line prediction based on a linear model. The algorithms are the well-known gradient descent ($\GD$) algorithm and a new algorithm, which we call $\EGpm$. They both maintain a weight vector using simple updates. For the $\GD$ algorithm, the update is based on subtracting the gradient of the squared error made on a prediction. The $\EGpm$ algorithm uses the components of the gradient in the exponents of factors that are used in updating the weight vector multiplicatively. We present worst-case loss bounds for $\EGpm$ and compare them to previously known bounds for the $\GD$ algorithm. The bounds suggest that the losses of the algorithms are in general incomparable, but $\EGpm$ has a much smaller loss if only few components of the input are relevant for the predictions. We have performed experiments, which show that our worst-case upper bounds are quite tight already on simple artificial data. -------------------------------------------------------------------- ***************** ACCESS INSTRUCTIONS ****************** The Report NC-TR-96-001 can be accessed and printed as follows % ftp ftp.dcs.rhbnc.ac.uk (134.219.96.1) Name: anonymous password: your full email address ftp> cd pub/neurocolt/tech_reports ftp> binary ftp> get nc-tr-96-001.ps.Z ftp> bye % zcat nc-tr-96-001.ps.Z | lpr -l Similarly for the other technical reports. Uncompressed versions of the postscript files have also been left for anyone not having an uncompress facility. In some cases there are two files available, for example, nc-tr-96-002-title.ps.Z nc-tr-96-002-body.ps.Z The first contains the title page while the second contains the body of the report. The single command, ftp> mget nc-tr-96-002* will prompt you for the files you require. A full list of the currently available Technical Reports in the Series is held in a file `abstracts' in the same directory. The files may also be accessed via WWW starting from the NeuroCOLT homepage (note that this is undergoing some corrections and may be temporarily inaccessible): http://www.dcs.rhbnc.ac.uk/neural/neurocolt.html Best wishes John Shawe-Taylor From tgc at kcl.ac.uk Fri Feb 2 05:05:31 1996 From: tgc at kcl.ac.uk (Trevor Clarkson) Date: Fri, 02 Feb 1996 10:05:31 +0000 Subject: NEuroFuzzy workshop in Prague Message-ID: I N F O R M A T I O N B U L L E T I N IEEE European Workshop on Computational Intelligence NEuroFuzzy'96 and Neuronet'96 April 16 - 18, 1996 Prague, Czech Republic NEuroFuzzy'96 ORGANIZED BY IEEE (in cooperation with) Institute of Computer Science of the Academy of Sciences of the Czech Republic Action M Agency Faculty of Transportation, Czech Technical University Prague SPONSORED BY IEEE UK&RI Neural Networks Regional Interest Group IEEE UK&RI Communications Chapter Czechoslovakia Section IEEE Czech Society for Energetics Prague AIMS The purpose of the workshop is to encourage high-quality research in all branches of Computational Intelligence, Artificial Neural Networks and Fuzzy Neural Systems and to provide an opportunity to bring together specialists active in the fields. INTERNATIONAL PROGRAM COMMITTEE M. NOVAK (Czech Republic) - chairman V. BEIU (Romania) A.B. BULSARI (Finland) T. CLARKSON (United Kingdom) G. DREYFUS (France) A. FROLOV (Russia) V. GUSTIN (Slovenia) C. JEDRZEJEK (Poland) K. HORNIK (Austria) S.V. KARTALOPOULOS (USA) E.J.H. KERCKHOFFS (The Netherlands) S. NORDBOTTEN (Norway) G. PIERONNI (Italy) T. ROSKA (Hungary) F. SANDOVAL (Spain) J.G. TAYLOR (United Kingdom) H.G. ZIMMERMANN (Germany) ORGANIZING COMMITTEE Hana Bilkova (ICS), Mirko Novak (ICS), Stanislav Rizek (ICS), Lucie Vachova (Action M Agency), Milena Zeithamlova (Action M Agency) LOCAL ARRANGEMENTS Action M Agency Milena Zeithamlova Vrsovicka 68 101 00 Prague 10 Phone: (422) 6731 2333-4 Fax: (422) 6731 0503 E-mail: actionm at cuni.cz G E N E R A L I N F O R M A T I O N LOCATION NEuroFuzzy'96 - the Workshop on Computational Intelligence will take place at the Bethlehem Palace, Betlemske nam. (Bethlehem Square, next to the Bethlehem Chapel, Old Town, Prague 1). The workshop site is situated just right in the heart of the historical part of Prague. It is located nearby Narodni Street and Metro station (line B - Narodni) and it is reachable by tram No 6, 9, 18, 22. REGISTRATION Those wishing to participate in NEuroFuzzy'96 are requested to pay the registration fees and the accommodation deposit, and to complete (in full) the enclosed Registration Form and fax or mail it to the NEurofuzzy'96 Local Arrangements Agency (Action M Agency) as soon as possible, but not later than March 1, 1996. REGISTRATION DESK The Registration Desk will be open at the Entrance Hall at the Bethlehem Palace Monday, April 15, 1996 1:00 p.m. - 8:00 p.m. Tuesday, April 16, 1996 8:00 a.m. - 1:00 p.m. Wednesday, April 17, 1996 8:30 a.m. - 1:00 p.m. Thursday, April 18, 1996 8:30 a.m. - 1:00 p.m. WORKSHOP FEES Early / Late Non-members IEEE members Full Registration Fee DM 370 / DM 440 DM 330 / DM 400 Students Fee DM 270 / DM 320 DM 220 / DM 260 East European Students Fee DM 150 / DM 180 DM 130 / DM 150 Accompanying Person Fee DM 90 Lunches DM 75 Walking Tour of Prague DM 15 Organ Concert DM 20 A Night with Mozart DM 25 Early means payment made until March 15, 1996. Late means payment made after March 15, 1996. Full Registration Fee, Students Fee and East European Students Fee include workshop materials, proceedings, attendance of all scientific and poster sessions as well as participation at Welcome Party and refreshment during coffee breaks. Accompanying person fee covers attendance at Welcome Party, participation in the Opening and Closing Ceremonies and the assistance in the individual art and music requests. Lunches include meals served during lunch breaks on Tuesday, Wednesday and Thursday, April 16-18, 1996. PAYMENTS All payments of registration fees, accommodation deposit and accommodation balance can be made either by a credit card (MasterCard, EuroCard, VISA, JCB, Diners Club) or by bank transfer to the Czech Republic, Prague, Komercni banka Prague 10, M. Zeithamlova, SWIFT code: KOMB CZ PP, Bank code: 0100, Account number: 221442-101/0100, NEUROFUZZY Registration Fee & Accommodation fees. The equivalent of payment in USD is acceptable. The Agency will send a receipt in acknowledgement once the payment has been registered in the Agency's account, at the latest upon your arrival in Prague at the Registration Desk. ACCOMMODATION We are happy to assist you in arranging the accommodation in Prague. Should you be interested in Action M Agency making the hotel reservations for you, select the hotel or hostel of your preference and indicate your choice on the Registration Form. 1st Category ***** 1. Renaissance Prague Hotel V Celnici 1, Praha 1 - Nove Mesto phone (422) 2481 0396, fax (422) 2481 1687 The new modern Renaissance Prague Hotel is located in the city centre, close to the well-known Prasna brana (the Prague Tower). The Namesti Republiky Metro station (line B) is only a few steps around the corner. In order to get to the Bethlehem Palace, you could walk 20 minutes on the Road of the Czech Kings, which begins at the Prague Tower, or you can take Metro for two stations. 2. Forum Hotel Kongresova 1, Praha 4 - Pankrac phone (422) 6119 1111, fax (422) 42 06 84 The Forum Hotel was built in 1988 and offers the high standard of services typical for the Inter-Continental Hotel Group. The Hotel overlooks the Nuselske udoli (The Nusle Valley) with the old Vysehrad castle. Being located on a hill, the hotel offers a spectacular view of the Prague Castle and the bridges over Vltava river. The Vysehradska Metro station (line C) is near the hotel entrance. The Metro (with one transfer) would take you to the workshop site in about 15 minutes. 2nd Category *** 3. Betlem Club Praha Betlemske nam. 9, Praha 1 - Stare Mesto phone (422) 2421 6872, fax (422) 26 38 85 The small stylish hotel is located in the historical part of Prague in the building partially from 13 century in Romanesque-Gothic style. It is situated just opposite the Bethlehem Palace. 4. SAX Hotel Jansky vrsek 328/3, Praha 1 - Mala Strana phone (422) 53 84 22, fax (422) 53 84 98 The hotel offers a unique atmosphere of its inner atrium and a beautiful view of Mala Strana and Prague Castle. It is located 20 minutes from Bethlehem Palace, when you wish to walk across the famous Charles Bridge, or 10 minutes by tram No.22 (Karmelitska stop). 5. U zlateho stromu Hotel Karlova 6, Praha 1 - Stare Mesto phone/fax (422) 2422 1385 The "Golden Tree" hotel is placed right in the historical centre of Prague, nearby Charles Bridge and 10 minutes walk to the workshop site. 3rd Category 6. U Sladku Pension Belohorska 130, Praha 6 - Brevnov phone/fax (422) 2051 13457 U Sladku Pension is situated in a quite residential quarter within walking distance of Prague Castle. Rooms are equipped with a shower, phone and satellite TV. By tram No.22 (U Kastanu stop) you can get to the workshop site in 25 minutes. 7. Petrska Hostel Petrska 3, Praha 1 - Nove Mesto phone (422) 23 16 430 The hostel is situated in the centre of Prague, nearby Renaissance Prague Hotel. Two rooms share one bathroom. 8. Mazanka Hostel Davidkova 84, Praha 8 - Liben phone (422) 688 59 58, fax (422) 688 42 42 The hostel belongs to the Academy of Sciences of the Czech Republic and is located near the Institute of Computer Science. It takes 40 minutes to get to the city centre by tram No.17 (Davidkova stop). Again two or three rooms share one bathroom. You will receive the hotel (hostel) voucher from Action M Agency in advance by fax or post after having paid your accommodation deposit. You can accommodate yourself with the voucher of your chosen hotel at the day of your arrival from 2.p.m. ACCOMMODATION DESK All questions regarding your accommodation in Prague during the workshop will be answered at the Registration Desk in the Bethlehem Palace. ACCOMMODATION FEES Prices per person/night in: single room double room 1. Renaissance Prague Hotel DM 285 DM 160 2. Forum Hotel DM 245 DM 140 3. Betlem Club Praha DM 165 DM 93 4. SAX Hotel DM 155 DM 90 5. U Zlateho stromu Hotel DM 115 DM 80 6. U Sladku Pension DM 65 DM 50 7. Petrska Hostel DM 50 DM 30 8. Mazanka Hostel DM 45 DM 28 Single room means also double room for single use. Double room supposes two persons. All prices include breakfast. Please, indicate the exact dates of the selected accommodation in the Registration Form. Please, note that the number of reserved rooms is limited and four nights (April 15-19) have been preliminary reserved for you. ACCOMMODATION DEPOSIT The accommodation deposit of DM 300 is required from participants wishing to stay in hotels of 1st and 2nd category (Renaissance Prague Hotel, Forum Hotel, Betlem Club Praha, SAX, Hotel, U Zlateho stromu Hotel). For the 3rd category (U Sladku Pension, Petrska Hostel, Mazanka Hostel) the accommodation deposit of DM 150 is required. Accommodation deposit must be paid together with the registration fees. ACCOMMODATION BALANCE Accommodation balance is the difference between the price of your required hotel and the paid accommodation deposit. After having received your deposit, Action M Agency will confirm your choice and tell you the amount of accommodation balance that has to be paid. The accommodation balance will be payable by a credit card or bank transfer (see PAYMENT). In the case of accommodation balance payments by cash, upon your arrival at the Registration Desk in Prague we require the equivalent amount in Czech Crowns at the current exchange rate. CANCELLATION Refunds of the Registration Fees will be granted for all written cancellations postmarked no later than March 15, 1996. From March 15, 1996 until March 31, 1996 the 50% cancellation fee will be charged. No refunds can be granted for fee cancellations postmarked after March 31, 1996. Refund of the hotel payment will be granted in full for all written cancellation postmarked no later than March 15, 1996. Any cancellation after that date will result in one-night deposit charge. MEALS Light snacks can be obtained in nearby bistros and restaurants. However, the workshop site being in a part of the city frequented by many visitors, the agency highly recommends the use of a prearranged service. For those interested, lunch will be served in the Club Restaurant of Architectures located 50 m from the Bethlehem Palace. A menu will include entree or soup, main course, salad and dessert. A vegetarian main course will be available. SOCIAL PROGRAM All participants and accompanying persons are invited to take part in the following activities, especially at Welcome Party on Tuesday, April 16, 1996 at 7:30 p.m. Walking Tour of Prague - on Monday, April 15, 1996 at 3:00 p.m. to 6:00 p.m. Guided tour of the Old Town, Prague Castle and other historical sites, starting from the Bethlehem Palace. The meeting point will be at 2:45 p.m. at the Registration Desk. Organ Concert - on Wednesday, April 17, 1996 at 7:00 p.m. The special concert for workshop participants in St. Climent Church. A Night with Mozart - on Thursday, April 18, 1996 at 8:00 p.m. The performance full of Mozart's lovely melodies in the Mozart Museum (Villa Bertramka) will introduce the atmosphere of Prague music life 200 years ago. Art and music - the Agency will assist you regarding current art exhibitions, theatre performances and concerts of contemporary and classical music. Social Program Fees can be paid in advance with Registration Fees or at the Registration Desk. PROCEEDINGS The proceedings will be published in Neural Network World. Expanded versions of selected papers are to be reprinted by IEEE as a book. S C I E N T I F I C P R O G R A M TUESDAY - April 16, 1996 9.00 - 9.30 OPENING CEREMONY 9.30 - 10.30 INVITED PLENARY LECTURE T.G.Clarkson (UK) Introduction to Neural Networks 10.30 - 11.00 C o f f e e B r e a k SHORT CONTRIBUTIONS - SECTION A 11.00 - 11.20 L. Reznik (Australia) Controller Design: The Combination of Techniques 11.20 - 11.40 A. Serac, H. Roth (Germany) Design of a Complex Rule-Based Controller for a Biotechnological Process 11.40 - 12.00 K. Althoefer, D.A. Fraser (UK) Fuzzy Obstacle Avoidance for Robotic Manipulators SHORT CONTRIBUTIONS - SECTION B 11.00 - 11.20 V. Kurkova (Czech Republic) Trade-Off between the Size of Parameters and the Number of Units in One-Hidden-Layer-Networks 11.20 - 11.40 D.A. Sprecher (USA) A Numerical Construction of a Universal Function for Kolmogorov's Superpositions 11.40 - 12.00 K. Hlavackova (Czech Republic) Dependence of the Rate of Approximation in a Feedforward Network on its Activation Function 12.00 - 14.00 L u n c h 14.00 - 15.00 INVITED PLENARY LECTURE L. Pecen (Czech Republic) Non-linear Mathematical Interpretation of the Medical Data 15.30 - 16.00 C o f f e e B r e a k SHORT CONTRIBUTIONS - SECTION A 15.30 - 15.50 R. Andonie (Romania) The New Computational Power of Neural Networks 15.50 - 16.10 S. Coombes, S.H. Doole, C. Campbell (UK) Central Pattern Generation in a Model Neuronal Network with Post Inhibitory Rebound and Reciprocal Inhibition 16.10 - 16.30 H. Nagashino, Y. Kinouchi (Japan) Control of Oscillation in a Plastic Neural Oscillator 16.30 - 16.50 U. Latsch (Germany) Neural Decoding of Block Coded Data in Colored Noise 16.50 - 17.10 G. de Tremiolles, K. Madani, P. Tannhof (France) A New Approach to Radial Basis Function's Like Artificial Neural Networks 17.10 - 17.30 S. Draghici (Italy) Improving the Speed of Some Constructive Algorithms by Using a Locking Detection Mechanism SHORT CONTRIBUTIONS - SECTION B 15.30 - 15.50 S. Taraglio, F. Di Fonzo, P. Burrascano (Italy) Training Data Representation in a Neural Based Robot Position Estimation System 15.50 - 16.10 L. Frangu, C. Tudorie, C. Gregoretti, D. Cornei (Romania) Simple Learning Pattern Recognition Predictor and Controller Using Neural Networks 16.10 - 16.30 M. Li (UK) Knowledge-Based Planning in a Simulated Robot World 16.30 - 16.50 S. Rizek (Czech Republic), A. Frolov (Russia) Influence of Feedback upon Learning of Differential Neurocontroller 16.50 - 17.10 V. Roschin, A. Frolov (Russia) Multidimensional Dynamic Differential Neurocontrol 17.10 - 17.30 H.B. Kazemian, E.M. Scharf (UK) An Application of Multi-Input Multi-Output Self Organizing Fuzzy Controller for a Robot-Arm 19.30 W e l c o m e P a r t y Wednesday - April 17, 1996 9.00 - 10.00 INVITED PLENARY LECTURE G. Dorffner (Austria) Neural Networks for Time-Series Analysis 10.00 - 10.30 C o f f e e B r e a k SHORT CONTRIBUTIONS - SECTION A 10.30 - 10.50 Y. Ding, T.G. Clarkson (UK) Fingerprint Recognition by pRAM Neural Networks 10.50 - 11.10 V. Beiu (UK) Entropy Bounds for Classification Algorithms - draft version 11.10 - 11.30 R. Jaitly, D.A. Fraser (UK) Automated 3D Object Recognition and Library Entry System 11.30 - 11.50 F. Hamker, H.M. Gross (Germany) Region Finding for Attention Control in Consideration of Subgoals SHORT CONTRIBUTIONS - SECTION B 10.30 - 10.50 Z.Q. Wu, C.J. Harris (UK) Indirect Adaptive Neurofuzzy Estimation of Nonlinear Time Series 10.50 - 11.10 L.A. Ludwig, A. Grauel (Germany) Designing a Fuzzy Rule Base for Time Series Analysis 11.10 - 11.30 A. Prochazka, M. Mudrova, J. Fiala (Czech Republic) Nonlinear Time-Series Modelling and Prediction 11.30 - 11.50 J. Castellanos, S. Leiva, L.F. Mingo, J. Rios (Spain) Long-Term Trajectory and Signal Behaviour Prediction 11.50 - 14.00 L u n c h 14.00 - 15.30 INVITED PLENARY LECTURE T. Roska (Hungary) Cellular Neural Network - a Paradigm behind a Visual Microprocessor 15.30 - 16.00 C o f f e e B r e a k SHORT CONTRIBUTIONS - SECTION A 16.00 - 16.20 S. Behnke (Germany), N.B. Karayiannis (USA) Competitive Neural Trees for Vector Quantization 16.20 - 16.40 L. Cieplinski, C. Jedrzejek (Poland) Block-Based Rate-Constrained Motion Estimation Using Hopfield Neural Network 16.40 - 17.00 Y. Won, B.H. Lee (Korea) Fuzzy-Morphological-Feature-Based Neural Network for Recognition of Targets in IR imagery 17.00 - 17.20 V. Alexopoulos, S. Kollias (Greece) An Intelligent Action-Based Image Recognition System 17.20 - 17.40 A. Cichocki, W. Kasprzak (Poland) Nonlinear Learning Algorithms for Blind Separation of Natural Images SHORT CONTRIBUTIONS - SECTION B 16.00 - 16.20 C. Schaeffer, R. Kersch, D. Schroeder (Germany) Stable Learning of Out-Of-Roundness with Neural Network 16.20 - 16.40 A. Prochazka, M. Slama, E. Pelikan (Czech Republic) Bayesian Estimators Use in Signal Processing 16.40 - 17.00 O. M. Boaghe (UK) Theoretical and Practical Considerations over Neural Networks Trained with Kalman Filtering 17.00 - 17.20 W. Skrabek, A. Cichocki, W. Kasprzak (Poland) Principal Subspace Analysis for Incomplete Image Data in One Learning Epoch 17.20 - 17.40 C. Thornton (UK) A Selection Principle for Machine Learning Methods THURSDAY - April 18, 1996 9.00 - 10.00 INVITED PLENARY LECTURE S.V. Kartalopoulos (USA) Applications of Fuzzy Logic and Neural Networks in Communications 10.00 - 10.30 C o f f e e B r e a k SHORT CONTRIBUTIONS - SECTION A 10.30 - 10.50 G.S. Wang, Y.H. Song, A.T. Johns (UK), P.Y. Wang, Z.Y. Hu (China) Fuzzy Logic Controlled Learning Algorithm for Training Multilayer Feedforward Neural Networks 10.50 - 11.10 R. Dogaru, A.T. Murgan (Romania), M. Glesner, S. Ortmann (Germany) Computation with Chaos in Discrete-Time Neuro-Fuzzy Networks 11.10 - 11.30 S.I. Vatlin (Republic of Belarus) The Covariant Monotonic Improvement of Fuzzy Classifier is Impossible 11.30 - 11.50 M. Sabau (Romania) A Fuzzy Logic Approach to an Analog to Digital Converter SHORT CONTRIBUTIONS - SECTION B 10.30 - 10.50 D. Gorse, D.A. Romano-Critchley, J.G. Taylor (UK) A Modular pRAM Architecture for the Classification of TESPAR-Encoded Speech Signals 10.50 - 11.10 A.H. El-Mousa, T.G. Clarkson (UK) Multi-Configurable pRAM Based Neuro Computer 11.10 - 11.30 P.J.L. Adeodato, J.G. Taylor (UK) Storage Capacity of RAM-based Neural Networks: Pyramids 11.30 - 11.50 A. Gabrielli, E. Gandolfi, M. Masetti (Italy) VLSI Fuzzy Chip Design that Processes 2-4 Inputs every 160-300 ns whichever is the Fuzzy System 11.50 - 14.00 L u n c h 14.00 - 15.00 POSTER PRESENTATIONS P. Barson, S. Field, N. Davey, G. McAskie, R. Frank (UK) The Detection of Fraud in Mobile Phone Networks G. Baptist (UK), F.C. Lin (USA), J. Nelson (USA) Note on the Long Term Trend of the Dow Jones Industrial Average L. Cieplinski, C. Jedrzejek (Poland) Performance Comparison of Neural Networks VS AD-HOC Heuristic Algorithm for the Traffic Control Problem in Multistage Interconnection Networks N.A. Dubrovskiy, L.K. Rimskaya-Korsakova (Russia) Modelling of Auditory Neurons Own Periodicity Influence on its Frequent Selectivity A. Frolov (Russia), D. Husek (Czech Republic), I. Muraviev (Russia) Statistical Neurodynamics of Sparsely Encoded Hopfield-like Associative Memory M. Jirina (Czech Republic) Neuro-Fuzzy Network Using Extended Kohonen's Map B. Krekelberg, J.G.Taylor (UK) Nitric Oxide and the Development of Long-Range Horizontal Connectivity L.B. Litinsky (Russia) Neural Networks and Factor Analysis L. Nikolic, M. Kejhar, P. Simandl, P. Holoubek (Czech Republic) Analog CMOS Blocks for VLSI Implementation of Programmable Neural Networks I. Silkis (Russia) The Model of Long-Term Modification (LTD,LTP) in the Efficacy of Excitatory and Inhibitory Transmission to Cerebellar Purkinje Neurons W. Skrabek, K. Ignasiak (Poland) Fast VQ Codebook Search in KLT Space K. Zacek (Czech Republic) Modelling of Electron Devices with Mapping Neural Networks 15.30 - 16.00 C o f f e e B r e a k SHORT CONTRIBUTIONS - SECTION A 15.30 - 15.50 S.V. Kartalopoulos (USA) Fuzzy Logic in the Time Domain 15.50 - 16.10 S. Piramuthu (USA) Feature Selection and Neuro-Fuzzy Systems 16.10 - 16.30 A.K. Tsadiras, K.G. Margaritis (Greece) Using Certainty Neurons in Fuzzy Cognitive Maps SHORT CONTRIBUTIONS - SECTION B 15.30 - 15.50 G. Murzina, I. Silkis (Russia) Computational Model of Simultaneous Long-Term Modifications in the Efficacy of Excitatory and Inhibitory Inputs to the Hippocampal Pyramidal Neuron 15.50 - 16.10 H. Djennane, M. Dufosse, A. Kaladjian (France) A Connectionist Hypothesis about the "Mysterious" Origin of Both Error Signals Sent to the Cerebellum & Reinforcement to the Striatum 16.10 - 16.30 I.R. Rezova, A.A. Frolov, V.A. Markevich (Russia) The Influence of a Long-Term Potentiation of the CA1 Hippocampal Field on the Theta Activity and Some Model Notions about the Role of CA1 Field in the Orienting Behaviour of the Animal 16.30 - 17.00 CLOSING CEREMONY All additional requirements and questions related to the scientific program may be addressed to: Institute of Computer Science Academy of Sciences of the Czech Republic Hana Bilkova, secretary Pod Vodarenskou vezi 2 182 07 Prague 8 Phone: (422) 6605 3201, 6605 3220 Fax: (422) 8585789 E-mail: neufuzzy at uivt.cas.cz NEUROFUZZY '95 - REGISTRATION FORM To be faxed or mailed before March 1, 1996 to the Action M Agency, M. Zeithamlova, Vrsovicka 68, 101 00 Prague 10, Czech Republic, fax: (422) 6731 0503 Please, fill in the block letters. ..... ................. ................................ Ms./Mr. First name Surname Institution ................................................. Mailing Address ............................................. ........................................................... .............. ................. ...................... phone fax e-mail IEEE member: YES NO Name of accompanying person ................................. Special needs (vegetarian meals, etc.) ...................... Means of transport (car, train, airplane) ................... Date/time of arrival ........................................ Date of departure ........................................... Accommodation List of available accommodation with required deposit Prices for person/night Single room Double room Deposit 1. Renaissance Prague Hotel DM 285 DM 160 DM 300 2. Forum Hotel DM 245 DM 140 DM 300 3. Betlem Club Praha DM 165 DM 93 DM 300 4. SAX Hotel DM 155 DM 90 DM 300 5. U Zlateho stromu Hotel DM 115 DM 80 DM 300 6. U Sladku Pension DM 65 DM 50 DM 150 7. Petrska Hostel DM 50 DM 30 DM 150 8. Mazanka Hostel DM 45 DM 28 DM 150 Please, reserve the accommodation preferably at: 1st choice ................................................... 2nd choice ................................................... 3rd choice ................................................... Number of nights ............................................... Type of room (single, double) .................................. Name of person sharing the room ................................ PAYMENTS Early payment until March 15,1996 Late payment after March 15,1996 early / late Full Registration Fee, Non-member DM 370 / DM 440 ............ Full Registration Fee, Member IEEE DM 330 / DM 400 ............ Students Fee, Non-member DM 270 / DM 320 ............ Students Fee, Member IEEE DM 220 / DM 260 ............ East Europ. Stud. Fee, Non-member DM 150 / DM 180 ............ East Europ. Stud. Fee, Member IEEE DM 130 / DM 150 ............ Accompanying Person Fee DM 90 ............ Accommodation Deposit DM 300 ............ Accommodation Deposit DM 150 ............ Lunches DM 75 ............ Walking Tour of Prague DM 18 ............ Organ Concert DM 20 ............ A Night with Mozart DM 25 ............ Total Amount ............ For payment by a credit card (MasterCard, EuroCard, VISA, JCB, Diner Club) Type of credit card ........................................... Credit card No. ..................... Expiration .............. I, the undersigned, give the authorization to Action M Agency to withdraw from my account the equivalent in Czech Crowns of the total amount of DM ........................................ Signature ................... I agree to withdraw from my credit card the accommodation balance Signature ................... For payment by bank Name of bank ................................................. Date of payment .............................................. Date ...................... Signature .......................... ------------------------------------------------------------------- NEuroFuzzy'96 Hana Bilkova - secretary Institute of Computer Science, AS CR Pod vodarenskou vezi 2 182 07 Praha 8, Czech Republic phone:(+422) 66052080, 66053201, 66053220 fax: (+422) 8585789 ------------------------------------------------------------ Professor Trevor Clarkson ./././ ./././ ./././ Communications Research Group ./ ./ ./ ./ Dept of Electronic & Electrical Eng ./ ./././ ./ /./ King's College London ./ ./ ./ ./ ./ Strand, London WC2R 2LS, UK ./././ ./ ./ ./././ Tel: +44 171 873 2367 Fax: +44 171 836 4781 Email: tgc at kcl.ac.uk WWW: http://crg.eee.kcl.ac.uk/ ------------------------------------------------------------ From lehr at simoon.Stanford.EDU Mon Feb 5 06:48:18 1996 From: lehr at simoon.Stanford.EDU (Mike Lehr) Date: Mon, 5 Feb 96 03:48:18 PST Subject: Dissertation available: Scaled Stochastic Methods for Training Neural Networks Message-ID: <9602051148.AA13302@simoon.Stanford.EDU> My PhD dissertation is available for electronic retrieval. Retrieval information appears at the bottom of this message. This thesis deals with the problem of training large nonlinear feedforward neural networks using practical stochastic descent methods. Four major topics are explored: (1) An O(N) statistical method that determines a reasonable estimate of the optimal time-varying learning parameter for the stochastic backpropagation procedure (last half of Chapter 4 and parts of Chapter 7). (2) An accelerated O(N) learning procedure that performs an optimal stochastic update along an arbitrary search direction (Chapters 5 and 7 and Appendix I). (3) An O(N) stochastic method which generates an estimate of the Newton direction (Chapters 6 and 7). (4) Various O(N) methods that generate second-order information and other essential information about a neural network's Sum Square Error and Mean Square Error surfaces (Appendices J and K and parts of Chapter 7). An abstract follows. --------------------------------------------------- Scaled Stochastic Methods for Training Neural Networks Michael A. Lehr Department of Electrical Engineering Stanford University Supervisor: Bernard Widrow The performance surfaces of large neural networks contain ravines, ``flat spots,'' nonconvex regions, and other features that make weight optimization difficult. Although a variety of sophisticated alternatives are available, the simple on-line backpropagation procedure remains the most popular method for adapting the weights of these systems. This approach, which performs stochastic (or incremental) steepest descent, is significantly hampered by the character of the performance surface. Backpropagation's principal advantage over alternate methods rests in its ability to perform an update following each pattern presentation, while maintaining time and space demands that grow only linearly with the number of adaptive weights. In this dissertation, we explore new stochastic methods that improve on the learning speed of the backpropagation algorithm, while retaining its linear complexity. We begin by examining the convergence properties of two deterministic steepest descent methods. Corresponding scaled stochastic algorithms are then developed from an analysis of the neural network's Expected Mean Square Error (EMSE) sequence in the neighborhood of a local minimum of the performance surface. To maintain stable behavior over broad conditions, this development uses a general statistical model for the neural network's instantaneous Hessian matrix. For theoretical performance comparisons, however, we require a more specialized statistical framework. The corresponding analysis helps reveal the complementary convergence properties of the two updates---a relationship we exploit by combining the updates to form a family of dual-update procedures. Effective methods are established for generating a slowly varying sequence of search direction vectors and all required scaling information. The result is a practical algorithm which performs robustly when the weight vector of a large neural network is placed at arbitrary initial positions. The two weight updates are scaled by parameters computed from recursive estimates of five scalar sequences: the first and second moments of the trace of the instantaneous Hessian matrix, the first and second moments of the instantaneous gradient vector's projection along the search direction, and the first moment of the instantaneous Hessian's ``projection'' along the same direction. ----------------------------------------------------------- RETRIEVAL INFORMATION: The thesis available at the URL: http://www-isl.stanford.edu/people/lehr in four gzipped postscript files: thesis1.ps.gz, thesis2.ps.gz, thesis3.ps.gz, and thesis4.ps.gz. The corresponding uncompressed postscript files will be available at the same location for the time being. Including front matter, the thesis contains 405 pages formatted for two-sided printing (size: 2.2M compressed, 10.3M uncompressed). The files can also be obtained by anonymous ftp from the directory /pub/lehr/thesis on either simoon.stanford.edu (36.10.0.209), boreas.stanford.edu (36.60.0.210), or zephyrus.stanford.edu (36.60.0.211). Sorry, hardcopies are not available. From degaris at hip.atr.co.jp Tue Feb 6 14:38:11 1996 From: degaris at hip.atr.co.jp (Hugo de Garis) Date: Tue, 6 Feb 96 14:38:11 JST Subject: 2 POSTDOCS REQUIRED AT ATR's BRAIN BUILDER GROUP Message-ID: <9602060538.AA05031@cam8> 2 POSTDOCS REQUIRED AT ATR's BRAIN BUILDER GROUP, EVOLUTIONARY SYSTEMS DEPT, KYOTO, JAPAN ATR's Brain Builder Group, Kyoto, Japan, needs 2 US postdocs in the fields of A) Mini-Robotics/Mechatronics (to build a robot kitten for ATR's Artificial Brain) B) Evolvable Hardware (to apply Genetic Algorithms to FPGAs (Field Programmable Gate Arrays)) (e.g. Xilinx's XC6200) ATR's Evolutionary Systems Dept (ESD) is (arguably) the strongest ALife group in the world with people such as Tom Ray (of Tierra fame) and Chris Langton (father of ALife, and regular ESD visitor and collaborator). One of the highlights of the ESD is the CAM-Brain Project, which builds/grows/evolves a billion neuron artificial brain using cellular automata based neural modules which will grow inside our cellular automata machine (a hundred billion cell updates a second). This artificial brain requires a body to house it, hence our group needs a body builder. If you have extensive experience in building minirobots with off the shelf components, then you might like to join our brain builder group. Ideally, we want to grow/evolve our neural circuits directly in hardware at hardware speeds. We are looking for a second postdoc in the new field of evolvable hardware. If you have extensive experience in FPGA use, and are familiar with genetic algorithms and neural networks, then please join us. Applicants should have a PhD, be US citizens (or have a green card). The working period is from 3 months to 2 years, preferably 2 years, granted by the US NSF (National Science Foundation). The actual money comes from the Japanese "Japan Foundation" and their Center for Global Partnership. The grants cover salary, airfare, rent, but not research costs. Selection will be a two phase process. The first is to be recommended by us. Then your application has to be sent to the NSF in Washington DC by April 1 1996. (Applications are received twice yearly, April 1 and November 1). The NSF people say that if the candidate and the project are good, the odds of selection are 50%. Probable starting date in Japan would be about September 1996. If you do a good job, there's a possibility that you could stay on at ATR on a long term basis. Sabbatical leave grants are also possible for more senior candidates. The type of candidates we are looking for need to be big egoed dreamers with strong vision and creativity. The senior members of ESD are all pioneers. If you are a CD type (i.e. competent dullard, meaning high in analytical skills, but lacking in vision and creativity), then this spot is not for you). If you are interested, please send your resume by email to - Dr. Hugo de Garis, Brain Builder Group, Evolutionary Systems Dept., ATR Human Information Processing Research Labs, 2-2 Hikari-dai, Seika-cho, Soraku-gun, Kansai Science City, Kyoto-fu, 619-02, Japan. tel. + 81 774 95 1079, fax. + 81 774 59 1008, email. degaris at hip.atr.co.jp For more information from the NSF, contact - email. info at nsf.gov tel. 703 306 1234 or 703 306 0090 web. http://www.nsf.gov/ If you have friends you might be interested, please forward this to them. From bishopc at helios.aston.ac.uk Tue Feb 6 08:50:32 1996 From: bishopc at helios.aston.ac.uk (Prof. Chris Bishop) Date: Tue, 06 Feb 1996 13:50:32 +0000 Subject: Reprint of New Book Message-ID: <20582.9602061350@sun.aston.ac.uk> "Neural Networks for Pattern Recognition" Oxford University Press Christopher M. Bishop http://neural-server.aston.ac.uk/NNPR/ Several people have reported difficulty in obtaining copies of this book. In fact demand was much higher than the publishers anticipated and the first print run sold out very quickly. It has now been reprinted and the supply problems should now be resolved. Chris Bishop Aston From horvitz at u.washington.edu Tue Feb 6 23:16:11 1996 From: horvitz at u.washington.edu (Eric Horvitz) Date: Tue, 6 Feb 1996 20:16:11 -0800 (PST) Subject: cfp: UAI '96 (Uncertainty in Artificial Intelligence) Message-ID: ========================================================= C A L L F O R P A P E R S (Note Revised Dates) ========================================================= ** U A I 96 ** THE TWELFTH ANNUAL CONFERENCE ON UNCERTAINTY IN ARTIFICIAL INTELLIGENCE August 1-3, 1996 Reed College Portland, Oregon, USA ======================================= See the UAI-96 WWW page at http://cuai-96.microsoft.com/ CALL FOR PAPERS The effective handling of uncertainty is critical in designing, understanding, and evaluating computational systems tasked with making intelligent decisions. For over a decade, the Conference on Uncertainty in Artificial Intelligence (UAI) has served as the central meeting on advances in methods for reasoning under uncertainty in computer-based systems. The conference is the annual international forum for exchanging results on the use of principled uncertain-reasoning methods to solve difficult challenges in AI. Theoretical and empirical contributions first presented at UAI have continued to have significant influence on the direction and focus of the larger community of AI researchers. The scope of UAI covers a broad spectrum of approaches to automated reasoning and decision making under uncertainty. Contributions to the proceedings address topics that advance theoretical principles or provide insights through empirical study of applications. Interests include quantitative and qualitative approaches, and traditional as well as alternative paradigms of uncertain reasoning. Innovative applications of automated uncertain reasoning have spanned a broad spectrum of tasks and domains, including systems that make autonomous decisions and those designed to support human decision making through interactive use. We encourage submissions of papers for UAI-96 that report on advances in the core areas of representation, inference, learning, and knowledge acquisition, as well as on insights derived from building or using applications of uncertain reasoning. Topics of interest include (but are not limited to): >> Foundations * Theoretical foundations of uncertain belief and decision * Uncertainty and models of causality * Representation of uncertainty and preference * Generalization of semantics of belief * Conceptual relationships among alternative calculi * Models of confidence in model structure and belief >> Principles and Methods * Planning under uncertainty * Temporal reasoning * Markov processes and decisions under uncertainty * Qualitative methods and models * Automated construction of decision models * Abstraction in representation and inference * Representing intervention and persistence * Uncertainty and methods for learning and datamining * Computation and action under limited resources * Control of computational processes under uncertainty * Time-dependent utility and time-critical decisions * Uncertainty and economic models of problem solving * Integration of logical and probabilistic inference * Statistical methods for automated uncertain reasoning * Synthesis of Bayesian and neural net techniques * Algorithms for uncertain reasoning * Advances in diagnosis, troubleshooting, and test selection >> Empirical Study and Applications * Empirical validation of methods for planning, learning, and diagnosis * Enhancing the human--computer interface with uncertain reasoning * Uncertain reasoning in embedded, situated systems (e.g., softbots) * Automated explanation of results of uncertain reasoning * Nature and performance of architectures for real-time reasoning * Experimental studies of inference strategies * Experience with knowledge-acquisition methods * Comparison of repres. and inferential adequacy of different calculi * Uncertain reasoning and information retrieval For papers focused on applications in specific domains, we suggest that the following issues be addressed in the submission: - Why was it necessary to represent uncertainty in your domain? - What are the distinguishing properties of the domain and problem? - What kind of uncertainties does your application address? - Why did you decide to use your particular uncertainty formalism? - What theoretical problems, if any, did you encounter? - What practical problems did you encounter? - Did users/clients of your system find the results useful? - Did your system lead to improvements in decision making? - What approaches were effective (ineffective) in your domain? - What methods were used to validate the effectiveness of the systems? ================================= SUBMISSION AND REVIEW OF PAPERS ================================= Papers submitted for review should represent original, previously unpublished work (details on policy on submission uniqueness are available at the UAI 96 www homepage). Submitted papers will be evaluated on the basis of originality, significance, technical soundness, and clarity of exposition. Papers may be accepted for presentation in plenary or poster sessions. All accepted papers will be included in the Proceedings of the Twelfth Conference on Uncertainty in Artificial Intelligence, published by Morgan Kaufmann Publishers. Outstanding student papers will be selected for special distinction. Submitted papers must be at most 20 pages of 12pt Latex article style or equivalent (about 4500 words). See the UAI-96 homepage for additional details about UAI submission policies. We strongly encourage the electronic submission of papers. To submit a paper electronically, send an email message to uai at microsoft.com that includes the following information (in this order): * Paper title (plain text) * Author names, including student status (plain text) * Surface mail and email address for a contact author (plain text) * A short abstract including keywords or topic indicators (plain text) An electronic version of the paper (Postscript format) should be submitted simultaneously via ftp to: cuai-96.microsoft.com/incoming. Files should be named $.ps, where $ is an identifier created from the first five letters of the last name of the first author, followed by the first initial of the author's first name. Multiple submissions by the same first author should be indicated by adding a number (e.g., pearlj2.ps) to the end of the identifier. Authors will receive electronic confirmation of the successful receipt of their articles. Authors unable to access ftp should electronically mail the first four items and the Postscript file of their paper to uai at microsoft.com. Authors unable to submit Postscript versions of their paper should send the first four items in email and 5 copies of the complete paper to one of the Program Chairs at the addresses listed below. ++++++++++++++++++++++++++++++ Important Dates (Note revisions) ++++++++++++++++++++++++++++++ >> Submissions must be received by 5PM local time: March 1, 1996 >> Notification of acceptance on or before: April 19, 1996 >> Camera-ready copy due: May 15, 1996 ========================== Program Cochairs: ================= Eric Horvitz Microsoft Research, 9S Redmond, WA 98052 Phone: (206) 936 2127 Fax: (206) 936 0502 Email: horvitz at microsoft.com WWW: http://www.research.microsoft.com/research/dtg/horvitz/ Finn Jensen Department of Mathematics and Computer Science Aalborg University Fredrik Bajers Vej 7,E DK-9220 Aalborg OE Denmark Phone: +45 98 15 85 22 (ext. 5024) Fax: +45 98 15 81 29 Email: fvj at iesd.auc.dk WWW: http://www.iesd.auc.dk/cgi-bin/photofinger?fvj General Conference Chair (General conference inquiries): ======================== Steve Hanks Department of Computer Science and Engineering, FR-35 University of Washington Seattle, WA 98195 Tel: (206) 543 4784 Fax: (206) 543 2969 Email: hanks at cs.washington.edu Program Committee =================== Fahiem Bacchus (U Waterloo) * Salem Benferhat (U Paul Sabatier) * Mark Boddy (Honeywell) * Piero Bonissone (GE) * Craig Boutilier (U Brit Columbia) * Jack Breese (Microsoft) * Wray Buntine (Thinkbank) * Luis M. de Campos * (U Granada) * Enrique Castillo (U Cantabria) * Eugene Charniak (Brown) * Greg Cooper (U Pittsburgh) * Bruce D'Ambrosio (Oregon State) * Paul Dagum (Stanford) * Adnan Darwiche (Rockwell) * Tom Dean (Brown) * Denise Draper (Rockwell) * Marek Druzdzel (U Pittsburgh) * Didier Dubois (Paul Sabatier) * Ward Edwards (USC) * Kazuo Ezawa (ATT Labs) * Robert Fung (Prevision) * Linda van der Gaag (Utrecht U) * Hector Geffner (Simon Bolivar) * Dan Geiger (Technion) * Lluis Godo (Barcelona) * Robert Goldman (Honeywell) * Moises Goldszmidt (Rockwell) * Adam Grove (NEC) * Peter Haddawy (U Wisc-Milwaukee) * Petr Hajek (Czech Acad Sci) * Joseph Halpern (IBM) * Steve Hanks (U Wash) * Othar Hansson (Berkeley) * Peter Hart (Ricoh) * David Heckerman (Microsoft) * Max Henrion (Lumina) * Frank Jensen (Hugin) * Michael Jordan (MIT) * Leslie Pack Kaelbling (Brown) * Keiji Kanazawa (Microsoft) * Uffe Kjaerulff (U Aalborg) * Daphne Koller (Stanford) * Paul Krause (Imp. Cancer Rsch Fund) * Rudolf Kruse (U Braunschweig) * Henry Kyburg (U Rochester) * Jerome Lang (U Paul Sabatier) * Kathryn Laskey (George Mason) * Paul Lehner (George Mason) * John Lemmer (Rome Lab) * Tod Levitt (IET) * Ramon Lopez de Mantaras (Spanish Sci. Rsch Council) * David Madigan (U Wash) * Eric Neufeld (U Saskatchewan) * Ann Nicholson (Monash U) * Nir Friedman (Stanford) * Judea Pearl (UCLA) * Mark Peot (Stanford) * Kim Leng Poh, (Natl U Singapore) * David Poole (U Brit Columbia) * Henri Prade (U Paul Sabatier) * Greg Provan (Inst. Learning Sys) * Enrique Ruspini (SRI) * Romano Scozzafava (Dip. Mo. Met., Rome) * Ross Shachter (Stanford) * Prakash Shenoy (U Kansas) * Philippe Smets (U Bruxelles) * David Spiegelhalter (Cambridge U) * Peter Spirtes (CMU) * Milan Studeny (Czech Acad Sci) * Sampath Srinivas (Microsoft) * Jaap Suermondt (HP Labs) * Marco Valtorta (U S.Carolina) * Michael Wellman (U Michigan) * Nic Wilson (Oxford Brookes U) * Y. Xiang (U Regina) * Hong Xu (U Bruxelles) * John Yen (Texas A&M) * Lian Wen Zhang, (Hong Kong U) * --------------- UAI-96 will occur right before KDD-96, AAAI-96, and the AAAI workshops, and will be in close proximity to these meetings. * * * UAI 96 will include a full-day tutorial program on uncertain reasoning on the day before the main UAI 96 conference (Wednesday, July 31) at Reed College. Details on the tutorials are available on the UAI 96 www homepage. * * * Refer to the UAI-96 WWW home page for late-breaking information: http://cuai-96.microsoft.com/ From ling at cs.hku.hk Tue Feb 6 22:32:37 1996 From: ling at cs.hku.hk (Charles X. Ling) Date: Wed, 7 Feb 1996 11:32:37 +0800 Subject: AAAI-96 Workshop: Computational Cognitive Modeling Message-ID: <9602070332.AA12863@sparc419> We are looking forward to a productive meeting. We seek for a balance between different models (such as connectionists and symbolic models). Submissions from cognitive scientists, AI researchers, and psychologists are warmly welcome. Charles Ling ************************ Computational Cognitive Modeling: Source of the Power AAAI-96 Workshop (During AAAI'96, IAAI 96 and KDD 96. August 4-8, 1996. Portland, Oregon) URL: http://www.cs.hku.hk/~ling for updated information. CALL FOR PAPERS AND PARTICIPATION Aims of the Workshop ==================== Computational models for various cognitive tasks have been extensively studied by cognitive scientists, AI researchers, and psychologists. These tasks include -- language acquisition (learning past tense, word reading and naming, learning grammar, etc.) -- cognitive skill acquisition (subconscious learning, learning sequences) -- cognitive development (the balance scale and learning arithmetic) -- conceptual development; reasoning (commonsense, analogical) We attempt to bring researchers from different backgrounds together, and to examine how and why computational models (connectionist, symbolic, memory-based, or others) are successful in terms of the source of power. The possible sources of power include: -- Representation of the task; -- General properties of the learning algorithm; -- Data sampling/selection; -- Parameters of the learning algorithms. The workshop will focus on, but not be limited to, the following topics, all of which should be discussed in relation to the source of power: -- Proper criteria for judging success or failure of a model. -- Methods for recognizing the source of power. -- Analyses of the success or failure of existing models. -- Presentation of new cognitive models. We hope that our workshop will shed new light on these questions, stimulate lively discussions on the topics, as well as generate new research ideas. Format of the Workshop: ====================== The Workshop will consist of invited talks, presentations, and a poster session. All accepted papers (presentation or poster) will be included in the Workshop Working Notes. A pannel will summarize and debate at the end of the Workshop. Submission information: ====================== Submissions from AI researchers, cognitive scientists and psychologists are welcome. We encourage submissions from people of divergent backgrounds. Potential presenters should submit a paper (maximum 12 pages total, 12 point font). We strongly encourage email submissions of text/postscript files; or you may also send 4 paper copies to one workshop co-chair: Charles Ling (co-chair) Ron Sun (co-chair) Department of Computer Science Department of Computer Science University of Hong Kong University of Alabama Hong Kong Tuscaloosa, AL 35487 ling at cs.hku.hk rsun at cs.ua.edu (On leave from University of Western Ontario) Researchers interested in attending Workshop only should send a short description of interests to one co-chair by deadline. Deadline for submission: March 18, 1996. Notification of acceptance: April 15, 1996. Submission of final versions: May 13, 1996. Program Committee: ================= Charles Ling (co-chair), University of Hong Kong, ling at cs.hku.hk Ron Sun (co-chair), University of Alabama, rsun at cs.ua.edu Pat Langley, Stanford University, langley at flamingo.Stanford.EDU Mike Pazzani, UC Irvine, pazzani at super-pan.ICS.UCI.EDU Tom Shultz, McGill University, shultz at psych.mcgill.ca Paul Thagard, Univ. of Waterloo, pthagard at watarts.uwaterloo.ca Kurt VanLehn, Univ. of Pittsburgh, vanlehn+ at pitt.edu Invited Speakers (NEW): ====================== We are glad to have the following confirmed invited speakers to present their work at the Workshop: Jeff Elman Mike Pazzani Aaron Sloman Denis Mareschal From terry at salk.edu Thu Feb 8 18:05:54 1996 From: terry at salk.edu (Terry Sejnowski) Date: Thu, 8 Feb 96 15:05:54 PST Subject: Neural Computation 8:2 Message-ID: <9602082305.AA23531@salk.edu> Neural Computation Volume 8, Issue 2, February 15, 1996 Article Encoding with Bursting, Subthreshold Oscillations and Noise in Mammalian Cold Receptors Andre Longtin and Karin Hinzer Notes Associative Memory with Uncorrelated Inputs Ronald Michaels Statistical Independence and Novelty Detection with Information Preserving Nonlinear Maps Lucas Parra, Gustavo Deco and Stefan Miesbach Letters Neural Network Models of Perceptual Learning of Angle Discrimination G. Mato and H. Sompolinsky Directional Filling-In Karl Frederick Arrington Binary-Oscillator Networks: Bridging a Gap between Experimental and Abstract Modeling of Neural Networks Wei-Ping Wang Annealed Competition of Experts for a Segmentation and Classification of Switching Dynamics Klaus Pawelzik, Jens Kohlmorgen and Klaus-Robert Muller A Recurrent Network Implementation of Time Series Classification Vasilios Petridis and Athanasios Kehagias Temporal Segmentation in a Neural Dynamical System David Horn and Irit Opher Circular Nodes in Neural Networks Michael J. Kirby and Rick Miranda The Computational Power of Discrete Hopfield Nets with Hidden Units Pekka Orponen A Self-Organizing Neural Network for the Traveling Salesman Problem That Is Competitive with Simulated Annealing Marco Budinich Hierarchical, Unsupervised Learning with Growing Via Phase Transitions David Miller and Kenneth Rose The Interchangeability of Learning Rate and Gain in Backpropagation Neural Networks Georg Thimm , Perry Moerland and Emile Fiesler ----- ABSTRACTS - http://www-mitpress.mit.edu/jrnls-catalog/neural.html SUBSCRIPTIONS - 1996 - VOLUME 8 - 8 ISSUES ______ $50 Student and Retired ______ $78 Individual ______ $220 Institution Add $28 for postage and handling outside USA (+7% GST for Canada). (Back issues from Volumes 1-7 are regularly available for $28 each to institutions and $14 each for individuals Add $5 for postage per issue outside USA (+7% GST for Canada) mitpress-orders at mit.edu MIT Press Journals, 55 Hayward Street, Cambridge, MA 02142. Tel: (617) 253-2889 FAX: (617) 258-6779 ----- From ericwan at choosh.eeap.ogi.edu Thu Feb 8 20:48:53 1996 From: ericwan at choosh.eeap.ogi.edu (Eric A. Wan) Date: Fri, 9 Feb 1996 09:48:53 +0800 Subject: FIR/TDNN Toolbox for MATLAB Message-ID: <9602091748.AA04337@choosh.eeap.ogi.edu> ***************************************************************** * * * FIR/TDNN Toolbox for MATLAB * * * ***************************************************************** ***************************************************************** DESCRIPTION: Beta version of a toolbox for FIR (Finite Impulse Response) and TD (Time Delay) Neural Networks. For efficient stochastic implementation, algorithms are written as MEX compatible c-code which can be called as primitive functions from within MATLAB. Both source and compiled functions are available. LOCATION: http://www.eeap.ogi.edu/~ericwan/fir.html +----------------------------------------------------------------------------+ | Eric A. Wan | Dept. of Electrical Engineering and Applied Physics | | | Oregon Graduate Institute of Science & Technology | +----------------------+-----------------------------------------------------+ | ericwan at eeap.ogi.edu | Mailing: | Shipping: | | tel (503) 690-1164 | P.O. Box 91000 | 20000 N.W. Walker Road | | fax (503) 690-1406 | Portland, OR 97291-1000 | Beaverton, OR 97006 | +----------------------------------------------------------------------------+ | Home page: http://www.cse.ogi.edu/~ericwan | +----------------------------------------------------------------------------+ From kak at ee.lsu.edu Fri Feb 9 17:15:59 1996 From: kak at ee.lsu.edu (Subhash Kak) Date: Fri, 9 Feb 96 16:15:59 CST Subject: Papers Message-ID: <9602092215.AA25252@ee.lsu.edu> The following two papers are available: 1. Information, Physics and Computation by Subhash C. Kak appearing in FOUNDATIONS OF PHYSICS, vol 26, 1996 ftp://gate.ee.lsu.edu/pub/kak/inf.ps.Z 2. Can we have different levels of artificial intelligence? by Subhash C. Kak appearing in JOURNAL OF INTELLIGENT SYSTEMS, vol 6, 1996 ftp://gate.ee.lsu.edu/pub/kak/ai.ps.Z ------------------------------------------------------------ Abstracts: --------- 1. Information, Physics and Computation The paper presents several observations on the connections between information, physics and computation. In particular, the computing power of quantum computers is examined. Quantum theory is characterized by superimposed states and non-local interactions. It is argued that recently studied quantum computers, which are based on local interactions, cannot simulate quantum physics. 2. Can we have different levels of artificial intelligence? This paper argues for a graded approach to the study of artificial intelligence. In contrast to the Turing test, such an approach permits the measurement of incremental progress in AI research. Results on the conceptual abilities of pigeons are summarized. These abilities far exceed the generalization abilities of current AI programs. It is argued that matching the conceptual abilities of animals would require new approaches to AI. Defining graded levels of intelligence would permit the identification of resources needed for implementation. From Dave_Touretzky at DST.BOLTZ.CS.CMU.EDU Fri Feb 9 23:35:22 1996 From: Dave_Touretzky at DST.BOLTZ.CS.CMU.EDU (Dave_Touretzky@DST.BOLTZ.CS.CMU.EDU) Date: Fri, 09 Feb 96 23:35:22 EST Subject: undergrad summer research program Message-ID: <16034.823926922@DST.BOLTZ.CS.CMU.EDU> NPC (Neural Processes in Cognition) Undergraduate Summer Program The Neural Processes in Cognition Training Program at the University of Pittsburgh and Carnegie Mellon University has several positions available for undergraduates interested in studying cognitive or computational neuroscience. These are growing interdisciplinary areas of study (see Science, 1993, v. 261, pp 1805-7) that interpret cognitive functions in terms of neuroanatomical and neurophysiological data and computer simulations. Undergraduate students participating in the summer program will have the opportunity to spend ten weeks of intensive involvement in laboratory research supervised by one of the program's faculty. The summer program also includes weekly journal clubs and a series of informal lectures. Students selected for the program will receive a $2500 stipend. The program is funded by a grant from the National Science Foundation and by the joint CMU/University of Pittsburgh Center for the Neural Basis of Cognition. Each student's research program will be determined in consultation with the training program's Director. Potential laboratory environments include single unit recording, neuroanatomy, computer simulation of biological and cognitive effects, robot control, neuropsychological assessment, behavioral assessment, and brain imaging. How to Apply to the NPC Undergraduate Summer Program: Applications are encouraged from highly motivated undergraduate students with interests in biology, psychology, engineering, physics, mathematics or computer science. Application deadline is March 15, 1996. To apply, request application materials by email at neurocog at vms.cis.pitt.edu, phone 412-624-7064, or write to the address below. The materials include a listing of faculty research areas to consider. Applicants are asked to supply a statement of their research interests, a recent school transcript, one faculty letter of recommendation, and a selection of one or two research areas which they would like to explore. Applicants are strongly encouraged to identify a particular faculty member with whom they want to work. Send requests and application materials to: Professor Walter Schneider, Program Director University of Pittsburgh Neural Processes in Cognition Program 3939 O'Hara Street Pittsburgh, PA 15260 Email: neurocog at vms.cis.pitt.edu Note: the Neural Processes in Cognition program also offers pre- and post-doctoral training. To find out more about the program or the Center for the Neural Basis of Cognition, visit our web sites: http://www.cs.cmu.edu/Web/Groups/CNBC http://neurocog.lrdc.pitt.edu/npc/npc.html From payman at uw-isdl.ee.washington.edu Sun Feb 11 17:48:03 1996 From: payman at uw-isdl.ee.washington.edu (Payman Arabshahi) Date: Sun, 11 Feb 1996 14:48:03 -0800 (PST) Subject: IEEE NNC Homepage - Call for Submissions Message-ID: <199602112248.OAA11919@uw-isdl.ee.washington.edu> The IEEE Neural Network Council's Homepage (http://www.ieee.org/nnc) is seeking information about Computational Intelligence Research Programs Worldwide, both in academia and industry. If you know of a relevant research group's homepage or would like a link to your homepage from the IEEE NNC research page, please let me know. At present we maintain links to some 64 Neural Computing Programs, and are especially seeking information on programs in Canada, Central and South America, Asia and the Middle East, Eastern Europe, the former Soviet Union, and Africa. Thank you for your cooperation. -- Payman Arabshahi Tel : (205) 895-6380 Dept. of Electrical & Computer Eng. Fax : (205) 895-6803 University of Alabama in Huntsville payman at ebs330.eb.uah.edu Huntsville, AL 35899 http://www.eb.uah.edu/ece/ From tony at discus.anu.edu.au Mon Feb 12 22:30:29 1996 From: tony at discus.anu.edu.au (Tony BURKITT) Date: Tue, 13 Feb 1996 14:30:29 +1100 (EST) Subject: ACNN'96 registration information Message-ID: <199602130330.OAA24815@discus.anu.edu.au> REGISTRATION INFORMATION ACNN'96 SEVENTH AUSTRALIAN CONFERENCE ON NEURAL NETWORKS 10th - 12th APRIL 1996 Australian National University Canberra, Australia ADVANCE REGISTRATION REMINDER To receive a 20% discount for registration at ACNN'96, you must post a registration form before Friday, February 16th. ACNN'96 The seventh Australian conference on neural networks will be held in Canberra on April 10th - 12th 1996 at the Australian National University. ACNN'96 is the annual national meeting of the Australian neural network community. It is a multi-disciplinary meeting, with contributions from Neuroscientists, Engineers, Computer Scientists, Mathematicians, Physicists and Psychologists. The program will include keynote talks, lecture presentations and poster sessions. Proceedings will be printed and distributed to the attendees. Invited Keynote Speakers ACNN'96 will feature two keynote speakers: Professor Wolfgang Maass, Institute for Theoretical Computer Science, Technical University Graz, "Networks of spiking neurons: The third generation of neural network models," and Professor Steve Redman, John Curtin School of Medical Research, Australian National University, "Modelling versus measuring: a perspective on computational neuroscience." Pre-Conference Workshops There will be two Pre-Conference Workshops on Tuesday 9th April: Memory, time, change and structure in ANNs: Distilling cognitive models into their functional components Convenors: Janet Wiles and J. Devin McAuley, Depts of Computer Science and Psychology, University of Queensland janet at psy.uq.edu.au devin at psy.uq.edu.au The workshop aim is to identify the functional roles of artificial neural network (ANN) components and to understand how they combine to explain cognitive phenomena. Existing ANN models will be distilled into their functional components through case study analysis, targeting three traditional strengths of ANNs - mechanisms for memory, time and change; and one area of weakness - mechanisms for structure. (see http://psy.uq.edu.au/CogPsych/acnn96/workshop.html) Neural Networks in the Australian Public Service Convenor: Andrew Freeman phone: (06) 264 3698 fax: (06) 264 4717 afreeman at pcug.org.au This workshop will provide a venue for an informal exchange of views and experiences between researchers, users, and suppliers of neural technologies for forms processing in the Australian Public Service. (see http://www.pcug.org.au/~afreeman/otsig.html) Special Poster Session ACNN'96 will include a special poster session devoted to recent work and work-in-progress. Abstracts are solicited for this session (1 page limit), and may be submitted up to one week before the commencement of the conference. They will not be refereed or included in the proceedings, but will be distributed to attendees upon arrival. Students are especially encouraged to submit abstracts for this session. Venue Huxley Lecture Theatre, Leonard Huxley Building, Mills Road, Australian National University, Canberra, Australia Further Information For more information on the conference (including the list of accepted papers, pointers to information on pre-conference workshops and tutorials, registration and accomodation information, and the registration form), see the ACNN96 web page: http://wwwsyseng.anu.edu.au/acnn96/ or contact: ACNN'96 Secretariat Department of Engineering FEIT Australian National University Canberra, ACT 0200 Australia Phone: +61 6 249 5645 ftp site: syseng.anu.edu.au:pub/acnn96 email: acnn96 at anu.edu.au ------------------------------------------------------------------------------ ACNN'96 Seventh Australian Conference on Neural Networks Registration Form Title & Name: ___________________________________________________________ Organisation: ___________________________________________________________ Department: _____________________________________________________________ Occupation: _____________________________________________________________ Address: ________________________________________________________________ State: ____________________ Post Code: _____________ Country: ___________ Tel: ( ) __________________________ Fax: ( ) _____________________ E-mail: _________________________________________________________________ [ ] Find enclosed a cheque for the sum of @: ______________________ [ ] Charge my credit card for the sum of # :________________________ Mastercard/Visa/Bankcard# Number : _____________________________ Valid until: ________ Signature: __________________ Date: ______ ------------------------------------------------------------------------------ To register, please fill in this form and return it together with payment to : ACNN'96 Secretariat L. P. O. Box 228 Australian National University Canberra, ACT 2601 Australia ------------------------------------------------------------------------------ @ Registration fees: Before 16 Feb 96 After 16 Feb 96 Full Time Students A$ 96.00 A$120.00 Academics A$208.00 A$260.00 Other A$304.00 A$380.00 # Please encircle type of card ------------------------------------------------------------------------------ From mozer at neuron.cs.colorado.edu Tue Feb 13 14:03:20 1996 From: mozer at neuron.cs.colorado.edu (Mike Mozer) Date: Tue, 13 Feb 1996 12:03:20 -0700 Subject: NIPS*96 CALL FOR PAPERS Message-ID: <199602131903.MAA29791@neuron.cs.colorado.edu> [ Moderator's note: Below is the NIPS*96 call for papers. I would like to remind people that many of the papers from NIPS*95 are now accessible online via the NIPS web site; the URL is given below. Also, NIPS*95 t-shirts and mousepads with the Wizard of Oz theme can now be ordered by mail at heavily discounted prices; see the NIPS web site for details. -- Dave Touretzky ] CALL FOR PAPERS Neural Information Processing Systems -- Natural and Synthetic Monday December 2 - Saturday December 7, 1996 Denver, Colorado This is the tenth meeting of an interdisciplinary conference which brings together cognitive scientists, computer scientists, engineers, neuro- scientists, physicists, and mathematicians interested in all aspects of neural processing and computation. The conference will include invited talks and oral and poster presentations of refereed papers. The conference is single track and is highly selective. Preceding the main session, there will be one day of tutorial presentations (Dec. 2), and following will be two days of focused workshops on topical issues at a nearby ski area (Dec. 6-7). Major categories for paper submission, with example subcategories, are as follows: Algorithms and Architectures: supervised and unsupervised learning algorithms, constructive/pruning algorithms, decision trees, localized basis functions, layered networks, recurrent networks, Monte Carlo algorithms, combinatorial optimization, performance comparisons Applications: database mining, DNA/protein sequence analysis, expert systems, fault diagnosis, financial analysis, medical diagnosis, music processing, time-series prediction Artificial Intelligence and Cognitive Science: perception, natural language, human learning and memory, problem solving, decision making, inductive reasoning, hybrid symbolic-subsymbolic systems Control, Navigation, and Planning: robotic motor control, process control, navigation, path planning, exploration, dynamic programming, reinforcement learning Implementation: analog and digital VLSI, optical neurocomputing systems, novel neuro-devices, simulation tools, parallelism Neuroscience: systems physiology, signal and noise analysis, oscillations, synchronization, mechanisms of inhibition and neuromodulation, synaptic plasticity, computational models Speech, Handwriting, and Signal Processing: speech recognition, coding, and synthesis, handwriting recognition, adaptive equalization, nonlinear noise removal, auditory scene analysis Theory: computational learning theory, complexity theory, dynamical systems, statistical mechanics, probability and statistics, approximation and estimation theory Visual Processing: image processing, image coding and classification, object recognition, stereopsis, motion detection and tracking, visual psychophysics Review Criteria: All submitted papers will be thoroughly refereed on the basis of technical quality, significance, and clarity. Novelty of the work is also a strong consideration in paper selection, but, to encourage interdisciplinary contributions, we will consider work which has been submitted or presented in part elsewhere, if it is unlikely to have been seen by the NIPS audience. Authors should not be dissuaded from submitting recent work, as there will be an opportunity after the meeting to revise accepted manuscripts before submitting final camera-ready copy. Paper Format: Submitted papers may be up to seven pages in length, including figures and references, using a font no smaller than 10 point. Submissions failing to follow these guidelines will not be considered. Authors are encouraged to use the NIPS LaTeX style files obtainable by anonymous FTP at the site given below. Papers must indicate (1) physical and e-mail addresses of all authors; (2) one of the nine major categories listed above, and, if desired, a subcategory; (3) if the work, or any substantial part thereof, has been submitted to or has appeared in other scientific conferences; (4) the authors' preference, if any, for oral or poster presentation; this preference will play no role in paper acceptance; and (5) author to whom correspondence should be addressed. Submission Instructions: Send six copies of submitted papers to the address below; electronic or FAX submission is not acceptable. Include one additional copy of the abstract only, to be used for preparation of the abstracts booklet distributed at the meeting. SUBMISSIONS MUST BE RECEIVED BY MAY 24, 1996. From within the U.S., submissions will be accepted if mailed first class and postmarked by May 21, 1996. Mail submissions to: Michael Jordan NIPS*96 Program Chair Department of Brain and Cognitive Sciences, E10-034D Massachusetts Institute of Technology 79 Amherst Street Cambridge, MA 02139 USA Mail general inquiries and requests for registration material to: NIPS*96 Registration Conference Consulting Associates 451 N. Sycamore Monticello, IA 52310 fax: (319) 465-6709 (attn: Denise Prull) e-mail: nipsinfo at salk.edu Copies of the LaTeX style files for NIPS are available via anonymous ftp at ftp.cs.cmu.edu (128.2.206.173) in /afs/cs/Web/Groups/NIPS/formatting The style files and other conference information may also be retrieved via World Wide Web at http://www.cs.cmu.edu/Web/Groups/NIPS NIPS*96 Organizing Committee: General Chair, Michael Mozer, U. Colorado; Program Chair, Michael Jordan, MIT; Publications Chair, Thomas Petsche, Siemens; Tutorial Chair, John Lazzaro, Berkeley; Workshops Co-Chairs, Michael Perrone, IBM, and Steven Nowlan, Lexicus; Publicity Chair, Suzanna Becker, McMaster; Local Arrangements, Marijke Augusteijn, U. Colorado; Treasurer, Eric Mjolsness, UCSD; Government/Corporate Liaison, John Moody, OGI; Contracts, Steve Hanson, Siemens, Scott Kirkpatrick, IBM, Gerry Tesauro, IBM. Conference arrangements by Conference Consulting Associates, Monticello, IA. DEADLINE FOR RECEIPT OF SUBMISSIONS IS MAY 24, 1996 - please post - From iconip at cs.cuhk.hk Wed Feb 14 05:13:26 1996 From: iconip at cs.cuhk.hk (ICONIP96) Date: Wed, 14 Feb 1996 18:13:26 +0800 Subject: ICONIP'96 EXTENSION OF PAPER SUBMISSION DEADLINE Message-ID: <199602141013.SAA00808@cs.cuhk.hk> ====================================================================== We apologize should you receive multiple copies of this CFP from different sources. ====================================================================== ************************************************ ICONIP'96 EXTENSION OF PAPER SUBMISSION DEADLINE ************************************************ 1996 International Conference on Neural Information Processing The Annual Conference of the Asian Pacific Neural Network Assembly ICONIP'96, September 24 - 27, 1996 Hong Kong Convention and Exhibition Center, Wan Chai, Hong Kong In cooperation with IEEE / NNC --IEEE Neural Networks Council INNS - International Neural Network Society ENNS - European Neural Network Society JNNS - Japanese Neural Network Society CNNC - China Neural Networks Council ====================================================================== In consideration of many requests from Europe, USA as well as Asia Pacific region for a possible extension of paper submission deadline for ICONIP'96, the ICONIP'96 Organizing Committee has decided to extend the paper submission deadline. ----- The Extended Paper Submission Deadline : March 10, 1996 ---- The goal of ICONIP'96 is to provide a forum for researchers and engineers from academia and industry to meet and to exchange ideas on the latest developments in neural information processing. The conference also further serves to stimulate local and regional interests in neural information processing and its potential applications to industries indigenous to this region. The conference consists of two tracks. One is SCIENTIFIC TRACK for the latest results on Theories, Technologies, Methods, Architectures and Algorithms in neural information processing. The other is APPLICATION TRACK for various neural network applications in any engineering/technical field and any business/service sector. There will be a one-day tutorial on the neural networks for capital markets which reflects Hong Kong's local interests on financial services. In addition, there will be several invited lectures in the main conference. Hong Kong is one of the most dynamic cities in the world with world-class facilities, easy accessibility, exciting entertainment, and high levels of service and professionalism. Come to Hong Kong! Visit this Eastern Pearl in this historical period before Hong Kong's return to China in 1997. ********************* CONFERENCE'S SCHEDULE ********************* Submission of paper (extended) March 10, 1996 Notification of acceptance May 1, 1996 Tutorial on Financial Engineering Sept. 24, 1996 Conference Sept. 25-27, 1996 *** *** *** The Conference Proceedings will be published by Springer Verlag. *** *** *** Registration forms, detailed tutorial information, invited talks and other related information will be available on the WWW site in due course. ********************************** Tutorials On Financial Engineering ********************************** 1. Professor John Moody, Oregon Graduate Institute, USA "Time Series Modeling: Classical and Nonlinear Approaches" 2. Professor Halbert White, University California, San Diego, USA "Option Pricing In Modern Finance Theory and the Relevance Of Artificial Neural Networks" 3. Professor A-P. N. Refenes, London Business School, UK "Neural Networks in Financial Engineering" ************* Keynote Talks ************* 1. Professor Shun-ichi Amari, Tokyo University. "Information Geometry of Neural Networks" 2. Professor Yaser Abu-Mostafa, California Institute of Technology, USA "The Bin Model for Learning and Generalization" 3. Professor Leo Breiman, University California, Berkeley, USA "Democratizing Predictors" 4. Professor Christoph von der Malsburg, Ruhr-Universitat Bochum, Germany "Scene Analysis Based on Dynamic Links" (tentatively) 5. Professor Erkki Oja, Helsinki University of Technology, Finland "Blind Signal Separation by Neural Networks " ************** Honored Talks ************** 1. Rolf Eckmiller, University of Bonn, Germany "Concerning the Development of Retina Implants with Neural Nets" 2. Mitsuo Kawato, ATR Human Information Processing Research Lab, Japan "Generalized Linear Model Analysis of Cerebellar Motor Learning" 3. Kunihiko Fukushima, Osaka University, Japan "To be announced" *** PLUS AROUND 20 INVITED PAPERS GIVEN BY WELL KNOWN RESEARCHERS IN THE FIELD. *** ***************** CONFERENCE TOPICS ***************** SCIENTIFIC TRACK: APPLICATION TRACK: ----------------- ------------------ * Theory * Foreign Exchange * Algorithms & Architectures * Equities & Commodities * Supervised Learning * Risk Management * Unsupervised Learning * Options & Futures * Hardware Implementations * Forecasting & Strategic Planning * Hybrid Systems * Government and Services * Neurobiological Systems * Geophysical Sciences * Associative Memory * Telecommunications * Visual & Speech Processing * Control & Modeling * Intelligent Control & Robotics * Manufacturing * Cognitive Science & AI * Chemical Engineering * Recurrent Net & Dynamics * Transportation * Image Processing * Environmental Engineering * Pattern Recognition * Remote Sensing * Computer Vision * Defense * Time Series Prediction * Multimedia Systems * Optimization * Document Processing * Fuzzy Logic * Medical Imaging * Evolutionary Computing * Biomedical Application * Other Related Areas * Other Related Applications ********************** SUBMISSION INFORMATION ********************** Authors are invited to submit one camera-ready original (do not staple) and five copies of the manuscript written in English on A4-format (or letter) white paper with 25 mm (1 inch) margins on all four sides, in one column format, no more than six pages (four pages preferred) including figures and references, single-spaced, in Times-Roman or similar font of 10 points or larger, without page numbers, and printed on one side of the page only. Electronic or fax submission is not acceptable. Additional pages will be charged at USD $50 per page. Centered at the top of the first page should be the complete title, author(s), affiliation, mailing, and email addresses, followed by an abstract (no more than 150 words) and the text. Each submission should be accompanied by a cover letter indicating the contacting author, affiliation, mailing and email addresses, telephone and fax number, and preference of track, technical session(s), and format of presentation, either oral or poster. All submitted papers will be refereed by experts in the field based on quality, clarity, originality, and significance. Authors may also retrieve the ICONIP style, "iconip.tex" and "iconip.sty" files for the conference by anonymous FTP at ftp.cs.cuhk.hk in the directory /pub/iconip96. The address for paper submissions, registration and information inquiries: ICONIP'96 Secretariat Department of Computer Science and Engineering The Chinese University of Hong Kong Shatin, N.T., Hong Kong Fax (852) 2603-5024 E-mail: iconip96 at cs.cuhk.hk http://www.cs.cuhk.hk/iconip96 ***************************** CONFERENCE'S REGISTRATION FEE ***************************** On & Before July 1, 1996, Member HKD $2,800 On & Before July 1, 1996, Non-Member HKD $3,200 Late & On-Site, Member HKD $3,200 Late & On-Site, Non-Member HKD $3,600 Student Registration Fee HKD $1,000 ====================================================================== General Co-Chairs ================= Omar Wing, CUHK Shun-ichi Amari, Tokyo U. Advisory Committee ================== International ------------- Yaser Abu-Mostafa, Caltech Michael Arbib, U. Southern Cal. Leo Breiman, UC Berkeley Jack Cowan, U. Chicago Rolf Eckmiller, U. Bonn Jerome Friedman, Stanford U. Stephen Grossberg, Boston U. Robert Hecht-Nielsen, HNC Geoffrey Hinton, U. Toronto Anil Jain, Michigan State U. Teuvo Kohonen, Helsinki U. of Tech. Sun-Yuan Kung, Princeton U. Robert Marks, II, U. Washington Thomas Poggio, MIT Harold Szu, US Naval SWC John Taylor, King's College London David Touretzky, CMU C. v. d. Malsburg, Ruhr-U. Bochum David Willshaw, Edinburgh U. Lofti Zadeh, UC Berkeley Asia-Pacific Region ------------------- Marcelo H. Ang Jr, NUS, Singapore Sung-Yang Bang, POSTECH, Pohang Hsin-Chia Fu, NCTU., Hsinchu Toshio Fukuda, Nagoya U., Nagoya Kunihiko Fukushima, Osaka U., Osaka Zhenya He, Southeastern U., Nanjing Marwan Jabri, U. Sydney, Sydney Nikola Kasabov, U. Otago, Dunedin Yousou Wu, Tsinghua U., Beijing Organizing Committee ==================== L.W. Chan (Co-Chair), CUHK K.S. Leung (Co-Chair), CUHK D.Y. Yeung (Finance), HKUST C.K. Ng (Publication), CityUHK A. Wu (Publication), CityUHK B.T. Low (Publicity), CUHK M.W. Mak (Local Arr.), HKPU C.S. Tong (Local Arr.), HKBU T. Lee (Registration), CUHK K.P. Chan (Tutorial), HKU H.T. Tsui (Industry Liaison), CUHK I. King (Secretary), CUHK Program Committee ================= Co-Chairs --------- Lei Xu, CUHK Michael Jordan, MIT Erkki Oja, Helsinki U. of Tech. Mitsuo Kawato, ATR Members ------- Yoshua Bengio, U. Montreal Jim Bezdek, U. West Florida Chris Bishop, Aston U. Leon Bottou, Neuristique Gail Carpenter, Boston U. Laiwan Chan, CUHK Huishen Chi, Peking U. Peter Dayan, MIT Kenji Doya, ATR Scott Fahlman, CMU Francoise Fogelman, SLIGOS Lee Giles, NEC Research Inst. Michael Hasselmo, Harvard U. Kurt Hornik, Technical U. Wien Yu Hen Hu, U. Wisconsin - Madison Jeng-Neng Hwang, U. Washington Nathan Intrator, Tel-Aviv U. Larry Jackel, AT&T Bell Lab Adam Kowalczyk, Telecom Australia Soo-Young Lee, KAIST Todd Leen, Oregon Grad. Inst. Cheng-Yuan Liou, National Taiwan U. David MacKay, Cavendish Lab Eric Mjolsness, UC San Diego John Moody, Oregon Grad. Inst. Nelson Morgan, ICSI Steven Nowlan, Synaptics Michael Perrone, IBM Watson Lab Ting-Chuen Pong, HKUST Paul Refenes, London Business School David Sanchez, U. Miami Hava Siegelmann, Technion Ah Chung Tsoi, U. Queensland Benjamin Wah, U. Illinois Andreas Weigend, Colorado U. Ronald Williams, Northeastern U. John Wyatt, MIT Alan Yuille, Harvard U. Richard Zemel, CMU Jacek Zurada, U. Louisville From lawrence at s4.elec.uq.edu.au Wed Feb 14 10:02:07 1996 From: lawrence at s4.elec.uq.edu.au (Steve Lawrence) Date: Thu, 15 Feb 1996 01:02:07 +1000 (EST) Subject: Paper on neural network simulation available Message-ID: <199602141502.BAA05378@s4.elec.uq.edu.au> It has been estimated that 85% of neural network researchers write their own simulation software. The following paper deals with correctness and efficiency in neural network simulation. We present several techniques which we have used in the implementation of our own simulator. We welcome your comments. http://www.elec.uq.edu.au/~lawrence - Australia http://www.neci.nj.nec.com/homepages/lawrence - USA Correctness, Efficiency, Extendability and Maintainability in Neural Network Simulation ABSTRACT A large number of neural network simulators are publicly available to researchers, many free of charge. However, when a new paradigm is being developed, as is often the case, the advantages of using existing simulators decrease, causing most researchers to write their own software. It has been estimated that 85% of neural network researchers write their own simulators. We present techniques and principles for the implementation of neural network simulators. First and foremost, we discuss methods for ensuring the correctness of results - avoiding duplication, automating common tasks, using assertions liberally, implementing reverse algorithms, employing multiple algorithms for the same task, and using extensive visualization. Secondly, we discuss efficiency concerns, including using appropriate granularity object-oriented programming, and pre-computing information whenever possible. From ronnyk at starry.engr.sgi.com Thu Feb 15 17:58:02 1996 From: ronnyk at starry.engr.sgi.com (Ronny Kohavi) Date: Thu, 15 Feb 1996 14:58:02 -0800 Subject: MLC++ : Machine learning library in C++ Message-ID: <199602152258.OAA15759@starry.engr.sgi.com> MLC++ is a machine learning library developed in C++. MLC++ is public domain and can be used free of charge, including use of the source code. MLC++ contains common induction algorithms, such as ID3, nearest-neighbors, naive-bayes, oneR (Holte), winnow, and decision tables, all written under a single framework. MLC++ also contains interfaces to common algorithms, such as C4.5, PEBLS, IB1-4, OC1, CN2. MLC++ contains wrappers to wrap around algorithms. These include: feature selection, discretization filters, automatic parameter setting for C4.5, bagging/combinining classifiers, and more. Finally, MLC++ contains common accuracy estimation methods, such as holdout, cross-validation, and bootstrap .632. Interfaces to existing algorithms are not hard to create and implementing new algorithms in MLC++ is possible with added benefits (some procedures work only on induction algorithms implemented in MLC++ as opposed to interfaced ones). Object code for MLC++ utilities is provided for Silicon Graphic machines running Irix 5.3. To contact us, send e-mail to: mlc at postofc.corp.sgi.com Visit our web page at: http://www.sgi.com/Technology/mlc/ -- Ronny Kohavi (ronnyk at sgi.com, http://robotics.stanford.edu/~ronnyk) From bogus@does.not.exist.com Thu Feb 15 14:41:41 1996 From: bogus@does.not.exist.com () Date: Thu, 15 Feb 1996 20:41:41 +0100 Subject: No subject Message-ID: <9602151941.AA06742@ti-doz10.fbe.fh-weingarten.de> From erik at bbf.uia.ac.be Thu Feb 15 08:38:40 1996 From: erik at bbf.uia.ac.be (Erik De Schutter) Date: Thu, 15 Feb 1996 13:38:40 GMT Subject: Crete Course in Computational Neuroscience Message-ID: <199602151338.NAA02343@kuifje.bbf.uia.ac.be> SECOND CALL CRETE COURSE IN COMPUTATIONAL NEUROSCIENCE AUGUST 25 - SEPTEMBER 20, 1996 CRETE, GREECE DIRECTORS: Erik De Schutter (University of Antwerp, Belgium) Idan Segev (Hebrew University, Jerusalem, Israel) Jim Bower (California Institute of Technology, USA) Adonis Moschovakis (University of Crete, Greece) The Crete Course in Computational Neuroscience introduces students to the practical application of computational methods in neuroscience, in particular how to create biologically realistic models of neurons and networks. The course consists of two complimentary parts. A distinguished international faculty gives morning lectures on topics in experimental and computational neuroscience. The rest of the day is spent learning how to use simulation software and how to implement a model of the system the student wishes to study. The first week of the course introduces students to the most important techniques in modeling single cells, networks and neural systems. Students learn how to use the GENESIS, NEURON, XPP and other software packages on their individual unix workstations. During the following three weeks the lectures will be more general, moving from modeling single cells and subcellular processes through the simulation of simple circuits and large neuronal networks and, finally, to system level models of the cortex and the brain. The course ends with a presentation of the student modeling projects. The Crete Course in Computational Neuroscience is designed for advanced graduate students and postdoctoral fellows in a variety of disciplines, including neurobiology, physics, electrical engineering, computer science and psychology. Students are expected to have a basic background in neurobiology as well as some computer experience. A total of 25 students will be accepted, the majority of whom will be from the European Union and affiliated countries. A tuition fee of 500 ECU ($700) covers lodging, travel from EC countries to Crete and all course-related expenses for European nationals. We specifically encourage applications from researchers younger than 35, from researchers who work in less-favoured regions, from women and from researchers from industry. We encourage students from the Far East and the USA to also apply to this inter- national course. More information and application forms can be obtained: - WWW access: http://bbf-www.uia.ac.be/CRETE/Crete_index.html - by mail: Prof. E. De Schutter Born-Bunge Foundation University of Antwerp - UIA, Universiteitsplein 1 B2610 Antwerp Belgium FAX: +32-3-8202541 - email: crete_course at bbf.uia.ac.be APPLICATION DEADLINE: April 10th, 1996. Applicants will be notified of the results of the selection procedures before May 1st. FACULTY: M. Abeles (Hebrew University, Jerusalem, Israel), D.J. Amit (University of Rome, Italy and Hebrew University, Israel), A. Berthoz (College de France, France), R.E. Burke (NIH, USA), C.E. Carr (University of Maryland, USA), A. Destexhe (Universite Laval, Canada), R.J. Douglas (Institute of Neuroinformatics, Zurich, Switzerland), T. Flash (Weizmann Institute, Rehovot, Israel), A. Grinvald (Weizmann Institute, Israel), J.J.B. Jack (Oxford University, England), C. Koch (California Institute of Technology, USA), H. Korn (Institut Pasteur, France), A. Lansner (Royal Institute Technology, Sweden), R. Llinas (New York University, USA), E. Marder (Brandeis University, USA), M. Nicolelis (Duke University, USA), J.M. Rinzel (NIH, USA), W. Singer (Max-Planck Institute, Frankfurt, Germany), S. Tanaka (RIKEN, Japan), A.M. Thomson (Royal Free Hospital, England), S. Ullman (Weizmann Institute, Israel), Y. Yarom (Hebrew University, Israel). The Crete Course in Computational Neuroscience is supported by the European Commission (4th Framework Training and Mobility of Researchers program) and by The Brain Science Foundation (Tokyo). Local administrative organization: the Institute of Applied and Computational Mathematics of FORTH (Crete, GR). From harnad at cogsci.soton.ac.uk Thu Feb 15 12:58:05 1996 From: harnad at cogsci.soton.ac.uk (Stevan Harnad) Date: Thu, 15 Feb 96 17:58:05 GMT Subject: Cerebellum and Sequences: BBS Call for Commentators Message-ID: <18060.9602151758@cogsci.ecs.soton.ac.uk> Below is the abstract of a forthcoming target article on: THE DETECTION AND GENERATION OF SEQUENCES AS A KEY TO CEREBELLAR FUNCTION. EXPERIMENTS AND THEORY by V. Braitenberg, D. Heck and F. Sultan This article has been accepted for publication in Behavioral and Brain Sciences (BBS), an international, interdisciplinary journal providing Open Peer Commentary on important and controversial current research in the biobehavioral and cognitive sciences. Commentators must be current BBS Associates or nominated by a current BBS Associate. To be considered as a commentator for this article, to suggest other appropriate commentators, or for information about how to become a BBS Associate, please send email to: bbs at ecs.soton.ac.uk or write to: Behavioral and Brain Sciences Department of Psychology University of Southampton Highfield, Southampton SO17 1BJ UNITED KINGDOM http://cogsci.ecs.soton.ac.uk/~harnad/bbs.html gopher://gopher.princeton.edu:70/11/.libraries/.pujournals ftp://ftp.princeton.edu/pub/harnad/BBS To help us put together a balanced list of commentators, please give some indication of the aspects of the topic on which you would bring your areas of expertise to bear if you were selected as a commentator. An electronic draft of the full text is available for inspection by anonymous ftp (or gopher or world-wide-web) according to the instructions that follow after the abstract. ____________________________________________________________________ THE DETECTION AND GENERATION OF SEQUENCES AS A KEY TO CEREBELLAR FUNCTION. EXPERIMENTS AND THEORY Valentino Braitenberg, Detlef Heck and Fahad Sultan Max-Planck-Institute for biological cybernetics Spemannstr. 38 72076 Tuebingen Germany KEYWORDS: Cerebellum; motor control; allometric relation; parallel fibers; synchronicity; spatio-temporal activity; sequence addressable memory; cerebro-cerebellar interaction. ABSTRACT:Starting from macroscopic and microscopic facts of cerebellar histology, we propose a new functional interpretation which may elucidate the role of the cerebellum in movement control. Briefly, the idea is that the cerebellum is a large collection of individual lines (Eccles' "beams") which respond specifically to certain sequences of events in the input and in turn produce sequences of signals in the output. We believe that the sequence in - sequence out mode operation is as typical for the cerebellar cortex as the transformation of sets into sets of active neurons is typical for the cerebral cortex, and that both the histological differences between the two and their reciprocal functional interactions become understandable in the light of this dichotomy. The response of Purkinje cells to sequences of stimuli in the mossy fiber system was shown experimentally by Heck on surviving slices of rat and guinea pig cerebellum. Sequential activation of a row of eleven stimulating electrodes in the granular layer, imitating a "movement" of the stimuli along the folium, produces a powerful volley in the parallel fibers which strongly excites Purkinje cells, as evidenced by intracellular recording. The volley, or "tidal wave" has maximal amplitude when the stimulus moves towards the recording site at the speed of conduction in parallel fibers, and much smaller amplitudes for lower or higher "velocities". The succession of stimuli has no effect when they "move" in the opposite direction. Synchronous activation of the stimulus electrodes also had hardly any effect. We believe that the sequences of mossy fiber activation which normally produce this effect in the intact cerebellum are a combination of motor planning, relayed to the cerebellum by the cerebral cortex, and information about ongoing movement, reaching the cerebellum from the spinal cord. The output elicited by the specific sequence to which a "beam" is tuned may well be a succession of well timed inhibitory volleys "sculpting" the motor sequences so as to adapt them to the complicated requirements of the physics of a multi-jointed system. -------------------------------------------------------------- To help you decide whether you would be an appropriate commentator for this article, an electronic draft is retrievable by anonymous ftp from ftp.princeton.edu according to the instructions below (the filename is bbs.braitenberg). Please do not prepare a commentary on this draft. Just let us know, after having inspected it, what relevant expertise you feel you would bring to bear on what aspect of the article. ------------------------------------------------------------- These files are also on the World Wide Web and the easiest way to retrieve them is with Netscape, Mosaic, gopher, archie, veronica, etc. Here are some of the URLs you can use to get to the BBS Archive: http://www.princeton.edu/~harnad/bbs.html http://cogsci.ecs.soton.ac.uk/~harnad/bbs.html gopher://gopher.princeton.edu:70/11/.libraries/.pujournals ftp://ftp.princeton.edu/pub/harnad/BBS/bbs.braitenberg ftp://cogsci.ecs.soton.ac.uk/pub/harnad/BBS/bbs.braitenberg To retrieve a file by ftp from an Internet site, type either: ftp ftp.princeton.edu or ftp 128.112.128.1 When you are asked for your login, type: anonymous Enter password as queried (your password is your actual userid: yourlogin at yourhost.whatever.whatever - be sure to include the "@") cd /pub/harnad/BBS To show the available files, type: ls Next, retrieve the file you want with (for example): get bbs.braitenberg When you have the file(s) you want, type: quit ---------- Where the above procedure is not available there are two fileservers: ftpmail at decwrl.dec.com and bitftp at pucc.bitnet that will do the transfer for you. To one or the other of them, send the following one line message: help for instructions (which will be similar to the above, but will be in the form of a series of lines in an email message that ftpmail or bitftp will then execute for you). ------------------------------------------------------------- From goldfarb at unb.ca Fri Feb 16 13:30:54 1996 From: goldfarb at unb.ca (Lev Goldfarb) Date: Fri, 16 Feb 1996 14:30:54 -0400 (AST) Subject: Workshop: What is inductive learning? Message-ID: Dear connectionists: The following workshop should be of particular interest to the connectionist community, since not only the topic itself was motivated by the resent developments in cognitive science and AI as they are being affected by connectionist movement, but also one of the main arguments that is going to be presented at the workshop is that to capture the main goals of the "connectionist movement" one needs to change fundamentally the underlying architectures from the numeric to the appropriately redefined "symbolic" architectures. My apologies if you receive multiple copies of this message. Please, post it. %*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*% Call for extended abstracts: WHAT IS INDUCTIVE LEARNING? On the foundations of AI and Cognitive Science Toronto - Canada May 20 - 21, 1996 A workshop in conjunction with the 11th Biennial Canadian AI Conference to be held at the Holiday Inn on King, Toronto during 21 - 24 May 1996 This workshop is a long overdue attempt to look at the inductive learning process (ILP) as the central process generating various representations of objects (events). To this end one needs, first of all, to have a working definition of the ILP, which has been lacking. Here is a starting point: ILP is the process that constructs class representation on the basis of a (small) finite set of examples, i.e. it constructs the INDUCTIVE class representation. This class representation must, in essence, provide INDUCTIVE definition (or construction) of the class. The constructed class representation, in turn, modifies the earlier representation of the objects (within the context specified by the ILP). Thus, any subsequent processes, e.g. pattern recognition, recall, problem solving, are performed on the basis of the newly constructed object (event) representations. To put it somewhat strongly, there are only inductive representations. Two main and strongly related reasons why ILPs have not been perceived as the very central processes are a lack of their adequate understanding and a lack of their satisfactory formal model. Most of the currently popular formal models of ILPs (including connectionist models) do not provide satisfactory inductive class representations. One can show that inductive class representations (in other words, representations of concepts and categories) cannot be adequately specified within the classical (numeric) mathematical models. We encourage all researchers (including graduate students) seriously interested in the foundations of the above areas to participate in the workshop. Both theoretical and applied contributions are welcomed (including, of course, those related to vision, speech, and language). While extended abstracts will be available at the workshop, we are planning to publish the expanded and reviewed versions of the presentations as a special issue of journal Pattern Recognition. EXTENDED ABSTRACT SUBMISSION ---------------------------- Submit a copy (or e-mail version) of a 3-4 page extended abstract to Lev Goldfarb ILP Workshop Chair Faculty of Computer Science University of New Brunswick P.O. Box 4400 E-mail: goldfarb at unb.ca Fredericton, N.B. E3B 5A3 Tel: 506-453-4566 Canada Fax: 506-453-3566 E-mail submissions are encouraged. Important dates: ---------------- Extended abstract due: Monday, March 25, 1996. Notification & review back to the author: Friday April 5, 1996. Final extended abstract due: Monday April 22, 1996. For more information about the Canadian AI Conference which is held in conjunction with two other conferences (Vision Interface and Graphics Interface) see: http://ai.iit.nrc.ca/cscsi/conferences/ai96.html %*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*% -- Lev Goldfarb http://wwwos2.cs.unb.ca/profs/goldfarb/goldfarb.htm Please e-mail to me: _____________________________________________________________________________ I intend to submit an abstract __ I plan to attend the workshop __ _____________________________________________________________________________ From mozer at neuron.cs.colorado.edu Fri Feb 16 16:28:57 1996 From: mozer at neuron.cs.colorado.edu (Michael C. Mozer) Date: Fri, 16 Feb 1996 14:28:57 -0700 Subject: NIPS*96 CALL FOR WORKSHOP PROPOSALS Message-ID: <199602162128.OAA09088@neuron.cs.colorado.edu> CALL FOR PROPOSALS NIPS*96 Post Conference Workshops December 6 and 7, 1996 Snowmass, Colorado Following the regular program of the Neural Information Processing Systems 1996 conference, workshops on current topics in neural information processing will be held on December 6 and 7, 1996, in Snowmass, Colorado. Proposals by qualified individuals interested in chairing one of these workshops are solicited. Past topics have included: Active Learning, Architectural Issues, Attention, Audition, Bayesian Analysis, Bayesian Networks, Benchmarking, Computational Complexity, Computational Molecular Biology, Control, Neuroscience, Genetic Algorithms, Grammars, Hybrid HMM/ANN Systems, Implementations, Music, Neural Hardware, Network Dynamics, Neurophysiology, On-Line Learning, Optimization, Recurrent Nets, Robot Learning, Rule Extraction, Self-Organization, Sensory Biophysics, Signal Processing, Symbolic Dynamics, Speech, Time Series, Topological Maps, and Vision. The goal of the workshops is to provide an informal forum for researchers to discuss important issues of current interest. There will be two workshop sessions a day, for a total of six hours, with free time in between for ongoing individual exchange or outdoor activities. Concrete open and/or controversial issues are encouraged and preferred as workshop topics. Representation of alternative viewpoints and panel-style discussions are particularly encouraged. Workshop organizers will have responsibilities including: 1) coordinating workshop participation and content, which involves arranging short informal presentations by experts working in an area, arranging for expert commentators to sit on a discussion panel and formulating a set of discussion topics, etc. 2) moderating or leading the discussion and reporting its high points, findings, and conclusions to the group during evening plenary sessions 3) writing a brief summary and/or coordinating submitted material for post-conference electronic dissemination. Submission Instructions ----------------------- Interested parties should submit via e-mail a short proposal for a workshop of interest by May 20, 1996. Proposals should include a title, a description of what the workshop is to address and accomplish, the proposed length of the workshop (one day or two days), the planned format (mini-conference, panel discussion, or group discussion, combinations of the above, etc), and the proposed number of speakers. Where possible, please also indicate potential invitees (particularly for panel discussions). Please note that this year we are looking for fewer "mini-conference" workshops and greater variety of workshop formats. Also, the time allotted to workshops has been increased to six hours each day. We strongly encourage that the organizers reserve a significant portion of time for open discussion. The proposal should motivate why the topic is of interest or controversial, why it should be discussed and who the targeted group of participants is. In addition, please send a brief resume of the prospective workshop chair, a list of publications, and evidence of scholarship in the field of interest. Submissions should include contact name, address, e-mail address, phone number and fax number if available. Proposals should be mailed electronically to mpp at watson.ibm.com. All proposals must be RECEIVED by May 20, 1996. If e-mail is unavailable, mail so as to arrive by the deadline to: NIPS*96 Workshops c/o Michael P. Perrone IBM T. J. Watson Research Center P.O. Box 218, 36-207 Yorktown Heights, NY 10598 Questions may be addressed to either of the Workshop Co-Chairs: Michael P. Perrone Steven J. Nowlan IBM T.J. Watson Research Center Motorola, Lexicus Division mpp at watson.ibm.com steven at lexicus.mot.com PROPOSALS MUST BE RECEIVED BY MAY 20, 1996 -Please Post- From steven at strontium.lexicus.mot.com Fri Feb 16 19:24:10 1996 From: steven at strontium.lexicus.mot.com (Steve Nowlan) Date: Fri, 16 Feb 1996 16:24:10 -0800 Subject: Position Available Message-ID: <9602161624.ZM14139@strontium.lexicus.mot.com> Please respond to the address at the end of the article, thank you. Motorola, Lexicus Division is currently seeking qualified applicants for the position of Recognition Scientist. Lexicus, a Division of Motorola, specializes in handwriting and speech recognition products for the mobile and wireless markets. Located in Palo Alto, California, across from Stanford University, the company numbers about 50 people, and is continuing to grow. JOB DESCRIPTION This individual will join a team of scientists working on technology for hand writing and speech recognition applications. Application areas range from low-cost embedded systems to high-end work station applications. The technology areas of interest include pattern recognition, vector quantization, neural networks, hidden Markov models and statistical language modeling. EDUCATION & EXPERIENCE The ideal candidate has an established record of significant contributions to research efforts in an academic or industrial environment and a strong background in statistical methods of pattern recognition. Specific skills include: * knowledge of current state of the art algorithms for pattern recognition and mathematical and statistical analysis/modeling techniques * the ability to work within a group to quickly implement and evaluate algorithms in a UNIX/C/C++ environment. Asian Language Skills also desirable. Applicants should be interested in developing state of the art technology and moving that technology into products in a rapid research/development cycle. SALARY: Depending on experience Please send resumes and a list of references (and include the title of the position) to: Human Resources Lexicus, a Division of Motorola 490 California Avenue, Suite 300 fax: (415) 323-4772 Palo Alto, CA 94306 email: hr at lexicus.mot.com Debbie Mayer Human Resources Motorola, Lexicus Division 490 California Ave, Suite #300 Palo Alto, CA 94306 Tel: (415) 617-1115 Fax: (415) 323-4772 From istvan at psych.ualberta.ca Mon Feb 19 19:23:42 1996 From: istvan at psych.ualberta.ca (Istvan Berkeley) Date: Mon, 19 Feb 1996 17:23:42 -0700 Subject: Intl. NN Workshop Announcement Message-ID: APPLICATIONS OF CONNECTIONISM IN COGNITIVE SCIENCE: AN INTERNATIONAL WORKSHOP On May 25-27, 1996, there will be a major international workshop at Carleton University, Ottawa, Canada, on the latest connectionist modelling techniques in cognitive science. The list of presenters, given below, includes some of the founders of contemporary PDP techniques, as well as younger researchers whose work stands at the forefront of new approaches and applications. Along with formal presentations, mornings will be devoted to demonstrations of the newest PDP software of potential interest to cognitive scientists, for which purpose each participant will have access to a workstation. Principal speakers: David E. Rummelhart, Stanford University Jerome A. Feldman, University of California at Berkeley Paul Skokowski, Stanford University Christopher Thornton, University of Sussex John Bullinaria, Edinburgh University Malcolm Forster, University of Wisconsin at Madison Istvan Berkeley, University of Alberta Each talk will be followed by an arranged commentary, and general discussion. For further information on registration procedures, fees and accommodations, please contact either Andrew Brook Department of Interdisciplinary Studies Carleton University Ottawa, Ontario CANADA K1S 5B6 or Don Ross Department of Philosophy Morisset Hall University of Ottawa Ottawa, Ontario CANADA K1N 6N5 Istvan S. N. Berkeley, email: istvan at psych.ualberta.ca Biological Computation Project & Department of Philosophy, c/o 4-108 Humanities Center University of Alberta Edmonton, Alberta Tel: +1 403 436 4182 T6G 2E5, Canada Fax: +1 403 492 9160 From imlm at tuck.cs.fit.edu Mon Feb 19 22:12:15 1996 From: imlm at tuck.cs.fit.edu (IMLM Workshop (pkc)) Date: Mon, 19 Feb 1996 22:12:15 -0500 Subject: 2nd CFP: AAAI-96 Workshop on Integrating Multiple Learned Models Message-ID: <199602200312.WAA13801@tuck.cs.fit.edu> ********************************************************************* Paper submission deadline: March 18, 1996 ********************************************************************* CALL FOR PAPERS/PARTICIPATION INTEGRATING MULTIPLE LEARNED MODELS FOR IMPROVING AND SCALING MACHINE LEARNING ALGORITHMS to be held in conjunction with AAAI 1996 (collocated with KDD-96, UAI-96, and IAAI-96) Portland, Oregon August 1996 Most modern machine learning research uses a single model or learning algorithm at a time, or at most selects one model from a set of candidate models. Recently however, there has been considerable interest in techniques that integrate the collective predictions of a set of models in some principled fashion. With such techniques often the predictive accuracy and/or the training efficiency of the overall system can be improved, since one can "mix and match" among the relative strengths of the models being combined. The goal of this workshop is to gather researchers actively working in the area of integrating multiple learned models, to exchange ideas and foster collaborations and new research directions. In particular, we seek to bring together researchers interested in this topic from the fields of Machine Learning, Knowledge Discovery in Databases, and Statistics. Any aspect of integrating multiple models is appropriate for the workshop. However we intend the focus of the workshop to be improving prediction accuracies, and improving training performance in the context of large training databases. More precisely, submissions are sought in, but not limited to, the following topics: 1) Techniques that generate and/or integrate multiple learned models. In particular, techniques that do so by: * using different training data distributions (in particular by training over different partitions of the data) * using different output classification schemes (for example using output codes) * using different hyperparameters or training heuristics (primarily as a tool for generating multiple models) 2) Systems and architectures to implement such strategies. In particular: * parallel and distributed multiple learning systems * multi-agent learning over inherently distributed data A paper need not be submitted to participate in the workshop, but space may be limited so contact the organizers as early as possible if you wish to participate. The workshop format is planned to encompass a full day of half hour presentations with discussion periods, ending with a brief period for summary and discussion of future activities. Notes or proceedings for the workshop may be provided, depending on the submissions received. Submission requirements: i) A short paper of not more than 2000 words detailing recent research results must be received by March 18, 1996. ii) The paper should include an abstract of not more than 150 words, and a list of keywords. Please include the name(s), email address(es), address(es), and phone number(s) of the author(s) on the first page. The first author will be the primary contact unless otherwise stated. iii) Electronic submissions in postscript or ASCII via email are preferred. Three printed copies (preferrably double-sided) of your submission are also accepted. iv) Please also send the title, name(s) and email address(es) of the author(s), abstract, and keywords in ASCII via email. Submission address: imlm at cs.fit.edu Philip Chan IMLM Workshop Computer Science Florida Institute of Technology 150 W. University Blvd. Melbourne, FL 32901-6988 407-768-8000 x7280 (x8062) 407-984-8461 (fax) Important Dates: Paper submission deadline: March 18, 1996 Notification of acceptance: April 15, 1996 Final copy: May 13, 1996 Chairs: Salvatore Stolfo, Columbia University sal at cs.columbia.edu David Wolpert, Santa Fe Institute dhw at santafe.edu Philip Chan, Florida Institute of Technology pkc at cs.fit.edu General Inquiries: Please address general inquiries to one of the chairs or send them to: imlm at cs.fit.edu Up-to-date workshop information is maintained on WWW at: http://www.cs.fit.edu/~imlm/ or http://cs.fit.edu/~imlm/ From zhuh at helios.aston.ac.uk Tue Feb 20 11:48:58 1996 From: zhuh at helios.aston.ac.uk (zhuh) Date: Tue, 20 Feb 1996 16:48:58 +0000 Subject: Paper available: cross validation Message-ID: <8961.9602201648@sun.aston.ac.uk> FTP-host: cs.aston.ac.uk FTP-file: neural/zhuh/nflcv.ps.Z URL: ftp://cs.aston.ac.uk/neural/zhuh/nflcv.ps.Z To appear in Neural Computation. ================================================================= No Free Lunch For Cross Validation Huaiyu Zhu and Richard Rohwer Neural Computing Research Group Aston University, Birmingham B4 7ET, UK Abstract -------- It is known theoretically that an algorithm cannot be good for an arbitrary prior. We show that in practical terms this also applies to the technique of ``cross validation'', which has been widely regarded as defying this general rule. Numerical examples are analysed in detail. Their implications to researches on learning algorithms are discussed. ================================================================= -- Huaiyu Zhu, PhD email: H.Zhu at aston.ac.uk Neural Computing Research Group http://neural-server.aston.ac.uk/People/zhuh Dept of Computer Science ftp://cs.aston.ac.uk/neural/zhuh and Applied Mathematics tel: +44 121 359 3611 x 5427 Aston University, fax: +44 121 333 6215 Birmingham B4 7ET, UK From scheier at ifi.unizh.ch Mon Feb 19 04:28:35 1996 From: scheier at ifi.unizh.ch (Christian Scheier) Date: Mon, 19 Feb 1996 10:28:35 +0100 Subject: Papers on Categorization in Autonomous Agents using Neural Networks Message-ID: The following papers deal with the problem of categorization/object recognition in autonomous agents (mobile robots). The papers can be retrieved from: ftp://claude.ifi.unizh.ch/pub/institute/ailab/techreports/ 96_01.ps.gz: Categorization in a real-world agent using haptic exploration and active perception Scheier, C. and Lambrinos, D. ABSTRACT An agent in the real world has to be able to make distinctions between different types of objects, i.e. it must have the competence of categorization. In mobile agents categorization is hard to achieve because there is a large variation in proximal sensory stimulation originating from the same object. In this paper we extend previous work on adaptive categorization in autonomous agents. The main idea of our approach is to include the agent's own actions into the classification process. In the experiments presented in this paper an agent equipped with an active vision and an arm-gripper system has to collect certain types of objects. The agent learns about the objects by actively exploring them. This exploration results in visual and haptic information that is used for learning. In essence, the categorization comes about via evolving reentrant connections between the haptic and the visual system. Results on the behavioral performance as well as the underlying internal dynamics are presented. 95_12.ps.gz: Adaptive Classification in Autonomous Agents Scheier, C. and Lambrinos, D. ABSTRACT One of the fundamental tasks facing autonomous robots is to reduce the many degrees of freedom of the input space by some sorts of classification mechanism. The sensory stimulation caused by one and the same object, for instance, varies enormously depending on lighting conditions, distance from object, orientation and so on. Efforts to solve this problem, say in classical computer vision, have only had limited success. In this paper a new approach towards classification in autonomous robots is proposed. It's cornerstone is the integration of the robots own actions into the classification process. More specifically, correlations through time-linked independent samples of sensory stimuli and of kinesthetic signals produced by self-motion of the system form the basis of the category learning. Thus, it is suggested that classification should not be seen as an isolated perceptual (sub-)system but rather as a {\it sensory-motor coordination} which comes about through a self-organizing process. These ideas are illustrated with a case study of an autonomous system that has to learn to distinguish between different types of objects. 95_05.ps.gz: Classification as Sensory-Motor Coordination: A Case Study on Autonomous Agents. Scheier, C. and Pfeifer, R. ABSTRACT In psychology classification is studied as a separate cognitive capacity. In the field of autonomous agents the robots are equipped with perceptual mechanisms for classifying objects in the environment, either by preprogramming or by some sorts of learning mechanisms. One of the well-known hard and fundamental problems is the one of perceptual aliasing, i.e. that the sensory stimulation caused by one and the same object varies enormously depending on distance from object, orientation, lighting conditions, etc. Efforts to solve this problem, say in classical computer vision, have only had limited success. In this paper we argue that classification cannot be viewed as a separate perceptual capacity of an agent but should be seen as a sensory-motor coordination which comes about through a self-organizing process. This implies that the whole organism is involved, not only sensors and neural circuitry. In this perspective, ``action selection'' becomes an integral part of classification. These ideas are illustrated with a case study of a robot that learns to distinguish between graspable and non-graspable pegs. For further informations and papers contact: -- __________________________________________________________________________ Christian Scheier Computer Science Department AI Lab University of Zurich tel: +41-1-257-4575 Winterthurerstrasse 190 fax: +41-1-363-0035 CH-8057 Switzerland http://josef.ifi.unizh.ch/groups/ailab/people/scheier.html ______________________________________ ____________________________________ From guy at taco.mpik-tueb.mpg.de Wed Feb 21 08:29:37 1996 From: guy at taco.mpik-tueb.mpg.de (Guy M. Wallis) Date: Wed, 21 Feb 1996 13:29:37 +0000 Subject: Papers available: "Object recognition and unsupervised learning" Message-ID: <9602211329.ZM182@taco.mpik-tueb.mpg.de> FTP-host: archive.cis.ohio-state.edu FTP-filename: /pub/neuroprose/wallisgm.ittrain.ps.Z FTP-filename: /pub/neuroprose/wallisgm.temporalobjrec1.ps.Z FTP-filename: /pub/neuroprose/wallisgm.temporalobjrec2.ps.Z ** Three papers available on the unsupervised ** ** learning of invariant object recognition ** The three papers listed above are now available for retrieval from the Neuroprose repository. All three papers discuss learning to associate different views of objects on the basis of their appearance in time as well as their spatial appearance. The papers are also available directly from my home page, along with a copy of my PhD thesis which, I should warn you, is rather long: http://www.mpik-tueb.mpg.de/people/personal/guy/guy.html -------------------------------------------------------------------------------- PaperI: A Model of Invariant Object Recognition in the Visual System ABSTRACT Neurons in the ventral stream of the primate visual system exhibit responses to the images of objects which are invariant with respect to natural transformations such as translation, size, and view. Anatomical and neurophysiological evidence suggests that this is achieved through a series of hierarchical processing areas. In an attempt to elucidate the manner in which such representations are established, we have constructed a model of cortical visual processing which seeks to parallel many features of this system, specifically the multi-stage hierarchy with its topologically constrained convergent connectivity. Each stage is constructed as a competitive network utilising a modified Hebb-like learning rule, called the trace rule, which incorporates previous as well as current neuronal activity. The trace rule enables neurons to learn about whatever is invariant over short time periods (e.g. 0.5 s) in the representation of objects as the objects transform in the real world. The trace rule enables neurons to learn the statistical invariances about objects during their transformations, by associating together representations which occur close together in time. We show that by using the trace rule training algorithm the model can indeed learn to produce transformation invariant responses to natural stimuli such as faces. Submitted to Journal of Computational Neuroscience 32 pages 1.6 Mb compressed ftp://archive.cis.ohio-state.edu/pub/neuroprose/wallisgm.ittrain.ps.Z or ftp://ftp.mpik-tueb.mpg.de/pub/guy/jcns7.ps.Z -------------------------------------------------------------------------------- PaperII: Optimal, Unsupervised Learning in Invariant Object Recognition ABSTRACT A means for establishing transformation invariant representations of objects at the single cell level is proposed and analysed. The association of views of objects is achieved by using both the temporal order of the presentation of these views, as well as their spatial similarity. Assuming knowledge of the distribution of presentation times, an optimal linear learning rule is derived. If we assume that objects are viewed with presentation times that are approximately Jeffrey's distributed, then the optimal learning rule is very well approximated using a simple exponential temporal trace. Simulations of a competitive network trained on a character recognition task are then used to highlight the success of this learning rule in relation to simple Hebbian learning, and to show that the theory can give quantitative predictions for the optimal parameters for such networks. Submitted to Neural Computation 15 pages 180 Kb compressed ftp://archive.cis.ohio-state.edu/pub/neuroprose/wallisgm.temporalobjrec1.ps.Z or ftp://ftp.mpik-tueb.mpg.de/pub/guy/nc.ps.Z -------------------------------------------------------------------------------- PaperIII: Using Spatio-Temporal Correlations to Learn Invariant Object Recognition ABSTRACT A competitive network is described which learns to classify objects on the basis of temporal as well as spatial correlations. This is achieved by using a Hebb-like learning rule which is dependent upon prior as well as current neural activity. The rule is shown to be capable of outperforming a supervised rule on the cross-validation test of an invariant character recognition task, given a relatively small training set. It is also shown to outperform the supervised version of Fukushima's Neocognitron, on a larger training set. Submitted to Neural Networks 13 pages 110 Kb compressed ftp://archive.cis.ohio-state.edu/pub/neuroprose/wallisgm.temporalobjrec2.ps.Z or ftp://ftp.mpik-tueb.mpg.de/pub/guy/nn.ps.Z -------------------------------------------------------------------------------- -- ----------------------------------------------------------- _/ _/ _/_/_/ _/_/_/ Guy Wallis _/_/ _/_/ _/ _/ _/ Max-Planck Institut f"ur _/ _/ _/ _/_/_/ _/ Biologische Kybernetik _/ _/ _/ _/ Spemannstr. 38 _/ _/ _/ _/_/_/ 72076 T"ubingen, Germany http://www.mpik-tueb.mpg.de/ TEL: +49-7071/601-630 Email: guy at mpik-tueb.mpg.de FAX: +49-7071/601-575 ----------------------------------------------------------- From jfeldman at ICSI.Berkeley.EDU Wed Feb 21 11:12:42 1996 From: jfeldman at ICSI.Berkeley.EDU (Jerry Feldman) Date: Wed, 21 Feb 1996 08:12:42 -0800 Subject: shift invariance Message-ID: <9602210812.ZM26438@ICSI.Berkeley.edu> Shift invariance is the ability of a neural system to recognize a pattern independent of where appears on the retina. It is generally understood that this property can not be learned by neural network methods, but I have not seen a published proof. A "local" learning rule is one that updates the input weights of a unit as a function of the unit's own activity and some performance measure for the network on the training example. All biologically plausible learning rules, as well as all backprop variants, are local in this sense. It is easy to show that no local rule can learn shift invariance. Consider learning binary strings with one occurrence of the sequence 101 and otherwise all zeros. First consider length 5; there are only 3 positive examples: 10100 01010 00101 Suppose that the training data does not include the middle example. The two positive training examples have no 1's in common with the withheld example. There will be no positive examples with a 1 in position 2 or 4 and many negative examples. Thus there is no correlation between a 1 in position 2 or 4 and a good example so no local training rule will learn the correct classification. A similar argument extends to binary strings of arbitrary length so an arbitrarily small fraction of the training data can be omitted and still no local updating rules will suffice to learn shift invariance. The one dimensional case of shift invariance can be handled by treating each string as a sequence and learning a finite-state acceptor. But the methods that work for this are not local or biologically plausible and don't extend to two dimensions. The unlearnability of shift invarince is not a problem in practice because people use preprocessing, weight sharing or other techniques to get shift invariance where it is known to be needed. However, it does pose a problem for the brain and for theories that are overly dependent on learning. From rolf at cs.rug.nl Thu Feb 22 06:56:06 1996 From: rolf at cs.rug.nl (rolf@cs.rug.nl) Date: Thu, 22 Feb 1996 12:56:06 +0100 Subject: shift invariance Message-ID: Dear Jerry, a short comment regarding your posting. It is no wonder that you have not seen a proof because it is simply not true that neural networks cannot do shift invariant recognition (SIR). If SIR is formalized it is most probably a computable function and ANNs can at least approximate all computable functions. No problem on a fundamental level here. K. Fukushima has shown a long time ago that a network can be wired up to do it. Nevertheless, invariant recognition seems to be a fundamental property of the visual system. It is not a _natural_ property of ANNs in the sense that you just give them a lot of natural stimuli and they develop the capability on their own. That, of course, makes one think if ANNs are a good model of the visual system, or if there is still a major point missing. I do not quite know what to make of your remark about ``theories overly dependent on learning''. If you have a better concept to offer than learning from experience I will be glad to hear about it. OK, you can say that there is ingenious machinery that does it, and the wiring is done by the genetic code, and we can go on and think about different things, but I do not consider that satisfactory explanation. I am looking forward to a discussion on the topic. Rolf +---------------------------------------------------------------------------+ | Rolf P. W"urtz | mailto:rolf at cs.rug.nl | URL: http://www.cs.rug.nl/~rolf/ | | Department of Computing Science, University of Groningen, The Netherlands | +---------------------------------------------------------------------------+ From wimw at mbfys.kun.nl Thu Feb 22 09:18:27 1996 From: wimw at mbfys.kun.nl (Wim Wiegerinck) Date: Thu, 22 Feb 1996 15:18:27 +0100 (MET) Subject: Paper Available: How Dependencies between Successive Examples Affect On-Line Learning. Message-ID: <199602221418.PAA04038@septimius.mbfys.kun.nl> A non-text attachment was scrubbed... Name: not available Type: text Size: 2015 bytes Desc: not available Url : https://mailman.srv.cs.cmu.edu/mailman/private/connectionists/attachments/00000000/8f607bc7/attachment.ksh From robbie at bcs.rochester.edu Thu Feb 22 10:04:38 1996 From: robbie at bcs.rochester.edu (Robbie Jacobs) Date: Thu, 22 Feb 1996 10:04:38 -0500 Subject: undergraduate summer workshop Message-ID: <199602221504.KAA04738@broca.bcs.rochester.edu> Below is an announcement for a three-day summer workshop for undergraduate students interested in the brain and cognitive sciences. It would be appreciated if you could bring this to the attention of your students. Robert Jacobs =================================================================== UNDERGRADUATE WORKSHOP ON PERCEPTION, PLASTICITY, AND ACTION (FELLOWSHIPS AVAILABLE) August 8-10, 1996 University of Rochester Department of Brain and Cognitive Sciences, Center for Visual Science, and the Program in Neuroscience The University of Rochester will host a Summer Workshop for Undergraduates on August 8-10, 1996. The Workshop will consist of lectures and laboratory demonstrations by the Rochester faculty on the coordination of perceptual mechanisms with those that control movement. It will also focus on how this coordination is established during development and modified by experience. These issues will be approached from neural, behavioral, and computational perspectives. Fellowships covering travel and living expenses will be provided for 20 students. Preference will be given to Juniors who plan to pursue advanced study in the behavioral and neural sciences. Request an application: * by email: judy at cvs.rochester.edu * by phone: (716) 275-2459 or fax: (716) 271-3043 * in writing: Dr. David R. Williams, Brain and Cognitive Sciences, Meliora Hall, University of Rochester, Rochester, NY 14627 * or apply electronically by visiting the web site at: http://www.bcs.rochester.edu/ug_workshop/ From hinton at cs.toronto.edu Thu Feb 22 10:25:32 1996 From: hinton at cs.toronto.edu (Geoffrey Hinton) Date: Thu, 22 Feb 1996 10:25:32 -0500 Subject: shift invariance In-Reply-To: Your message of Wed, 21 Feb 1996 11:12:42 -0500. Message-ID: <96Feb22.102548edt.882@neuron.ai.toronto.edu> Dear Jerry, Its a long time since we had a really good disagreement. Contrary to your assertions, shift invariance can be learned by backpropagation. It was one of the problems that I tried when fiddling about with backprop in the mid 80's. I published a paper demonstrating this in an obscure conference proceedings: Hinton, G.~E. (1987) Learning translation invariant recognition in a massively parallel network. In Goos, G. and Hartmanis, J., editors, PARLE: Parallel Architectures and Languages Europe, pages~1--13, Lecture Notes in Computer Science, Springer-Verlag, Berlin. So far as I know, this is also the first paper to demonstrate that weight decay can make a really big difference in generalization performance. It reduced the error rate from about 45% to about 6%, though I must confess that the amount of weight decay was determined by using the test set (as was usual in our sloppy past). I used a one dimensional "retina" with 12 pixels. On this retina there was one instance of a shape at a time. The "shape" consisted of two bright "boundary" pixels with 4 pixels between them. The 4 pixels in the sandwich could have any of the 16 binary patterns, so there were 16 very confusable shapes. For example, here are two instances of the shape corresponding to the binary number 0011 (strip off the boundary bits before reading the number): 000100111000 010011100000 The retina had wraparound so that each shape could occur in 12 different positions. This seems to me to be exactly the kind of data that you think cannot be learned. In other words, if I train a neural network to identitfy some instances of the shapes you think that it couldnt possibly generalize to instances in other positions. Of course, once you understand why it can generalize, you will decide on a way to exclude this kind of example, but so far it seems to me to fit your assertion about what cannot be done. The network had two hidden layers. In the first hidden layer there were 60 units divided into 12 groups of 5 with local receptive fields. So we are telling it about locality, but not about translation. Within each group, all 5 units receive input from the same 6 adjacent input units. In the next hidden layer there is a bottleneck of only 6 units (I didnt dare use 4), so all the information used to make the final decision has to be represented in a distributed pattern of activity in the bottleneck. There are 16 output units for the 16 shapes. The idea behind the network is as follows: Shapes are composed of features that they share with other shapes. Although we may not have seen a particular shape in a novel position, we will presumably have seen its features in those positions before. So if we have already developed translation invariant feature detectors, and if we represent our knowledge of the shape in terms of these detectors, we can generalize across translation. The "features" in this example are the values of the four pixels inside the sandwich. A hidden unit in the first hidden layer can see the whole sandwich, so it could learn to respond to the conjunction of the two boundary pixels and ONE of the four internal "shape feature" pixels. Its weights might look like this: ...+.+..+.. It would then be a position-dependent feature detector. In each location we have five such units to enable the net to develop all 4 position-dependent feature detectors (or funny combinations of them that span the space). In the next layer, we simply perform an OR for the 12 different copies of the same feature detector in the 12 different positions. So in the next layer we have position-independent feature detectors. Finally the outgoing connections from this layer represent the identity of a shape in terms of its position-independent feature detectors. Notice that the use of an OR should encourage the net to choose equivalent position-dependent feature detectors in the different locations, even though there is no explicit weight sharing. The amazing thing is that simply using backprop on the shape identities is sufficent to create this whole structure (or rather one of the zillions of mangled versions of it that uses hard-to-decipher distributed representations). Thanks to kevin lang for writing the Convex code that made this simulation possible in 1987. Please note that the local connectivity was NOT NECESSARY to get generalization. Without it the net still got 20/32 correct (guessing would be 2/32). Now, I dont for a moment believe that human shape perception is learned entirely by backpropagating from object identity labels. The simulation was simply intended to answer the philosophical point about whether this was impossible. Its taken nearly a decade for someone to come out and publicly voice the widely held belief that there is no way in hell a network could learn this. Thanks Geoff PS: I do think that there may be some biological mileage in the idea that local, position-dependent feature detectors are encouraged to break symmetry in the same way in order for later stages of processing to be able to achieve position independence by just performing an OR. An effect like this ought to occur in slightly less unrealistic models like Helmholtz machines. From pf2 at st-andrews.ac.uk Thu Feb 22 10:39:25 1996 From: pf2 at st-andrews.ac.uk (Peter Foldiak) Date: Thu, 22 Feb 1996 15:39:25 GMT Subject: shift invariance Message-ID: <199602221539.PAA24901@psych.st-andrews.ac.uk> > Shift invariance is the ability of a neural system to recognize a pattern > independent of where appears on the retina. It is generally understood that > this property can not be learned by neural network methods, but I have > not seen a published proof. Minsky & Papert: Perceptons, 1969, MIT Press, p 54 : "... order-1 predicates invariant under the usual geometric groups can do nothing more than define simple ">=m"-type inequalities on the size or "area" of the figures. In particular, taking the translation group G we see that no first-order perceptron can distinguish the A's in the figure on p. 46 from some other translation-invarian set of figures of the same area." This doesn't say anything about multi-layer nets, i.e. you can combine feaures in a way that will be invariant. Peter Foldiak From goldfarb at unb.ca Thu Feb 22 16:56:11 1996 From: goldfarb at unb.ca (Lev Goldfarb) Date: Thu, 22 Feb 1996 17:56:11 -0400 (AST) Subject: On the structure of connectionist models Message-ID: Dear connectionists: Since my posting of the workshop announcement (What is inductive learning?) several days ago, I was asked to clarify what I meant when I said that "one can show that inductive class representations (in other words, representations of concepts and categories) cannot be adequately specified within the classical (numeric) mathematical models" including, of course, connectionist models. Here are some general ideas from the paper which will be presented at the workshop. The following observations about the STRUCTURE of inductive learning models strongly suggest why the classical (numeric) mathematical models will be able to support only "weak" inductive learning models, i.e. the models that can perform reasonably only in VERY rigidly delineated environments. The questions I'm going to address in this posting on the one hand lay at the very foundations of connectionism and on the other hand are relatively simple, provided one keeps in mind that we are discussing the overall FORMAL STRUCTURE of the learning models (which requires a relatively high level of abstraction). Let's look at the structure of connectionist models through the very basic problem of inductive learning. In order to arrive at a useful formulation of the inductive learning problem and, at the same time, at a useful framework for solving the problem, I propose to proceed as follows. First and foremost, the inductive learning involves a finite set of data (objects from the class C) labeled either (C+, C-), positive and negative examples, or, more generally, simply C', examples. Since we want to compare quite different classes of models (e.g. symbolic and numeric), let us focus only on very general assumptions about the nature of the object representation (input) space: Postulate 1. Input space S satisfies a finite set A of axioms. (S, in fact, provide a formal specifications of all the necessary data properties; compare with the concept of abstract data type in computer science). Thus, for example, the vector (linear) space is defined by means of the well known set of axioms for vector addition and scalar multiplication. Next, let us attach the name "inductive class representation" (ICR) to the formal description (specification) of the class C obtained in a chosen model as a result of an inductive learning process: Postulate 2. In a learning model, ICR is specified in some (which?) formal manner. --------------------------------------------------------------------- | My first main point connects Postulate 2 to Postulate 1: ICR | | should be expressed in the "language" of the axioms from set A. | --------------------------------------------------------------------- For example, in a vector space ICR should be specified only in terms of the given data set plus the operations in the vector space, i.e. we are restricted to the spanned affine subspace or its approximation. The reason is quite simple: the only relationships that can be (mathematically) legitimately extracted from the input data are those that are expressible in the language of the input space S. Otherwise, we are, in fact, IMPLICITLY postulating some other relationships not specified in the input space by Postulate 1, and, therefore, the "discovery" of such implicit relationships in the data during the learning process is an illusion: such relationships are not "visible" in S. Thus, for example, "non-linear" relationships cannot be discovered from a finite data in a vector space, simply because a non-linear relationship is not part of the linear structure and, therefore, cannot be (mathematically) legitimately extracted from the finite input set of vectors in the vector space. What is happening (of necessity) in a typical connectionist model is that in addition to the set A of vector space axioms, some additional non-linear structure (determined by the class of non-linear functions chosen for the internal nodes of the NN) is being postulated IMPLICITLY from the beginning. Question: What does this additional non-linear structure has to do with the finite input set of vectors? (In fact, there are uncountably many such non-linear structures and, typically, none of them is directly related to the structure of the vector space or the input set of vectors.) ----------------------------------------------------------------------- | My second main point is this: if S is a vector space, in both cases, | | whether we do or don't postulate in addition to the vector space | | axioms some non-linear structure (for the internal nodes), we are | | faced with the following important question. What are we learning | | during the learning process? Certainly, we are not learning any | | interesting ICR: the entire STRUCTURE is fixed before the learning | | process. | ----------------------------------------------------------------------- It appears, that this situation is inevitable if we choose one of the classical (numeric) mathematical structures to model the input space S. However, in an appropriately defined symbolic setting (i.e. with an appropriate dynamic metric structure, see my home page) the situation changes fundamentally. To summarize (but not everything is before your eyes), the "strong" (symbolic) inductive learning models offer the ICRs that are much more flexible than those offered by the classical (numeric) models. In other words, the appropriate symbolic models offer true INDUCTIVE class representations. [The latter is given by a subset of objects + the constructed finite set of (weighted) operations that can transform objects into objects.] Lev Goldfarb http://wwwos2.cs.unb.ca/profs/goldfarb/goldfarb.htm From dorffner at cns.bu.edu Thu Feb 22 21:05:34 1996 From: dorffner at cns.bu.edu (dorffner@cns.bu.edu) Date: Thu, 22 Feb 1996 21:05:34 -0500 (EST) Subject: shift invariance Message-ID: <199602230205.VAA13842@bucnsd> Hi fellow connectionists, I must say I'm a little puzzled by this discussion about shift invariance. It was started by Jerry Feldman by saying > Shift invariance is the ability of a neural system to recognize a pattern > independent of where appears on the retina. It is generally understood that > this property can not be learned by neural network methods, but I have > not seen a published proof. A "local" learning rule is one that updates the > input weights of a unit as a function of the unit's own activity and some > performance measure for the network on the training example. All biologically > plausible learning rules, as well as all backprop variants, are local in this > sense. Now I always thought that this is so obvious that it didn't need any proof. Geoff Hinton responded by disagreeing: > Contrary to your assertions, shift invariance can be learned by > backpropagation. It was one of the problems that I tried when fiddling about > with backprop in the mid 80's. I published a paper demonstrating this in an > obscure conference proceedings: He describes a model based on feature detectors and subsequent backpropagation that can actually generalize over different positions. He finishes by saying > The simulation was > simply intended to answer the philosophical point about whether this was > impossible. Its taken nearly a decade for someone to come out and publicly > voice the widely held belief that there is no way in hell a network could > learn this. > IMHO, there seems to be a misunderstanding of what the topic of discussion is here. I don't think that Jerry meant that no model consisting of neural network components could ever learn shift invariance. After all, there are many famous examples in visual recognition with neural networks (such as the Neocognitron, as Rolf W"urtz pointed out), and if this impossibility were the case, we would have to give up neural network research in perceptual modeling altogether. What I think Jerry meant is that any cascade of fully-connected feed-forward connection schemes between layers (including the perceptron and the MLP) cannot learn shift invariance. Now besides being obvious, this does raise some important questions, possibly weakening the fundamentals of connectionism. Let me explain why: - state spaces in connectionist layers (based on the assumption that activation patterns are viewed as vectors) span a Euclidean space, with each connection scheme that transfers patterns into another layer applying a certain kind of metric defining similarity. This metric is non-trivial, especially in MLPs, but it restricts the ways of what such a basic neural network component (i.e. fully connected feedforward) can view as similar. Patterns that are close in this space according to a distance measure, or patterns that have large orthogonal projections onto each other (in my analysis the basic similarity measure in MLPs) are similar according to this metric. Different patterns with a sub-pattern in different positions are obviously NOT. Neither are patterns which share common differences between components (e.g. the patterns (0.8 0.3) and (0.6 0.1)), and a whole bunch of other examples. That's why we have to be so careful about the right kind of preprocessing when we apply neural networks in engineering, and why we have to be equally careful in choosing the appropriate representations in connectionist cognitive modeling. - introducing feature detectors and other complex connectivities and learning schemes (weight sharing, or the OR Geoff mentioned) is a way of translating the original pattern space into a space where the similarity structures which we expect obey the said metric in state space again. It's the same thing we do in preprocessing (e.g. we apply an FFT to signals, since we cannot expect that the network can extract invariances in the frequency domain). - Geoff's model, necognitron, and many others do exactly that. Each single component (e.g. one feature detector) is restricted by the similarity metric mentioned above. But by applying non-linear functions, and by combining their output in a clever way they translate the original patterns into a new pattern space, where similarity corresponds to this metric again (e.g. for the final backprop network Geoff introduced). Now obviously, when we look at the human visual system, the brain does seem to do some kind of preprocessing, such as applying feature detectors, as well. So we're kind of safe here. But the above observation does make one think, whether the similarity metric a neural network basically applies is actually the right kind of basis for cognitive modeling. Think about it: By introducing complex wiring and learning schemes, and by carefully choosing representations, we go a long way to finally satisfy the actual neural network that has to do the job of extracting information from the patterns. Visual recognition is but one, although prominent, example. Now what makes us sure that deeper processes ARE of the kind a fully connected feedforward network can handle (i.e. that processes DO work on the said restricted kind of similarity metric)? Now I do not really think that we have a problem here. But some people recently have raised serious doubts. Some have suggested that perhaps replacing connectionist state spaces by the "space" that is spanned by attractors in a dynamical systems gives one a more appropriate metric. I am not suggesting this, I am just proposing that connectionists have to be alert in this respect, and keep questioning themselves whether the fundamental assumptions we're making are the appropriate ones. In this way I think Jerry's point is worth discussing. Just my 2 c. worth, Georg (email: georg at ai.univie.ac.at) P.S: I would like to acknowledge F.G. Winkler from Vienna for some of the ideas expressed in this message. From hicks at cs.titech.ac.jp Thu Feb 22 21:21:25 1996 From: hicks at cs.titech.ac.jp (hicks@cs.titech.ac.jp) Date: Fri, 23 Feb 1996 11:21:25 +0900 Subject: shift invariance In-Reply-To: Jerry Feldman's message of Wed, 21 Feb 1996 08:12:42 -0800 <9602210812.ZM26438@ICSI.Berkeley.edu> Message-ID: <199602230221.LAA04831@euclid.cs.titech.ac.jp> jfeldman at ICSI.Berkeley.EDU Wed, 21 Feb 1996 08:12:42 wrote:: >Shift invariance is the ability of a neural system to recognize a pattern >independent of where appears on the retina. It is generally understood that >this property can not be learned by neural network methods I disagree. As you state later, the network of neurons need only "share" weights. Sharing can be forced, or can occur by independent learnign of the same (but translated) data. > The unlearnability of shift invarince is not a problem in practice because >people use preprocessing, weight sharing or other techniques to get shift >invariance where it is known to be needed. However, it does pose a problem for >the brain and for theories that are overly dependent on learning. There are two obvious ways in which shift invariance could occur in "biological" or other learning systems. 1) Nature Some of part of the patterns of connectivity in the low level vision system are decided genetically and replicated automatically (like many cellular structures throughout the boody); in effect, a kind of weight sharing. Natural selection (learning through genetics) favors patterns of connectivity which will detect frequently appearing patterns; in effect weight learning. 2) Nurture The strengths of neuronal connections in the low level vision system self-adjust in some Hebbian scheme to frequently occuring patterns. The distribution of these patterns, in actual fact, is shift invariant. (If they aren't shift invariant then there's not much point in learning them as though they were shift invariant.) Respectfully Yours, Craig Hicks From juergen at idsia.ch Fri Feb 23 03:15:27 1996 From: juergen at idsia.ch (Juergen Schmidhuber) Date: Fri, 23 Feb 96 09:15:27 +0100 Subject: shifts Message-ID: <9602230815.AA04599@fava.idsia.ch> Jerry Feldman writes: >>> Shift invariance is the ability of a neural system to recognize a pattern independent of where appears on the retina. It is generally understood that this property can not be learned by neural network methods, but I have not seen a published proof. [...] It is easy to show that no local rule can learn shift invariance. [...] The one dimensional case of shift invariance can be handled by treating each string as a sequence and learning a finite-state acceptor. But the methods that work for this are not local or biologically plausible and don't extend to two dimensions. <<< It might be of interest to note that the situation changes if the neural system includes a controller that is able to generate retina- movements (to change the position of the image on the retina). There are gradient-based controllers that (in certain cases) can *learn* appropriate, 2-dimensional retina shifts. They are `local' to the extent backprop through time is `local'. See, e.g., Schmidhuber & Huber (1991): Learning to generate fovea trajectories for target detection. Int. Journal of Neural Systems, 2(1 & 2):135-141. Juergen Schmidhuber, IDSIA From karim at ax1303.physik.uni-marburg.de Fri Feb 23 09:59:31 1996 From: karim at ax1303.physik.uni-marburg.de (Karim Mohraz) Date: Fri, 23 Feb 1996 15:59:31 +0100 Subject: FlexNet - a flexible neural network construction algorithm Message-ID: <9602231459.AA28657@ax1303.physik.uni-marburg.de> The following paper is available via WWW FlexNet - a flexible neural network construction algorithm Abstract Dynamic neural network algorithms are used for automatic network design in order to avoid time consuming search for finding an appropriate network topology with trial & error methods. The new FlexNet algorithm, unlike other network construction algorithms, does not underlie any constraints regarding the number of hidden layers and hidden units. In addition different connection strategies are available, together with candidate pool training and the option of freezing weights. Test results on 3 different benchmarks showed higher generalization rates for FlexNet compared to Cascade-Correlation and optimized MLP networks. Keywords: network construction, generalization, Cascade-Correlation. This paper has been accepted for publication in the European Symposium on Artificial Neural Networks (ESANN), Bruges, Belgium , April, 96. http://www.physik.uni-marburg.de/bio/mitarbei/karim/flexnet.ps (6 pages) Sorry, no hardcopies available ``` (o o) +-------------------oOO--(_)--OOo---------------------------------------------------+ Karim Mohraz Bereich Neuronale Netze & Fuzzy Logik Bayerisches Forschungszentrum fuer Wissensbasierte Systeme F O R W I S S Erlangen New address: AG Neurophysik, Universitaet Marburg, Germany Email: karim at bio.physik.uni-marburg.de WWW: http://www.physik.uni-marburg.de/bio/mitarbei/karim.html _ +-------------------o00--( )--00o---------------------------------------------------+ (o o) ''' From listerrj at helios.aston.ac.uk Fri Feb 23 09:59:39 1996 From: listerrj at helios.aston.ac.uk (Richard Lister) Date: Fri, 23 Feb 1996 14:59:39 +0000 Subject: Neural Computing Research Programmer post Message-ID: <8199.199602231459@sun.aston.ac.uk> ---------------------------------------------------------------------- Neural Computing Research Group ------------------------------- Dept of Computer Science and Applied Mathematics Aston University, Birmingham, UK Research Programmer ------------------- * Full details at http://www.ncrg.aston.ac.uk/ * Applications are invited for the post of Research Programmer within the Neural Computing Research Group (NCRG) at Aston University. The NCRG is now the largest academic research group in this area in the UK, and has an extensive and lively programme of research ranging from the theoretical foundations of neural computing and pattern recognition through to industrial and commercial applications. The Group is based in spacious accommodation in the University's Main Building, and is well equipped with its own network of Silicon Graphics and Sun workstations, supported by a full-time system administrator. The successful candidate will have the opportunity to contribute in the following areas: * development of real-world applications of neural networks in connection with a wide variety of industrial and commercial research contracts * providing software contributions in support of basic research projects * production of demonstration software for use in teaching a variety of neural network courses Most of the software will be developed in C++ and Matlab, on a high- power Silicon Graphics workstation with access to the Group's SGI Challenge supercomputer. The ideal candidate will have: * a good first degree in a numerate discipline * expertise in software development (preferably in C and C++) * a good understanding of neural networks * working knowledge of basic mathematics such as calculus and linear algebra * experience of working in a UNIX environment * willingness to undertake complex and challenging problems This post provides an excellent opportunity to learn new skills within an exciting team environment Conditions of Service --------------------- The appointment will be for an initial period of one year, with the possibility of subsequent renewal. Initial salary will be on the academic 1A or 1B scales up to 15,986. How to Apply ------------ If you wish to be considered for this position, please send a full CV, together with the names and addresses of at least 3 referees, to: Hanni Sondermann Neural Computing Research Group Department of Computer Science and Applied Mathematics Aston University Birmingham B4 7ET, U.K. Tel: (+44 or 01) 21 333 4631 Fax: (+44 or 01) 21 333 6215 e-mail: h.e.sondermann at aston.ac.uk ---------------------------------------------------------------------- ~~~~~~~~~~~~~~ Richard J. Lister r.j.lister at aston.ac.uk ~~~~~~~~~~~~~~~~ Research Assistant, Neural Computing Research Group Aston University, Birmingham B4 7ET, UK ~~~~~~~~~~~~~~~~~ http://www.ncrg.aston.ac.uk/~listerrj/ ~~~~~~~~~~~~~~~~~~ From kak at ee.lsu.edu Fri Feb 23 10:06:38 1996 From: kak at ee.lsu.edu (Subhash Kak) Date: Fri, 23 Feb 96 09:06:38 CST Subject: shift invariance Message-ID: <9602231506.AA12042@ee.lsu.edu> For feedback neural networks here are some more references on shift invariant learning: T. Maxwell et al, Transformation invariance using high order correlations in neural net architectures. Plasma Preprint UMLPF #88-125. Univ of Maryland, 1988. W. Widrow and R. Winter, Neural nets for adaptive filtering and adaptive pattern recognition. Computer, 21 (3):25-39, 1988. D.L. Prados and S.C. Kak, Shift invariant associative memory. IN VLSI for Artificial Intelligence, J.G. Delgado-Frias and W.R. Moore (eds.), pp. 185-197, Kluwer Academic Publishers, 1989. -Subhash Kak From franco at cim.mcgill.ca Fri Feb 23 10:43:43 1996 From: franco at cim.mcgill.ca (Francesco Callari) Date: Fri, 23 Feb 1996 15:43:43 GMT Subject: Paper on active 3D object recognition and sensor planning via ANN Message-ID: <199602231543.PAA15318@Poptart.McRCIM.McGill.EDU> The following paper deals with the problems of combining 3D shape information and class priors for the purposes of active, model-based object recognition and sensor planning. The proposed system correlates estimates of shape and class uncertainty to determine those sensor locations that best disambiguate the objects. The class and class sensitivity estimates are computed by an MLP network, trained using MacKay's "evidence" framework and put in the planning feedback loop of a mobile robot. FTP-host: ftp.cim.mcgill.ca FTP-file: /pub/people/franco/ambiguity96.ps.gz Active Recognition: Using Uncertainty to Reduce Ambiguity Francesco G. Callari and Frank P. Ferrie Centre for Intelligent Machines, McGill University 3480 University St., Montre\'al, Que., Canada, H3A 2A7 email: franco at cim.mcgill.ca, ferrie at cim.mcgill.ca Keywords: Active Vision, Control of Perception, Learning in Computer Vision ABSTRACT Ambiguity in scene information, due to noisy measurements and uncertain object models, can be quantified and actively used by an autonomous agent to efficiently gather new data and improve its information about the environment. In this work an information-based utility measure is used to derive from a learned classification of shape models an efficient data collection strategy, specifically aimed at increasing classification confidence when recognizing uncertain shapes. Promising experimental results with real data are reported. Submitted to: ICPR96. From oby at cs.tu-berlin.de Fri Feb 23 12:07:06 1996 From: oby at cs.tu-berlin.de (Klaus Obermayer) Date: Fri, 23 Feb 1996 18:07:06 +0100 Subject: No subject Message-ID: <199602231707.SAA29613@pollux.cs.tu-berlin.de> GRADUATE STUDENT POSITION A graduate student position is available at the CS-department of the Technical University of Berlin to study models of neural development. A major objective is to construct and investigate models for the formation of ocular dominance and orientation selectivity in striate cortex. The candidate is expected to join a close collaboration between theorists and experimentalists. Candidates should have experience in computational modelling. The position is available initially for two years. Salary is commensurate with BAT II a/2. Applicants should send their CV, list of publications, a letter describing their interest, and name, address and phone number of two references to: Prof. Klaus Obermayer phone: 49-30-314-73442 FR2-1, KON, Informatik 49-30-314-73120 Technische Universitaet Berlin fax: 49-30-314-73121 Franklinstrasse 28/29 e-mail: oby at cs.tu-berlin.de 10587 Berlin, Germany http://www.cs.tu-berlin.de/fachgebiete/kon/ From kak at ee.lsu.edu Fri Feb 23 18:55:23 1996 From: kak at ee.lsu.edu (Subhash Kak) Date: Fri, 23 Feb 96 17:55:23 CST Subject: Paper Message-ID: <9602232355.AA19148@ee.lsu.edu> The following paper =========================================== ON BIOLOGICAL CYCLES AND INTELLIGENCE By Subhash C. Kak may be obtained by ftp from the following location: ftp://gate.ee.lsu.edu/pub/kak/bio.ps.Z -------------------------------------------- Abstract: If intelligence is taken as a measure of the organism's ability to adapt to its environment, the question of the organism's sensitivity to the rhythms of the environment becomes important. In this paper we provide a summary, including a brief historical resume, of this question of timing. We arge that artificial connectionist systems designed to range in natural environments will have to incorporate a system of inner clocks. --------------------------------------------- -Subhash Kak From carmesin at schoner.physik.uni-bremen.de Sat Feb 24 06:47:16 1996 From: carmesin at schoner.physik.uni-bremen.de (Hans-Otto Carmesin) Date: Sat, 24 Feb 1996 12:47:16 +0100 Subject: SIR: shift invariant recognition Message-ID: <199602241147.MAA09674@schoner.physik.uni-bremen.de> Dear Rolf and Jerry. The question raised by Rolf Wurtz is, how SIR ( shift invariant recognition) might be processed in the visual system. There is a biologically reasonable candidate network: I proposed it for experiments on so-called stroboscopic alternative motion (SAM). The most simple instance is established by TWO light dots, one of which is elicited at a time in an alternating manner. At adequate frequency an observer perceives ONE dot moving back and forth. The experiment becomes quite non-trivial with four dots at the corners of a square, two elicited at a time at diagonal positions and in an alternating manner. An observer perceives either two dots moving horizontally or two dots moving vertically (roughly speaking). The network represents each dot by a formal neuron; these neurons project to inner neurons that tend to fire in accordance with the stimulation and that are coupled with rapid formal couplings (similar to dynamic links) with a local coupling dynamics reminescent of the Hebb-rule [1-4]. A motion percept is established by the emerging nonzero couplings. It turns out that each active neuron at a time t is coupled to exactly one active neuron at a later time, t+t' say. Moreover there are prestabilized coupling weights (modeling synaptic densities) that prefer short distances in space and time. As a result: If a pattern is presented at a time t and a shifted pattern is presented at a time t+t', then the dots of the first pattern are coupled to the corresponding dots of the second pattern. This network is understood very well [3,4]: It can be solved analytically and exhibits an effective potential dynamics in coupling space. I predicted [3] a continuous phase transition and measured it together with experimental psychologists later. Another indication of biological relevance: Formally the network is very similar to networks with retinotopy emergence [5]. References: [1] H.-O. Carmesin: Statistical neurodynamics: A model for universal properties of EEG-data and perception. Acta Physica Slovaca, 44:311--330, 1994. [2] H.-O. Carmesin and S. Arndt: Neuronal self-organization of motion percepts. Technical Report 6/95, ZKW Universitt Bremen, Bremen, 1995. [3] H.-O. Carmesin: Theorie neuronaler Adaption. (Kster, Berlin, 1994. ISBN 3-89574-020-9). [4] H.-O. Carmesin: Neuronal Adaptation Theory. (Peter Lang, Frankfurt am Main, 1996. ISBN 3-631-30039-5). [5] H.-O. Carmesin: Topology preservation emergence by Hebb rule with infinitesimal short range signals. Phys. Rev. E, 53(1):993--1003, 1996. For details see: WWW: http://schoner.physik.uni-bremen.de/~carmesin/ From scheler at informatik.tu-muenchen.de Sat Feb 24 07:50:42 1996 From: scheler at informatik.tu-muenchen.de (Gabriele Scheler) Date: Sat, 24 Feb 1996 13:50:42 +0100 Subject: Shift Invariance Message-ID: <96Feb24.135055+0100_met.116444+24@papa.informatik.tu-muenchen.de> There should be a difference made between shift-invariance, i.e. distinguishing between T1: {[a,b,c,d,e], [b,c,d,e,a], [c,d,e,a,b]} T2: {[a,d,b,c,d], [a,d,c,b,e] etc.} which is more of a purely mathematical problem, and translational invariance, i.e. detecting a pattern on a plane, no matter where it occurs. For the latter goal it is sufficient to develop a set of features in the first layer to detect that pattern in a local field, and to develop an invariant detector in the next layer, which is ON for any of the lower-level features. (develop means train for ANN). In the domain of neural networks the obvious solution to the mathematical problem would be to train a level of units as sequence encoders: A1 B1 C1 D1 ----- ---- a b ------- c ------- d and classify patterns then on how many of the sequence encoders a-d are ON. Of course this may be rather wasteful. In another learning approach called adaptive distance measures, we can reduce training effort considerably when we use a distance measure which is specifically tuned to problems of shift invariance. Of course this is nothing else than to have a class of networks with pre-trained sequence encoders available. The question here as often is not, which NN can learn this task (backprop can, Fukushima's Neocognitron can), but which is most economical in its resources - without requiring too much knowledge on the type of function to be learned. From lpratt at fennel.mines.edu Sat Feb 24 13:15:45 1996 From: lpratt at fennel.mines.edu (Lorien Y. Pratt) Date: Sat, 24 Feb 1996 11:15:45 -0700 (MST) Subject: Call for papers -- please post Message-ID: <199602241815.LAA01740@fennel.mines.edu> ------------------------------------------------------------------------------- Call for papers (please post) Special Issue of the Machine Learning Journal on Inductive Transfer ------------------------------------------------------------------------------- Lorien Pratt and Sebastian Thrun, Guest Editors ------------------------------------------------------------------------------- Many recent machine learning efforts are focusing on the question of how to learn in an environment in which more than one task is performed by a system. As in human learning, related tasks can build on one another, tasks that are learned simultaneously can cross-fertilize, and learning can occur at multiple levels, where the learning process itself is a learned skill. Learning in such an environment can have a number of benefits, including speedier learning of new tasks, a reduced number of training examples for new tasks, and improved accuracy. These benefits are especially apparent in complex applied tasks, where the combinatorics of learning are often otherwise prohibitive. Current efforts in this quickly growing research area include investigation of methods that facilitate learning multiple tasks simultaneously, those that determine the degree to which two related tasks can benefit from each other, and methods that extract and apply abstract representations from a source task to a new, related, target task. The situation where the target task is a specialization of the source task is an important special case. The study of such methods has broad application, including a natural fit to data mining systems, which extract regularities from heterogeneous data sources under the guidance of a human user, and can benefit from the additional bias afforded by inductive transfer. We solicit papers on inductive transfer and learning to learn for an upcoming Special Issue of the Machine Learning Journal. Please send six (6) copies of your manuscript postmarked by June 1, 1996 to: Dr. Lorien Pratt MCS Dept. CSM Golden, CO 80401 USA One (1) additional copy should be mailed to: Karen Cullen Attn: Special Issue on Inductive Transfer MACHINE LEARNING Editorial Office Kluwer Academic Publishers 101 Philip Drive Assinippi Park Norwell, MA 02061 USA Manuscripts should be limited to at most 12000 words. Please also note that Machine Learning is now accepting submission of final copy in electronic form. Authors may want to adhere to the journal formatting standards for paper submissions as well. There is a latex style file and related files available via anonymous ftp from ftp.std.com. Look in Kluwer/styles/journals for the files README, kbsfonts.sty, kbsjrnl.ins, kbsjrnl.sty, kbssamp.tex, and kbstmpl.tex, or the file kbsstyles.tar.Z, which contains them all. Please see http://vita.mines.edu:3857/1s/lpratt/transfer.html for more information on inductive transfer. Papers will be quickly reviewed for a target publication date in the first quarter of 1997. -- Dr. Lorien Y. Pratt Department of Mathematical and Computer Sciences lpratt at mines.edu Colorado School of Mines (303) 273-3878 (work) 402 Stratton (303) 278-4552 (home) Golden, CO 80401, USA Vita, photographs, all publications, all course materials available from my web page: http://vita.mines.edu:3857/1s/lpratt From clee at it.wustl.edu Sat Feb 24 22:57:36 1996 From: clee at it.wustl.edu (Christopher Lee) Date: Sat, 24 Feb 1996 21:57:36 -0600 Subject: shift invariance In-Reply-To: <9602210812.ZM26438@ICSI.Berkeley.edu> References: <9602210812.ZM26438@ICSI.Berkeley.edu> Message-ID: <199602250357.VAA23373@it> >>>>> "Jerry" == Jerry Feldman writes: Jerry> Shift invariance is the ability of a neural system to Jerry> recognize a pattern independent of where appears on the Jerry> retina. It is generally understood that this property can Jerry> not be learned by neural network methods, but I have not Jerry> seen a published proof. A "local" learning rule is one that Jerry> updates the input weights of a unit as a function of the Jerry> unit's own activity and some performance measure for the Jerry> network on the training example. All biologically plausible Jerry> learning rules, as well as all backprop variants, are local Jerry> in this sense. Jerry's communique has certainly certainly sparked discussion, but I feel as if his reference to "neural network methods" needs more precise definition. Perhaps Jerry could state more specifically the class of network architectures and neurons he wishes to consider? (E.g., Minsky and Papert restricted their proof to order one perceptrons.) What sort of resource limitations would you put on this network relative to the complexity of the task? (To give an absurd example of why this is important: for a small "test problem"-like space, if given an appropriate number of nodes a network could simply "memorize" all the configurations of an object at all locations. Clearly, this isn't what one would normally considering "learning" shift invariance.) On another vein that might be of interest, it's clear that shift invariance is a fundamental to the primate visual system in some way, and a fair amount of interest exists in the neurophysiology community concerning how this problem is solved; one hypothesis involves the role of attentional mechanisms in scale and translational invariance (Olshausen, Anderson, Van Essen, J. of Neuroscience. 13(11):4700-19, 1993). It is not obvious to me that anything along the lines of Jerry's proof could be applied to their (the Olshausen et al.) network model. Christopher Lee -- Washington University Department of Anatomy and Neurobiology email: clee at v1.wustl.edu From hinton at cs.toronto.edu Sun Feb 25 13:00:36 1996 From: hinton at cs.toronto.edu (Geoffrey Hinton) Date: Sun, 25 Feb 1996 13:00:36 -0500 Subject: obvious, but false Message-ID: <96Feb25.130037edt.525@neuron.ai.toronto.edu> In response to may email about a network that learns shift invariance, Dorffner says: > there seems to be a misunderstanding of what the topic of discussion > is here. I don't think that Jerry meant that no model consisting of neural > network components could ever learn shift invariance. After all, there are > many famous examples in visual recognition with neural networks (such as > the Neocognitron, as Rolf W"urtz pointed out), and if this impossibility > were the case, we would have to give up neural network research in > perceptual modeling altogether. > > What I think Jerry meant is that any cascade of fully-connected feed-forward > connection schemes between layers (including the perceptron and the MLP) > cannot learn shift invariance. > Now besides being obvious, this does raise some > important questions, possibly weakening the fundamentals of connectionism. I agree that this is what Jerry meant. What Jerry said was actually very reasonable. He did NOT say it was obviously impossible. He just said that it was generally understood to be impossible and he would like to see a proof. I think Jerry was right in the sense that most people I have talked to believed it to be impossible. I'd like to apologize to Jerry for the antagonistic tone of my previous message. Dorffner takes the impossibility for granted. My simulation conclusively demonstrates that translation invariance can be learned with no built in bias towards translation invariance. The only requirement is that the shapes should share features, and this is a requirement on the data, not on the network. At the risk of looking very silly, I bet that it really cannot be done if shapes do not share features. My simulation did not have built in preprocessing or weight-sharing as Dorffner seems to imply. So, unlike the neocognitron, it had no innate bias towards translation invariance. It got the "raw" retinal inputs and its desired outputs were shape identities. The version with local connectivity worked best, but as I pointed out, it also worked without local connectivity. So that version exactly fitted Dorffner's definition of what cannot be done. Geoff PS: As I noted in the paper and others have pointed out in their responses, Minsky and Papert's group invariance theorem really does prove that this task cannot be done without hidden layers (using conventional units). From geva at fit.qut.edu.au Sun Feb 25 19:05:33 1996 From: geva at fit.qut.edu.au (Shlomo Geva) Date: Mon, 26 Feb 1996 10:05:33 +1000 (EST) Subject: shift invariance In-Reply-To: <199602230205.VAA13842@bucnsd.qut.edu.au> Message-ID: Regarding shift invariance: One might learn something about the problem by looking at the Fourier Transform in the context of shift invariance. One may perform a Discrete Fourier Transform (DFT), and take advantage of the shift invariance properties of the magnitudes components, discarding the phase and representing objects by a feature vector consisting of the the magnitudes in frequency domain alone. (This is not new and also extends to higher dimensionalities) This approach will solve many practical problems, but has an in-principle difficulty in that this procedure does not produce a unique mapping from objects to invariant features. For example, start from any object and obtain its invariant representation as above. By choosing arbitrary phase components and performing an inverse DFT we can get arbitrarily many object representations. Note that these objects are extremely unlikely to look like an original shifted object! If by chance - and it may be very remote - two of the objects you wish to recognize with shift invariance, have identical magnitudes in the frequency domain then this method will obviously fail. Now I'd like to make a conjecture. It appears to make sense to assume that this difficulty is inherent to the shift invariance requirement itself. If this is so then unless you have an additional constraint imposed on objects - they cannot be allowed to be identical under the invariant feature extraction transformation you wish to employ - then you cannot solve the problem. In other words, one needs a guarantee that all permissible objects are uniquely transformed by the procedure. It seems to follow that no general procedure, that does not take into account the nature of the objects for which the procedure is intended, can exist. I am wondering if anyone could clarify whether this is a valid argument. Shlomo Geva s.geva at qut.edu.au From hinton at cs.toronto.edu Sun Feb 25 13:33:46 1996 From: hinton at cs.toronto.edu (Geoffrey Hinton) Date: Sun, 25 Feb 1996 13:33:46 -0500 Subject: yet more on shift invariance Message-ID: <96Feb25.133354edt.525@neuron.ai.toronto.edu> The argument that Jerry Feldman gave for the difficulty of learning shift invariance illustrates a nice point about learning. He talked about learning to recognize a SINGLE shape in different locations. I think he is probably right that this is impossible without built in prejudices. But the implication was that if it was true for ONE shape then it would be true for a bunch of shapes. This is where the argument breaks down. Its like the point being made by the life-time learning people, except that here the separate tasks are not presented one after another but jumbled together. Geoff From juergen at idsia.ch Mon Feb 26 03:00:00 1996 From: juergen at idsia.ch (Juergen Schmidhuber) Date: Mon, 26 Feb 96 09:00:00 +0100 Subject: neural text compression Message-ID: <9602260800.AA24691@fava.idsia.ch> Now available online: SEQUENTIAL NEURAL TEXT COMPRESSION (9 pages, 68 K) IEEE Transactions on Neural Networks, 7(1):142-146, 1996 Juergen Schmidhuber, IDSIA Stefan Heil, TUM http://www.idsia.ch/~juergen Abstract: Neural nets may be promising tools for loss-free data compression. We combine predictive neural nets and statistical coding techniques to compress text files. We apply our methods to short newspaper articles and obtain compression ratios exceeding those of widely used Lempel-Ziv algorithms (the basis of UNIX functions `compress' and `gzip'). The main disadvantage of our methods is: on conventional machines they are about three orders of magnitude slower than standard methods. To obtain a copy, cut and paste this: netscape ftp://ftp.idsia.ch/pub/juergen/textcompression.ps.gz ------------ P.S.: Have you got a question on recent work on "learning to learn" and "incremental self-improvement"? Stewart Wilson asked me to place the corresponding paper "Environment-independent reinforcement acceleration" in his NetQ web site. Now it is sitting there and waiting for a friendly question or two (questions may be anonymous): netscape http://netq.rowland.org Juergen Schmidhuber, IDSIA From giles at research.nj.nec.com Mon Feb 26 13:01:25 1996 From: giles at research.nj.nec.com (Lee Giles) Date: Mon, 26 Feb 96 13:01:25 EST Subject: Shift Invariance Message-ID: <9602261801.AA26373@alta> We and others [1, 2, 3, 4] showed that invariances, actually affine transformations, could directly be encoded into feedforward higher-order (sometimes called polynomial, sigma-pi, gated, ...) neural nets such that these networks are invariant to shift, scale, and rotation of individual patterns. As mentioned previously, similar invariant encodings can be had for associative memories in autonomous recurrent networks. Interestingly, this idea of encoding geometric invariances into neural networks is an old one [5]. [1] C.L. Giles, T. Maxwell,``Learning, Invariance, and Generalization in High-Order Neural Networks'', Applied Optics, 26(23), p 4972, 1987. Reprinted in ``Artificial Neural Networks: Concepts and Theory,'' eds. P. Mehra and B. W. Wah, IEEE Computer Society Press, Los Alamitos, CA. 1992. [2] C.L. Giles, R.D. Griffin, T. Maxwell,``Encoding Geometric Invariances in Higher-Order Neural Networks'', Neural Information Processing Systems, ed. D.Z. Anderson, Am. Inst. of Physics, N.Y., N.Y., p 301-309, 1988. [3] S.J. Perantonis, P.J.G. Lisboa, ``Translation, Rotation, and Scale Invariant Pattern Recognition by Higher-Order Neural Networks and Moment Classifiers'', IEEE Transactions on Neural Networks'', 3(2), p 241, 1992. [4] L. Spirkovska, M.B. Reid,``Higher-Order Neural Networks Applied to 2D and 3D Object Recognition'', Machine Learning, 15(2), p. 169-200, 1994. [5] W. Pitts, W.S. McCulloch, ``How We Know Universals: The Perception of Auditory and Visual Forms'', Bulletin of Mathematical Biophysics, vol 9, p. 127, 1947. Bibtex entries for the above can be found in: ftp://external.nj.nec.com/pub/giles/papers/high-order.bib -- C. Lee Giles / Computer Sciences / NEC Research Institute / 4 Independence Way / Princeton, NJ 08540, USA / 609-951-2642 / Fax 2482 www.neci.nj.nec.com/homepages/giles.html == From kim.plunkett at psy.ox.ac.uk Mon Feb 26 09:47:50 1996 From: kim.plunkett at psy.ox.ac.uk (Kim Plunkett) Date: Mon, 26 Feb 1996 14:47:50 +0000 Subject: LCP.txt Message-ID: <9602261447.AA50643@mac17.psych.ox.ac.uk> REMINDER: Manuscript submissions are invited for inclusion in a Special Issue of Language and Cognitive Processes on Connectionist Approaches to Language Development. It is anticipated that most of the papers in the special issue will describe previously unpublished work on some aspect of language development (first or second language learning in either normal or disordered populations) that incorporates a neural network modelling component. However, theoretical papers discussing the general enterprise of connectionist modelling within the domain of language development are also welcome. The deadline for submissions is 1st April 1996. Manuscripts should be sent to the guest editor for this special issue: Kim Plunkett Department of Experimental Psychology Oxford University South Parks Road Oxford, OX1 3UD UK email: plunkett at psy.ox.ac.uk FAX: 1865-310447 All manuscripts will be submitted to the usual Language and Cognitive Processes peer review process. From perso at DI.Unipi.IT Mon Feb 26 06:36:04 1996 From: perso at DI.Unipi.IT (Alessandro Sperduti) Date: Mon, 26 Feb 1996 12:36:04 +0100 (MET) Subject: ICML'96 W/S on EC&ML Message-ID: <199602261136.MAA11154@neuron.di.unipi.it> ICML'96 Workshop on EVOLUTIONARY COMPUTING AND MACHINE LEARNING to be held in Bari, Italy, July 2-3, 1996, at the International Conference on Machine Learning. http://zen.btc.uwe.ac.uk/evol/cfp.html In the last decade, research concentrating on the interaction between evolutionary computing and machine learning has developed from the study and use of genetic algorithms and reinforcement learning in rule based systems (i.e. classifier systems) to a variety of learning systems such as neural networks, fuzzy systems and hybrid symbolic/evolutionary systems. Many kinds of learning process are now being integrated with evolutionary algorithms, e.g. supervised, unsupervised, reinforcement, on/off-line and incremental. The aim of this workshop is to bring together people involved and interested in this field to share common theory and practice, and to represent the state of the art. Submissions are invited on topics related to: machine learning using evolutionary algorithms, the artificial evolution of machine learning systems, systems exploring the interaction between evolution and learning, systems integrating evolutionary and machine learning algorithms and on applications that make use of both machine learning and evolutionary algorithms. Contributions that argue a position, give an overview, give a review, or report recent work are all encouraged. Copies of extended abstracts or full papers no longer than 15 pages should be sent (by April 23 1996) to: Terry Fogarty Faculty of Computer Studies and Mathematics University of the West of England Frenchay Phone: (+44) 117 965 6261 Bristol BS16 1QY Fax: (+44) 117 975 0416 UK Email: tcf at btc.uwe.ac.uk or: Gilles Venturini Ecole d'Ingenieurs en Informatique pour l'Industrie Universite de Tours, 64, Avenue Jean Portalis, Phone: (+33)-47-36-14-33 Technopole Boite No 4, Fax: (+33)-47-36-14-22 37913 Tours Cedex 9 Email: venturi at lri.fr FRANCE venturini at univ-tours.fr Accepted papers will constitute the workshop notes and will be refereed by the program committee for inclusion in the post-workshop proceedings in the light of scientific progress made at the workshop. Program committee: F. Baiardi, University of Pisa, Italy. H. Bersini, Universite Libre de Bruxelles, Belgium. L.B. Booker, MITRE Corporation, USA. D. Cliff, University of Sussex, UK. M. Colombetti, Politecnico di Milano, Italy. K. De Jong, George Mason University, USA. M. Dorigo, Universite Libre de Bruxelles, Belgium. T.C. Fogarty, University of the West of England, UK. A. Giordana, University of Torino, Italy. J.G. Grefenstette, Naval Research Laboratory, USA. J.A. Meyer, Ecole Normale Superieure, France. M. Patel, University of Newcastle, UK. M. Schoenauer, Ecole Polytechnique, France. R.E. Smith, University of Alabama, USA. G. Venturini, University of Tours, France. S.W. Wilson, Rowland Institute for Science, USA. Important Dates: April 23: Extended abstracts and papers due May 14: Notification of acceptance June 4: Camera-ready copy for workshop notes due July 2-3: Workshop Prof. Fabrizio Baiardi Dip. di Informatica, Universita di Pisa C. Italia 40, 56123 Pisa, Italy ph: +39/50/887262 email: baiardi at di.unipi.it From stefan.kremer at crc.doc.ca Mon Feb 26 14:35:00 1996 From: stefan.kremer at crc.doc.ca (Dr. Stefan C. Kremer) Date: Mon, 26 Feb 1996 14:35:00 -0500 Subject: shift invariance and recurrent networks Message-ID: <2.2.32.19960226193500.00694f68@digame.dgcd.doc.ca> At 08:12 96-02-21 -0800, Jerry Feldman wrote: > The one dimensional case of shift invariance can be handled by treating >each string as a sequence and learning a finite-state acceptor. But the >methods that work for this are not local or biologically plausible and >don't extend to two dimensions. Recently, many recurrent connectionist networks have been applied to the problem of grammatical induction (i.e. inducing a grammar, or equivalently a finite state acceptor for a given set of example strings) [see, for example: Giles (1990)]. These types of networks are capable of learning many types of regular grammars (e.g. (0)*(101)(0)*). Learning of context-free grammars by connectionist networks has also been studied elsewhere [Das (1993)]. The resulting trained networks work only on the basis of local (both spatially and temporally) interactions among neighbouring processing elements. There are a variety of learning algorithms for these networks. Some like backpropagation through time [Rumelhart, 1986] are spatially local, but temporally global, some like real-time recurrent learning [Williams, 1989] are temporally local and spatially global, and some are both temporally and spatially local like Elman's truncated gradient descent [Elman, 1990] and various locally recurrent networks [Tsoi, 1994]. Don't these types of networks can handle shift invariance problems using local processing? (I'd agree that they're not biologically plausible... ;) ). > The unlearnability of shift invarince is not a problem in practice because >people use preprocessing, weight sharing or other techniques to get shift >invariance where it is known to be needed. However, it does pose a problem for >the brain and for theories that are overly dependent on learning. I'm not sure I understand this last part. Are you saying that "preprocessing" and "weight sharing" can handle shift invariance problems because they are a type of non-local processing? -Stefan P.S. Here's the refs: @INPROCEEDINGS{giles90p, AUTHOR = "C.L. Giles and G.Z. Sun and H.H. Chen and Y.C. Lee and D. Chen", TITLE = "Higher Order Recurrent Networks & Grammatical Inference", BOOKTITLE = "Advances in Neural Information Processing Systems~2", YEAR = "1990", EDITOR = "D.S. Touretzky", PUBLISHER = "Morgan Kaufmann Publishers", ADDRESS = "San Mateo, CA", PAGES = "380-387"} @INPROCEEDINGS{das93p, AUTHOR = "S. Das and C.L. Giles and G.Z. Sun ", TITLE = "Using Prior Knowledge in a NNPDA to Learn Context-Free Languages", BOOKTITLE = "Advances in Neural Information Processing Systems 5", PUBLISHER = "Morgan Kaufmann Publishers", EDITOR = "S.J. Hanson and J.D. Cowan and C.L. Giles", PAGES = "65--72", ADDRESS = "San Mateo, CA" YEAR = "1993"} @BOOK{rumelhart86b1, EDITOR = "J. L. McClelland, D.E. Rumelhart and the P.D.P. Group (Eds.)", AUTHOR = "D. Rumberlhart, G. Hinton, R. Williams", TITLE = "Learning Internal Representation by Error Propagation", VOLUME = "1: Foundations", BOOKTITLE = "Parallel Distributed Processing: Explorations in the Microstructure of Cognition", YEAR = "1986", PUBLISHER = "MIT Press", ADDRESS = "Cambridge, MA"} @ARTICLE{williams89j1, AUTHOR = "R.J. Williams and D. Zipser", TITLE = "A Learning Algorithm for Continually Running Fully Recurrent Neural Networks", JOURNAL = "Neural Computation", YEAR = "1989", VOLUME = "1", NUMBER = "2", PAGES = "270-280"} @ARTICLE{elman90j, AUTHOR = "J.L. Elman", TITLE = "Finding Structure in Time", JOURNAL = "Cognitive Science", YEAR = "1990", VOLUME = "14", PAGES = "179-211"} @ARTICLE{tsoi94j, AUTHOR = "A.C. Tsoi and A. Back", TITLE = "Locally Recurrent Globally Feedforward Networks, A Critical Review of Architectures", JOURNAL = "IEEE Transactions on Neural Networks", VOLUME = "5", NUMBER = "2", PAGES = "229-239", YEAR = "1994"} -- Dr. Stefan C. Kremer, Neural Network Research Scientist, Communications Research Centre, 3701 Carling Avenue, P.O. Box 11490, Station H Ottawa, Ontario K2H 8S2 # Tel: (613)990-8175 Fax: (613)990-8369 E-mail: Stefan.Kremer at crc.doc.ca # WWW: http://digame.dgcd.doc.ca/~kremer/ From jamie at atlas.ex.ac.uk Mon Feb 26 15:35:05 1996 From: jamie at atlas.ex.ac.uk (jamie@atlas.ex.ac.uk) Date: Mon, 26 Feb 96 20:35:05 GMT Subject: shift invariance Message-ID: <1708.9602262035@sirius.dcs.exeter.ac.uk> Jerry Feldman writes: > Shift invariance is the ability of a neural system to recognize a pattern >independent of where appears on the retina. It is generally understood that >this property can not be learned by neural network methods, I agree with Jerry that connectionist networks cannot actually learn shift invariance. A connectionist network can exhibit shift invariant behavior by either being exhaustively trained on a set of patterns that happens to have this property, or by having shift invariance somehow wired into the network before the learning occurs. However, neither of these situation constitutes "learning shift invariance". On the other hand, we are still left with the problem of explaining shift invariant behavior. Some of the responses so far imply training on all patterns in all positions (exhaustive training). I don't find this approach interesting, since it doesn't address the basic issue of generalization ability. Thus the question seems to be how shift invariance can be wired in while still using a learning rule that is local, biologically plausible, etc. Geoff Hinton writes: >shift invariance can be learned by backpropagation. It was one of the >problems that I tried when fiddling about with backprop in the mid 80's. Geoff Hinton's experiment clearly does not train the network exhaustively on all patterns in all positions, so (by the above argument) I have to claim that shift invariance is wired in. The network does not use weight sharing, which would be the most direct way of wiring in shift invariance. However, it does appear to use "error sharing". As I understood Geoff's description, the weights between the position-dependent and position-independent hidden layers are not modified by learning. Each position-independent feature detector is connected to all its associated position-dependent feature detectors with links of the same weight (in particular, they are ORed). Using backprop, this has the effect of distributing the same error signal to each of these position-dependent feature detectors. Thus they all tend to converge to the same feature. In this way, fixing the weights between the two hidden layers to equal values makes the feature detectors learned in one position tend to generalize to other positions. I suspect "tend to" may be an important caveat here, but in essence the equivalence of all positions has been wired in. This equivalence is simply shift invariance. On the other hand, the learning rule is still local, so in that sense it does seem to meet Jerry's challenge. Jerry Feldman also writes: > The one dimensional case of shift invariance can be handled by treating >each string as a sequence and learning a finite-state acceptor. But the >methods that work for this are not local or biologically plausible and >don't extend to two dimensions. I have to disagree with this dismissal of recurrent networks for handling shift invariance. I'm not in a position to judge biological plausibility, but I would take issue with the claim that methods of training recurrent networks are nonlocal and can't be generalized to higher dimensions. These learning methods are to some extent temporally nonlocal, but some degree of temporal nonlocality is necessary for any computation that extends over time. The important thing is that they are just as spatially local as the feedforward methods they are based on. Jerry's own definition of locality is spatial locality: >A "local" learning rule is one that updates the input weights of a unit as a >function of the unit's own activity and some performance measure for the >network on the training example. Now finally I get to my primary gripe. Contrary to Jerry's claim, learning methods for recurrent networks can be generalized to more than one dimension. The issues for two dimensions are entirely the same as those for one. All that is necessary to extend recurrence to two dimensions is units that pulse periodically. In engineering terms, a single network is time-multiplexed across one dimension while being sequenced across the other. Conceptually, learning can be done by unfolding the network over one time dimension, then unfolding the result over the other time dimension, then using a feedforward method. The idea of using time to represent two different dimensions has in fact already been proposed. At an abstract level, this dual use of the time dimension is the core idea behind temporal synchrony variable binding (TSVB) (Shastri and Ajjanagadde, 1993). Recurrent networks use the time dimension to represent position in the input sequence (or computation sequence). TSVB also uses the time dimension to represent variables. Because the same network is being used at every point in the input sequence, recurrent networks inherently generalize things learned in one input sequence position to other input sequence positions. In this way shift invariance is "wired in". Exactly analogously, because the same network is being used for every variable, TSVB networks inherently generalize things learned using one variable to other variables. I argue in (Henderson, submitted) that this results in a network that exhibits systematicity. Replacing the labels "sequence position" and "variable" with the labels "horizontal position" and "vertical position" does not change this basic ability to generalize across both dimensions. Work on applying learning to TSVB networks is being done by both Shastri and myself. (Note that this description of TSVB is at a very abstract level. Issues of biological plausibility are addressed in (Shastri and Ajjanagadde, 1993) and the papers cited there.) Shastri, L. and Ajjanagadde, V. (1993). From simple associations to systematic reasoning: A connectionist representation of rules, variables, and dynamic bindings using temporal synchrony. Behavioral and Brain Sciences, 16:417--451. Henderson, J. (submitted). A connectionist architecture with inherent systematicity. Submitted to the Eighteenth Annual Conference of the Cognitive Science Society. - Jamie ------------------------------ Dr James Henderson Department of Computer Science University of Exeter Exeter EX4 4PT, U.K. ------------------------------ From johnd at saturn.sdsu.edu Mon Feb 26 16:35:10 1996 From: johnd at saturn.sdsu.edu (John Donald) Date: Mon, 26 Feb 1996 13:35:10 -0800 (PST) Subject: shift invariance Message-ID: <199602262135.NAA15297@saturn.sdsu.edu> Abu-Mostafa's "learning from hints" is a very simple approach to learning neural net representations of functions that satisfy global constraints such as shift invariance. Cf Scientific American April 1995 (!) and the references therein, eg. "Learning from hints", Yaser Abu-Mostafa, J. of Complexity 10 (165-178), 1994. His idea is to add to the training examples additional invented examples that represent the global properties. He claims significant speed-up, eg in training nets to learn an even (global property) shift invariant (global) function. From wimw at mbfys.kun.nl Tue Feb 27 06:27:31 1996 From: wimw at mbfys.kun.nl (Wim Wiegerinck) Date: Tue, 27 Feb 1996 12:27:31 +0100 (MET) Subject: Paper Available Message-ID: <199602271127.MAA16094@septimius.mbfys.kun.nl> A non-text attachment was scrubbed... Name: not available Type: text Size: 2250 bytes Desc: not available Url : https://mailman.srv.cs.cmu.edu/mailman/private/connectionists/attachments/00000000/696d56a4/attachment.ksh From edelman at wisdom.weizmann.ac.il Tue Feb 27 08:19:18 1996 From: edelman at wisdom.weizmann.ac.il (Edelman Shimon) Date: Tue, 27 Feb 1996 13:19:18 GMT Subject: shift invariance In-Reply-To: (message from Shlomo Geva on Mon, 26 Feb 1996 10:05:33 +1000 (EST)) Message-ID: <199602271319.NAA05288@lachesis.wisdom.weizmann.ac.il> > Date: Mon, 26 Feb 1996 10:05:33 +1000 (EST) > From: Shlomo Geva > > Regarding shift invariance: > > [some stuff omitted] > > Now I'd like to make a conjecture. > It appears to make sense to assume that this difficulty is inherent > to the shift invariance requirement itself. If this is so > then unless you have an additional constraint imposed on objects - > they cannot be allowed to be identical under the invariant feature > extraction transformation you wish to employ - > then you cannot solve the problem. In other words, one needs a guarantee that > all permissible objects are uniquely transformed by the procedure. > It seems to follow that > no general procedure, that does not take into account the nature of the objects > for which the procedure is intended, can exist. > > I am wondering if anyone could clarify whether this is a valid argument. > > Shlomo Geva A number of people (see the refs below) have proved in the past that no universal invariants with respect to viewpoint exist for objects represented as point sets in 3D. The proofs hinged on the possibility of two different objects having the same 2D projection. Offhand, it seems that a similar argument could be used to prove Shlomo's conjecture. -Shimon Dr. Shimon Edelman, Applied Math. & Computer Science Weizmann Institute of Science, Rehovot 76100, Israel The Web: http://eris.wisdom.weizmann.ac.il/~edelman fax: (+972) 8 344122 tel: 8 342856 sec: 8 343545 @inproceedings{MosesUllman92, author="Y. Moses and S. Ullman", title="Limitations of non model-based recognition schemes", booktitle="Proc. 2nd European Conf. on Computer Vision, Lecture Notes in Computer Science", volume="588", pages="820-828", editor="G. Sandini", publisher="Springer Verlag", addredd="Berlin", year="1992" } @article{BurWeiRis93, author="J.B. Burns and R. Weiss and E. Riseman", title="View variation of point-set and line segment features", journal=pami, volume="15", pages = "51-68", year = 1993 } From wiskott at salk.edu Tue Feb 27 20:14:15 1996 From: wiskott at salk.edu (Laurenz Wiskott) Date: Tue, 27 Feb 1996 17:14:15 -0800 Subject: shift invariance Message-ID: <199602280114.RAA21840@bayes.salk.edu> Dear connectionists, here is an attempt to put the arguments so far into order (and to to add a bit). I have put it into the form of a list of statements, which are, of course, subjective. You can skip the indented comments in the first reading. --------------------------------------------------------------------------- 1) With respect to shift invariance, there are two types of artificial neural nets (ANNs): a) ANNs with a build in concept of spatial order (e.g. neocognitron and other weight sharing ANNs, neural shifter circuits, dynamic link matching), let me call these ANNs structured. b) ANNs without any build in concept of spatial order (e.g. fully connected back-propagation), let me call these ANNs isotropic. (This distinction is important. For instance, Jerry Feldman's statement referred to isotropic ANNs while Rolf W"urtz' disagreement was based on structured ANNs.) 2) Structured ANNs can DO shift invariant pattern discrimination, but they do not LEARN it. (It is important to note that structured ANNs for shift invariant pattern recognition usually do NOT LEARN the shift invariance. It is already build in (I'd be glad to see a counterexample). What they learn is pattern discrimination, under the constraint that, whatever they do, it is shift invariant.) 3) The isotropic ANNs can learn shift invariant pattern recognition, given that during training the patterns are presented at ALL locations. This is not surprising and not what we are asking for. (This is what Christopher Lee pointed out:>>... for a small "test problem"-like space, if given an appropriate number of nodes a network could simply "memorize" all the configurations of an object at all locations. Clearly, this isn't what one would normally considering "learning" shift invariance.<<) 4) What we are asking for is generalization. I see two types of generalization: a) generalization of pattern recognition from one part of the input plain to another. b) generalization of shift invariance from one pattern to another. (4a example: training on patterns A {101} and B {011} in the left half-plane, i.e. {101000, 010100, 011000, 001100}, and testing on patterns A and B in the right half-plane, e.g. {000101, 000011}. 4b example: training on some patterns {111, 011, 010} in all possible locations, i.e. {111000, 011100, 001110, 000111, 011000, ..., 000010}, and on pattern A {101} in the left half-plane, i.e {101000, 010100}, and testing on pattern A in the right half-plane, e.g. {000101}. This is again an important distinction. For instance, Jerry Feldman's statement referred to generalization 4a and Geoffrey Hinton's disagreement referred to generalization 4b.) 5) Generalization 4a seems to be impossible for an isotropic ANN. (This was illustrated by Jerry Feldman and, more elaborately, by Georg Dorffner.) 6) Generalization 4b is possible. (This has been demonstrated by the model of Geoffrey Hinton.) 7) Models which generalize according to 4b usually loose discriminative power, because patterns with the same set of features but in different spatial order get confused. (This has been pointed out by Shlomo Geva. This also holds for some structured ANNs, such as the neocognitron and other weight sharing ANNs, but does not hold for the neural shifter circuits and dynamic link matching. The loss of discriminative power can be avoided by using sufficiently complex features, which has its own drawbacks.) --------------------------------------------------------------------------- Best regards, Laurenz Wiskott. =========================================================================== .---. .---. Dr. L a u r e n z W i s k o t t | |. S A L K .| | Computational Neurobiology Laboratory | ||. C N L .|| | The Salk Institute for Biological Studies | ||| L W ||| | mail: PO Box 85800, San Diego, CA 92186-5800 | |||=======||| | street, parcels: 10010 North Torrey Pines Road `---''' " ```---' La Jolla, CA 92037 phone: +1 (619) 453-4100 ext 1463; fax: +1 (619) 587-0417; email: wiskott at salk.edu; WWW: http://www.cnl.salk.edu/~wiskott/ =========================================================================== From sontag at control.rutgers.edu Wed Feb 28 11:01:03 1996 From: sontag at control.rutgers.edu (Eduardo Sontag) Date: Wed, 28 Feb 1996 11:01:03 -0500 Subject: TR available - classification of points in general position Message-ID: <199602281601.LAA19838@control.rutgers.edu> SHATTERING ALL SETS OF k POINTS IN GENERAL POSITION REQUIRES (k-1)/2 PARAMETERS Rutgers Center for Systems and Control (SYCON) Report 96-01 Eduardo D. Sontag Department of Mathematics, Rutgers University The generalization capabilities of neural networks are often quantified in terms of the maximal number of possible binary classifications that could be obtained, by means of weight assignments, on any given set of input patterns. The Vapnik-Chervonenkis (VC) dimension is the size of the largest set of inputs that can be shattered (i.e, arbitrary binary labeling is possible). Recent results show that the VC dimension grows at least as fast as the square n**2 of the number of adjustable weights n in the net, and this number might grow as fast as n**4. These results are quite pessimistic, since they imply that the number of samples required for reliable generalization, in the sense of PAC learning, is very high. On the other hand, it is conceivable that those sets of input patterns which can be shattered are all in some sense ``special'' and that if we ask instead, as done in the classical literature in pattern recognition, for the shattering of all sets in ``general position,'' then an upper bound of O(n) might hold. This paper shows a linear upper bound for arbitrary sigmoidal (as well as threshold) neural nets, proving that in that sense the classical results can be recovered in a strong sense (up to a factor of two). Specifically, for classes of concepts defined by certain classes of analytic functions depending on n parameters, it is shown that there are nonempty open sets of samples of length 2n+2 which cannot be shattered. ============================================================================ The paper is available starting from Eduardo Sontag's WWW HomePage at URL: http://www.math.rutgers.edu/~sontag/ (follow links to FTP archive, file generic-vc.ps.gz) or directly via anonymous FTP: ftp math.rutgers.edu login: anonymous cd pub/sontag bin get generic-vc.ps.gz Once file is retrieved, use gunzip to uncompress and then print as postscript. ============================================================================ Comments welcome. From andre at icmsc.sc.usp.br Wed Feb 28 14:52:21 1996 From: andre at icmsc.sc.usp.br ( Andre Carlos P. de Leon F. de Carvalho ) Date: Wed, 28 Feb 96 14:52:21 EST Subject: II Workshop on Cybernetic Vision Message-ID: <9602281752.AA01880@xavante> ====================================================== First Call for Contributions II Workshop on Cybernetic Vision Instituto de Fisica e Quimica de Sao Carlos Universidade de Sao Paulo Sao Carlos, SP, Brazil 9-11 December 1996 ====================================================== As stated in his classical book *Cybernetics*, Norbert Wiener believed that the most interesting and exciting possibilities for original research were to be found in the interface between the major scientific areas. In a tribute to Wiener's visionary approach, the term CYBERNETIC VISION has been proposed to express those research activities lying in the natural/artificial vision interface. It is believed that not only more powerful and versatile artificial visual systems can be obtained through the incorporation of biological insights, but also that our understanding of the natural visual systems can benefit from advances in artificial vision research, thus sustaining a positive feedback. The I WORKSHOP ON CYBERNETIC VISION took place at the Brazilian town of Sao Carlos, SP, in 1994 and attracted the attention of many researchers from the most diverse areas. The second issue of this meeting is aimed at providing another opportunity for scientific interchange as well as the beginning of new collaborations between the related communities. Quality papers related to any of the areas below are welcomed. Prospective authors should send (through FAX, conventional mail, or e-mail) an abstract of approximately 500 words no later than 15th April 1996 to: Prof Dr Luciano da Fontoura Costa Cybernetic Vision Research Group IFSC-USP Av Dr Carlos Botelho, 1465 Caixa Postal 369 Sao Carlos, SP 13560-970 Brazil FAX: +55 162 71 3616 Electronic-mail submission of the abstracts, to be sent to the address below, is strongly encouraged. Luciano at olive.ifqsc.sc.usp.br Upon acceptance of the proposed abstracts, the authors will be asked to prepare the full manuscript (full paper or communication), for further assessment. Accepted contributions will be included in the Workshop Proceedings. The abstracts of the accepted papers will be eventually incorporated into a WWW page. Although contributions in the Portuguese language are welcomed, preference will be given to manuscript in the English language. Subjected to the author's consent, accepted papers in the English language may also be considered for publication in international journals in some of the areas covered. The organizing committee will do its best towards providing a limited number of grants. Areas covered include, but are by no means limited to: - Active Vision - Anatomy and Histology - Eletrophysiology - Ethology - Fuzzy Models - Image Analysis and Computer Vision - Medical Imaging - Modeling and Simulation of Biological Visual Systems - Neural Networks (natural and artificial) - Neurogenesys - Neuromorphometry - Neuroscience - Optical Computing - Psychophysics - Robotics - Scientific Visualization - Vertebrate and Invertebrate Vision - Vision and Consciousness ====================================================== Important Dates: * A 500-word abstract by April 15 * Feedback to author on abstract by May 17 * Three copies of the complete version of the paper by July 26 * Notification of accepted papers by September 6 * WORKSHOP December 9-11 ====================================================== Organizing Committee: Luciano da F. Costa, CVRG, IFSC-USP (Coordinator) Sergio Fukusima, USP-Rib. Preto Roland Koberle, CVRG, IFSC-USP Rafael Linden, UFRJ Valentin O. Roda, CVRG, IFSC-USP Jan F. W. Slaets, IFSC-USP ====================================================== Program Committee (preliminary): Arnaldo Albuquerque, UFMG Junior Barrera, IME-USP Paulo E. Cruvinel, EMBRAPA-CNPDIA Antonio Francisco, INPE Annie F. Frere, EESC-USP Sergio Fukusima, USP-Rib. Preto Andre P. de Leon, ICMSC-USP Rafael Linden, UFRJ Roberto de A. Lotufo, DCA-UNICAMP Ricardo Machado, IBM Joaquim P. Brasil Neto, UnB Nelson D. D. Mascarenhas, UFSCar Valdir F. Pessoa, UnB Anna H. R. C. Rillo, PCS-EPUSP ======================================================= From jfeldman at ICSI.Berkeley.EDU Wed Feb 28 13:00:39 1996 From: jfeldman at ICSI.Berkeley.EDU (Jerry Feldman) Date: Wed, 28 Feb 1996 10:00:39 -0800 Subject: shift invariance Message-ID: <9602281000.ZM15421@ICSI.Berkeley.edu> There seem to be three separate threads arising from my cryptic post and it might be useful to separate them. 1) The capabilities of spatial feedforward nets and backprop(ffbp) Everyone seems to now agree that conventional feedforward nets and backprop (ffbp) will not learn the simple 0*1010* languages of my posting. Of course any formal technique has limitations; the interesting point is that shift invariance is a basic property of apparent biological significance. Geoff Hinton's series of messages asserts that the world and experience are (could be?) structured such that ffbp will learn shift invariance in practice because patterns overlap and are dense enough in space. My inference is that Geoff would like to extend this claim (the world makes ffbp work) to everything of biological importance. Results along these lines would be remarkable indeed. 2) Understanding how the visual system achieves shift invariance. This thread has been non-argumentative. The problem of invariances and constancies in the visual system remains central in visual science. I can't think of any useful message-sized summary, but this is an area where connectionist models should play a crucial role in expressing and testing theories. But, as several people have pointed out, we can't expect much from tabula rasa learning. 3) Shift invariance in time and recurrent networks. I threw in some (even more cryptic) comments on this anticipating that some readers would morph the original task into this form. The 0*1010* problem is an easy one for FSA induction and many simple techniques might work for this. But consider a task that is only slightly more general, and much more natural. Suppose the task is to learn any FSL from the class b*pb* where b and p are fixed for each case and might overlap. Any learning technique that just tried to predict (the probability of) successors will fail because there are three distinct regimes and the learning algorithm needs to learn this. I don't have a way to characterize all recurrent net learning algorithms to show that they can't do this and it will be interesting to see if one can. There are a variety on non-connectionist FSA induction methods that can effectively learn such languages, but they all depend on some overall measure of simplicity of the machine and its fit to the data - and are thus non-local. The remark about not extending to two dimensions referred to the fact that we have no formal grammar for two dimensional patterns (although several proposals for one) and, a fortiori, no algorithm for learning same. One can, as Jamie Henderson suggests, try to linearize two-dimensional problems. But no one has done this successfully for shift (rotation, scale, etc.) invariance and it doesn't seem to me a promising approach to these issues. Jerry F. From pjs at aig.jpl.nasa.gov Wed Feb 28 14:18:41 1996 From: pjs at aig.jpl.nasa.gov (Padhraic J. Smyth) Date: Wed, 28 Feb 1996 11:18:41 -0800 (PST) Subject: TR on HMMs and graphical models Message-ID: <199602281918.LAA03228@amorgos.jpl.nasa.gov> The following technical report is available online at: ftp://aig.jpl.nasa.gov/pub/smyth/papers/TR-96-03.ps.Z PROBABILISTIC INDEPENDENCE NETWORKS FOR HIDDEN MARKOV PROBABILITY MODELS Padhraic Smyth [a], David Heckerman [b], and Michael Jordan [c] [a] Jet Propulsion Laboratory and Department of Information and Computer Science, UCI [b] Microsoft Research [c] Department of Brain and Cognitive Sciences, MIT Abstract Graphical techniques for modeling the dependencies of random variables have been explored in a variety of different areas including statistics, statistical physics, artificial intelligence, speech recognition, image processing, and genetics. Formalisms for manipulating these models have been developed relatively independently in these research communities. In this paper we explore hidden Markov models (HMMs) and related structures within the general framework of probabilistic independence networks (PINs). The paper contains a self-contained review of the basic principles of PINs. It is shown that the well-known forward-backward (F-B) and Viterbi algorithms for HMMs are special cases of more general inference algorithms for arbitrary PINs. Furthermore, the existence of inference and estimation algorithms for more general graphical models provides a set of analysis tools for HMM practitioners who wish to explore a richer class of HMM structures. Examples of relatively complex models to handle sensor fusion and coarticulation in speech recognition are introduced and treated within the graphical model framework to illustrate the advantages of the general approach. This TR is available as Microsoft Research Technical Report TR-96-03, Microsoft Research, Redmond, WA. and as AI Lab Memo AIM-1565, Massachusetts Institute of Technology, Cambridge, MA. From clee at it.wustl.edu Wed Feb 28 16:33:05 1996 From: clee at it.wustl.edu (Christopher Lee) Date: Wed, 28 Feb 1996 15:33:05 -0600 Subject: paper available-Nonlinear Hebbian learning Message-ID: <199602282133.PAA01001@it> Announcing the availability of a paper that may be of relevance to those who have been following the recent discussion of shift-invariance. Key words: Hebbian learning, disparity, nonlinear systems, random-dot stereograms. --------------------------------------------------------------------- A nonlinear Hebbian network that learns to detect disparity in random-dot stereograms. C.W. Lee and B.A. Olshausen An intrinsic limitation of linear, Hebbian networks is that they are capable of learning only from the linear pairwise correlations within an input stream. In order to explore what higher forms of structure could be learned with a nonlinear Hebbian network, we have constructed a model network containing a simple form of nonlinearity and we have applied the network to the problem of learning to detect the disparities present in random-dot stereograms. The network consists of three layers, with nonlinear, sigmoidal activation functions in the second layer units. The nonlinearities allow the second layer to transform the pixel-based representation in the input into a new representation based on coupled pairs of left-right inputs. The third layer of the network then clusters patterns occurring on the second layer outputs according to their disparity via a standard competitive learning rule. Analysis of the network dynamics shows that the second-layer units' nonlinearities interact with the Hebbian learning rule to expand the region over which pairs of left-right inputs are stable. The learning rule is neurobiologically inspired and plausible, and the model may shed light on how the nervous system learns to use coincidence detection in general. (To appear in Neural Computation 8:3) This paper is available via World Wide Web at: http://v1.wustl.edu/chris/chris.html Hard copies are available upon request from clee at v1.wustl.edu, or write to: Chris Lee Campus Box 8108 Washington University 660 S. Euclid Ave St. Louis, MO 63110. From giles at research.nj.nec.com Thu Feb 29 10:03:15 1996 From: giles at research.nj.nec.com (Lee Giles) Date: Thu, 29 Feb 96 10:03:15 EST Subject: shift invariance Message-ID: <9602291503.AA29190@alta> We and others [1, 2, 3, 4] showed that invariances, actually affine transformations, could directly be encoded into feedforward higher-order (sometimes called polynomial, sigma-pi, gated, ...) neural nets such that these networks are invariant to shift, scale, and rotation of individual patterns. As mentioned previously, similar invariant encodings can be had for associative memories in autonomous recurrent networks. Interestingly, this idea of encoding geometric invariances into neural networks is an old one [5]. [1] C.L. Giles, T. Maxwell, ``Learning, Invariance, and Generalization in High-Order Neural Networks'', Applied Optics, 26(23), p 4972, 1987. Reprinted in ``Artificial Neural Networks: Concepts and Theory,'' eds. P. Mehra and B. W. Wah, IEEE Computer Society Press, Los Alamitos, CA. 1992. [2] C.L. Giles, R.D. Griffin, T. Maxwell,``Encoding Geometric Invariances in Higher-Order Neural Networks'', Neural Information Processing Systems, Eds. D.Z. Anderson, Am. Inst. of Physics, N.Y., N.Y., p 301-309, 1988. [3] S.J. Perantonis, P.J.G. Lisboa, ``Translation, Rotation, and Scale Invariant Pattern Recognition by Higher-Order Neural Networks and Moment Classifiers'', IEEE Transactions on Neural Networks, 3(2), p 241, 1992. [4] L. Spirkovska, M.B. Reid,``Higher-Order Neural Networks Applied to 2D and 3D Object Recognition'', Machine Learning, 15(2), p. 169-200, 1994. [5] W. Pitts, W.S. McCulloch, ``How We Know Universals: The Perception of Auditory and Visual Forms'', Bulletin of Mathematical Biophysics, vol 9, p. 127, 1947. A bibtex entry for the above references can be found in: ftp://external.nj.nec.com/pub/giles/papers/high-order.bib -- C. Lee Giles / Computer Sciences / NEC Research Institute / 4 Independence Way / Princeton, NJ 08540, USA / 609-951-2642 / Fax 2482 www.neci.nj.nec.com/homepages/giles.html ==  From tp-temp at ai.mit.edu Thu Feb 29 20:37:36 1996 From: tp-temp at ai.mit.edu (Tomaso Poggio) Date: Thu, 29 Feb 96 20:37:36 EST Subject: Shift Invariance In-Reply-To: Lee Giles's message of Mon, 26 Feb 96 13:01:25 EST <9602261801.AA26373@alta> Message-ID: <9603010137.AA00462@corpus-callosum.ai.mit.edu> A footnote to Lee Giles msg. Polynomial networks, analog perceptrons, Kolmogorov theorem and invariances were described in an old paper with Werner Reichardt. ``On the representation of multi-input systems: computational properties of polynomial algorithms,'' (T. Poggio and W. Reichardt). {\it Biol. Cyber.}, {\bf 37}, 167-186, 1980.  From janet at psy.uq.oz.au Thu Feb 1 02:46:37 1996 From: janet at psy.uq.oz.au (Janet Wiles) Date: Thu, 1 Feb 1996 17:46:37 +1000 (EST) Subject: Cognitive Modelling Workshop (Virtual and Physical) Message-ID: CALL FOR PAPERS We are pleased to announce a COGNITIVE MODELLING WORKSHOP in conjunction with the Australian Conference on Neural Networks (ACNN'96) and the electronic cognitive science journal Noetica This announcement can be accessed on the WWW at URL: http://psy.uq.edu.au/CogPsych/acnn96/cfp.html -------------------------------------------------------------------- Noetica/ACNN'96 Cognitive Modelling Workshop: Call For Papers VIRTUAL WORKSHOP February 1 to March 26, 1996 PHYSICAL WORKSHOP Canberra Australia April 9, 1996 Memory, time, change and structure in ANNs: Distilling cognitive models into their functional components Organisers: Janet Wiles and J. Devin McAuley Departments of Computer Science and Psychology University of Queensland QLD 4072 Australia janet at psy.uq.edu.au devin at psy.uq.edu.au Call For Papers Web Page http://psy.uq.edu.au/CogPsych/acnn96/cfp.html Workshop Web Page http://psy.uq.edu.au/CogPsych/acnn96/workshop.html Aim The Workshop aim is to identify the functional roles of artificial neural network (ANN) components and to understand how they combine to explain cognitive phenomena. Existing ANN models will be distilled into their functional components through case study analysis, targeting three traditional strengths of ANNs - mechanisms for memory, time and change; and one area of weakness - mechanisms for structure (see below for details of these four areas). Workshop Format The Workshop will be held in two parts: a virtual workshop via the World Wide Web followed by the physical workshop at ACNN'96. February - March 1996 (Part 1 - Virtual Workshop): All members of the cognitive modelling and ANN research communities are invited to submit case studies of neural-network-based cognitive models (their own or established models from the literature) and commentaries on the workshop issues and case studies. Submissions judged appropriate for the workshop will be posted to the Workshop Web Page as they arrive, and will be collated into a Special Issue of the electronic journal Noetica. Multiple case studies of an ANN model may be accepted if they address different cognitive phenomena. It is OK to participate in the virtual workshop without attending the physical workshop. April 9, 1996 (Part 2 - Physical Workshop): A physical workshop will be held as part of the 1996 Australian Conference on Neural Networks (ACNN'96) in Canberra Australia. At the workshop, the collection of case studies and commentaries will be available in hard copy form. The physical workshop will be 90 minutes long, beginning with an introduction (review of the issues); then presentation of submitted and invited Case Studies; and closing with a discussion of what's missing from the list of available mechanisms. A summary of the issues raised in the discussion will be compiled afterwards, and made available via the workshop web page. Further details on presentations will be announced closer to the date of the workshop. Rationale For many ANN models of cognitive phenomena, interesting behaviour appears to arise from the model as a total package, and it is often a challenge to understand how aspects of the behaviour are supported by components of the ANN. The goal of this workshop is to further such understanding: Specifically, to identify the functional roles of ANN components, the link from the component to the overall behaviour and how they combine to explain cognitive phenomena. The four target areas (memory, time, change and structure) are not disjoint, but rather, provide overlapping viewpoints from which to examine models. In essence, we believe these target areas are where to look for the ``sources of power'' in a model. We use the term "distillation" to refer to the process of identifying the functional components of a model with respect to the four target areas. The task of distillation requires exploring the details of a model in order to clarify its source of power, stripping away other aspects. It focuses on the computational properties of the model formalism, providing a method for: (1) understanding the computational components of a newly presented model, and how they give rise to its behavior, (2) discerning novel computational components that may prove useful in model development, and (3) comparing ANN models that target similar cognitive tasks. The workshop is specifically intended for cognitive modellers who use ANNs, but we anticipate that it will be of interest to the wider ANN community. The case study format grounds the analysis of the functional components of ANNs in the cognitive modelling literature, and focuses on phenomena that do admit a computational explanation. The workshop is intended as much as a learning experience as a communicative one. ---------------------------------------------------------------------------- Submission Details Each Case Study should be based on a published ANN simulation in an area of cognitive science. It should address all four target areas of memory, time, change and structure, with the order of sections and choice of sub-headings up to the individual. Some models may have little to say about one or more of these areas - make this explicit. Include simulation details relevant to understanding the functional components of the model but complete replication details are not necessary. Issues beyond the scope of the functional components and the behaviour they support should not be included in the case study itself, but may be appropriate as commentary. Maximum length is 2500 words including references. Where possible, papers should be submitted in html format (but ascii and postscript will also be accepted). The URL for each submission or the source document can be emailed to the organisers at janet at psy.uq.edu.au or devin at psy.uq.edu.au between Feb 1 and March 26, 1996. Case Study Format See Case Study #1 , "Elman's SRN and the discovery of lexical classes" as a guide: http://psych.psy.uq.oz.au/CogPsych/acnn96/case1.html Target paper: Give the full reference to the original paper and relevant simulation. Introduction: In this section introduce the task addressed by the model and the ANN used. Distinguish between the cognitive task of the model and its instantiation in the input/output task of the network. Is there a gap between the the cognitive task and the input/output task of the ANN? For some studies this mismatch may be intentional, as the cognitive task can be viewed as a by-product of another process (e.g., discovery of lexical classes via the prediction task in Elman's SRN). In others, the mismatch between cognitive task and input/output task may be less benign, obscuring the contribution of the model towards understanding the phenomenon. For the ANN task, consider the following questions: What are the inputs and outputs of the model? How is the task information encoded in the input representation? How is the model's response encoded by the output representation? How does the network address the cognitive task? Memory: Identify the information to be stored in memory, then describe the mechanisms. There have been a range of mechanisms proposed for storing and retrieving memories in neural networks: such as implicit long-term coding of memories distributed in the weights; memory as an attractor; short-term memory as transient decay of activations; limit-cycle encoding of memories with synchrony as a method of retrieval. Consider the what and how of memory storage and retrieval: What information needs memory? How is it stored and retrieved? Time: Describe how time is treated in the data, processing and parameters of the network. For example, does the network consider time as an absolute measure in which events in the input are time-stamped with reference to an external clock, as a sequence in which only the order of events is specified, or as a relative measure in which durations are ratios of one another? Methods for processing temporal information with neural networks have included: using a fixed or sliding time window which maps time into space; learning of time delays in the network weights; sequential processing of time slices; and encoding time as the phase angle of an oscillator. What measures of time are used by the network? How are they represented and what are the underlying mechanisms? Change: In this section, consider the types of changes that occur in the neural network, parameters, data, etc, over a range of timescales: * evolutionary change such as a genetic algorithm operating over network parameters; * generational change such as networks training the next generation of networks; * development and aging such as adding or removing units; * learning such as changing weights based on training data; * transient behavior such as activation equation dynamics What types of changes occur in the selected model and how are they implemented in the network's mechanisms? (Note that few of these aspects are expected to apply to any one model, with many case studies focusing on change as learning.) Structure: In this section, consider how structured information in the environment is represented as structured information in the network (e.g., an implicit grammar in training data can be encoded in the hidden-unit space of a recurrent net; and higher-order bindings can be stored using tensors or phase synchrony). What structure is coded directly into the architecture of the network? Is the network partitioned into modules to directly encode structure? What generalization is the network capable of? Is the generalization due to direct coding of structure or does it learn it from the training data? There has been an ongoing debate in the ANN literature on the generalization abilities of networks. Are there important aspects of structure that cannot be represented, learned or generalized by the network? Where possible, identify structure in the environment that may be expected to be reflected in the ANN model but is not. Discussion and Conclusions: In the final section, discuss how the functional components reviewed in the previous sections combine to explain the cognitive phenomena targeted by the case study. Commentary Format The commentary section of the workshop is provided as an outlet for interpretation, elaboration, and substantive criticism of case studies. It is included as part of the workshop format to complement the case studies, which are intended to be compact and focussed on the workshop aim of distilling ANN models into their functional components. Each commentary should discuss one or more case studies and have a maximum length of 1000 words including references. ---------------------------------------------------------------------------- From pjs at aig.jpl.nasa.gov Thu Feb 1 12:29:16 1996 From: pjs at aig.jpl.nasa.gov (Padhraic J. Smyth) Date: Thu, 1 Feb 1996 09:29:16 -0800 (PST) Subject: CFP for special issue of Machine Learning Journal Message-ID: <199602011729.JAA08415@amorgos.jpl.nasa.gov> CALL FOR PAPERS SPECIAL ISSUE OF THE MACHINE LEARNING JOURNAL ON LEARNING WITH PROBABILISTIC REPRESENTATIONS Guest editors: Pat Langley (ISLE/Stanford University) Gregory Provan (Rockwell Science Center/ISLE) Padhraic Smyth (JPL/University of California, Irvine) In recent years, probabilistic formalisms for representing knowledge and inference techniques for using such knowledge have come to play an important role in artificial intelligence. The further development of algorithms for inducing such probabilistic knowledge from experience has resulted in novel approaches to machine learning. To increase awareness of such probabilistic methods, including their relation to each other and to other induction techniques, Machine Learning will publish a special issue on this topic. We encourage submission of papers that address all aspects of learning with probabilistic representations, including but not limited to: Bayesian networks, probabilistic concept hierarchies, naive Bayesian classifiers, mixture models, (hidden) Markov models, and stochastic context-free grammars. We consider any work on learning over representations with explicit probabilistic semantics to fall within the scope of this issue. Submissions should describe clearly the learning task, the representation of data and learned knowledge, the performance element that uses this knowledge, and the induction algorithm itself. Moreover, we encourage authors to decompose their characterization of learning into the processes of (i) selecting a model (or family of models): what are the properties of the model representation ? (ii) selecting a method for evaluating the quality of a fitted model: given a particular parametrization of the model what is the performance criterion by which one can judge its quality ? and (iii) the algorithmic specification of how to search over parameter and model space. An ideal paper will specify these three items clearly and relatively independently. Papers should also evaluate the proposed methods using techniques acknowledged in the machine learning literature, including but not limited to: experimental studies of algorithm behavior on natural and synthetic data (but not the latter alone), theoretical analyses of algorithm behavior, ability to model psychological phenomena, and evidence of successful application in real-world contexts. We especially encourage comparisons that clarify relations among different probabilistic methods or to nonprobabilistic techniques. Papers should meet the standard submission requirements given in the Machine Learning instructions to authors, including having length between 8,000 and 12,000 words. Hardcopies of each submission should be mailed to: Karen Cullen (5 copies) Pat Langley (1 copy) Kluwer Academic Publishers Institute for the Study 101 Philip Drive of Learning and Expertise Assinippi Park 2164 Staunton Court Norwell, MA 02061 Palo Alto, CA 94306 by the submission deadline, July 1, 1996. The review process will take into account the usual criteria, including clarity of presentation, originality of the contribution, and quality of evaluation. We encourage potential authors to contact Pat Langley (langley at cs.stanford.edu), Gregory Provan (provan at jupiter.risc.rockwell.com), or Padhraic Smyth (pjs at aig.jpl.nasa.gov) prior to submission if they have questions. From john at dcs.rhbnc.ac.uk Fri Feb 2 11:45:57 1996 From: john at dcs.rhbnc.ac.uk (John Shawe-Taylor) Date: Fri, 02 Feb 96 16:45:57 +0000 Subject: Technical Report Series in Neural and Computational Learning Message-ID: <199602021645.QAA32350@platon.cs.rhbnc.ac.uk> The European Community ESPRIT Working Group in Neural and Computational Learning Theory (NeuroCOLT) has produced a set of new Technical Reports available from the remote ftp site described below. They cover topics in real valued complexity theory, computational learning theory, and analysis of the computational power of continuous neural networks. Abstracts are included for the titles. *** Please note that the location of the files was changed at the beginning of ** the year, so that any copies you have of the previous instructions should be * discarded. The new location and instructions are given at the end of the list. ---------------------------------------- NeuroCOLT Technical Report NC-TR-96-030: ---------------------------------------- Exponentially many local minima for single neurons by Peter Auer, University of California, Santa Cruz, USA, Mark Herbster, University of California, Santa Cruz, USA, Manfred K. Warmuth, University of California, Santa Cruz, USA Abstract: We show that for a single neuron with the logistic function as the transfer function the number of local minima of the error function based on the square loss can grow exponentially in the dimension. ---------------------------------------- NeuroCOLT Technical Report NC-TR-96-031: ---------------------------------------- An Efficient Implementation of Sigmoidal Neural Nets in Temporal Coding with Noisy Spiking Neurons by Wolfgang Maass, Institute for Theoretical Computer Science, Technische Universitaet Graz, Austria Abstract: We show that networks of rather realistic models for biological neurons can in principle simulate arbitrary feedforward sigmoidal neural nets in a way which has previously not been considered. This new approach is based on temporal coding by single spikes (respectively by the timing of synchronous firing in pools of neurons), rather than on the traditional interpretation of analog variables in terms of firing rates. The resulting new simulation is substantially faster and apparently more consistent with experimental results about fast information processing in cortical neural systems. As a consequence we can show that networks of noisy spiking neurons are "universal approximators" in the sense that they can approximate with regard to temporal coding {\it any} given continuous function of several variables. Our new proposal for the possible organization of computations in biological neural systems has some interesting consequences for the type of learning rules that would be needed to explain the self-organization of such neural circuits. Finally, our fast and noise-robust implementation of sigmoidal neural nets via temporal coding points to possible new ways of implementing sigmoidal neural nets with pulse stream VLSI. ---------------------------------------- NeuroCOLT Technical Report NC-TR-96-032: ---------------------------------------- A Framework for Stuctural Risk Minimisation by John Shawe-Taylor, Royal Holloway, University of London, UK Peter Bartlett, RSISE, Australian National University, Australia Robert Williamson, Australian National University, Australia Martin Anthony, London School of Economics, UK Abstract: The paper introduces a framework for studying structural risk minimisation. The model views structural risk minimisation in a PAC context. It then generalises to the case when the hierarchy of classes is chosen in response to the data, hence explaining the impressive performance of the maximal margin hyperplane algorithm of Vapnik. ---------------------------------------- NeuroCOLT Technical Report NC-TR-96-033: ---------------------------------------- Learning to Compress Ergodic Sources by Jonathan Baxter, Royal Holloway, University of London and London School of Economics, UK John Shawe-Taylor, Royal Holloway, University of London, UK Abstract: We present an adaptive coding technique which is shown to achieve the optimal coding in the limit as the size of the text grows, while the data structures associated with the code only grow linearly with the text. The approach relies on Huffman codes which are generated relative to the context in which a particular character occurs. The Huffman codes themselves are inferred from the data that has already been seen. A key part of the paper involves showing that the loss per character incurred by the learning process tends to zero as the size of the text tends to infinity. ---------------------------------------- NeuroCOLT Technical Report NC-TR-96-034: ---------------------------------------- Theory and Applications of Agnostic PAC-Learning with Small Decision Trees by Peter Auer, University of California, Santa Cruz, USA, Mark Herbster, University of California, Santa Cruz, USA, Robert C. Holte, University of Ottawa, Canada, Wolfgang Maass, Technische Universitaet Graz, Austria Abstract: We exhibit a theoretically founded algorithm $\Tii$ for agnostic PAC-learning of decision trees of at most 2 levels, whose computation time is almost linear in the size of the training set. We evaluate the performance of this learning algorithm $\Tii$ on 15 common ``real-world'' datasets, and show that for most of these datasets $\Tii$ provides simple decision trees with little or no loss in predictive power (compared with C4.5). In fact, for datasets with continuous attributes its error rate tends to be lower than that of C4.5. To the best of our knowledge this is the first time that a PAC-learning algorithm is shown to be applicable to ``real-world'' classification problems. Since one can {\em prove} that $\Tii$ is an agnostic PAC-learning algorithm, $\Tii$ is {\em guaranteed} to produce close to optimal 2-level decision trees from sufficiently large training sets for {\em any} (!) distribution of data. In this regard $\Tii$ differs strongly from all other learning algorithms that are considered in applied machine learning, for which no {\em guarantee} can be given about their performance on {\em new } datasets. We also demonstrate that this algorithm $\Tii$ can be used as a diagnostic tool for the investigation of the expressive limits of 2-level decision trees. Finally, T2, in combination with new bounds on the VC-dimension of decision trees of bounded depth that we derive, provides us now for the first time with the tools necessary for comparing learning curves of decision trees for ``real-world'' datasets with the theoretical estimates of PAC-learning theory. ---------------------------------------- NeuroCOLT Technical Report NC-TR-96-035: ---------------------------------------- A recurrent network that performs a context-sensitive prediction task by Mark Steijvers, Indiana University, Peter Gr\"{u}nwald, CWI, Dept. of Algorithmics, The Netherlands Abstract: We address the problem of processing a context-sensitive language with a recurrent neural network (RN). So far, the language processing capabilities of RNs have only been investigated for regular and context-free languages. We present an extremely simple RN with only one parameter for its two hidden nodes that can perform a prediction task on sequences of symbols from the language $\{ (ba^{k})^n \mid k \geq 0, n > 0 \}$, a language that is context-sensitive but not context-free. The input to the RN consists of any string of the language, one symbol at a time. The network should then, at all times, predict the symbol that should follow. This means that the network must be able to count the number of $a$'s in the first subsequence and to retain this number for future use. Our network can solve the task for $k=1$ up to $k=120$. The network represents the count of $a$'s in the subsequence by having different limit cycles for every different number of $a$'s counted. The limit cycles are related in such a way that the representation of network states in which an $a$ should be predicted are linearly separable from those in which a $b$ should be predicted. Our work shows that connectionism in general can handle more complex formal languages than was previously known. ---------------------------------------- NeuroCOLT Technical Report NC-TR-96-036: ---------------------------------------- Tight worst-case loss bounds for predicting with expert advice by David Haussler, University of California, Santa Cruz, USA, Jyrki Kivinen, University of Helsinki, Finland, Manfred K. Warmuth, University of California, Santa Cruz, USA Abstract: We consider on-line algorithms for predicting binary or continuous-valued outcomes, when the algorithm has available the predictions made by $N$ experts. For a sequence of trials, we compute total losses for both the algorithm and the experts under a loss function. At the end of the trial sequence, we compare the total loss of the algorithm to the total loss of the best expert, \ie, the expert with the least loss on the particular trial sequence. For a large class of loss functions, with binary outcomes the total loss of the algorithm proposed by Vovk exceeds the total loss of the best expert at most by the amount $c\ln N$, where $c$ is a constant determined by the loss function. This upper bound does not depend on any assumptions on how the experts' predictions or the outcomes are generated, and the trial sequence can be arbitrarily long. We give a straightforward method for finding the correct value $c$ and show by a lower bound that for this value of $c$, the upper bound is asymptotically tight. The lower bound is based on a probabilistic adversary argument. The class of loss functions for which the $c\ln N$ upper bound holds includes the square loss, the logarithmic loss, and the Hellinger loss. We also consider another class of loss functions, including the absolute loss, for which we have an $\Omega\left(\sqrt{\ell\log N}\right)$ lower bound, where $\ell$ is the number of trials. We show that for the square and logarithmic loss functions, Vovk's algorithm achieves the same worst-case upper bounds with continuous-valued outcomes as with binary outcomes. For the absolute loss, we show how bounds earlier achieved for binary outcomes can be achieved with continuous-valued outcomes using a slightly more complicated algorithm. ---------------------------------------- NeuroCOLT Technical Report NC-TR-96-037: ---------------------------------------- Exponentiated Gradient Versus Gradient Descent for Linear Predictors by Jyrki Kivinen, University of Helsinki, Finland, Manfred K. Warmuth, University of California, Santa Cruz, USA Abstract: We consider two algorithm for on-line prediction based on a linear model. The algorithms are the well-known gradient descent ($\GD$) algorithm and a new algorithm, which we call $\EGpm$. They both maintain a weight vector using simple updates. For the $\GD$ algorithm, the update is based on subtracting the gradient of the squared error made on a prediction. The $\EGpm$ algorithm uses the components of the gradient in the exponents of factors that are used in updating the weight vector multiplicatively. We present worst-case loss bounds for $\EGpm$ and compare them to previously known bounds for the $\GD$ algorithm. The bounds suggest that the losses of the algorithms are in general incomparable, but $\EGpm$ has a much smaller loss if only few components of the input are relevant for the predictions. We have performed experiments, which show that our worst-case upper bounds are quite tight already on simple artificial data. -------------------------------------------------------------------- ***************** ACCESS INSTRUCTIONS ****************** The Report NC-TR-96-001 can be accessed and printed as follows % ftp ftp.dcs.rhbnc.ac.uk (134.219.96.1) Name: anonymous password: your full email address ftp> cd pub/neurocolt/tech_reports ftp> binary ftp> get nc-tr-96-001.ps.Z ftp> bye % zcat nc-tr-96-001.ps.Z | lpr -l Similarly for the other technical reports. Uncompressed versions of the postscript files have also been left for anyone not having an uncompress facility. In some cases there are two files available, for example, nc-tr-96-002-title.ps.Z nc-tr-96-002-body.ps.Z The first contains the title page while the second contains the body of the report. The single command, ftp> mget nc-tr-96-002* will prompt you for the files you require. A full list of the currently available Technical Reports in the Series is held in a file `abstracts' in the same directory. The files may also be accessed via WWW starting from the NeuroCOLT homepage (note that this is undergoing some corrections and may be temporarily inaccessible): http://www.dcs.rhbnc.ac.uk/neural/neurocolt.html Best wishes John Shawe-Taylor From tgc at kcl.ac.uk Fri Feb 2 05:05:31 1996 From: tgc at kcl.ac.uk (Trevor Clarkson) Date: Fri, 02 Feb 1996 10:05:31 +0000 Subject: NEuroFuzzy workshop in Prague Message-ID: I N F O R M A T I O N B U L L E T I N IEEE European Workshop on Computational Intelligence NEuroFuzzy'96 and Neuronet'96 April 16 - 18, 1996 Prague, Czech Republic NEuroFuzzy'96 ORGANIZED BY IEEE (in cooperation with) Institute of Computer Science of the Academy of Sciences of the Czech Republic Action M Agency Faculty of Transportation, Czech Technical University Prague SPONSORED BY IEEE UK&RI Neural Networks Regional Interest Group IEEE UK&RI Communications Chapter Czechoslovakia Section IEEE Czech Society for Energetics Prague AIMS The purpose of the workshop is to encourage high-quality research in all branches of Computational Intelligence, Artificial Neural Networks and Fuzzy Neural Systems and to provide an opportunity to bring together specialists active in the fields. INTERNATIONAL PROGRAM COMMITTEE M. NOVAK (Czech Republic) - chairman V. BEIU (Romania) A.B. BULSARI (Finland) T. CLARKSON (United Kingdom) G. DREYFUS (France) A. FROLOV (Russia) V. GUSTIN (Slovenia) C. JEDRZEJEK (Poland) K. HORNIK (Austria) S.V. KARTALOPOULOS (USA) E.J.H. KERCKHOFFS (The Netherlands) S. NORDBOTTEN (Norway) G. PIERONNI (Italy) T. ROSKA (Hungary) F. SANDOVAL (Spain) J.G. TAYLOR (United Kingdom) H.G. ZIMMERMANN (Germany) ORGANIZING COMMITTEE Hana Bilkova (ICS), Mirko Novak (ICS), Stanislav Rizek (ICS), Lucie Vachova (Action M Agency), Milena Zeithamlova (Action M Agency) LOCAL ARRANGEMENTS Action M Agency Milena Zeithamlova Vrsovicka 68 101 00 Prague 10 Phone: (422) 6731 2333-4 Fax: (422) 6731 0503 E-mail: actionm at cuni.cz G E N E R A L I N F O R M A T I O N LOCATION NEuroFuzzy'96 - the Workshop on Computational Intelligence will take place at the Bethlehem Palace, Betlemske nam. (Bethlehem Square, next to the Bethlehem Chapel, Old Town, Prague 1). The workshop site is situated just right in the heart of the historical part of Prague. It is located nearby Narodni Street and Metro station (line B - Narodni) and it is reachable by tram No 6, 9, 18, 22. REGISTRATION Those wishing to participate in NEuroFuzzy'96 are requested to pay the registration fees and the accommodation deposit, and to complete (in full) the enclosed Registration Form and fax or mail it to the NEurofuzzy'96 Local Arrangements Agency (Action M Agency) as soon as possible, but not later than March 1, 1996. REGISTRATION DESK The Registration Desk will be open at the Entrance Hall at the Bethlehem Palace Monday, April 15, 1996 1:00 p.m. - 8:00 p.m. Tuesday, April 16, 1996 8:00 a.m. - 1:00 p.m. Wednesday, April 17, 1996 8:30 a.m. - 1:00 p.m. Thursday, April 18, 1996 8:30 a.m. - 1:00 p.m. WORKSHOP FEES Early / Late Non-members IEEE members Full Registration Fee DM 370 / DM 440 DM 330 / DM 400 Students Fee DM 270 / DM 320 DM 220 / DM 260 East European Students Fee DM 150 / DM 180 DM 130 / DM 150 Accompanying Person Fee DM 90 Lunches DM 75 Walking Tour of Prague DM 15 Organ Concert DM 20 A Night with Mozart DM 25 Early means payment made until March 15, 1996. Late means payment made after March 15, 1996. Full Registration Fee, Students Fee and East European Students Fee include workshop materials, proceedings, attendance of all scientific and poster sessions as well as participation at Welcome Party and refreshment during coffee breaks. Accompanying person fee covers attendance at Welcome Party, participation in the Opening and Closing Ceremonies and the assistance in the individual art and music requests. Lunches include meals served during lunch breaks on Tuesday, Wednesday and Thursday, April 16-18, 1996. PAYMENTS All payments of registration fees, accommodation deposit and accommodation balance can be made either by a credit card (MasterCard, EuroCard, VISA, JCB, Diners Club) or by bank transfer to the Czech Republic, Prague, Komercni banka Prague 10, M. Zeithamlova, SWIFT code: KOMB CZ PP, Bank code: 0100, Account number: 221442-101/0100, NEUROFUZZY Registration Fee & Accommodation fees. The equivalent of payment in USD is acceptable. The Agency will send a receipt in acknowledgement once the payment has been registered in the Agency's account, at the latest upon your arrival in Prague at the Registration Desk. ACCOMMODATION We are happy to assist you in arranging the accommodation in Prague. Should you be interested in Action M Agency making the hotel reservations for you, select the hotel or hostel of your preference and indicate your choice on the Registration Form. 1st Category ***** 1. Renaissance Prague Hotel V Celnici 1, Praha 1 - Nove Mesto phone (422) 2481 0396, fax (422) 2481 1687 The new modern Renaissance Prague Hotel is located in the city centre, close to the well-known Prasna brana (the Prague Tower). The Namesti Republiky Metro station (line B) is only a few steps around the corner. In order to get to the Bethlehem Palace, you could walk 20 minutes on the Road of the Czech Kings, which begins at the Prague Tower, or you can take Metro for two stations. 2. Forum Hotel Kongresova 1, Praha 4 - Pankrac phone (422) 6119 1111, fax (422) 42 06 84 The Forum Hotel was built in 1988 and offers the high standard of services typical for the Inter-Continental Hotel Group. The Hotel overlooks the Nuselske udoli (The Nusle Valley) with the old Vysehrad castle. Being located on a hill, the hotel offers a spectacular view of the Prague Castle and the bridges over Vltava river. The Vysehradska Metro station (line C) is near the hotel entrance. The Metro (with one transfer) would take you to the workshop site in about 15 minutes. 2nd Category *** 3. Betlem Club Praha Betlemske nam. 9, Praha 1 - Stare Mesto phone (422) 2421 6872, fax (422) 26 38 85 The small stylish hotel is located in the historical part of Prague in the building partially from 13 century in Romanesque-Gothic style. It is situated just opposite the Bethlehem Palace. 4. SAX Hotel Jansky vrsek 328/3, Praha 1 - Mala Strana phone (422) 53 84 22, fax (422) 53 84 98 The hotel offers a unique atmosphere of its inner atrium and a beautiful view of Mala Strana and Prague Castle. It is located 20 minutes from Bethlehem Palace, when you wish to walk across the famous Charles Bridge, or 10 minutes by tram No.22 (Karmelitska stop). 5. U zlateho stromu Hotel Karlova 6, Praha 1 - Stare Mesto phone/fax (422) 2422 1385 The "Golden Tree" hotel is placed right in the historical centre of Prague, nearby Charles Bridge and 10 minutes walk to the workshop site. 3rd Category 6. U Sladku Pension Belohorska 130, Praha 6 - Brevnov phone/fax (422) 2051 13457 U Sladku Pension is situated in a quite residential quarter within walking distance of Prague Castle. Rooms are equipped with a shower, phone and satellite TV. By tram No.22 (U Kastanu stop) you can get to the workshop site in 25 minutes. 7. Petrska Hostel Petrska 3, Praha 1 - Nove Mesto phone (422) 23 16 430 The hostel is situated in the centre of Prague, nearby Renaissance Prague Hotel. Two rooms share one bathroom. 8. Mazanka Hostel Davidkova 84, Praha 8 - Liben phone (422) 688 59 58, fax (422) 688 42 42 The hostel belongs to the Academy of Sciences of the Czech Republic and is located near the Institute of Computer Science. It takes 40 minutes to get to the city centre by tram No.17 (Davidkova stop). Again two or three rooms share one bathroom. You will receive the hotel (hostel) voucher from Action M Agency in advance by fax or post after having paid your accommodation deposit. You can accommodate yourself with the voucher of your chosen hotel at the day of your arrival from 2.p.m. ACCOMMODATION DESK All questions regarding your accommodation in Prague during the workshop will be answered at the Registration Desk in the Bethlehem Palace. ACCOMMODATION FEES Prices per person/night in: single room double room 1. Renaissance Prague Hotel DM 285 DM 160 2. Forum Hotel DM 245 DM 140 3. Betlem Club Praha DM 165 DM 93 4. SAX Hotel DM 155 DM 90 5. U Zlateho stromu Hotel DM 115 DM 80 6. U Sladku Pension DM 65 DM 50 7. Petrska Hostel DM 50 DM 30 8. Mazanka Hostel DM 45 DM 28 Single room means also double room for single use. Double room supposes two persons. All prices include breakfast. Please, indicate the exact dates of the selected accommodation in the Registration Form. Please, note that the number of reserved rooms is limited and four nights (April 15-19) have been preliminary reserved for you. ACCOMMODATION DEPOSIT The accommodation deposit of DM 300 is required from participants wishing to stay in hotels of 1st and 2nd category (Renaissance Prague Hotel, Forum Hotel, Betlem Club Praha, SAX, Hotel, U Zlateho stromu Hotel). For the 3rd category (U Sladku Pension, Petrska Hostel, Mazanka Hostel) the accommodation deposit of DM 150 is required. Accommodation deposit must be paid together with the registration fees. ACCOMMODATION BALANCE Accommodation balance is the difference between the price of your required hotel and the paid accommodation deposit. After having received your deposit, Action M Agency will confirm your choice and tell you the amount of accommodation balance that has to be paid. The accommodation balance will be payable by a credit card or bank transfer (see PAYMENT). In the case of accommodation balance payments by cash, upon your arrival at the Registration Desk in Prague we require the equivalent amount in Czech Crowns at the current exchange rate. CANCELLATION Refunds of the Registration Fees will be granted for all written cancellations postmarked no later than March 15, 1996. From March 15, 1996 until March 31, 1996 the 50% cancellation fee will be charged. No refunds can be granted for fee cancellations postmarked after March 31, 1996. Refund of the hotel payment will be granted in full for all written cancellation postmarked no later than March 15, 1996. Any cancellation after that date will result in one-night deposit charge. MEALS Light snacks can be obtained in nearby bistros and restaurants. However, the workshop site being in a part of the city frequented by many visitors, the agency highly recommends the use of a prearranged service. For those interested, lunch will be served in the Club Restaurant of Architectures located 50 m from the Bethlehem Palace. A menu will include entree or soup, main course, salad and dessert. A vegetarian main course will be available. SOCIAL PROGRAM All participants and accompanying persons are invited to take part in the following activities, especially at Welcome Party on Tuesday, April 16, 1996 at 7:30 p.m. Walking Tour of Prague - on Monday, April 15, 1996 at 3:00 p.m. to 6:00 p.m. Guided tour of the Old Town, Prague Castle and other historical sites, starting from the Bethlehem Palace. The meeting point will be at 2:45 p.m. at the Registration Desk. Organ Concert - on Wednesday, April 17, 1996 at 7:00 p.m. The special concert for workshop participants in St. Climent Church. A Night with Mozart - on Thursday, April 18, 1996 at 8:00 p.m. The performance full of Mozart's lovely melodies in the Mozart Museum (Villa Bertramka) will introduce the atmosphere of Prague music life 200 years ago. Art and music - the Agency will assist you regarding current art exhibitions, theatre performances and concerts of contemporary and classical music. Social Program Fees can be paid in advance with Registration Fees or at the Registration Desk. PROCEEDINGS The proceedings will be published in Neural Network World. Expanded versions of selected papers are to be reprinted by IEEE as a book. S C I E N T I F I C P R O G R A M TUESDAY - April 16, 1996 9.00 - 9.30 OPENING CEREMONY 9.30 - 10.30 INVITED PLENARY LECTURE T.G.Clarkson (UK) Introduction to Neural Networks 10.30 - 11.00 C o f f e e B r e a k SHORT CONTRIBUTIONS - SECTION A 11.00 - 11.20 L. Reznik (Australia) Controller Design: The Combination of Techniques 11.20 - 11.40 A. Serac, H. Roth (Germany) Design of a Complex Rule-Based Controller for a Biotechnological Process 11.40 - 12.00 K. Althoefer, D.A. Fraser (UK) Fuzzy Obstacle Avoidance for Robotic Manipulators SHORT CONTRIBUTIONS - SECTION B 11.00 - 11.20 V. Kurkova (Czech Republic) Trade-Off between the Size of Parameters and the Number of Units in One-Hidden-Layer-Networks 11.20 - 11.40 D.A. Sprecher (USA) A Numerical Construction of a Universal Function for Kolmogorov's Superpositions 11.40 - 12.00 K. Hlavackova (Czech Republic) Dependence of the Rate of Approximation in a Feedforward Network on its Activation Function 12.00 - 14.00 L u n c h 14.00 - 15.00 INVITED PLENARY LECTURE L. Pecen (Czech Republic) Non-linear Mathematical Interpretation of the Medical Data 15.30 - 16.00 C o f f e e B r e a k SHORT CONTRIBUTIONS - SECTION A 15.30 - 15.50 R. Andonie (Romania) The New Computational Power of Neural Networks 15.50 - 16.10 S. Coombes, S.H. Doole, C. Campbell (UK) Central Pattern Generation in a Model Neuronal Network with Post Inhibitory Rebound and Reciprocal Inhibition 16.10 - 16.30 H. Nagashino, Y. Kinouchi (Japan) Control of Oscillation in a Plastic Neural Oscillator 16.30 - 16.50 U. Latsch (Germany) Neural Decoding of Block Coded Data in Colored Noise 16.50 - 17.10 G. de Tremiolles, K. Madani, P. Tannhof (France) A New Approach to Radial Basis Function's Like Artificial Neural Networks 17.10 - 17.30 S. Draghici (Italy) Improving the Speed of Some Constructive Algorithms by Using a Locking Detection Mechanism SHORT CONTRIBUTIONS - SECTION B 15.30 - 15.50 S. Taraglio, F. Di Fonzo, P. Burrascano (Italy) Training Data Representation in a Neural Based Robot Position Estimation System 15.50 - 16.10 L. Frangu, C. Tudorie, C. Gregoretti, D. Cornei (Romania) Simple Learning Pattern Recognition Predictor and Controller Using Neural Networks 16.10 - 16.30 M. Li (UK) Knowledge-Based Planning in a Simulated Robot World 16.30 - 16.50 S. Rizek (Czech Republic), A. Frolov (Russia) Influence of Feedback upon Learning of Differential Neurocontroller 16.50 - 17.10 V. Roschin, A. Frolov (Russia) Multidimensional Dynamic Differential Neurocontrol 17.10 - 17.30 H.B. Kazemian, E.M. Scharf (UK) An Application of Multi-Input Multi-Output Self Organizing Fuzzy Controller for a Robot-Arm 19.30 W e l c o m e P a r t y Wednesday - April 17, 1996 9.00 - 10.00 INVITED PLENARY LECTURE G. Dorffner (Austria) Neural Networks for Time-Series Analysis 10.00 - 10.30 C o f f e e B r e a k SHORT CONTRIBUTIONS - SECTION A 10.30 - 10.50 Y. Ding, T.G. Clarkson (UK) Fingerprint Recognition by pRAM Neural Networks 10.50 - 11.10 V. Beiu (UK) Entropy Bounds for Classification Algorithms - draft version 11.10 - 11.30 R. Jaitly, D.A. Fraser (UK) Automated 3D Object Recognition and Library Entry System 11.30 - 11.50 F. Hamker, H.M. Gross (Germany) Region Finding for Attention Control in Consideration of Subgoals SHORT CONTRIBUTIONS - SECTION B 10.30 - 10.50 Z.Q. Wu, C.J. Harris (UK) Indirect Adaptive Neurofuzzy Estimation of Nonlinear Time Series 10.50 - 11.10 L.A. Ludwig, A. Grauel (Germany) Designing a Fuzzy Rule Base for Time Series Analysis 11.10 - 11.30 A. Prochazka, M. Mudrova, J. Fiala (Czech Republic) Nonlinear Time-Series Modelling and Prediction 11.30 - 11.50 J. Castellanos, S. Leiva, L.F. Mingo, J. Rios (Spain) Long-Term Trajectory and Signal Behaviour Prediction 11.50 - 14.00 L u n c h 14.00 - 15.30 INVITED PLENARY LECTURE T. Roska (Hungary) Cellular Neural Network - a Paradigm behind a Visual Microprocessor 15.30 - 16.00 C o f f e e B r e a k SHORT CONTRIBUTIONS - SECTION A 16.00 - 16.20 S. Behnke (Germany), N.B. Karayiannis (USA) Competitive Neural Trees for Vector Quantization 16.20 - 16.40 L. Cieplinski, C. Jedrzejek (Poland) Block-Based Rate-Constrained Motion Estimation Using Hopfield Neural Network 16.40 - 17.00 Y. Won, B.H. Lee (Korea) Fuzzy-Morphological-Feature-Based Neural Network for Recognition of Targets in IR imagery 17.00 - 17.20 V. Alexopoulos, S. Kollias (Greece) An Intelligent Action-Based Image Recognition System 17.20 - 17.40 A. Cichocki, W. Kasprzak (Poland) Nonlinear Learning Algorithms for Blind Separation of Natural Images SHORT CONTRIBUTIONS - SECTION B 16.00 - 16.20 C. Schaeffer, R. Kersch, D. Schroeder (Germany) Stable Learning of Out-Of-Roundness with Neural Network 16.20 - 16.40 A. Prochazka, M. Slama, E. Pelikan (Czech Republic) Bayesian Estimators Use in Signal Processing 16.40 - 17.00 O. M. Boaghe (UK) Theoretical and Practical Considerations over Neural Networks Trained with Kalman Filtering 17.00 - 17.20 W. Skrabek, A. Cichocki, W. Kasprzak (Poland) Principal Subspace Analysis for Incomplete Image Data in One Learning Epoch 17.20 - 17.40 C. Thornton (UK) A Selection Principle for Machine Learning Methods THURSDAY - April 18, 1996 9.00 - 10.00 INVITED PLENARY LECTURE S.V. Kartalopoulos (USA) Applications of Fuzzy Logic and Neural Networks in Communications 10.00 - 10.30 C o f f e e B r e a k SHORT CONTRIBUTIONS - SECTION A 10.30 - 10.50 G.S. Wang, Y.H. Song, A.T. Johns (UK), P.Y. Wang, Z.Y. Hu (China) Fuzzy Logic Controlled Learning Algorithm for Training Multilayer Feedforward Neural Networks 10.50 - 11.10 R. Dogaru, A.T. Murgan (Romania), M. Glesner, S. Ortmann (Germany) Computation with Chaos in Discrete-Time Neuro-Fuzzy Networks 11.10 - 11.30 S.I. Vatlin (Republic of Belarus) The Covariant Monotonic Improvement of Fuzzy Classifier is Impossible 11.30 - 11.50 M. Sabau (Romania) A Fuzzy Logic Approach to an Analog to Digital Converter SHORT CONTRIBUTIONS - SECTION B 10.30 - 10.50 D. Gorse, D.A. Romano-Critchley, J.G. Taylor (UK) A Modular pRAM Architecture for the Classification of TESPAR-Encoded Speech Signals 10.50 - 11.10 A.H. El-Mousa, T.G. Clarkson (UK) Multi-Configurable pRAM Based Neuro Computer 11.10 - 11.30 P.J.L. Adeodato, J.G. Taylor (UK) Storage Capacity of RAM-based Neural Networks: Pyramids 11.30 - 11.50 A. Gabrielli, E. Gandolfi, M. Masetti (Italy) VLSI Fuzzy Chip Design that Processes 2-4 Inputs every 160-300 ns whichever is the Fuzzy System 11.50 - 14.00 L u n c h 14.00 - 15.00 POSTER PRESENTATIONS P. Barson, S. Field, N. Davey, G. McAskie, R. Frank (UK) The Detection of Fraud in Mobile Phone Networks G. Baptist (UK), F.C. Lin (USA), J. Nelson (USA) Note on the Long Term Trend of the Dow Jones Industrial Average L. Cieplinski, C. Jedrzejek (Poland) Performance Comparison of Neural Networks VS AD-HOC Heuristic Algorithm for the Traffic Control Problem in Multistage Interconnection Networks N.A. Dubrovskiy, L.K. Rimskaya-Korsakova (Russia) Modelling of Auditory Neurons Own Periodicity Influence on its Frequent Selectivity A. Frolov (Russia), D. Husek (Czech Republic), I. Muraviev (Russia) Statistical Neurodynamics of Sparsely Encoded Hopfield-like Associative Memory M. Jirina (Czech Republic) Neuro-Fuzzy Network Using Extended Kohonen's Map B. Krekelberg, J.G.Taylor (UK) Nitric Oxide and the Development of Long-Range Horizontal Connectivity L.B. Litinsky (Russia) Neural Networks and Factor Analysis L. Nikolic, M. Kejhar, P. Simandl, P. Holoubek (Czech Republic) Analog CMOS Blocks for VLSI Implementation of Programmable Neural Networks I. Silkis (Russia) The Model of Long-Term Modification (LTD,LTP) in the Efficacy of Excitatory and Inhibitory Transmission to Cerebellar Purkinje Neurons W. Skrabek, K. Ignasiak (Poland) Fast VQ Codebook Search in KLT Space K. Zacek (Czech Republic) Modelling of Electron Devices with Mapping Neural Networks 15.30 - 16.00 C o f f e e B r e a k SHORT CONTRIBUTIONS - SECTION A 15.30 - 15.50 S.V. Kartalopoulos (USA) Fuzzy Logic in the Time Domain 15.50 - 16.10 S. Piramuthu (USA) Feature Selection and Neuro-Fuzzy Systems 16.10 - 16.30 A.K. Tsadiras, K.G. Margaritis (Greece) Using Certainty Neurons in Fuzzy Cognitive Maps SHORT CONTRIBUTIONS - SECTION B 15.30 - 15.50 G. Murzina, I. Silkis (Russia) Computational Model of Simultaneous Long-Term Modifications in the Efficacy of Excitatory and Inhibitory Inputs to the Hippocampal Pyramidal Neuron 15.50 - 16.10 H. Djennane, M. Dufosse, A. Kaladjian (France) A Connectionist Hypothesis about the "Mysterious" Origin of Both Error Signals Sent to the Cerebellum & Reinforcement to the Striatum 16.10 - 16.30 I.R. Rezova, A.A. Frolov, V.A. Markevich (Russia) The Influence of a Long-Term Potentiation of the CA1 Hippocampal Field on the Theta Activity and Some Model Notions about the Role of CA1 Field in the Orienting Behaviour of the Animal 16.30 - 17.00 CLOSING CEREMONY All additional requirements and questions related to the scientific program may be addressed to: Institute of Computer Science Academy of Sciences of the Czech Republic Hana Bilkova, secretary Pod Vodarenskou vezi 2 182 07 Prague 8 Phone: (422) 6605 3201, 6605 3220 Fax: (422) 8585789 E-mail: neufuzzy at uivt.cas.cz NEUROFUZZY '95 - REGISTRATION FORM To be faxed or mailed before March 1, 1996 to the Action M Agency, M. Zeithamlova, Vrsovicka 68, 101 00 Prague 10, Czech Republic, fax: (422) 6731 0503 Please, fill in the block letters. ..... ................. ................................ Ms./Mr. First name Surname Institution ................................................. Mailing Address ............................................. ........................................................... .............. ................. ...................... phone fax e-mail IEEE member: YES NO Name of accompanying person ................................. Special needs (vegetarian meals, etc.) ...................... Means of transport (car, train, airplane) ................... Date/time of arrival ........................................ Date of departure ........................................... Accommodation List of available accommodation with required deposit Prices for person/night Single room Double room Deposit 1. Renaissance Prague Hotel DM 285 DM 160 DM 300 2. Forum Hotel DM 245 DM 140 DM 300 3. Betlem Club Praha DM 165 DM 93 DM 300 4. SAX Hotel DM 155 DM 90 DM 300 5. U Zlateho stromu Hotel DM 115 DM 80 DM 300 6. U Sladku Pension DM 65 DM 50 DM 150 7. Petrska Hostel DM 50 DM 30 DM 150 8. Mazanka Hostel DM 45 DM 28 DM 150 Please, reserve the accommodation preferably at: 1st choice ................................................... 2nd choice ................................................... 3rd choice ................................................... Number of nights ............................................... Type of room (single, double) .................................. Name of person sharing the room ................................ PAYMENTS Early payment until March 15,1996 Late payment after March 15,1996 early / late Full Registration Fee, Non-member DM 370 / DM 440 ............ Full Registration Fee, Member IEEE DM 330 / DM 400 ............ Students Fee, Non-member DM 270 / DM 320 ............ Students Fee, Member IEEE DM 220 / DM 260 ............ East Europ. Stud. Fee, Non-member DM 150 / DM 180 ............ East Europ. Stud. Fee, Member IEEE DM 130 / DM 150 ............ Accompanying Person Fee DM 90 ............ Accommodation Deposit DM 300 ............ Accommodation Deposit DM 150 ............ Lunches DM 75 ............ Walking Tour of Prague DM 18 ............ Organ Concert DM 20 ............ A Night with Mozart DM 25 ............ Total Amount ............ For payment by a credit card (MasterCard, EuroCard, VISA, JCB, Diner Club) Type of credit card ........................................... Credit card No. ..................... Expiration .............. I, the undersigned, give the authorization to Action M Agency to withdraw from my account the equivalent in Czech Crowns of the total amount of DM ........................................ Signature ................... I agree to withdraw from my credit card the accommodation balance Signature ................... For payment by bank Name of bank ................................................. Date of payment .............................................. Date ...................... Signature .......................... ------------------------------------------------------------------- NEuroFuzzy'96 Hana Bilkova - secretary Institute of Computer Science, AS CR Pod vodarenskou vezi 2 182 07 Praha 8, Czech Republic phone:(+422) 66052080, 66053201, 66053220 fax: (+422) 8585789 ------------------------------------------------------------ Professor Trevor Clarkson ./././ ./././ ./././ Communications Research Group ./ ./ ./ ./ Dept of Electronic & Electrical Eng ./ ./././ ./ /./ King's College London ./ ./ ./ ./ ./ Strand, London WC2R 2LS, UK ./././ ./ ./ ./././ Tel: +44 171 873 2367 Fax: +44 171 836 4781 Email: tgc at kcl.ac.uk WWW: http://crg.eee.kcl.ac.uk/ ------------------------------------------------------------ From lehr at simoon.Stanford.EDU Mon Feb 5 06:48:18 1996 From: lehr at simoon.Stanford.EDU (Mike Lehr) Date: Mon, 5 Feb 96 03:48:18 PST Subject: Dissertation available: Scaled Stochastic Methods for Training Neural Networks Message-ID: <9602051148.AA13302@simoon.Stanford.EDU> My PhD dissertation is available for electronic retrieval. Retrieval information appears at the bottom of this message. This thesis deals with the problem of training large nonlinear feedforward neural networks using practical stochastic descent methods. Four major topics are explored: (1) An O(N) statistical method that determines a reasonable estimate of the optimal time-varying learning parameter for the stochastic backpropagation procedure (last half of Chapter 4 and parts of Chapter 7). (2) An accelerated O(N) learning procedure that performs an optimal stochastic update along an arbitrary search direction (Chapters 5 and 7 and Appendix I). (3) An O(N) stochastic method which generates an estimate of the Newton direction (Chapters 6 and 7). (4) Various O(N) methods that generate second-order information and other essential information about a neural network's Sum Square Error and Mean Square Error surfaces (Appendices J and K and parts of Chapter 7). An abstract follows. --------------------------------------------------- Scaled Stochastic Methods for Training Neural Networks Michael A. Lehr Department of Electrical Engineering Stanford University Supervisor: Bernard Widrow The performance surfaces of large neural networks contain ravines, ``flat spots,'' nonconvex regions, and other features that make weight optimization difficult. Although a variety of sophisticated alternatives are available, the simple on-line backpropagation procedure remains the most popular method for adapting the weights of these systems. This approach, which performs stochastic (or incremental) steepest descent, is significantly hampered by the character of the performance surface. Backpropagation's principal advantage over alternate methods rests in its ability to perform an update following each pattern presentation, while maintaining time and space demands that grow only linearly with the number of adaptive weights. In this dissertation, we explore new stochastic methods that improve on the learning speed of the backpropagation algorithm, while retaining its linear complexity. We begin by examining the convergence properties of two deterministic steepest descent methods. Corresponding scaled stochastic algorithms are then developed from an analysis of the neural network's Expected Mean Square Error (EMSE) sequence in the neighborhood of a local minimum of the performance surface. To maintain stable behavior over broad conditions, this development uses a general statistical model for the neural network's instantaneous Hessian matrix. For theoretical performance comparisons, however, we require a more specialized statistical framework. The corresponding analysis helps reveal the complementary convergence properties of the two updates---a relationship we exploit by combining the updates to form a family of dual-update procedures. Effective methods are established for generating a slowly varying sequence of search direction vectors and all required scaling information. The result is a practical algorithm which performs robustly when the weight vector of a large neural network is placed at arbitrary initial positions. The two weight updates are scaled by parameters computed from recursive estimates of five scalar sequences: the first and second moments of the trace of the instantaneous Hessian matrix, the first and second moments of the instantaneous gradient vector's projection along the search direction, and the first moment of the instantaneous Hessian's ``projection'' along the same direction. ----------------------------------------------------------- RETRIEVAL INFORMATION: The thesis available at the URL: http://www-isl.stanford.edu/people/lehr in four gzipped postscript files: thesis1.ps.gz, thesis2.ps.gz, thesis3.ps.gz, and thesis4.ps.gz. The corresponding uncompressed postscript files will be available at the same location for the time being. Including front matter, the thesis contains 405 pages formatted for two-sided printing (size: 2.2M compressed, 10.3M uncompressed). The files can also be obtained by anonymous ftp from the directory /pub/lehr/thesis on either simoon.stanford.edu (36.10.0.209), boreas.stanford.edu (36.60.0.210), or zephyrus.stanford.edu (36.60.0.211). Sorry, hardcopies are not available. From degaris at hip.atr.co.jp Tue Feb 6 14:38:11 1996 From: degaris at hip.atr.co.jp (Hugo de Garis) Date: Tue, 6 Feb 96 14:38:11 JST Subject: 2 POSTDOCS REQUIRED AT ATR's BRAIN BUILDER GROUP Message-ID: <9602060538.AA05031@cam8> 2 POSTDOCS REQUIRED AT ATR's BRAIN BUILDER GROUP, EVOLUTIONARY SYSTEMS DEPT, KYOTO, JAPAN ATR's Brain Builder Group, Kyoto, Japan, needs 2 US postdocs in the fields of A) Mini-Robotics/Mechatronics (to build a robot kitten for ATR's Artificial Brain) B) Evolvable Hardware (to apply Genetic Algorithms to FPGAs (Field Programmable Gate Arrays)) (e.g. Xilinx's XC6200) ATR's Evolutionary Systems Dept (ESD) is (arguably) the strongest ALife group in the world with people such as Tom Ray (of Tierra fame) and Chris Langton (father of ALife, and regular ESD visitor and collaborator). One of the highlights of the ESD is the CAM-Brain Project, which builds/grows/evolves a billion neuron artificial brain using cellular automata based neural modules which will grow inside our cellular automata machine (a hundred billion cell updates a second). This artificial brain requires a body to house it, hence our group needs a body builder. If you have extensive experience in building minirobots with off the shelf components, then you might like to join our brain builder group. Ideally, we want to grow/evolve our neural circuits directly in hardware at hardware speeds. We are looking for a second postdoc in the new field of evolvable hardware. If you have extensive experience in FPGA use, and are familiar with genetic algorithms and neural networks, then please join us. Applicants should have a PhD, be US citizens (or have a green card). The working period is from 3 months to 2 years, preferably 2 years, granted by the US NSF (National Science Foundation). The actual money comes from the Japanese "Japan Foundation" and their Center for Global Partnership. The grants cover salary, airfare, rent, but not research costs. Selection will be a two phase process. The first is to be recommended by us. Then your application has to be sent to the NSF in Washington DC by April 1 1996. (Applications are received twice yearly, April 1 and November 1). The NSF people say that if the candidate and the project are good, the odds of selection are 50%. Probable starting date in Japan would be about September 1996. If you do a good job, there's a possibility that you could stay on at ATR on a long term basis. Sabbatical leave grants are also possible for more senior candidates. The type of candidates we are looking for need to be big egoed dreamers with strong vision and creativity. The senior members of ESD are all pioneers. If you are a CD type (i.e. competent dullard, meaning high in analytical skills, but lacking in vision and creativity), then this spot is not for you). If you are interested, please send your resume by email to - Dr. Hugo de Garis, Brain Builder Group, Evolutionary Systems Dept., ATR Human Information Processing Research Labs, 2-2 Hikari-dai, Seika-cho, Soraku-gun, Kansai Science City, Kyoto-fu, 619-02, Japan. tel. + 81 774 95 1079, fax. + 81 774 59 1008, email. degaris at hip.atr.co.jp For more information from the NSF, contact - email. info at nsf.gov tel. 703 306 1234 or 703 306 0090 web. http://www.nsf.gov/ If you have friends you might be interested, please forward this to them. From bishopc at helios.aston.ac.uk Tue Feb 6 08:50:32 1996 From: bishopc at helios.aston.ac.uk (Prof. Chris Bishop) Date: Tue, 06 Feb 1996 13:50:32 +0000 Subject: Reprint of New Book Message-ID: <20582.9602061350@sun.aston.ac.uk> "Neural Networks for Pattern Recognition" Oxford University Press Christopher M. Bishop http://neural-server.aston.ac.uk/NNPR/ Several people have reported difficulty in obtaining copies of this book. In fact demand was much higher than the publishers anticipated and the first print run sold out very quickly. It has now been reprinted and the supply problems should now be resolved. Chris Bishop Aston From horvitz at u.washington.edu Tue Feb 6 23:16:11 1996 From: horvitz at u.washington.edu (Eric Horvitz) Date: Tue, 6 Feb 1996 20:16:11 -0800 (PST) Subject: cfp: UAI '96 (Uncertainty in Artificial Intelligence) Message-ID: ========================================================= C A L L F O R P A P E R S (Note Revised Dates) ========================================================= ** U A I 96 ** THE TWELFTH ANNUAL CONFERENCE ON UNCERTAINTY IN ARTIFICIAL INTELLIGENCE August 1-3, 1996 Reed College Portland, Oregon, USA ======================================= See the UAI-96 WWW page at http://cuai-96.microsoft.com/ CALL FOR PAPERS The effective handling of uncertainty is critical in designing, understanding, and evaluating computational systems tasked with making intelligent decisions. For over a decade, the Conference on Uncertainty in Artificial Intelligence (UAI) has served as the central meeting on advances in methods for reasoning under uncertainty in computer-based systems. The conference is the annual international forum for exchanging results on the use of principled uncertain-reasoning methods to solve difficult challenges in AI. Theoretical and empirical contributions first presented at UAI have continued to have significant influence on the direction and focus of the larger community of AI researchers. The scope of UAI covers a broad spectrum of approaches to automated reasoning and decision making under uncertainty. Contributions to the proceedings address topics that advance theoretical principles or provide insights through empirical study of applications. Interests include quantitative and qualitative approaches, and traditional as well as alternative paradigms of uncertain reasoning. Innovative applications of automated uncertain reasoning have spanned a broad spectrum of tasks and domains, including systems that make autonomous decisions and those designed to support human decision making through interactive use. We encourage submissions of papers for UAI-96 that report on advances in the core areas of representation, inference, learning, and knowledge acquisition, as well as on insights derived from building or using applications of uncertain reasoning. Topics of interest include (but are not limited to): >> Foundations * Theoretical foundations of uncertain belief and decision * Uncertainty and models of causality * Representation of uncertainty and preference * Generalization of semantics of belief * Conceptual relationships among alternative calculi * Models of confidence in model structure and belief >> Principles and Methods * Planning under uncertainty * Temporal reasoning * Markov processes and decisions under uncertainty * Qualitative methods and models * Automated construction of decision models * Abstraction in representation and inference * Representing intervention and persistence * Uncertainty and methods for learning and datamining * Computation and action under limited resources * Control of computational processes under uncertainty * Time-dependent utility and time-critical decisions * Uncertainty and economic models of problem solving * Integration of logical and probabilistic inference * Statistical methods for automated uncertain reasoning * Synthesis of Bayesian and neural net techniques * Algorithms for uncertain reasoning * Advances in diagnosis, troubleshooting, and test selection >> Empirical Study and Applications * Empirical validation of methods for planning, learning, and diagnosis * Enhancing the human--computer interface with uncertain reasoning * Uncertain reasoning in embedded, situated systems (e.g., softbots) * Automated explanation of results of uncertain reasoning * Nature and performance of architectures for real-time reasoning * Experimental studies of inference strategies * Experience with knowledge-acquisition methods * Comparison of repres. and inferential adequacy of different calculi * Uncertain reasoning and information retrieval For papers focused on applications in specific domains, we suggest that the following issues be addressed in the submission: - Why was it necessary to represent uncertainty in your domain? - What are the distinguishing properties of the domain and problem? - What kind of uncertainties does your application address? - Why did you decide to use your particular uncertainty formalism? - What theoretical problems, if any, did you encounter? - What practical problems did you encounter? - Did users/clients of your system find the results useful? - Did your system lead to improvements in decision making? - What approaches were effective (ineffective) in your domain? - What methods were used to validate the effectiveness of the systems? ================================= SUBMISSION AND REVIEW OF PAPERS ================================= Papers submitted for review should represent original, previously unpublished work (details on policy on submission uniqueness are available at the UAI 96 www homepage). Submitted papers will be evaluated on the basis of originality, significance, technical soundness, and clarity of exposition. Papers may be accepted for presentation in plenary or poster sessions. All accepted papers will be included in the Proceedings of the Twelfth Conference on Uncertainty in Artificial Intelligence, published by Morgan Kaufmann Publishers. Outstanding student papers will be selected for special distinction. Submitted papers must be at most 20 pages of 12pt Latex article style or equivalent (about 4500 words). See the UAI-96 homepage for additional details about UAI submission policies. We strongly encourage the electronic submission of papers. To submit a paper electronically, send an email message to uai at microsoft.com that includes the following information (in this order): * Paper title (plain text) * Author names, including student status (plain text) * Surface mail and email address for a contact author (plain text) * A short abstract including keywords or topic indicators (plain text) An electronic version of the paper (Postscript format) should be submitted simultaneously via ftp to: cuai-96.microsoft.com/incoming. Files should be named $.ps, where $ is an identifier created from the first five letters of the last name of the first author, followed by the first initial of the author's first name. Multiple submissions by the same first author should be indicated by adding a number (e.g., pearlj2.ps) to the end of the identifier. Authors will receive electronic confirmation of the successful receipt of their articles. Authors unable to access ftp should electronically mail the first four items and the Postscript file of their paper to uai at microsoft.com. Authors unable to submit Postscript versions of their paper should send the first four items in email and 5 copies of the complete paper to one of the Program Chairs at the addresses listed below. ++++++++++++++++++++++++++++++ Important Dates (Note revisions) ++++++++++++++++++++++++++++++ >> Submissions must be received by 5PM local time: March 1, 1996 >> Notification of acceptance on or before: April 19, 1996 >> Camera-ready copy due: May 15, 1996 ========================== Program Cochairs: ================= Eric Horvitz Microsoft Research, 9S Redmond, WA 98052 Phone: (206) 936 2127 Fax: (206) 936 0502 Email: horvitz at microsoft.com WWW: http://www.research.microsoft.com/research/dtg/horvitz/ Finn Jensen Department of Mathematics and Computer Science Aalborg University Fredrik Bajers Vej 7,E DK-9220 Aalborg OE Denmark Phone: +45 98 15 85 22 (ext. 5024) Fax: +45 98 15 81 29 Email: fvj at iesd.auc.dk WWW: http://www.iesd.auc.dk/cgi-bin/photofinger?fvj General Conference Chair (General conference inquiries): ======================== Steve Hanks Department of Computer Science and Engineering, FR-35 University of Washington Seattle, WA 98195 Tel: (206) 543 4784 Fax: (206) 543 2969 Email: hanks at cs.washington.edu Program Committee =================== Fahiem Bacchus (U Waterloo) * Salem Benferhat (U Paul Sabatier) * Mark Boddy (Honeywell) * Piero Bonissone (GE) * Craig Boutilier (U Brit Columbia) * Jack Breese (Microsoft) * Wray Buntine (Thinkbank) * Luis M. de Campos * (U Granada) * Enrique Castillo (U Cantabria) * Eugene Charniak (Brown) * Greg Cooper (U Pittsburgh) * Bruce D'Ambrosio (Oregon State) * Paul Dagum (Stanford) * Adnan Darwiche (Rockwell) * Tom Dean (Brown) * Denise Draper (Rockwell) * Marek Druzdzel (U Pittsburgh) * Didier Dubois (Paul Sabatier) * Ward Edwards (USC) * Kazuo Ezawa (ATT Labs) * Robert Fung (Prevision) * Linda van der Gaag (Utrecht U) * Hector Geffner (Simon Bolivar) * Dan Geiger (Technion) * Lluis Godo (Barcelona) * Robert Goldman (Honeywell) * Moises Goldszmidt (Rockwell) * Adam Grove (NEC) * Peter Haddawy (U Wisc-Milwaukee) * Petr Hajek (Czech Acad Sci) * Joseph Halpern (IBM) * Steve Hanks (U Wash) * Othar Hansson (Berkeley) * Peter Hart (Ricoh) * David Heckerman (Microsoft) * Max Henrion (Lumina) * Frank Jensen (Hugin) * Michael Jordan (MIT) * Leslie Pack Kaelbling (Brown) * Keiji Kanazawa (Microsoft) * Uffe Kjaerulff (U Aalborg) * Daphne Koller (Stanford) * Paul Krause (Imp. Cancer Rsch Fund) * Rudolf Kruse (U Braunschweig) * Henry Kyburg (U Rochester) * Jerome Lang (U Paul Sabatier) * Kathryn Laskey (George Mason) * Paul Lehner (George Mason) * John Lemmer (Rome Lab) * Tod Levitt (IET) * Ramon Lopez de Mantaras (Spanish Sci. Rsch Council) * David Madigan (U Wash) * Eric Neufeld (U Saskatchewan) * Ann Nicholson (Monash U) * Nir Friedman (Stanford) * Judea Pearl (UCLA) * Mark Peot (Stanford) * Kim Leng Poh, (Natl U Singapore) * David Poole (U Brit Columbia) * Henri Prade (U Paul Sabatier) * Greg Provan (Inst. Learning Sys) * Enrique Ruspini (SRI) * Romano Scozzafava (Dip. Mo. Met., Rome) * Ross Shachter (Stanford) * Prakash Shenoy (U Kansas) * Philippe Smets (U Bruxelles) * David Spiegelhalter (Cambridge U) * Peter Spirtes (CMU) * Milan Studeny (Czech Acad Sci) * Sampath Srinivas (Microsoft) * Jaap Suermondt (HP Labs) * Marco Valtorta (U S.Carolina) * Michael Wellman (U Michigan) * Nic Wilson (Oxford Brookes U) * Y. Xiang (U Regina) * Hong Xu (U Bruxelles) * John Yen (Texas A&M) * Lian Wen Zhang, (Hong Kong U) * --------------- UAI-96 will occur right before KDD-96, AAAI-96, and the AAAI workshops, and will be in close proximity to these meetings. * * * UAI 96 will include a full-day tutorial program on uncertain reasoning on the day before the main UAI 96 conference (Wednesday, July 31) at Reed College. Details on the tutorials are available on the UAI 96 www homepage. * * * Refer to the UAI-96 WWW home page for late-breaking information: http://cuai-96.microsoft.com/ From ling at cs.hku.hk Tue Feb 6 22:32:37 1996 From: ling at cs.hku.hk (Charles X. Ling) Date: Wed, 7 Feb 1996 11:32:37 +0800 Subject: AAAI-96 Workshop: Computational Cognitive Modeling Message-ID: <9602070332.AA12863@sparc419> We are looking forward to a productive meeting. We seek for a balance between different models (such as connectionists and symbolic models). Submissions from cognitive scientists, AI researchers, and psychologists are warmly welcome. Charles Ling ************************ Computational Cognitive Modeling: Source of the Power AAAI-96 Workshop (During AAAI'96, IAAI 96 and KDD 96. August 4-8, 1996. Portland, Oregon) URL: http://www.cs.hku.hk/~ling for updated information. CALL FOR PAPERS AND PARTICIPATION Aims of the Workshop ==================== Computational models for various cognitive tasks have been extensively studied by cognitive scientists, AI researchers, and psychologists. These tasks include -- language acquisition (learning past tense, word reading and naming, learning grammar, etc.) -- cognitive skill acquisition (subconscious learning, learning sequences) -- cognitive development (the balance scale and learning arithmetic) -- conceptual development; reasoning (commonsense, analogical) We attempt to bring researchers from different backgrounds together, and to examine how and why computational models (connectionist, symbolic, memory-based, or others) are successful in terms of the source of power. The possible sources of power include: -- Representation of the task; -- General properties of the learning algorithm; -- Data sampling/selection; -- Parameters of the learning algorithms. The workshop will focus on, but not be limited to, the following topics, all of which should be discussed in relation to the source of power: -- Proper criteria for judging success or failure of a model. -- Methods for recognizing the source of power. -- Analyses of the success or failure of existing models. -- Presentation of new cognitive models. We hope that our workshop will shed new light on these questions, stimulate lively discussions on the topics, as well as generate new research ideas. Format of the Workshop: ====================== The Workshop will consist of invited talks, presentations, and a poster session. All accepted papers (presentation or poster) will be included in the Workshop Working Notes. A pannel will summarize and debate at the end of the Workshop. Submission information: ====================== Submissions from AI researchers, cognitive scientists and psychologists are welcome. We encourage submissions from people of divergent backgrounds. Potential presenters should submit a paper (maximum 12 pages total, 12 point font). We strongly encourage email submissions of text/postscript files; or you may also send 4 paper copies to one workshop co-chair: Charles Ling (co-chair) Ron Sun (co-chair) Department of Computer Science Department of Computer Science University of Hong Kong University of Alabama Hong Kong Tuscaloosa, AL 35487 ling at cs.hku.hk rsun at cs.ua.edu (On leave from University of Western Ontario) Researchers interested in attending Workshop only should send a short description of interests to one co-chair by deadline. Deadline for submission: March 18, 1996. Notification of acceptance: April 15, 1996. Submission of final versions: May 13, 1996. Program Committee: ================= Charles Ling (co-chair), University of Hong Kong, ling at cs.hku.hk Ron Sun (co-chair), University of Alabama, rsun at cs.ua.edu Pat Langley, Stanford University, langley at flamingo.Stanford.EDU Mike Pazzani, UC Irvine, pazzani at super-pan.ICS.UCI.EDU Tom Shultz, McGill University, shultz at psych.mcgill.ca Paul Thagard, Univ. of Waterloo, pthagard at watarts.uwaterloo.ca Kurt VanLehn, Univ. of Pittsburgh, vanlehn+ at pitt.edu Invited Speakers (NEW): ====================== We are glad to have the following confirmed invited speakers to present their work at the Workshop: Jeff Elman Mike Pazzani Aaron Sloman Denis Mareschal From terry at salk.edu Thu Feb 8 18:05:54 1996 From: terry at salk.edu (Terry Sejnowski) Date: Thu, 8 Feb 96 15:05:54 PST Subject: Neural Computation 8:2 Message-ID: <9602082305.AA23531@salk.edu> Neural Computation Volume 8, Issue 2, February 15, 1996 Article Encoding with Bursting, Subthreshold Oscillations and Noise in Mammalian Cold Receptors Andre Longtin and Karin Hinzer Notes Associative Memory with Uncorrelated Inputs Ronald Michaels Statistical Independence and Novelty Detection with Information Preserving Nonlinear Maps Lucas Parra, Gustavo Deco and Stefan Miesbach Letters Neural Network Models of Perceptual Learning of Angle Discrimination G. Mato and H. Sompolinsky Directional Filling-In Karl Frederick Arrington Binary-Oscillator Networks: Bridging a Gap between Experimental and Abstract Modeling of Neural Networks Wei-Ping Wang Annealed Competition of Experts for a Segmentation and Classification of Switching Dynamics Klaus Pawelzik, Jens Kohlmorgen and Klaus-Robert Muller A Recurrent Network Implementation of Time Series Classification Vasilios Petridis and Athanasios Kehagias Temporal Segmentation in a Neural Dynamical System David Horn and Irit Opher Circular Nodes in Neural Networks Michael J. Kirby and Rick Miranda The Computational Power of Discrete Hopfield Nets with Hidden Units Pekka Orponen A Self-Organizing Neural Network for the Traveling Salesman Problem That Is Competitive with Simulated Annealing Marco Budinich Hierarchical, Unsupervised Learning with Growing Via Phase Transitions David Miller and Kenneth Rose The Interchangeability of Learning Rate and Gain in Backpropagation Neural Networks Georg Thimm , Perry Moerland and Emile Fiesler ----- ABSTRACTS - http://www-mitpress.mit.edu/jrnls-catalog/neural.html SUBSCRIPTIONS - 1996 - VOLUME 8 - 8 ISSUES ______ $50 Student and Retired ______ $78 Individual ______ $220 Institution Add $28 for postage and handling outside USA (+7% GST for Canada). (Back issues from Volumes 1-7 are regularly available for $28 each to institutions and $14 each for individuals Add $5 for postage per issue outside USA (+7% GST for Canada) mitpress-orders at mit.edu MIT Press Journals, 55 Hayward Street, Cambridge, MA 02142. Tel: (617) 253-2889 FAX: (617) 258-6779 ----- From ericwan at choosh.eeap.ogi.edu Thu Feb 8 20:48:53 1996 From: ericwan at choosh.eeap.ogi.edu (Eric A. Wan) Date: Fri, 9 Feb 1996 09:48:53 +0800 Subject: FIR/TDNN Toolbox for MATLAB Message-ID: <9602091748.AA04337@choosh.eeap.ogi.edu> ***************************************************************** * * * FIR/TDNN Toolbox for MATLAB * * * ***************************************************************** ***************************************************************** DESCRIPTION: Beta version of a toolbox for FIR (Finite Impulse Response) and TD (Time Delay) Neural Networks. For efficient stochastic implementation, algorithms are written as MEX compatible c-code which can be called as primitive functions from within MATLAB. Both source and compiled functions are available. LOCATION: http://www.eeap.ogi.edu/~ericwan/fir.html +----------------------------------------------------------------------------+ | Eric A. Wan | Dept. of Electrical Engineering and Applied Physics | | | Oregon Graduate Institute of Science & Technology | +----------------------+-----------------------------------------------------+ | ericwan at eeap.ogi.edu | Mailing: | Shipping: | | tel (503) 690-1164 | P.O. Box 91000 | 20000 N.W. Walker Road | | fax (503) 690-1406 | Portland, OR 97291-1000 | Beaverton, OR 97006 | +----------------------------------------------------------------------------+ | Home page: http://www.cse.ogi.edu/~ericwan | +----------------------------------------------------------------------------+ From kak at ee.lsu.edu Fri Feb 9 17:15:59 1996 From: kak at ee.lsu.edu (Subhash Kak) Date: Fri, 9 Feb 96 16:15:59 CST Subject: Papers Message-ID: <9602092215.AA25252@ee.lsu.edu> The following two papers are available: 1. Information, Physics and Computation by Subhash C. Kak appearing in FOUNDATIONS OF PHYSICS, vol 26, 1996 ftp://gate.ee.lsu.edu/pub/kak/inf.ps.Z 2. Can we have different levels of artificial intelligence? by Subhash C. Kak appearing in JOURNAL OF INTELLIGENT SYSTEMS, vol 6, 1996 ftp://gate.ee.lsu.edu/pub/kak/ai.ps.Z ------------------------------------------------------------ Abstracts: --------- 1. Information, Physics and Computation The paper presents several observations on the connections between information, physics and computation. In particular, the computing power of quantum computers is examined. Quantum theory is characterized by superimposed states and non-local interactions. It is argued that recently studied quantum computers, which are based on local interactions, cannot simulate quantum physics. 2. Can we have different levels of artificial intelligence? This paper argues for a graded approach to the study of artificial intelligence. In contrast to the Turing test, such an approach permits the measurement of incremental progress in AI research. Results on the conceptual abilities of pigeons are summarized. These abilities far exceed the generalization abilities of current AI programs. It is argued that matching the conceptual abilities of animals would require new approaches to AI. Defining graded levels of intelligence would permit the identification of resources needed for implementation. From Dave_Touretzky at DST.BOLTZ.CS.CMU.EDU Fri Feb 9 23:35:22 1996 From: Dave_Touretzky at DST.BOLTZ.CS.CMU.EDU (Dave_Touretzky@DST.BOLTZ.CS.CMU.EDU) Date: Fri, 09 Feb 96 23:35:22 EST Subject: undergrad summer research program Message-ID: <16034.823926922@DST.BOLTZ.CS.CMU.EDU> NPC (Neural Processes in Cognition) Undergraduate Summer Program The Neural Processes in Cognition Training Program at the University of Pittsburgh and Carnegie Mellon University has several positions available for undergraduates interested in studying cognitive or computational neuroscience. These are growing interdisciplinary areas of study (see Science, 1993, v. 261, pp 1805-7) that interpret cognitive functions in terms of neuroanatomical and neurophysiological data and computer simulations. Undergraduate students participating in the summer program will have the opportunity to spend ten weeks of intensive involvement in laboratory research supervised by one of the program's faculty. The summer program also includes weekly journal clubs and a series of informal lectures. Students selected for the program will receive a $2500 stipend. The program is funded by a grant from the National Science Foundation and by the joint CMU/University of Pittsburgh Center for the Neural Basis of Cognition. Each student's research program will be determined in consultation with the training program's Director. Potential laboratory environments include single unit recording, neuroanatomy, computer simulation of biological and cognitive effects, robot control, neuropsychological assessment, behavioral assessment, and brain imaging. How to Apply to the NPC Undergraduate Summer Program: Applications are encouraged from highly motivated undergraduate students with interests in biology, psychology, engineering, physics, mathematics or computer science. Application deadline is March 15, 1996. To apply, request application materials by email at neurocog at vms.cis.pitt.edu, phone 412-624-7064, or write to the address below. The materials include a listing of faculty research areas to consider. Applicants are asked to supply a statement of their research interests, a recent school transcript, one faculty letter of recommendation, and a selection of one or two research areas which they would like to explore. Applicants are strongly encouraged to identify a particular faculty member with whom they want to work. Send requests and application materials to: Professor Walter Schneider, Program Director University of Pittsburgh Neural Processes in Cognition Program 3939 O'Hara Street Pittsburgh, PA 15260 Email: neurocog at vms.cis.pitt.edu Note: the Neural Processes in Cognition program also offers pre- and post-doctoral training. To find out more about the program or the Center for the Neural Basis of Cognition, visit our web sites: http://www.cs.cmu.edu/Web/Groups/CNBC http://neurocog.lrdc.pitt.edu/npc/npc.html From payman at uw-isdl.ee.washington.edu Sun Feb 11 17:48:03 1996 From: payman at uw-isdl.ee.washington.edu (Payman Arabshahi) Date: Sun, 11 Feb 1996 14:48:03 -0800 (PST) Subject: IEEE NNC Homepage - Call for Submissions Message-ID: <199602112248.OAA11919@uw-isdl.ee.washington.edu> The IEEE Neural Network Council's Homepage (http://www.ieee.org/nnc) is seeking information about Computational Intelligence Research Programs Worldwide, both in academia and industry. If you know of a relevant research group's homepage or would like a link to your homepage from the IEEE NNC research page, please let me know. At present we maintain links to some 64 Neural Computing Programs, and are especially seeking information on programs in Canada, Central and South America, Asia and the Middle East, Eastern Europe, the former Soviet Union, and Africa. Thank you for your cooperation. -- Payman Arabshahi Tel : (205) 895-6380 Dept. of Electrical & Computer Eng. Fax : (205) 895-6803 University of Alabama in Huntsville payman at ebs330.eb.uah.edu Huntsville, AL 35899 http://www.eb.uah.edu/ece/ From tony at discus.anu.edu.au Mon Feb 12 22:30:29 1996 From: tony at discus.anu.edu.au (Tony BURKITT) Date: Tue, 13 Feb 1996 14:30:29 +1100 (EST) Subject: ACNN'96 registration information Message-ID: <199602130330.OAA24815@discus.anu.edu.au> REGISTRATION INFORMATION ACNN'96 SEVENTH AUSTRALIAN CONFERENCE ON NEURAL NETWORKS 10th - 12th APRIL 1996 Australian National University Canberra, Australia ADVANCE REGISTRATION REMINDER To receive a 20% discount for registration at ACNN'96, you must post a registration form before Friday, February 16th. ACNN'96 The seventh Australian conference on neural networks will be held in Canberra on April 10th - 12th 1996 at the Australian National University. ACNN'96 is the annual national meeting of the Australian neural network community. It is a multi-disciplinary meeting, with contributions from Neuroscientists, Engineers, Computer Scientists, Mathematicians, Physicists and Psychologists. The program will include keynote talks, lecture presentations and poster sessions. Proceedings will be printed and distributed to the attendees. Invited Keynote Speakers ACNN'96 will feature two keynote speakers: Professor Wolfgang Maass, Institute for Theoretical Computer Science, Technical University Graz, "Networks of spiking neurons: The third generation of neural network models," and Professor Steve Redman, John Curtin School of Medical Research, Australian National University, "Modelling versus measuring: a perspective on computational neuroscience." Pre-Conference Workshops There will be two Pre-Conference Workshops on Tuesday 9th April: Memory, time, change and structure in ANNs: Distilling cognitive models into their functional components Convenors: Janet Wiles and J. Devin McAuley, Depts of Computer Science and Psychology, University of Queensland janet at psy.uq.edu.au devin at psy.uq.edu.au The workshop aim is to identify the functional roles of artificial neural network (ANN) components and to understand how they combine to explain cognitive phenomena. Existing ANN models will be distilled into their functional components through case study analysis, targeting three traditional strengths of ANNs - mechanisms for memory, time and change; and one area of weakness - mechanisms for structure. (see http://psy.uq.edu.au/CogPsych/acnn96/workshop.html) Neural Networks in the Australian Public Service Convenor: Andrew Freeman phone: (06) 264 3698 fax: (06) 264 4717 afreeman at pcug.org.au This workshop will provide a venue for an informal exchange of views and experiences between researchers, users, and suppliers of neural technologies for forms processing in the Australian Public Service. (see http://www.pcug.org.au/~afreeman/otsig.html) Special Poster Session ACNN'96 will include a special poster session devoted to recent work and work-in-progress. Abstracts are solicited for this session (1 page limit), and may be submitted up to one week before the commencement of the conference. They will not be refereed or included in the proceedings, but will be distributed to attendees upon arrival. Students are especially encouraged to submit abstracts for this session. Venue Huxley Lecture Theatre, Leonard Huxley Building, Mills Road, Australian National University, Canberra, Australia Further Information For more information on the conference (including the list of accepted papers, pointers to information on pre-conference workshops and tutorials, registration and accomodation information, and the registration form), see the ACNN96 web page: http://wwwsyseng.anu.edu.au/acnn96/ or contact: ACNN'96 Secretariat Department of Engineering FEIT Australian National University Canberra, ACT 0200 Australia Phone: +61 6 249 5645 ftp site: syseng.anu.edu.au:pub/acnn96 email: acnn96 at anu.edu.au ------------------------------------------------------------------------------ ACNN'96 Seventh Australian Conference on Neural Networks Registration Form Title & Name: ___________________________________________________________ Organisation: ___________________________________________________________ Department: _____________________________________________________________ Occupation: _____________________________________________________________ Address: ________________________________________________________________ State: ____________________ Post Code: _____________ Country: ___________ Tel: ( ) __________________________ Fax: ( ) _____________________ E-mail: _________________________________________________________________ [ ] Find enclosed a cheque for the sum of @: ______________________ [ ] Charge my credit card for the sum of # :________________________ Mastercard/Visa/Bankcard# Number : _____________________________ Valid until: ________ Signature: __________________ Date: ______ ------------------------------------------------------------------------------ To register, please fill in this form and return it together with payment to : ACNN'96 Secretariat L. P. O. Box 228 Australian National University Canberra, ACT 2601 Australia ------------------------------------------------------------------------------ @ Registration fees: Before 16 Feb 96 After 16 Feb 96 Full Time Students A$ 96.00 A$120.00 Academics A$208.00 A$260.00 Other A$304.00 A$380.00 # Please encircle type of card ------------------------------------------------------------------------------ From mozer at neuron.cs.colorado.edu Tue Feb 13 14:03:20 1996 From: mozer at neuron.cs.colorado.edu (Mike Mozer) Date: Tue, 13 Feb 1996 12:03:20 -0700 Subject: NIPS*96 CALL FOR PAPERS Message-ID: <199602131903.MAA29791@neuron.cs.colorado.edu> [ Moderator's note: Below is the NIPS*96 call for papers. I would like to remind people that many of the papers from NIPS*95 are now accessible online via the NIPS web site; the URL is given below. Also, NIPS*95 t-shirts and mousepads with the Wizard of Oz theme can now be ordered by mail at heavily discounted prices; see the NIPS web site for details. -- Dave Touretzky ] CALL FOR PAPERS Neural Information Processing Systems -- Natural and Synthetic Monday December 2 - Saturday December 7, 1996 Denver, Colorado This is the tenth meeting of an interdisciplinary conference which brings together cognitive scientists, computer scientists, engineers, neuro- scientists, physicists, and mathematicians interested in all aspects of neural processing and computation. The conference will include invited talks and oral and poster presentations of refereed papers. The conference is single track and is highly selective. Preceding the main session, there will be one day of tutorial presentations (Dec. 2), and following will be two days of focused workshops on topical issues at a nearby ski area (Dec. 6-7). Major categories for paper submission, with example subcategories, are as follows: Algorithms and Architectures: supervised and unsupervised learning algorithms, constructive/pruning algorithms, decision trees, localized basis functions, layered networks, recurrent networks, Monte Carlo algorithms, combinatorial optimization, performance comparisons Applications: database mining, DNA/protein sequence analysis, expert systems, fault diagnosis, financial analysis, medical diagnosis, music processing, time-series prediction Artificial Intelligence and Cognitive Science: perception, natural language, human learning and memory, problem solving, decision making, inductive reasoning, hybrid symbolic-subsymbolic systems Control, Navigation, and Planning: robotic motor control, process control, navigation, path planning, exploration, dynamic programming, reinforcement learning Implementation: analog and digital VLSI, optical neurocomputing systems, novel neuro-devices, simulation tools, parallelism Neuroscience: systems physiology, signal and noise analysis, oscillations, synchronization, mechanisms of inhibition and neuromodulation, synaptic plasticity, computational models Speech, Handwriting, and Signal Processing: speech recognition, coding, and synthesis, handwriting recognition, adaptive equalization, nonlinear noise removal, auditory scene analysis Theory: computational learning theory, complexity theory, dynamical systems, statistical mechanics, probability and statistics, approximation and estimation theory Visual Processing: image processing, image coding and classification, object recognition, stereopsis, motion detection and tracking, visual psychophysics Review Criteria: All submitted papers will be thoroughly refereed on the basis of technical quality, significance, and clarity. Novelty of the work is also a strong consideration in paper selection, but, to encourage interdisciplinary contributions, we will consider work which has been submitted or presented in part elsewhere, if it is unlikely to have been seen by the NIPS audience. Authors should not be dissuaded from submitting recent work, as there will be an opportunity after the meeting to revise accepted manuscripts before submitting final camera-ready copy. Paper Format: Submitted papers may be up to seven pages in length, including figures and references, using a font no smaller than 10 point. Submissions failing to follow these guidelines will not be considered. Authors are encouraged to use the NIPS LaTeX style files obtainable by anonymous FTP at the site given below. Papers must indicate (1) physical and e-mail addresses of all authors; (2) one of the nine major categories listed above, and, if desired, a subcategory; (3) if the work, or any substantial part thereof, has been submitted to or has appeared in other scientific conferences; (4) the authors' preference, if any, for oral or poster presentation; this preference will play no role in paper acceptance; and (5) author to whom correspondence should be addressed. Submission Instructions: Send six copies of submitted papers to the address below; electronic or FAX submission is not acceptable. Include one additional copy of the abstract only, to be used for preparation of the abstracts booklet distributed at the meeting. SUBMISSIONS MUST BE RECEIVED BY MAY 24, 1996. From within the U.S., submissions will be accepted if mailed first class and postmarked by May 21, 1996. Mail submissions to: Michael Jordan NIPS*96 Program Chair Department of Brain and Cognitive Sciences, E10-034D Massachusetts Institute of Technology 79 Amherst Street Cambridge, MA 02139 USA Mail general inquiries and requests for registration material to: NIPS*96 Registration Conference Consulting Associates 451 N. Sycamore Monticello, IA 52310 fax: (319) 465-6709 (attn: Denise Prull) e-mail: nipsinfo at salk.edu Copies of the LaTeX style files for NIPS are available via anonymous ftp at ftp.cs.cmu.edu (128.2.206.173) in /afs/cs/Web/Groups/NIPS/formatting The style files and other conference information may also be retrieved via World Wide Web at http://www.cs.cmu.edu/Web/Groups/NIPS NIPS*96 Organizing Committee: General Chair, Michael Mozer, U. Colorado; Program Chair, Michael Jordan, MIT; Publications Chair, Thomas Petsche, Siemens; Tutorial Chair, John Lazzaro, Berkeley; Workshops Co-Chairs, Michael Perrone, IBM, and Steven Nowlan, Lexicus; Publicity Chair, Suzanna Becker, McMaster; Local Arrangements, Marijke Augusteijn, U. Colorado; Treasurer, Eric Mjolsness, UCSD; Government/Corporate Liaison, John Moody, OGI; Contracts, Steve Hanson, Siemens, Scott Kirkpatrick, IBM, Gerry Tesauro, IBM. Conference arrangements by Conference Consulting Associates, Monticello, IA. DEADLINE FOR RECEIPT OF SUBMISSIONS IS MAY 24, 1996 - please post - From iconip at cs.cuhk.hk Wed Feb 14 05:13:26 1996 From: iconip at cs.cuhk.hk (ICONIP96) Date: Wed, 14 Feb 1996 18:13:26 +0800 Subject: ICONIP'96 EXTENSION OF PAPER SUBMISSION DEADLINE Message-ID: <199602141013.SAA00808@cs.cuhk.hk> ====================================================================== We apologize should you receive multiple copies of this CFP from different sources. ====================================================================== ************************************************ ICONIP'96 EXTENSION OF PAPER SUBMISSION DEADLINE ************************************************ 1996 International Conference on Neural Information Processing The Annual Conference of the Asian Pacific Neural Network Assembly ICONIP'96, September 24 - 27, 1996 Hong Kong Convention and Exhibition Center, Wan Chai, Hong Kong In cooperation with IEEE / NNC --IEEE Neural Networks Council INNS - International Neural Network Society ENNS - European Neural Network Society JNNS - Japanese Neural Network Society CNNC - China Neural Networks Council ====================================================================== In consideration of many requests from Europe, USA as well as Asia Pacific region for a possible extension of paper submission deadline for ICONIP'96, the ICONIP'96 Organizing Committee has decided to extend the paper submission deadline. ----- The Extended Paper Submission Deadline : March 10, 1996 ---- The goal of ICONIP'96 is to provide a forum for researchers and engineers from academia and industry to meet and to exchange ideas on the latest developments in neural information processing. The conference also further serves to stimulate local and regional interests in neural information processing and its potential applications to industries indigenous to this region. The conference consists of two tracks. One is SCIENTIFIC TRACK for the latest results on Theories, Technologies, Methods, Architectures and Algorithms in neural information processing. The other is APPLICATION TRACK for various neural network applications in any engineering/technical field and any business/service sector. There will be a one-day tutorial on the neural networks for capital markets which reflects Hong Kong's local interests on financial services. In addition, there will be several invited lectures in the main conference. Hong Kong is one of the most dynamic cities in the world with world-class facilities, easy accessibility, exciting entertainment, and high levels of service and professionalism. Come to Hong Kong! Visit this Eastern Pearl in this historical period before Hong Kong's return to China in 1997. ********************* CONFERENCE'S SCHEDULE ********************* Submission of paper (extended) March 10, 1996 Notification of acceptance May 1, 1996 Tutorial on Financial Engineering Sept. 24, 1996 Conference Sept. 25-27, 1996 *** *** *** The Conference Proceedings will be published by Springer Verlag. *** *** *** Registration forms, detailed tutorial information, invited talks and other related information will be available on the WWW site in due course. ********************************** Tutorials On Financial Engineering ********************************** 1. Professor John Moody, Oregon Graduate Institute, USA "Time Series Modeling: Classical and Nonlinear Approaches" 2. Professor Halbert White, University California, San Diego, USA "Option Pricing In Modern Finance Theory and the Relevance Of Artificial Neural Networks" 3. Professor A-P. N. Refenes, London Business School, UK "Neural Networks in Financial Engineering" ************* Keynote Talks ************* 1. Professor Shun-ichi Amari, Tokyo University. "Information Geometry of Neural Networks" 2. Professor Yaser Abu-Mostafa, California Institute of Technology, USA "The Bin Model for Learning and Generalization" 3. Professor Leo Breiman, University California, Berkeley, USA "Democratizing Predictors" 4. Professor Christoph von der Malsburg, Ruhr-Universitat Bochum, Germany "Scene Analysis Based on Dynamic Links" (tentatively) 5. Professor Erkki Oja, Helsinki University of Technology, Finland "Blind Signal Separation by Neural Networks " ************** Honored Talks ************** 1. Rolf Eckmiller, University of Bonn, Germany "Concerning the Development of Retina Implants with Neural Nets" 2. Mitsuo Kawato, ATR Human Information Processing Research Lab, Japan "Generalized Linear Model Analysis of Cerebellar Motor Learning" 3. Kunihiko Fukushima, Osaka University, Japan "To be announced" *** PLUS AROUND 20 INVITED PAPERS GIVEN BY WELL KNOWN RESEARCHERS IN THE FIELD. *** ***************** CONFERENCE TOPICS ***************** SCIENTIFIC TRACK: APPLICATION TRACK: ----------------- ------------------ * Theory * Foreign Exchange * Algorithms & Architectures * Equities & Commodities * Supervised Learning * Risk Management * Unsupervised Learning * Options & Futures * Hardware Implementations * Forecasting & Strategic Planning * Hybrid Systems * Government and Services * Neurobiological Systems * Geophysical Sciences * Associative Memory * Telecommunications * Visual & Speech Processing * Control & Modeling * Intelligent Control & Robotics * Manufacturing * Cognitive Science & AI * Chemical Engineering * Recurrent Net & Dynamics * Transportation * Image Processing * Environmental Engineering * Pattern Recognition * Remote Sensing * Computer Vision * Defense * Time Series Prediction * Multimedia Systems * Optimization * Document Processing * Fuzzy Logic * Medical Imaging * Evolutionary Computing * Biomedical Application * Other Related Areas * Other Related Applications ********************** SUBMISSION INFORMATION ********************** Authors are invited to submit one camera-ready original (do not staple) and five copies of the manuscript written in English on A4-format (or letter) white paper with 25 mm (1 inch) margins on all four sides, in one column format, no more than six pages (four pages preferred) including figures and references, single-spaced, in Times-Roman or similar font of 10 points or larger, without page numbers, and printed on one side of the page only. Electronic or fax submission is not acceptable. Additional pages will be charged at USD $50 per page. Centered at the top of the first page should be the complete title, author(s), affiliation, mailing, and email addresses, followed by an abstract (no more than 150 words) and the text. Each submission should be accompanied by a cover letter indicating the contacting author, affiliation, mailing and email addresses, telephone and fax number, and preference of track, technical session(s), and format of presentation, either oral or poster. All submitted papers will be refereed by experts in the field based on quality, clarity, originality, and significance. Authors may also retrieve the ICONIP style, "iconip.tex" and "iconip.sty" files for the conference by anonymous FTP at ftp.cs.cuhk.hk in the directory /pub/iconip96. The address for paper submissions, registration and information inquiries: ICONIP'96 Secretariat Department of Computer Science and Engineering The Chinese University of Hong Kong Shatin, N.T., Hong Kong Fax (852) 2603-5024 E-mail: iconip96 at cs.cuhk.hk http://www.cs.cuhk.hk/iconip96 ***************************** CONFERENCE'S REGISTRATION FEE ***************************** On & Before July 1, 1996, Member HKD $2,800 On & Before July 1, 1996, Non-Member HKD $3,200 Late & On-Site, Member HKD $3,200 Late & On-Site, Non-Member HKD $3,600 Student Registration Fee HKD $1,000 ====================================================================== General Co-Chairs ================= Omar Wing, CUHK Shun-ichi Amari, Tokyo U. Advisory Committee ================== International ------------- Yaser Abu-Mostafa, Caltech Michael Arbib, U. Southern Cal. Leo Breiman, UC Berkeley Jack Cowan, U. Chicago Rolf Eckmiller, U. Bonn Jerome Friedman, Stanford U. Stephen Grossberg, Boston U. Robert Hecht-Nielsen, HNC Geoffrey Hinton, U. Toronto Anil Jain, Michigan State U. Teuvo Kohonen, Helsinki U. of Tech. Sun-Yuan Kung, Princeton U. Robert Marks, II, U. Washington Thomas Poggio, MIT Harold Szu, US Naval SWC John Taylor, King's College London David Touretzky, CMU C. v. d. Malsburg, Ruhr-U. Bochum David Willshaw, Edinburgh U. Lofti Zadeh, UC Berkeley Asia-Pacific Region ------------------- Marcelo H. Ang Jr, NUS, Singapore Sung-Yang Bang, POSTECH, Pohang Hsin-Chia Fu, NCTU., Hsinchu Toshio Fukuda, Nagoya U., Nagoya Kunihiko Fukushima, Osaka U., Osaka Zhenya He, Southeastern U., Nanjing Marwan Jabri, U. Sydney, Sydney Nikola Kasabov, U. Otago, Dunedin Yousou Wu, Tsinghua U., Beijing Organizing Committee ==================== L.W. Chan (Co-Chair), CUHK K.S. Leung (Co-Chair), CUHK D.Y. Yeung (Finance), HKUST C.K. Ng (Publication), CityUHK A. Wu (Publication), CityUHK B.T. Low (Publicity), CUHK M.W. Mak (Local Arr.), HKPU C.S. Tong (Local Arr.), HKBU T. Lee (Registration), CUHK K.P. Chan (Tutorial), HKU H.T. Tsui (Industry Liaison), CUHK I. King (Secretary), CUHK Program Committee ================= Co-Chairs --------- Lei Xu, CUHK Michael Jordan, MIT Erkki Oja, Helsinki U. of Tech. Mitsuo Kawato, ATR Members ------- Yoshua Bengio, U. Montreal Jim Bezdek, U. West Florida Chris Bishop, Aston U. Leon Bottou, Neuristique Gail Carpenter, Boston U. Laiwan Chan, CUHK Huishen Chi, Peking U. Peter Dayan, MIT Kenji Doya, ATR Scott Fahlman, CMU Francoise Fogelman, SLIGOS Lee Giles, NEC Research Inst. Michael Hasselmo, Harvard U. Kurt Hornik, Technical U. Wien Yu Hen Hu, U. Wisconsin - Madison Jeng-Neng Hwang, U. Washington Nathan Intrator, Tel-Aviv U. Larry Jackel, AT&T Bell Lab Adam Kowalczyk, Telecom Australia Soo-Young Lee, KAIST Todd Leen, Oregon Grad. Inst. Cheng-Yuan Liou, National Taiwan U. David MacKay, Cavendish Lab Eric Mjolsness, UC San Diego John Moody, Oregon Grad. Inst. Nelson Morgan, ICSI Steven Nowlan, Synaptics Michael Perrone, IBM Watson Lab Ting-Chuen Pong, HKUST Paul Refenes, London Business School David Sanchez, U. Miami Hava Siegelmann, Technion Ah Chung Tsoi, U. Queensland Benjamin Wah, U. Illinois Andreas Weigend, Colorado U. Ronald Williams, Northeastern U. John Wyatt, MIT Alan Yuille, Harvard U. Richard Zemel, CMU Jacek Zurada, U. Louisville From lawrence at s4.elec.uq.edu.au Wed Feb 14 10:02:07 1996 From: lawrence at s4.elec.uq.edu.au (Steve Lawrence) Date: Thu, 15 Feb 1996 01:02:07 +1000 (EST) Subject: Paper on neural network simulation available Message-ID: <199602141502.BAA05378@s4.elec.uq.edu.au> It has been estimated that 85% of neural network researchers write their own simulation software. The following paper deals with correctness and efficiency in neural network simulation. We present several techniques which we have used in the implementation of our own simulator. We welcome your comments. http://www.elec.uq.edu.au/~lawrence - Australia http://www.neci.nj.nec.com/homepages/lawrence - USA Correctness, Efficiency, Extendability and Maintainability in Neural Network Simulation ABSTRACT A large number of neural network simulators are publicly available to researchers, many free of charge. However, when a new paradigm is being developed, as is often the case, the advantages of using existing simulators decrease, causing most researchers to write their own software. It has been estimated that 85% of neural network researchers write their own simulators. We present techniques and principles for the implementation of neural network simulators. First and foremost, we discuss methods for ensuring the correctness of results - avoiding duplication, automating common tasks, using assertions liberally, implementing reverse algorithms, employing multiple algorithms for the same task, and using extensive visualization. Secondly, we discuss efficiency concerns, including using appropriate granularity object-oriented programming, and pre-computing information whenever possible. From ronnyk at starry.engr.sgi.com Thu Feb 15 17:58:02 1996 From: ronnyk at starry.engr.sgi.com (Ronny Kohavi) Date: Thu, 15 Feb 1996 14:58:02 -0800 Subject: MLC++ : Machine learning library in C++ Message-ID: <199602152258.OAA15759@starry.engr.sgi.com> MLC++ is a machine learning library developed in C++. MLC++ is public domain and can be used free of charge, including use of the source code. MLC++ contains common induction algorithms, such as ID3, nearest-neighbors, naive-bayes, oneR (Holte), winnow, and decision tables, all written under a single framework. MLC++ also contains interfaces to common algorithms, such as C4.5, PEBLS, IB1-4, OC1, CN2. MLC++ contains wrappers to wrap around algorithms. These include: feature selection, discretization filters, automatic parameter setting for C4.5, bagging/combinining classifiers, and more. Finally, MLC++ contains common accuracy estimation methods, such as holdout, cross-validation, and bootstrap .632. Interfaces to existing algorithms are not hard to create and implementing new algorithms in MLC++ is possible with added benefits (some procedures work only on induction algorithms implemented in MLC++ as opposed to interfaced ones). Object code for MLC++ utilities is provided for Silicon Graphic machines running Irix 5.3. To contact us, send e-mail to: mlc at postofc.corp.sgi.com Visit our web page at: http://www.sgi.com/Technology/mlc/ -- Ronny Kohavi (ronnyk at sgi.com, http://robotics.stanford.edu/~ronnyk) From bogus@does.not.exist.com Thu Feb 15 14:41:41 1996 From: bogus@does.not.exist.com () Date: Thu, 15 Feb 1996 20:41:41 +0100 Subject: No subject Message-ID: <9602151941.AA06742@ti-doz10.fbe.fh-weingarten.de> From erik at bbf.uia.ac.be Thu Feb 15 08:38:40 1996 From: erik at bbf.uia.ac.be (Erik De Schutter) Date: Thu, 15 Feb 1996 13:38:40 GMT Subject: Crete Course in Computational Neuroscience Message-ID: <199602151338.NAA02343@kuifje.bbf.uia.ac.be> SECOND CALL CRETE COURSE IN COMPUTATIONAL NEUROSCIENCE AUGUST 25 - SEPTEMBER 20, 1996 CRETE, GREECE DIRECTORS: Erik De Schutter (University of Antwerp, Belgium) Idan Segev (Hebrew University, Jerusalem, Israel) Jim Bower (California Institute of Technology, USA) Adonis Moschovakis (University of Crete, Greece) The Crete Course in Computational Neuroscience introduces students to the practical application of computational methods in neuroscience, in particular how to create biologically realistic models of neurons and networks. The course consists of two complimentary parts. A distinguished international faculty gives morning lectures on topics in experimental and computational neuroscience. The rest of the day is spent learning how to use simulation software and how to implement a model of the system the student wishes to study. The first week of the course introduces students to the most important techniques in modeling single cells, networks and neural systems. Students learn how to use the GENESIS, NEURON, XPP and other software packages on their individual unix workstations. During the following three weeks the lectures will be more general, moving from modeling single cells and subcellular processes through the simulation of simple circuits and large neuronal networks and, finally, to system level models of the cortex and the brain. The course ends with a presentation of the student modeling projects. The Crete Course in Computational Neuroscience is designed for advanced graduate students and postdoctoral fellows in a variety of disciplines, including neurobiology, physics, electrical engineering, computer science and psychology. Students are expected to have a basic background in neurobiology as well as some computer experience. A total of 25 students will be accepted, the majority of whom will be from the European Union and affiliated countries. A tuition fee of 500 ECU ($700) covers lodging, travel from EC countries to Crete and all course-related expenses for European nationals. We specifically encourage applications from researchers younger than 35, from researchers who work in less-favoured regions, from women and from researchers from industry. We encourage students from the Far East and the USA to also apply to this inter- national course. More information and application forms can be obtained: - WWW access: http://bbf-www.uia.ac.be/CRETE/Crete_index.html - by mail: Prof. E. De Schutter Born-Bunge Foundation University of Antwerp - UIA, Universiteitsplein 1 B2610 Antwerp Belgium FAX: +32-3-8202541 - email: crete_course at bbf.uia.ac.be APPLICATION DEADLINE: April 10th, 1996. Applicants will be notified of the results of the selection procedures before May 1st. FACULTY: M. Abeles (Hebrew University, Jerusalem, Israel), D.J. Amit (University of Rome, Italy and Hebrew University, Israel), A. Berthoz (College de France, France), R.E. Burke (NIH, USA), C.E. Carr (University of Maryland, USA), A. Destexhe (Universite Laval, Canada), R.J. Douglas (Institute of Neuroinformatics, Zurich, Switzerland), T. Flash (Weizmann Institute, Rehovot, Israel), A. Grinvald (Weizmann Institute, Israel), J.J.B. Jack (Oxford University, England), C. Koch (California Institute of Technology, USA), H. Korn (Institut Pasteur, France), A. Lansner (Royal Institute Technology, Sweden), R. Llinas (New York University, USA), E. Marder (Brandeis University, USA), M. Nicolelis (Duke University, USA), J.M. Rinzel (NIH, USA), W. Singer (Max-Planck Institute, Frankfurt, Germany), S. Tanaka (RIKEN, Japan), A.M. Thomson (Royal Free Hospital, England), S. Ullman (Weizmann Institute, Israel), Y. Yarom (Hebrew University, Israel). The Crete Course in Computational Neuroscience is supported by the European Commission (4th Framework Training and Mobility of Researchers program) and by The Brain Science Foundation (Tokyo). Local administrative organization: the Institute of Applied and Computational Mathematics of FORTH (Crete, GR). From harnad at cogsci.soton.ac.uk Thu Feb 15 12:58:05 1996 From: harnad at cogsci.soton.ac.uk (Stevan Harnad) Date: Thu, 15 Feb 96 17:58:05 GMT Subject: Cerebellum and Sequences: BBS Call for Commentators Message-ID: <18060.9602151758@cogsci.ecs.soton.ac.uk> Below is the abstract of a forthcoming target article on: THE DETECTION AND GENERATION OF SEQUENCES AS A KEY TO CEREBELLAR FUNCTION. EXPERIMENTS AND THEORY by V. Braitenberg, D. Heck and F. Sultan This article has been accepted for publication in Behavioral and Brain Sciences (BBS), an international, interdisciplinary journal providing Open Peer Commentary on important and controversial current research in the biobehavioral and cognitive sciences. Commentators must be current BBS Associates or nominated by a current BBS Associate. To be considered as a commentator for this article, to suggest other appropriate commentators, or for information about how to become a BBS Associate, please send email to: bbs at ecs.soton.ac.uk or write to: Behavioral and Brain Sciences Department of Psychology University of Southampton Highfield, Southampton SO17 1BJ UNITED KINGDOM http://cogsci.ecs.soton.ac.uk/~harnad/bbs.html gopher://gopher.princeton.edu:70/11/.libraries/.pujournals ftp://ftp.princeton.edu/pub/harnad/BBS To help us put together a balanced list of commentators, please give some indication of the aspects of the topic on which you would bring your areas of expertise to bear if you were selected as a commentator. An electronic draft of the full text is available for inspection by anonymous ftp (or gopher or world-wide-web) according to the instructions that follow after the abstract. ____________________________________________________________________ THE DETECTION AND GENERATION OF SEQUENCES AS A KEY TO CEREBELLAR FUNCTION. EXPERIMENTS AND THEORY Valentino Braitenberg, Detlef Heck and Fahad Sultan Max-Planck-Institute for biological cybernetics Spemannstr. 38 72076 Tuebingen Germany KEYWORDS: Cerebellum; motor control; allometric relation; parallel fibers; synchronicity; spatio-temporal activity; sequence addressable memory; cerebro-cerebellar interaction. ABSTRACT:Starting from macroscopic and microscopic facts of cerebellar histology, we propose a new functional interpretation which may elucidate the role of the cerebellum in movement control. Briefly, the idea is that the cerebellum is a large collection of individual lines (Eccles' "beams") which respond specifically to certain sequences of events in the input and in turn produce sequences of signals in the output. We believe that the sequence in - sequence out mode operation is as typical for the cerebellar cortex as the transformation of sets into sets of active neurons is typical for the cerebral cortex, and that both the histological differences between the two and their reciprocal functional interactions become understandable in the light of this dichotomy. The response of Purkinje cells to sequences of stimuli in the mossy fiber system was shown experimentally by Heck on surviving slices of rat and guinea pig cerebellum. Sequential activation of a row of eleven stimulating electrodes in the granular layer, imitating a "movement" of the stimuli along the folium, produces a powerful volley in the parallel fibers which strongly excites Purkinje cells, as evidenced by intracellular recording. The volley, or "tidal wave" has maximal amplitude when the stimulus moves towards the recording site at the speed of conduction in parallel fibers, and much smaller amplitudes for lower or higher "velocities". The succession of stimuli has no effect when they "move" in the opposite direction. Synchronous activation of the stimulus electrodes also had hardly any effect. We believe that the sequences of mossy fiber activation which normally produce this effect in the intact cerebellum are a combination of motor planning, relayed to the cerebellum by the cerebral cortex, and information about ongoing movement, reaching the cerebellum from the spinal cord. The output elicited by the specific sequence to which a "beam" is tuned may well be a succession of well timed inhibitory volleys "sculpting" the motor sequences so as to adapt them to the complicated requirements of the physics of a multi-jointed system. -------------------------------------------------------------- To help you decide whether you would be an appropriate commentator for this article, an electronic draft is retrievable by anonymous ftp from ftp.princeton.edu according to the instructions below (the filename is bbs.braitenberg). Please do not prepare a commentary on this draft. Just let us know, after having inspected it, what relevant expertise you feel you would bring to bear on what aspect of the article. ------------------------------------------------------------- These files are also on the World Wide Web and the easiest way to retrieve them is with Netscape, Mosaic, gopher, archie, veronica, etc. Here are some of the URLs you can use to get to the BBS Archive: http://www.princeton.edu/~harnad/bbs.html http://cogsci.ecs.soton.ac.uk/~harnad/bbs.html gopher://gopher.princeton.edu:70/11/.libraries/.pujournals ftp://ftp.princeton.edu/pub/harnad/BBS/bbs.braitenberg ftp://cogsci.ecs.soton.ac.uk/pub/harnad/BBS/bbs.braitenberg To retrieve a file by ftp from an Internet site, type either: ftp ftp.princeton.edu or ftp 128.112.128.1 When you are asked for your login, type: anonymous Enter password as queried (your password is your actual userid: yourlogin at yourhost.whatever.whatever - be sure to include the "@") cd /pub/harnad/BBS To show the available files, type: ls Next, retrieve the file you want with (for example): get bbs.braitenberg When you have the file(s) you want, type: quit ---------- Where the above procedure is not available there are two fileservers: ftpmail at decwrl.dec.com and bitftp at pucc.bitnet that will do the transfer for you. To one or the other of them, send the following one line message: help for instructions (which will be similar to the above, but will be in the form of a series of lines in an email message that ftpmail or bitftp will then execute for you). ------------------------------------------------------------- From goldfarb at unb.ca Fri Feb 16 13:30:54 1996 From: goldfarb at unb.ca (Lev Goldfarb) Date: Fri, 16 Feb 1996 14:30:54 -0400 (AST) Subject: Workshop: What is inductive learning? Message-ID: Dear connectionists: The following workshop should be of particular interest to the connectionist community, since not only the topic itself was motivated by the resent developments in cognitive science and AI as they are being affected by connectionist movement, but also one of the main arguments that is going to be presented at the workshop is that to capture the main goals of the "connectionist movement" one needs to change fundamentally the underlying architectures from the numeric to the appropriately redefined "symbolic" architectures. My apologies if you receive multiple copies of this message. Please, post it. %*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*% Call for extended abstracts: WHAT IS INDUCTIVE LEARNING? On the foundations of AI and Cognitive Science Toronto - Canada May 20 - 21, 1996 A workshop in conjunction with the 11th Biennial Canadian AI Conference to be held at the Holiday Inn on King, Toronto during 21 - 24 May 1996 This workshop is a long overdue attempt to look at the inductive learning process (ILP) as the central process generating various representations of objects (events). To this end one needs, first of all, to have a working definition of the ILP, which has been lacking. Here is a starting point: ILP is the process that constructs class representation on the basis of a (small) finite set of examples, i.e. it constructs the INDUCTIVE class representation. This class representation must, in essence, provide INDUCTIVE definition (or construction) of the class. The constructed class representation, in turn, modifies the earlier representation of the objects (within the context specified by the ILP). Thus, any subsequent processes, e.g. pattern recognition, recall, problem solving, are performed on the basis of the newly constructed object (event) representations. To put it somewhat strongly, there are only inductive representations. Two main and strongly related reasons why ILPs have not been perceived as the very central processes are a lack of their adequate understanding and a lack of their satisfactory formal model. Most of the currently popular formal models of ILPs (including connectionist models) do not provide satisfactory inductive class representations. One can show that inductive class representations (in other words, representations of concepts and categories) cannot be adequately specified within the classical (numeric) mathematical models. We encourage all researchers (including graduate students) seriously interested in the foundations of the above areas to participate in the workshop. Both theoretical and applied contributions are welcomed (including, of course, those related to vision, speech, and language). While extended abstracts will be available at the workshop, we are planning to publish the expanded and reviewed versions of the presentations as a special issue of journal Pattern Recognition. EXTENDED ABSTRACT SUBMISSION ---------------------------- Submit a copy (or e-mail version) of a 3-4 page extended abstract to Lev Goldfarb ILP Workshop Chair Faculty of Computer Science University of New Brunswick P.O. Box 4400 E-mail: goldfarb at unb.ca Fredericton, N.B. E3B 5A3 Tel: 506-453-4566 Canada Fax: 506-453-3566 E-mail submissions are encouraged. Important dates: ---------------- Extended abstract due: Monday, March 25, 1996. Notification & review back to the author: Friday April 5, 1996. Final extended abstract due: Monday April 22, 1996. For more information about the Canadian AI Conference which is held in conjunction with two other conferences (Vision Interface and Graphics Interface) see: http://ai.iit.nrc.ca/cscsi/conferences/ai96.html %*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*%*% -- Lev Goldfarb http://wwwos2.cs.unb.ca/profs/goldfarb/goldfarb.htm Please e-mail to me: _____________________________________________________________________________ I intend to submit an abstract __ I plan to attend the workshop __ _____________________________________________________________________________ From mozer at neuron.cs.colorado.edu Fri Feb 16 16:28:57 1996 From: mozer at neuron.cs.colorado.edu (Michael C. Mozer) Date: Fri, 16 Feb 1996 14:28:57 -0700 Subject: NIPS*96 CALL FOR WORKSHOP PROPOSALS Message-ID: <199602162128.OAA09088@neuron.cs.colorado.edu> CALL FOR PROPOSALS NIPS*96 Post Conference Workshops December 6 and 7, 1996 Snowmass, Colorado Following the regular program of the Neural Information Processing Systems 1996 conference, workshops on current topics in neural information processing will be held on December 6 and 7, 1996, in Snowmass, Colorado. Proposals by qualified individuals interested in chairing one of these workshops are solicited. Past topics have included: Active Learning, Architectural Issues, Attention, Audition, Bayesian Analysis, Bayesian Networks, Benchmarking, Computational Complexity, Computational Molecular Biology, Control, Neuroscience, Genetic Algorithms, Grammars, Hybrid HMM/ANN Systems, Implementations, Music, Neural Hardware, Network Dynamics, Neurophysiology, On-Line Learning, Optimization, Recurrent Nets, Robot Learning, Rule Extraction, Self-Organization, Sensory Biophysics, Signal Processing, Symbolic Dynamics, Speech, Time Series, Topological Maps, and Vision. The goal of the workshops is to provide an informal forum for researchers to discuss important issues of current interest. There will be two workshop sessions a day, for a total of six hours, with free time in between for ongoing individual exchange or outdoor activities. Concrete open and/or controversial issues are encouraged and preferred as workshop topics. Representation of alternative viewpoints and panel-style discussions are particularly encouraged. Workshop organizers will have responsibilities including: 1) coordinating workshop participation and content, which involves arranging short informal presentations by experts working in an area, arranging for expert commentators to sit on a discussion panel and formulating a set of discussion topics, etc. 2) moderating or leading the discussion and reporting its high points, findings, and conclusions to the group during evening plenary sessions 3) writing a brief summary and/or coordinating submitted material for post-conference electronic dissemination. Submission Instructions ----------------------- Interested parties should submit via e-mail a short proposal for a workshop of interest by May 20, 1996. Proposals should include a title, a description of what the workshop is to address and accomplish, the proposed length of the workshop (one day or two days), the planned format (mini-conference, panel discussion, or group discussion, combinations of the above, etc), and the proposed number of speakers. Where possible, please also indicate potential invitees (particularly for panel discussions). Please note that this year we are looking for fewer "mini-conference" workshops and greater variety of workshop formats. Also, the time allotted to workshops has been increased to six hours each day. We strongly encourage that the organizers reserve a significant portion of time for open discussion. The proposal should motivate why the topic is of interest or controversial, why it should be discussed and who the targeted group of participants is. In addition, please send a brief resume of the prospective workshop chair, a list of publications, and evidence of scholarship in the field of interest. Submissions should include contact name, address, e-mail address, phone number and fax number if available. Proposals should be mailed electronically to mpp at watson.ibm.com. All proposals must be RECEIVED by May 20, 1996. If e-mail is unavailable, mail so as to arrive by the deadline to: NIPS*96 Workshops c/o Michael P. Perrone IBM T. J. Watson Research Center P.O. Box 218, 36-207 Yorktown Heights, NY 10598 Questions may be addressed to either of the Workshop Co-Chairs: Michael P. Perrone Steven J. Nowlan IBM T.J. Watson Research Center Motorola, Lexicus Division mpp at watson.ibm.com steven at lexicus.mot.com PROPOSALS MUST BE RECEIVED BY MAY 20, 1996 -Please Post- From steven at strontium.lexicus.mot.com Fri Feb 16 19:24:10 1996 From: steven at strontium.lexicus.mot.com (Steve Nowlan) Date: Fri, 16 Feb 1996 16:24:10 -0800 Subject: Position Available Message-ID: <9602161624.ZM14139@strontium.lexicus.mot.com> Please respond to the address at the end of the article, thank you. Motorola, Lexicus Division is currently seeking qualified applicants for the position of Recognition Scientist. Lexicus, a Division of Motorola, specializes in handwriting and speech recognition products for the mobile and wireless markets. Located in Palo Alto, California, across from Stanford University, the company numbers about 50 people, and is continuing to grow. JOB DESCRIPTION This individual will join a team of scientists working on technology for hand writing and speech recognition applications. Application areas range from low-cost embedded systems to high-end work station applications. The technology areas of interest include pattern recognition, vector quantization, neural networks, hidden Markov models and statistical language modeling. EDUCATION & EXPERIENCE The ideal candidate has an established record of significant contributions to research efforts in an academic or industrial environment and a strong background in statistical methods of pattern recognition. Specific skills include: * knowledge of current state of the art algorithms for pattern recognition and mathematical and statistical analysis/modeling techniques * the ability to work within a group to quickly implement and evaluate algorithms in a UNIX/C/C++ environment. Asian Language Skills also desirable. Applicants should be interested in developing state of the art technology and moving that technology into products in a rapid research/development cycle. SALARY: Depending on experience Please send resumes and a list of references (and include the title of the position) to: Human Resources Lexicus, a Division of Motorola 490 California Avenue, Suite 300 fax: (415) 323-4772 Palo Alto, CA 94306 email: hr at lexicus.mot.com Debbie Mayer Human Resources Motorola, Lexicus Division 490 California Ave, Suite #300 Palo Alto, CA 94306 Tel: (415) 617-1115 Fax: (415) 323-4772 From istvan at psych.ualberta.ca Mon Feb 19 19:23:42 1996 From: istvan at psych.ualberta.ca (Istvan Berkeley) Date: Mon, 19 Feb 1996 17:23:42 -0700 Subject: Intl. NN Workshop Announcement Message-ID: APPLICATIONS OF CONNECTIONISM IN COGNITIVE SCIENCE: AN INTERNATIONAL WORKSHOP On May 25-27, 1996, there will be a major international workshop at Carleton University, Ottawa, Canada, on the latest connectionist modelling techniques in cognitive science. The list of presenters, given below, includes some of the founders of contemporary PDP techniques, as well as younger researchers whose work stands at the forefront of new approaches and applications. Along with formal presentations, mornings will be devoted to demonstrations of the newest PDP software of potential interest to cognitive scientists, for which purpose each participant will have access to a workstation. Principal speakers: David E. Rummelhart, Stanford University Jerome A. Feldman, University of California at Berkeley Paul Skokowski, Stanford University Christopher Thornton, University of Sussex John Bullinaria, Edinburgh University Malcolm Forster, University of Wisconsin at Madison Istvan Berkeley, University of Alberta Each talk will be followed by an arranged commentary, and general discussion. For further information on registration procedures, fees and accommodations, please contact either Andrew Brook Department of Interdisciplinary Studies Carleton University Ottawa, Ontario CANADA K1S 5B6 or Don Ross Department of Philosophy Morisset Hall University of Ottawa Ottawa, Ontario CANADA K1N 6N5 Istvan S. N. Berkeley, email: istvan at psych.ualberta.ca Biological Computation Project & Department of Philosophy, c/o 4-108 Humanities Center University of Alberta Edmonton, Alberta Tel: +1 403 436 4182 T6G 2E5, Canada Fax: +1 403 492 9160 From imlm at tuck.cs.fit.edu Mon Feb 19 22:12:15 1996 From: imlm at tuck.cs.fit.edu (IMLM Workshop (pkc)) Date: Mon, 19 Feb 1996 22:12:15 -0500 Subject: 2nd CFP: AAAI-96 Workshop on Integrating Multiple Learned Models Message-ID: <199602200312.WAA13801@tuck.cs.fit.edu> ********************************************************************* Paper submission deadline: March 18, 1996 ********************************************************************* CALL FOR PAPERS/PARTICIPATION INTEGRATING MULTIPLE LEARNED MODELS FOR IMPROVING AND SCALING MACHINE LEARNING ALGORITHMS to be held in conjunction with AAAI 1996 (collocated with KDD-96, UAI-96, and IAAI-96) Portland, Oregon August 1996 Most modern machine learning research uses a single model or learning algorithm at a time, or at most selects one model from a set of candidate models. Recently however, there has been considerable interest in techniques that integrate the collective predictions of a set of models in some principled fashion. With such techniques often the predictive accuracy and/or the training efficiency of the overall system can be improved, since one can "mix and match" among the relative strengths of the models being combined. The goal of this workshop is to gather researchers actively working in the area of integrating multiple learned models, to exchange ideas and foster collaborations and new research directions. In particular, we seek to bring together researchers interested in this topic from the fields of Machine Learning, Knowledge Discovery in Databases, and Statistics. Any aspect of integrating multiple models is appropriate for the workshop. However we intend the focus of the workshop to be improving prediction accuracies, and improving training performance in the context of large training databases. More precisely, submissions are sought in, but not limited to, the following topics: 1) Techniques that generate and/or integrate multiple learned models. In particular, techniques that do so by: * using different training data distributions (in particular by training over different partitions of the data) * using different output classification schemes (for example using output codes) * using different hyperparameters or training heuristics (primarily as a tool for generating multiple models) 2) Systems and architectures to implement such strategies. In particular: * parallel and distributed multiple learning systems * multi-agent learning over inherently distributed data A paper need not be submitted to participate in the workshop, but space may be limited so contact the organizers as early as possible if you wish to participate. The workshop format is planned to encompass a full day of half hour presentations with discussion periods, ending with a brief period for summary and discussion of future activities. Notes or proceedings for the workshop may be provided, depending on the submissions received. Submission requirements: i) A short paper of not more than 2000 words detailing recent research results must be received by March 18, 1996. ii) The paper should include an abstract of not more than 150 words, and a list of keywords. Please include the name(s), email address(es), address(es), and phone number(s) of the author(s) on the first page. The first author will be the primary contact unless otherwise stated. iii) Electronic submissions in postscript or ASCII via email are preferred. Three printed copies (preferrably double-sided) of your submission are also accepted. iv) Please also send the title, name(s) and email address(es) of the author(s), abstract, and keywords in ASCII via email. Submission address: imlm at cs.fit.edu Philip Chan IMLM Workshop Computer Science Florida Institute of Technology 150 W. University Blvd. Melbourne, FL 32901-6988 407-768-8000 x7280 (x8062) 407-984-8461 (fax) Important Dates: Paper submission deadline: March 18, 1996 Notification of acceptance: April 15, 1996 Final copy: May 13, 1996 Chairs: Salvatore Stolfo, Columbia University sal at cs.columbia.edu David Wolpert, Santa Fe Institute dhw at santafe.edu Philip Chan, Florida Institute of Technology pkc at cs.fit.edu General Inquiries: Please address general inquiries to one of the chairs or send them to: imlm at cs.fit.edu Up-to-date workshop information is maintained on WWW at: http://www.cs.fit.edu/~imlm/ or http://cs.fit.edu/~imlm/ From zhuh at helios.aston.ac.uk Tue Feb 20 11:48:58 1996 From: zhuh at helios.aston.ac.uk (zhuh) Date: Tue, 20 Feb 1996 16:48:58 +0000 Subject: Paper available: cross validation Message-ID: <8961.9602201648@sun.aston.ac.uk> FTP-host: cs.aston.ac.uk FTP-file: neural/zhuh/nflcv.ps.Z URL: ftp://cs.aston.ac.uk/neural/zhuh/nflcv.ps.Z To appear in Neural Computation. ================================================================= No Free Lunch For Cross Validation Huaiyu Zhu and Richard Rohwer Neural Computing Research Group Aston University, Birmingham B4 7ET, UK Abstract -------- It is known theoretically that an algorithm cannot be good for an arbitrary prior. We show that in practical terms this also applies to the technique of ``cross validation'', which has been widely regarded as defying this general rule. Numerical examples are analysed in detail. Their implications to researches on learning algorithms are discussed. ================================================================= -- Huaiyu Zhu, PhD email: H.Zhu at aston.ac.uk Neural Computing Research Group http://neural-server.aston.ac.uk/People/zhuh Dept of Computer Science ftp://cs.aston.ac.uk/neural/zhuh and Applied Mathematics tel: +44 121 359 3611 x 5427 Aston University, fax: +44 121 333 6215 Birmingham B4 7ET, UK From scheier at ifi.unizh.ch Mon Feb 19 04:28:35 1996 From: scheier at ifi.unizh.ch (Christian Scheier) Date: Mon, 19 Feb 1996 10:28:35 +0100 Subject: Papers on Categorization in Autonomous Agents using Neural Networks Message-ID: The following papers deal with the problem of categorization/object recognition in autonomous agents (mobile robots). The papers can be retrieved from: ftp://claude.ifi.unizh.ch/pub/institute/ailab/techreports/ 96_01.ps.gz: Categorization in a real-world agent using haptic exploration and active perception Scheier, C. and Lambrinos, D. ABSTRACT An agent in the real world has to be able to make distinctions between different types of objects, i.e. it must have the competence of categorization. In mobile agents categorization is hard to achieve because there is a large variation in proximal sensory stimulation originating from the same object. In this paper we extend previous work on adaptive categorization in autonomous agents. The main idea of our approach is to include the agent's own actions into the classification process. In the experiments presented in this paper an agent equipped with an active vision and an arm-gripper system has to collect certain types of objects. The agent learns about the objects by actively exploring them. This exploration results in visual and haptic information that is used for learning. In essence, the categorization comes about via evolving reentrant connections between the haptic and the visual system. Results on the behavioral performance as well as the underlying internal dynamics are presented. 95_12.ps.gz: Adaptive Classification in Autonomous Agents Scheier, C. and Lambrinos, D. ABSTRACT One of the fundamental tasks facing autonomous robots is to reduce the many degrees of freedom of the input space by some sorts of classification mechanism. The sensory stimulation caused by one and the same object, for instance, varies enormously depending on lighting conditions, distance from object, orientation and so on. Efforts to solve this problem, say in classical computer vision, have only had limited success. In this paper a new approach towards classification in autonomous robots is proposed. It's cornerstone is the integration of the robots own actions into the classification process. More specifically, correlations through time-linked independent samples of sensory stimuli and of kinesthetic signals produced by self-motion of the system form the basis of the category learning. Thus, it is suggested that classification should not be seen as an isolated perceptual (sub-)system but rather as a {\it sensory-motor coordination} which comes about through a self-organizing process. These ideas are illustrated with a case study of an autonomous system that has to learn to distinguish between different types of objects. 95_05.ps.gz: Classification as Sensory-Motor Coordination: A Case Study on Autonomous Agents. Scheier, C. and Pfeifer, R. ABSTRACT In psychology classification is studied as a separate cognitive capacity. In the field of autonomous agents the robots are equipped with perceptual mechanisms for classifying objects in the environment, either by preprogramming or by some sorts of learning mechanisms. One of the well-known hard and fundamental problems is the one of perceptual aliasing, i.e. that the sensory stimulation caused by one and the same object varies enormously depending on distance from object, orientation, lighting conditions, etc. Efforts to solve this problem, say in classical computer vision, have only had limited success. In this paper we argue that classification cannot be viewed as a separate perceptual capacity of an agent but should be seen as a sensory-motor coordination which comes about through a self-organizing process. This implies that the whole organism is involved, not only sensors and neural circuitry. In this perspective, ``action selection'' becomes an integral part of classification. These ideas are illustrated with a case study of a robot that learns to distinguish between graspable and non-graspable pegs. For further informations and papers contact: -- __________________________________________________________________________ Christian Scheier Computer Science Department AI Lab University of Zurich tel: +41-1-257-4575 Winterthurerstrasse 190 fax: +41-1-363-0035 CH-8057 Switzerland http://josef.ifi.unizh.ch/groups/ailab/people/scheier.html ______________________________________ ____________________________________ From guy at taco.mpik-tueb.mpg.de Wed Feb 21 08:29:37 1996 From: guy at taco.mpik-tueb.mpg.de (Guy M. Wallis) Date: Wed, 21 Feb 1996 13:29:37 +0000 Subject: Papers available: "Object recognition and unsupervised learning" Message-ID: <9602211329.ZM182@taco.mpik-tueb.mpg.de> FTP-host: archive.cis.ohio-state.edu FTP-filename: /pub/neuroprose/wallisgm.ittrain.ps.Z FTP-filename: /pub/neuroprose/wallisgm.temporalobjrec1.ps.Z FTP-filename: /pub/neuroprose/wallisgm.temporalobjrec2.ps.Z ** Three papers available on the unsupervised ** ** learning of invariant object recognition ** The three papers listed above are now available for retrieval from the Neuroprose repository. All three papers discuss learning to associate different views of objects on the basis of their appearance in time as well as their spatial appearance. The papers are also available directly from my home page, along with a copy of my PhD thesis which, I should warn you, is rather long: http://www.mpik-tueb.mpg.de/people/personal/guy/guy.html -------------------------------------------------------------------------------- PaperI: A Model of Invariant Object Recognition in the Visual System ABSTRACT Neurons in the ventral stream of the primate visual system exhibit responses to the images of objects which are invariant with respect to natural transformations such as translation, size, and view. Anatomical and neurophysiological evidence suggests that this is achieved through a series of hierarchical processing areas. In an attempt to elucidate the manner in which such representations are established, we have constructed a model of cortical visual processing which seeks to parallel many features of this system, specifically the multi-stage hierarchy with its topologically constrained convergent connectivity. Each stage is constructed as a competitive network utilising a modified Hebb-like learning rule, called the trace rule, which incorporates previous as well as current neuronal activity. The trace rule enables neurons to learn about whatever is invariant over short time periods (e.g. 0.5 s) in the representation of objects as the objects transform in the real world. The trace rule enables neurons to learn the statistical invariances about objects during their transformations, by associating together representations which occur close together in time. We show that by using the trace rule training algorithm the model can indeed learn to produce transformation invariant responses to natural stimuli such as faces. Submitted to Journal of Computational Neuroscience 32 pages 1.6 Mb compressed ftp://archive.cis.ohio-state.edu/pub/neuroprose/wallisgm.ittrain.ps.Z or ftp://ftp.mpik-tueb.mpg.de/pub/guy/jcns7.ps.Z -------------------------------------------------------------------------------- PaperII: Optimal, Unsupervised Learning in Invariant Object Recognition ABSTRACT A means for establishing transformation invariant representations of objects at the single cell level is proposed and analysed. The association of views of objects is achieved by using both the temporal order of the presentation of these views, as well as their spatial similarity. Assuming knowledge of the distribution of presentation times, an optimal linear learning rule is derived. If we assume that objects are viewed with presentation times that are approximately Jeffrey's distributed, then the optimal learning rule is very well approximated using a simple exponential temporal trace. Simulations of a competitive network trained on a character recognition task are then used to highlight the success of this learning rule in relation to simple Hebbian learning, and to show that the theory can give quantitative predictions for the optimal parameters for such networks. Submitted to Neural Computation 15 pages 180 Kb compressed ftp://archive.cis.ohio-state.edu/pub/neuroprose/wallisgm.temporalobjrec1.ps.Z or ftp://ftp.mpik-tueb.mpg.de/pub/guy/nc.ps.Z -------------------------------------------------------------------------------- PaperIII: Using Spatio-Temporal Correlations to Learn Invariant Object Recognition ABSTRACT A competitive network is described which learns to classify objects on the basis of temporal as well as spatial correlations. This is achieved by using a Hebb-like learning rule which is dependent upon prior as well as current neural activity. The rule is shown to be capable of outperforming a supervised rule on the cross-validation test of an invariant character recognition task, given a relatively small training set. It is also shown to outperform the supervised version of Fukushima's Neocognitron, on a larger training set. Submitted to Neural Networks 13 pages 110 Kb compressed ftp://archive.cis.ohio-state.edu/pub/neuroprose/wallisgm.temporalobjrec2.ps.Z or ftp://ftp.mpik-tueb.mpg.de/pub/guy/nn.ps.Z -------------------------------------------------------------------------------- -- ----------------------------------------------------------- _/ _/ _/_/_/ _/_/_/ Guy Wallis _/_/ _/_/ _/ _/ _/ Max-Planck Institut f"ur _/ _/ _/ _/_/_/ _/ Biologische Kybernetik _/ _/ _/ _/ Spemannstr. 38 _/ _/ _/ _/_/_/ 72076 T"ubingen, Germany http://www.mpik-tueb.mpg.de/ TEL: +49-7071/601-630 Email: guy at mpik-tueb.mpg.de FAX: +49-7071/601-575 ----------------------------------------------------------- From jfeldman at ICSI.Berkeley.EDU Wed Feb 21 11:12:42 1996 From: jfeldman at ICSI.Berkeley.EDU (Jerry Feldman) Date: Wed, 21 Feb 1996 08:12:42 -0800 Subject: shift invariance Message-ID: <9602210812.ZM26438@ICSI.Berkeley.edu> Shift invariance is the ability of a neural system to recognize a pattern independent of where appears on the retina. It is generally understood that this property can not be learned by neural network methods, but I have not seen a published proof. A "local" learning rule is one that updates the input weights of a unit as a function of the unit's own activity and some performance measure for the network on the training example. All biologically plausible learning rules, as well as all backprop variants, are local in this sense. It is easy to show that no local rule can learn shift invariance. Consider learning binary strings with one occurrence of the sequence 101 and otherwise all zeros. First consider length 5; there are only 3 positive examples: 10100 01010 00101 Suppose that the training data does not include the middle example. The two positive training examples have no 1's in common with the withheld example. There will be no positive examples with a 1 in position 2 or 4 and many negative examples. Thus there is no correlation between a 1 in position 2 or 4 and a good example so no local training rule will learn the correct classification. A similar argument extends to binary strings of arbitrary length so an arbitrarily small fraction of the training data can be omitted and still no local updating rules will suffice to learn shift invariance. The one dimensional case of shift invariance can be handled by treating each string as a sequence and learning a finite-state acceptor. But the methods that work for this are not local or biologically plausible and don't extend to two dimensions. The unlearnability of shift invarince is not a problem in practice because people use preprocessing, weight sharing or other techniques to get shift invariance where it is known to be needed. However, it does pose a problem for the brain and for theories that are overly dependent on learning. From rolf at cs.rug.nl Thu Feb 22 06:56:06 1996 From: rolf at cs.rug.nl (rolf@cs.rug.nl) Date: Thu, 22 Feb 1996 12:56:06 +0100 Subject: shift invariance Message-ID: Dear Jerry, a short comment regarding your posting. It is no wonder that you have not seen a proof because it is simply not true that neural networks cannot do shift invariant recognition (SIR). If SIR is formalized it is most probably a computable function and ANNs can at least approximate all computable functions. No problem on a fundamental level here. K. Fukushima has shown a long time ago that a network can be wired up to do it. Nevertheless, invariant recognition seems to be a fundamental property of the visual system. It is not a _natural_ property of ANNs in the sense that you just give them a lot of natural stimuli and they develop the capability on their own. That, of course, makes one think if ANNs are a good model of the visual system, or if there is still a major point missing. I do not quite know what to make of your remark about ``theories overly dependent on learning''. If you have a better concept to offer than learning from experience I will be glad to hear about it. OK, you can say that there is ingenious machinery that does it, and the wiring is done by the genetic code, and we can go on and think about different things, but I do not consider that satisfactory explanation. I am looking forward to a discussion on the topic. Rolf +---------------------------------------------------------------------------+ | Rolf P. W"urtz | mailto:rolf at cs.rug.nl | URL: http://www.cs.rug.nl/~rolf/ | | Department of Computing Science, University of Groningen, The Netherlands | +---------------------------------------------------------------------------+ From wimw at mbfys.kun.nl Thu Feb 22 09:18:27 1996 From: wimw at mbfys.kun.nl (Wim Wiegerinck) Date: Thu, 22 Feb 1996 15:18:27 +0100 (MET) Subject: Paper Available: How Dependencies between Successive Examples Affect On-Line Learning. Message-ID: <199602221418.PAA04038@septimius.mbfys.kun.nl> A non-text attachment was scrubbed... Name: not available Type: text Size: 2015 bytes Desc: not available Url : https://mailman.srv.cs.cmu.edu/mailman/private/connectionists/attachments/00000000/8f607bc7/attachment-0001.ksh From robbie at bcs.rochester.edu Thu Feb 22 10:04:38 1996 From: robbie at bcs.rochester.edu (Robbie Jacobs) Date: Thu, 22 Feb 1996 10:04:38 -0500 Subject: undergraduate summer workshop Message-ID: <199602221504.KAA04738@broca.bcs.rochester.edu> Below is an announcement for a three-day summer workshop for undergraduate students interested in the brain and cognitive sciences. It would be appreciated if you could bring this to the attention of your students. Robert Jacobs =================================================================== UNDERGRADUATE WORKSHOP ON PERCEPTION, PLASTICITY, AND ACTION (FELLOWSHIPS AVAILABLE) August 8-10, 1996 University of Rochester Department of Brain and Cognitive Sciences, Center for Visual Science, and the Program in Neuroscience The University of Rochester will host a Summer Workshop for Undergraduates on August 8-10, 1996. The Workshop will consist of lectures and laboratory demonstrations by the Rochester faculty on the coordination of perceptual mechanisms with those that control movement. It will also focus on how this coordination is established during development and modified by experience. These issues will be approached from neural, behavioral, and computational perspectives. Fellowships covering travel and living expenses will be provided for 20 students. Preference will be given to Juniors who plan to pursue advanced study in the behavioral and neural sciences. Request an application: * by email: judy at cvs.rochester.edu * by phone: (716) 275-2459 or fax: (716) 271-3043 * in writing: Dr. David R. Williams, Brain and Cognitive Sciences, Meliora Hall, University of Rochester, Rochester, NY 14627 * or apply electronically by visiting the web site at: http://www.bcs.rochester.edu/ug_workshop/ From hinton at cs.toronto.edu Thu Feb 22 10:25:32 1996 From: hinton at cs.toronto.edu (Geoffrey Hinton) Date: Thu, 22 Feb 1996 10:25:32 -0500 Subject: shift invariance In-Reply-To: Your message of Wed, 21 Feb 1996 11:12:42 -0500. Message-ID: <96Feb22.102548edt.882@neuron.ai.toronto.edu> Dear Jerry, Its a long time since we had a really good disagreement. Contrary to your assertions, shift invariance can be learned by backpropagation. It was one of the problems that I tried when fiddling about with backprop in the mid 80's. I published a paper demonstrating this in an obscure conference proceedings: Hinton, G.~E. (1987) Learning translation invariant recognition in a massively parallel network. In Goos, G. and Hartmanis, J., editors, PARLE: Parallel Architectures and Languages Europe, pages~1--13, Lecture Notes in Computer Science, Springer-Verlag, Berlin. So far as I know, this is also the first paper to demonstrate that weight decay can make a really big difference in generalization performance. It reduced the error rate from about 45% to about 6%, though I must confess that the amount of weight decay was determined by using the test set (as was usual in our sloppy past). I used a one dimensional "retina" with 12 pixels. On this retina there was one instance of a shape at a time. The "shape" consisted of two bright "boundary" pixels with 4 pixels between them. The 4 pixels in the sandwich could have any of the 16 binary patterns, so there were 16 very confusable shapes. For example, here are two instances of the shape corresponding to the binary number 0011 (strip off the boundary bits before reading the number): 000100111000 010011100000 The retina had wraparound so that each shape could occur in 12 different positions. This seems to me to be exactly the kind of data that you think cannot be learned. In other words, if I train a neural network to identitfy some instances of the shapes you think that it couldnt possibly generalize to instances in other positions. Of course, once you understand why it can generalize, you will decide on a way to exclude this kind of example, but so far it seems to me to fit your assertion about what cannot be done. The network had two hidden layers. In the first hidden layer there were 60 units divided into 12 groups of 5 with local receptive fields. So we are telling it about locality, but not about translation. Within each group, all 5 units receive input from the same 6 adjacent input units. In the next hidden layer there is a bottleneck of only 6 units (I didnt dare use 4), so all the information used to make the final decision has to be represented in a distributed pattern of activity in the bottleneck. There are 16 output units for the 16 shapes. The idea behind the network is as follows: Shapes are composed of features that they share with other shapes. Although we may not have seen a particular shape in a novel position, we will presumably have seen its features in those positions before. So if we have already developed translation invariant feature detectors, and if we represent our knowledge of the shape in terms of these detectors, we can generalize across translation. The "features" in this example are the values of the four pixels inside the sandwich. A hidden unit in the first hidden layer can see the whole sandwich, so it could learn to respond to the conjunction of the two boundary pixels and ONE of the four internal "shape feature" pixels. Its weights might look like this: ...+.+..+.. It would then be a position-dependent feature detector. In each location we have five such units to enable the net to develop all 4 position-dependent feature detectors (or funny combinations of them that span the space). In the next layer, we simply perform an OR for the 12 different copies of the same feature detector in the 12 different positions. So in the next layer we have position-independent feature detectors. Finally the outgoing connections from this layer represent the identity of a shape in terms of its position-independent feature detectors. Notice that the use of an OR should encourage the net to choose equivalent position-dependent feature detectors in the different locations, even though there is no explicit weight sharing. The amazing thing is that simply using backprop on the shape identities is sufficent to create this whole structure (or rather one of the zillions of mangled versions of it that uses hard-to-decipher distributed representations). Thanks to kevin lang for writing the Convex code that made this simulation possible in 1987. Please note that the local connectivity was NOT NECESSARY to get generalization. Without it the net still got 20/32 correct (guessing would be 2/32). Now, I dont for a moment believe that human shape perception is learned entirely by backpropagating from object identity labels. The simulation was simply intended to answer the philosophical point about whether this was impossible. Its taken nearly a decade for someone to come out and publicly voice the widely held belief that there is no way in hell a network could learn this. Thanks Geoff PS: I do think that there may be some biological mileage in the idea that local, position-dependent feature detectors are encouraged to break symmetry in the same way in order for later stages of processing to be able to achieve position independence by just performing an OR. An effect like this ought to occur in slightly less unrealistic models like Helmholtz machines. From pf2 at st-andrews.ac.uk Thu Feb 22 10:39:25 1996 From: pf2 at st-andrews.ac.uk (Peter Foldiak) Date: Thu, 22 Feb 1996 15:39:25 GMT Subject: shift invariance Message-ID: <199602221539.PAA24901@psych.st-andrews.ac.uk> > Shift invariance is the ability of a neural system to recognize a pattern > independent of where appears on the retina. It is generally understood that > this property can not be learned by neural network methods, but I have > not seen a published proof. Minsky & Papert: Perceptons, 1969, MIT Press, p 54 : "... order-1 predicates invariant under the usual geometric groups can do nothing more than define simple ">=m"-type inequalities on the size or "area" of the figures. In particular, taking the translation group G we see that no first-order perceptron can distinguish the A's in the figure on p. 46 from some other translation-invarian set of figures of the same area." This doesn't say anything about multi-layer nets, i.e. you can combine feaures in a way that will be invariant. Peter Foldiak From goldfarb at unb.ca Thu Feb 22 16:56:11 1996 From: goldfarb at unb.ca (Lev Goldfarb) Date: Thu, 22 Feb 1996 17:56:11 -0400 (AST) Subject: On the structure of connectionist models Message-ID: Dear connectionists: Since my posting of the workshop announcement (What is inductive learning?) several days ago, I was asked to clarify what I meant when I said that "one can show that inductive class representations (in other words, representations of concepts and categories) cannot be adequately specified within the classical (numeric) mathematical models" including, of course, connectionist models. Here are some general ideas from the paper which will be presented at the workshop. The following observations about the STRUCTURE of inductive learning models strongly suggest why the classical (numeric) mathematical models will be able to support only "weak" inductive learning models, i.e. the models that can perform reasonably only in VERY rigidly delineated environments. The questions I'm going to address in this posting on the one hand lay at the very foundations of connectionism and on the other hand are relatively simple, provided one keeps in mind that we are discussing the overall FORMAL STRUCTURE of the learning models (which requires a relatively high level of abstraction). Let's look at the structure of connectionist models through the very basic problem of inductive learning. In order to arrive at a useful formulation of the inductive learning problem and, at the same time, at a useful framework for solving the problem, I propose to proceed as follows. First and foremost, the inductive learning involves a finite set of data (objects from the class C) labeled either (C+, C-), positive and negative examples, or, more generally, simply C', examples. Since we want to compare quite different classes of models (e.g. symbolic and numeric), let us focus only on very general assumptions about the nature of the object representation (input) space: Postulate 1. Input space S satisfies a finite set A of axioms. (S, in fact, provide a formal specifications of all the necessary data properties; compare with the concept of abstract data type in computer science). Thus, for example, the vector (linear) space is defined by means of the well known set of axioms for vector addition and scalar multiplication. Next, let us attach the name "inductive class representation" (ICR) to the formal description (specification) of the class C obtained in a chosen model as a result of an inductive learning process: Postulate 2. In a learning model, ICR is specified in some (which?) formal manner. --------------------------------------------------------------------- | My first main point connects Postulate 2 to Postulate 1: ICR | | should be expressed in the "language" of the axioms from set A. | --------------------------------------------------------------------- For example, in a vector space ICR should be specified only in terms of the given data set plus the operations in the vector space, i.e. we are restricted to the spanned affine subspace or its approximation. The reason is quite simple: the only relationships that can be (mathematically) legitimately extracted from the input data are those that are expressible in the language of the input space S. Otherwise, we are, in fact, IMPLICITLY postulating some other relationships not specified in the input space by Postulate 1, and, therefore, the "discovery" of such implicit relationships in the data during the learning process is an illusion: such relationships are not "visible" in S. Thus, for example, "non-linear" relationships cannot be discovered from a finite data in a vector space, simply because a non-linear relationship is not part of the linear structure and, therefore, cannot be (mathematically) legitimately extracted from the finite input set of vectors in the vector space. What is happening (of necessity) in a typical connectionist model is that in addition to the set A of vector space axioms, some additional non-linear structure (determined by the class of non-linear functions chosen for the internal nodes of the NN) is being postulated IMPLICITLY from the beginning. Question: What does this additional non-linear structure has to do with the finite input set of vectors? (In fact, there are uncountably many such non-linear structures and, typically, none of them is directly related to the structure of the vector space or the input set of vectors.) ----------------------------------------------------------------------- | My second main point is this: if S is a vector space, in both cases, | | whether we do or don't postulate in addition to the vector space | | axioms some non-linear structure (for the internal nodes), we are | | faced with the following important question. What are we learning | | during the learning process? Certainly, we are not learning any | | interesting ICR: the entire STRUCTURE is fixed before the learning | | process. | ----------------------------------------------------------------------- It appears, that this situation is inevitable if we choose one of the classical (numeric) mathematical structures to model the input space S. However, in an appropriately defined symbolic setting (i.e. with an appropriate dynamic metric structure, see my home page) the situation changes fundamentally. To summarize (but not everything is before your eyes), the "strong" (symbolic) inductive learning models offer the ICRs that are much more flexible than those offered by the classical (numeric) models. In other words, the appropriate symbolic models offer true INDUCTIVE class representations. [The latter is given by a subset of objects + the constructed finite set of (weighted) operations that can transform objects into objects.] Lev Goldfarb http://wwwos2.cs.unb.ca/profs/goldfarb/goldfarb.htm From dorffner at cns.bu.edu Thu Feb 22 21:05:34 1996 From: dorffner at cns.bu.edu (dorffner@cns.bu.edu) Date: Thu, 22 Feb 1996 21:05:34 -0500 (EST) Subject: shift invariance Message-ID: <199602230205.VAA13842@bucnsd> Hi fellow connectionists, I must say I'm a little puzzled by this discussion about shift invariance. It was started by Jerry Feldman by saying > Shift invariance is the ability of a neural system to recognize a pattern > independent of where appears on the retina. It is generally understood that > this property can not be learned by neural network methods, but I have > not seen a published proof. A "local" learning rule is one that updates the > input weights of a unit as a function of the unit's own activity and some > performance measure for the network on the training example. All biologically > plausible learning rules, as well as all backprop variants, are local in this > sense. Now I always thought that this is so obvious that it didn't need any proof. Geoff Hinton responded by disagreeing: > Contrary to your assertions, shift invariance can be learned by > backpropagation. It was one of the problems that I tried when fiddling about > with backprop in the mid 80's. I published a paper demonstrating this in an > obscure conference proceedings: He describes a model based on feature detectors and subsequent backpropagation that can actually generalize over different positions. He finishes by saying > The simulation was > simply intended to answer the philosophical point about whether this was > impossible. Its taken nearly a decade for someone to come out and publicly > voice the widely held belief that there is no way in hell a network could > learn this. > IMHO, there seems to be a misunderstanding of what the topic of discussion is here. I don't think that Jerry meant that no model consisting of neural network components could ever learn shift invariance. After all, there are many famous examples in visual recognition with neural networks (such as the Neocognitron, as Rolf W"urtz pointed out), and if this impossibility were the case, we would have to give up neural network research in perceptual modeling altogether. What I think Jerry meant is that any cascade of fully-connected feed-forward connection schemes between layers (including the perceptron and the MLP) cannot learn shift invariance. Now besides being obvious, this does raise some important questions, possibly weakening the fundamentals of connectionism. Let me explain why: - state spaces in connectionist layers (based on the assumption that activation patterns are viewed as vectors) span a Euclidean space, with each connection scheme that transfers patterns into another layer applying a certain kind of metric defining similarity. This metric is non-trivial, especially in MLPs, but it restricts the ways of what such a basic neural network component (i.e. fully connected feedforward) can view as similar. Patterns that are close in this space according to a distance measure, or patterns that have large orthogonal projections onto each other (in my analysis the basic similarity measure in MLPs) are similar according to this metric. Different patterns with a sub-pattern in different positions are obviously NOT. Neither are patterns which share common differences between components (e.g. the patterns (0.8 0.3) and (0.6 0.1)), and a whole bunch of other examples. That's why we have to be so careful about the right kind of preprocessing when we apply neural networks in engineering, and why we have to be equally careful in choosing the appropriate representations in connectionist cognitive modeling. - introducing feature detectors and other complex connectivities and learning schemes (weight sharing, or the OR Geoff mentioned) is a way of translating the original pattern space into a space where the similarity structures which we expect obey the said metric in state space again. It's the same thing we do in preprocessing (e.g. we apply an FFT to signals, since we cannot expect that the network can extract invariances in the frequency domain). - Geoff's model, necognitron, and many others do exactly that. Each single component (e.g. one feature detector) is restricted by the similarity metric mentioned above. But by applying non-linear functions, and by combining their output in a clever way they translate the original patterns into a new pattern space, where similarity corresponds to this metric again (e.g. for the final backprop network Geoff introduced). Now obviously, when we look at the human visual system, the brain does seem to do some kind of preprocessing, such as applying feature detectors, as well. So we're kind of safe here. But the above observation does make one think, whether the similarity metric a neural network basically applies is actually the right kind of basis for cognitive modeling. Think about it: By introducing complex wiring and learning schemes, and by carefully choosing representations, we go a long way to finally satisfy the actual neural network that has to do the job of extracting information from the patterns. Visual recognition is but one, although prominent, example. Now what makes us sure that deeper processes ARE of the kind a fully connected feedforward network can handle (i.e. that processes DO work on the said restricted kind of similarity metric)? Now I do not really think that we have a problem here. But some people recently have raised serious doubts. Some have suggested that perhaps replacing connectionist state spaces by the "space" that is spanned by attractors in a dynamical systems gives one a more appropriate metric. I am not suggesting this, I am just proposing that connectionists have to be alert in this respect, and keep questioning themselves whether the fundamental assumptions we're making are the appropriate ones. In this way I think Jerry's point is worth discussing. Just my 2 c. worth, Georg (email: georg at ai.univie.ac.at) P.S: I would like to acknowledge F.G. Winkler from Vienna for some of the ideas expressed in this message. From hicks at cs.titech.ac.jp Thu Feb 22 21:21:25 1996 From: hicks at cs.titech.ac.jp (hicks@cs.titech.ac.jp) Date: Fri, 23 Feb 1996 11:21:25 +0900 Subject: shift invariance In-Reply-To: Jerry Feldman's message of Wed, 21 Feb 1996 08:12:42 -0800 <9602210812.ZM26438@ICSI.Berkeley.edu> Message-ID: <199602230221.LAA04831@euclid.cs.titech.ac.jp> jfeldman at ICSI.Berkeley.EDU Wed, 21 Feb 1996 08:12:42 wrote:: >Shift invariance is the ability of a neural system to recognize a pattern >independent of where appears on the retina. It is generally understood that >this property can not be learned by neural network methods I disagree. As you state later, the network of neurons need only "share" weights. Sharing can be forced, or can occur by independent learnign of the same (but translated) data. > The unlearnability of shift invarince is not a problem in practice because >people use preprocessing, weight sharing or other techniques to get shift >invariance where it is known to be needed. However, it does pose a problem for >the brain and for theories that are overly dependent on learning. There are two obvious ways in which shift invariance could occur in "biological" or other learning systems. 1) Nature Some of part of the patterns of connectivity in the low level vision system are decided genetically and replicated automatically (like many cellular structures throughout the boody); in effect, a kind of weight sharing. Natural selection (learning through genetics) favors patterns of connectivity which will detect frequently appearing patterns; in effect weight learning. 2) Nurture The strengths of neuronal connections in the low level vision system self-adjust in some Hebbian scheme to frequently occuring patterns. The distribution of these patterns, in actual fact, is shift invariant. (If they aren't shift invariant then there's not much point in learning them as though they were shift invariant.) Respectfully Yours, Craig Hicks From juergen at idsia.ch Fri Feb 23 03:15:27 1996 From: juergen at idsia.ch (Juergen Schmidhuber) Date: Fri, 23 Feb 96 09:15:27 +0100 Subject: shifts Message-ID: <9602230815.AA04599@fava.idsia.ch> Jerry Feldman writes: >>> Shift invariance is the ability of a neural system to recognize a pattern independent of where appears on the retina. It is generally understood that this property can not be learned by neural network methods, but I have not seen a published proof. [...] It is easy to show that no local rule can learn shift invariance. [...] The one dimensional case of shift invariance can be handled by treating each string as a sequence and learning a finite-state acceptor. But the methods that work for this are not local or biologically plausible and don't extend to two dimensions. <<< It might be of interest to note that the situation changes if the neural system includes a controller that is able to generate retina- movements (to change the position of the image on the retina). There are gradient-based controllers that (in certain cases) can *learn* appropriate, 2-dimensional retina shifts. They are `local' to the extent backprop through time is `local'. See, e.g., Schmidhuber & Huber (1991): Learning to generate fovea trajectories for target detection. Int. Journal of Neural Systems, 2(1 & 2):135-141. Juergen Schmidhuber, IDSIA From karim at ax1303.physik.uni-marburg.de Fri Feb 23 09:59:31 1996 From: karim at ax1303.physik.uni-marburg.de (Karim Mohraz) Date: Fri, 23 Feb 1996 15:59:31 +0100 Subject: FlexNet - a flexible neural network construction algorithm Message-ID: <9602231459.AA28657@ax1303.physik.uni-marburg.de> The following paper is available via WWW FlexNet - a flexible neural network construction algorithm Abstract Dynamic neural network algorithms are used for automatic network design in order to avoid time consuming search for finding an appropriate network topology with trial & error methods. The new FlexNet algorithm, unlike other network construction algorithms, does not underlie any constraints regarding the number of hidden layers and hidden units. In addition different connection strategies are available, together with candidate pool training and the option of freezing weights. Test results on 3 different benchmarks showed higher generalization rates for FlexNet compared to Cascade-Correlation and optimized MLP networks. Keywords: network construction, generalization, Cascade-Correlation. This paper has been accepted for publication in the European Symposium on Artificial Neural Networks (ESANN), Bruges, Belgium , April, 96. http://www.physik.uni-marburg.de/bio/mitarbei/karim/flexnet.ps (6 pages) Sorry, no hardcopies available ``` (o o) +-------------------oOO--(_)--OOo---------------------------------------------------+ Karim Mohraz Bereich Neuronale Netze & Fuzzy Logik Bayerisches Forschungszentrum fuer Wissensbasierte Systeme F O R W I S S Erlangen New address: AG Neurophysik, Universitaet Marburg, Germany Email: karim at bio.physik.uni-marburg.de WWW: http://www.physik.uni-marburg.de/bio/mitarbei/karim.html _ +-------------------o00--( )--00o---------------------------------------------------+ (o o) ''' From listerrj at helios.aston.ac.uk Fri Feb 23 09:59:39 1996 From: listerrj at helios.aston.ac.uk (Richard Lister) Date: Fri, 23 Feb 1996 14:59:39 +0000 Subject: Neural Computing Research Programmer post Message-ID: <8199.199602231459@sun.aston.ac.uk> ---------------------------------------------------------------------- Neural Computing Research Group ------------------------------- Dept of Computer Science and Applied Mathematics Aston University, Birmingham, UK Research Programmer ------------------- * Full details at http://www.ncrg.aston.ac.uk/ * Applications are invited for the post of Research Programmer within the Neural Computing Research Group (NCRG) at Aston University. The NCRG is now the largest academic research group in this area in the UK, and has an extensive and lively programme of research ranging from the theoretical foundations of neural computing and pattern recognition through to industrial and commercial applications. The Group is based in spacious accommodation in the University's Main Building, and is well equipped with its own network of Silicon Graphics and Sun workstations, supported by a full-time system administrator. The successful candidate will have the opportunity to contribute in the following areas: * development of real-world applications of neural networks in connection with a wide variety of industrial and commercial research contracts * providing software contributions in support of basic research projects * production of demonstration software for use in teaching a variety of neural network courses Most of the software will be developed in C++ and Matlab, on a high- power Silicon Graphics workstation with access to the Group's SGI Challenge supercomputer. The ideal candidate will have: * a good first degree in a numerate discipline * expertise in software development (preferably in C and C++) * a good understanding of neural networks * working knowledge of basic mathematics such as calculus and linear algebra * experience of working in a UNIX environment * willingness to undertake complex and challenging problems This post provides an excellent opportunity to learn new skills within an exciting team environment Conditions of Service --------------------- The appointment will be for an initial period of one year, with the possibility of subsequent renewal. Initial salary will be on the academic 1A or 1B scales up to 15,986. How to Apply ------------ If you wish to be considered for this position, please send a full CV, together with the names and addresses of at least 3 referees, to: Hanni Sondermann Neural Computing Research Group Department of Computer Science and Applied Mathematics Aston University Birmingham B4 7ET, U.K. Tel: (+44 or 01) 21 333 4631 Fax: (+44 or 01) 21 333 6215 e-mail: h.e.sondermann at aston.ac.uk ---------------------------------------------------------------------- ~~~~~~~~~~~~~~ Richard J. Lister r.j.lister at aston.ac.uk ~~~~~~~~~~~~~~~~ Research Assistant, Neural Computing Research Group Aston University, Birmingham B4 7ET, UK ~~~~~~~~~~~~~~~~~ http://www.ncrg.aston.ac.uk/~listerrj/ ~~~~~~~~~~~~~~~~~~ From kak at ee.lsu.edu Fri Feb 23 10:06:38 1996 From: kak at ee.lsu.edu (Subhash Kak) Date: Fri, 23 Feb 96 09:06:38 CST Subject: shift invariance Message-ID: <9602231506.AA12042@ee.lsu.edu> For feedback neural networks here are some more references on shift invariant learning: T. Maxwell et al, Transformation invariance using high order correlations in neural net architectures. Plasma Preprint UMLPF #88-125. Univ of Maryland, 1988. W. Widrow and R. Winter, Neural nets for adaptive filtering and adaptive pattern recognition. Computer, 21 (3):25-39, 1988. D.L. Prados and S.C. Kak, Shift invariant associative memory. IN VLSI for Artificial Intelligence, J.G. Delgado-Frias and W.R. Moore (eds.), pp. 185-197, Kluwer Academic Publishers, 1989. -Subhash Kak From franco at cim.mcgill.ca Fri Feb 23 10:43:43 1996 From: franco at cim.mcgill.ca (Francesco Callari) Date: Fri, 23 Feb 1996 15:43:43 GMT Subject: Paper on active 3D object recognition and sensor planning via ANN Message-ID: <199602231543.PAA15318@Poptart.McRCIM.McGill.EDU> The following paper deals with the problems of combining 3D shape information and class priors for the purposes of active, model-based object recognition and sensor planning. The proposed system correlates estimates of shape and class uncertainty to determine those sensor locations that best disambiguate the objects. The class and class sensitivity estimates are computed by an MLP network, trained using MacKay's "evidence" framework and put in the planning feedback loop of a mobile robot. FTP-host: ftp.cim.mcgill.ca FTP-file: /pub/people/franco/ambiguity96.ps.gz Active Recognition: Using Uncertainty to Reduce Ambiguity Francesco G. Callari and Frank P. Ferrie Centre for Intelligent Machines, McGill University 3480 University St., Montre\'al, Que., Canada, H3A 2A7 email: franco at cim.mcgill.ca, ferrie at cim.mcgill.ca Keywords: Active Vision, Control of Perception, Learning in Computer Vision ABSTRACT Ambiguity in scene information, due to noisy measurements and uncertain object models, can be quantified and actively used by an autonomous agent to efficiently gather new data and improve its information about the environment. In this work an information-based utility measure is used to derive from a learned classification of shape models an efficient data collection strategy, specifically aimed at increasing classification confidence when recognizing uncertain shapes. Promising experimental results with real data are reported. Submitted to: ICPR96. From oby at cs.tu-berlin.de Fri Feb 23 12:07:06 1996 From: oby at cs.tu-berlin.de (Klaus Obermayer) Date: Fri, 23 Feb 1996 18:07:06 +0100 Subject: No subject Message-ID: <199602231707.SAA29613@pollux.cs.tu-berlin.de> GRADUATE STUDENT POSITION A graduate student position is available at the CS-department of the Technical University of Berlin to study models of neural development. A major objective is to construct and investigate models for the formation of ocular dominance and orientation selectivity in striate cortex. The candidate is expected to join a close collaboration between theorists and experimentalists. Candidates should have experience in computational modelling. The position is available initially for two years. Salary is commensurate with BAT II a/2. Applicants should send their CV, list of publications, a letter describing their interest, and name, address and phone number of two references to: Prof. Klaus Obermayer phone: 49-30-314-73442 FR2-1, KON, Informatik 49-30-314-73120 Technische Universitaet Berlin fax: 49-30-314-73121 Franklinstrasse 28/29 e-mail: oby at cs.tu-berlin.de 10587 Berlin, Germany http://www.cs.tu-berlin.de/fachgebiete/kon/ From kak at ee.lsu.edu Fri Feb 23 18:55:23 1996 From: kak at ee.lsu.edu (Subhash Kak) Date: Fri, 23 Feb 96 17:55:23 CST Subject: Paper Message-ID: <9602232355.AA19148@ee.lsu.edu> The following paper =========================================== ON BIOLOGICAL CYCLES AND INTELLIGENCE By Subhash C. Kak may be obtained by ftp from the following location: ftp://gate.ee.lsu.edu/pub/kak/bio.ps.Z -------------------------------------------- Abstract: If intelligence is taken as a measure of the organism's ability to adapt to its environment, the question of the organism's sensitivity to the rhythms of the environment becomes important. In this paper we provide a summary, including a brief historical resume, of this question of timing. We arge that artificial connectionist systems designed to range in natural environments will have to incorporate a system of inner clocks. --------------------------------------------- -Subhash Kak From carmesin at schoner.physik.uni-bremen.de Sat Feb 24 06:47:16 1996 From: carmesin at schoner.physik.uni-bremen.de (Hans-Otto Carmesin) Date: Sat, 24 Feb 1996 12:47:16 +0100 Subject: SIR: shift invariant recognition Message-ID: <199602241147.MAA09674@schoner.physik.uni-bremen.de> Dear Rolf and Jerry. The question raised by Rolf Wurtz is, how SIR ( shift invariant recognition) might be processed in the visual system. There is a biologically reasonable candidate network: I proposed it for experiments on so-called stroboscopic alternative motion (SAM). The most simple instance is established by TWO light dots, one of which is elicited at a time in an alternating manner. At adequate frequency an observer perceives ONE dot moving back and forth. The experiment becomes quite non-trivial with four dots at the corners of a square, two elicited at a time at diagonal positions and in an alternating manner. An observer perceives either two dots moving horizontally or two dots moving vertically (roughly speaking). The network represents each dot by a formal neuron; these neurons project to inner neurons that tend to fire in accordance with the stimulation and that are coupled with rapid formal couplings (similar to dynamic links) with a local coupling dynamics reminescent of the Hebb-rule [1-4]. A motion percept is established by the emerging nonzero couplings. It turns out that each active neuron at a time t is coupled to exactly one active neuron at a later time, t+t' say. Moreover there are prestabilized coupling weights (modeling synaptic densities) that prefer short distances in space and time. As a result: If a pattern is presented at a time t and a shifted pattern is presented at a time t+t', then the dots of the first pattern are coupled to the corresponding dots of the second pattern. This network is understood very well [3,4]: It can be solved analytically and exhibits an effective potential dynamics in coupling space. I predicted [3] a continuous phase transition and measured it together with experimental psychologists later. Another indication of biological relevance: Formally the network is very similar to networks with retinotopy emergence [5]. References: [1] H.-O. Carmesin: Statistical neurodynamics: A model for universal properties of EEG-data and perception. Acta Physica Slovaca, 44:311--330, 1994. [2] H.-O. Carmesin and S. Arndt: Neuronal self-organization of motion percepts. Technical Report 6/95, ZKW Universitt Bremen, Bremen, 1995. [3] H.-O. Carmesin: Theorie neuronaler Adaption. (Kster, Berlin, 1994. ISBN 3-89574-020-9). [4] H.-O. Carmesin: Neuronal Adaptation Theory. (Peter Lang, Frankfurt am Main, 1996. ISBN 3-631-30039-5). [5] H.-O. Carmesin: Topology preservation emergence by Hebb rule with infinitesimal short range signals. Phys. Rev. E, 53(1):993--1003, 1996. For details see: WWW: http://schoner.physik.uni-bremen.de/~carmesin/ From scheler at informatik.tu-muenchen.de Sat Feb 24 07:50:42 1996 From: scheler at informatik.tu-muenchen.de (Gabriele Scheler) Date: Sat, 24 Feb 1996 13:50:42 +0100 Subject: Shift Invariance Message-ID: <96Feb24.135055+0100_met.116444+24@papa.informatik.tu-muenchen.de> There should be a difference made between shift-invariance, i.e. distinguishing between T1: {[a,b,c,d,e], [b,c,d,e,a], [c,d,e,a,b]} T2: {[a,d,b,c,d], [a,d,c,b,e] etc.} which is more of a purely mathematical problem, and translational invariance, i.e. detecting a pattern on a plane, no matter where it occurs. For the latter goal it is sufficient to develop a set of features in the first layer to detect that pattern in a local field, and to develop an invariant detector in the next layer, which is ON for any of the lower-level features. (develop means train for ANN). In the domain of neural networks the obvious solution to the mathematical problem would be to train a level of units as sequence encoders: A1 B1 C1 D1 ----- ---- a b ------- c ------- d and classify patterns then on how many of the sequence encoders a-d are ON. Of course this may be rather wasteful. In another learning approach called adaptive distance measures, we can reduce training effort considerably when we use a distance measure which is specifically tuned to problems of shift invariance. Of course this is nothing else than to have a class of networks with pre-trained sequence encoders available. The question here as often is not, which NN can learn this task (backprop can, Fukushima's Neocognitron can), but which is most economical in its resources - without requiring too much knowledge on the type of function to be learned. From lpratt at fennel.mines.edu Sat Feb 24 13:15:45 1996 From: lpratt at fennel.mines.edu (Lorien Y. Pratt) Date: Sat, 24 Feb 1996 11:15:45 -0700 (MST) Subject: Call for papers -- please post Message-ID: <199602241815.LAA01740@fennel.mines.edu> ------------------------------------------------------------------------------- Call for papers (please post) Special Issue of the Machine Learning Journal on Inductive Transfer ------------------------------------------------------------------------------- Lorien Pratt and Sebastian Thrun, Guest Editors ------------------------------------------------------------------------------- Many recent machine learning efforts are focusing on the question of how to learn in an environment in which more than one task is performed by a system. As in human learning, related tasks can build on one another, tasks that are learned simultaneously can cross-fertilize, and learning can occur at multiple levels, where the learning process itself is a learned skill. Learning in such an environment can have a number of benefits, including speedier learning of new tasks, a reduced number of training examples for new tasks, and improved accuracy. These benefits are especially apparent in complex applied tasks, where the combinatorics of learning are often otherwise prohibitive. Current efforts in this quickly growing research area include investigation of methods that facilitate learning multiple tasks simultaneously, those that determine the degree to which two related tasks can benefit from each other, and methods that extract and apply abstract representations from a source task to a new, related, target task. The situation where the target task is a specialization of the source task is an important special case. The study of such methods has broad application, including a natural fit to data mining systems, which extract regularities from heterogeneous data sources under the guidance of a human user, and can benefit from the additional bias afforded by inductive transfer. We solicit papers on inductive transfer and learning to learn for an upcoming Special Issue of the Machine Learning Journal. Please send six (6) copies of your manuscript postmarked by June 1, 1996 to: Dr. Lorien Pratt MCS Dept. CSM Golden, CO 80401 USA One (1) additional copy should be mailed to: Karen Cullen Attn: Special Issue on Inductive Transfer MACHINE LEARNING Editorial Office Kluwer Academic Publishers 101 Philip Drive Assinippi Park Norwell, MA 02061 USA Manuscripts should be limited to at most 12000 words. Please also note that Machine Learning is now accepting submission of final copy in electronic form. Authors may want to adhere to the journal formatting standards for paper submissions as well. There is a latex style file and related files available via anonymous ftp from ftp.std.com. Look in Kluwer/styles/journals for the files README, kbsfonts.sty, kbsjrnl.ins, kbsjrnl.sty, kbssamp.tex, and kbstmpl.tex, or the file kbsstyles.tar.Z, which contains them all. Please see http://vita.mines.edu:3857/1s/lpratt/transfer.html for more information on inductive transfer. Papers will be quickly reviewed for a target publication date in the first quarter of 1997. -- Dr. Lorien Y. Pratt Department of Mathematical and Computer Sciences lpratt at mines.edu Colorado School of Mines (303) 273-3878 (work) 402 Stratton (303) 278-4552 (home) Golden, CO 80401, USA Vita, photographs, all publications, all course materials available from my web page: http://vita.mines.edu:3857/1s/lpratt From clee at it.wustl.edu Sat Feb 24 22:57:36 1996 From: clee at it.wustl.edu (Christopher Lee) Date: Sat, 24 Feb 1996 21:57:36 -0600 Subject: shift invariance In-Reply-To: <9602210812.ZM26438@ICSI.Berkeley.edu> References: <9602210812.ZM26438@ICSI.Berkeley.edu> Message-ID: <199602250357.VAA23373@it> >>>>> "Jerry" == Jerry Feldman writes: Jerry> Shift invariance is the ability of a neural system to Jerry> recognize a pattern independent of where appears on the Jerry> retina. It is generally understood that this property can Jerry> not be learned by neural network methods, but I have not Jerry> seen a published proof. A "local" learning rule is one that Jerry> updates the input weights of a unit as a function of the Jerry> unit's own activity and some performance measure for the Jerry> network on the training example. All biologically plausible Jerry> learning rules, as well as all backprop variants, are local Jerry> in this sense. Jerry's communique has certainly certainly sparked discussion, but I feel as if his reference to "neural network methods" needs more precise definition. Perhaps Jerry could state more specifically the class of network architectures and neurons he wishes to consider? (E.g., Minsky and Papert restricted their proof to order one perceptrons.) What sort of resource limitations would you put on this network relative to the complexity of the task? (To give an absurd example of why this is important: for a small "test problem"-like space, if given an appropriate number of nodes a network could simply "memorize" all the configurations of an object at all locations. Clearly, this isn't what one would normally considering "learning" shift invariance.) On another vein that might be of interest, it's clear that shift invariance is a fundamental to the primate visual system in some way, and a fair amount of interest exists in the neurophysiology community concerning how this problem is solved; one hypothesis involves the role of attentional mechanisms in scale and translational invariance (Olshausen, Anderson, Van Essen, J. of Neuroscience. 13(11):4700-19, 1993). It is not obvious to me that anything along the lines of Jerry's proof could be applied to their (the Olshausen et al.) network model. Christopher Lee -- Washington University Department of Anatomy and Neurobiology email: clee at v1.wustl.edu From hinton at cs.toronto.edu Sun Feb 25 13:00:36 1996 From: hinton at cs.toronto.edu (Geoffrey Hinton) Date: Sun, 25 Feb 1996 13:00:36 -0500 Subject: obvious, but false Message-ID: <96Feb25.130037edt.525@neuron.ai.toronto.edu> In response to may email about a network that learns shift invariance, Dorffner says: > there seems to be a misunderstanding of what the topic of discussion > is here. I don't think that Jerry meant that no model consisting of neural > network components could ever learn shift invariance. After all, there are > many famous examples in visual recognition with neural networks (such as > the Neocognitron, as Rolf W"urtz pointed out), and if this impossibility > were the case, we would have to give up neural network research in > perceptual modeling altogether. > > What I think Jerry meant is that any cascade of fully-connected feed-forward > connection schemes between layers (including the perceptron and the MLP) > cannot learn shift invariance. > Now besides being obvious, this does raise some > important questions, possibly weakening the fundamentals of connectionism. I agree that this is what Jerry meant. What Jerry said was actually very reasonable. He did NOT say it was obviously impossible. He just said that it was generally understood to be impossible and he would like to see a proof. I think Jerry was right in the sense that most people I have talked to believed it to be impossible. I'd like to apologize to Jerry for the antagonistic tone of my previous message. Dorffner takes the impossibility for granted. My simulation conclusively demonstrates that translation invariance can be learned with no built in bias towards translation invariance. The only requirement is that the shapes should share features, and this is a requirement on the data, not on the network. At the risk of looking very silly, I bet that it really cannot be done if shapes do not share features. My simulation did not have built in preprocessing or weight-sharing as Dorffner seems to imply. So, unlike the neocognitron, it had no innate bias towards translation invariance. It got the "raw" retinal inputs and its desired outputs were shape identities. The version with local connectivity worked best, but as I pointed out, it also worked without local connectivity. So that version exactly fitted Dorffner's definition of what cannot be done. Geoff PS: As I noted in the paper and others have pointed out in their responses, Minsky and Papert's group invariance theorem really does prove that this task cannot be done without hidden layers (using conventional units). From geva at fit.qut.edu.au Sun Feb 25 19:05:33 1996 From: geva at fit.qut.edu.au (Shlomo Geva) Date: Mon, 26 Feb 1996 10:05:33 +1000 (EST) Subject: shift invariance In-Reply-To: <199602230205.VAA13842@bucnsd.qut.edu.au> Message-ID: Regarding shift invariance: One might learn something about the problem by looking at the Fourier Transform in the context of shift invariance. One may perform a Discrete Fourier Transform (DFT), and take advantage of the shift invariance properties of the magnitudes components, discarding the phase and representing objects by a feature vector consisting of the the magnitudes in frequency domain alone. (This is not new and also extends to higher dimensionalities) This approach will solve many practical problems, but has an in-principle difficulty in that this procedure does not produce a unique mapping from objects to invariant features. For example, start from any object and obtain its invariant representation as above. By choosing arbitrary phase components and performing an inverse DFT we can get arbitrarily many object representations. Note that these objects are extremely unlikely to look like an original shifted object! If by chance - and it may be very remote - two of the objects you wish to recognize with shift invariance, have identical magnitudes in the frequency domain then this method will obviously fail. Now I'd like to make a conjecture. It appears to make sense to assume that this difficulty is inherent to the shift invariance requirement itself. If this is so then unless you have an additional constraint imposed on objects - they cannot be allowed to be identical under the invariant feature extraction transformation you wish to employ - then you cannot solve the problem. In other words, one needs a guarantee that all permissible objects are uniquely transformed by the procedure. It seems to follow that no general procedure, that does not take into account the nature of the objects for which the procedure is intended, can exist. I am wondering if anyone could clarify whether this is a valid argument. Shlomo Geva s.geva at qut.edu.au From hinton at cs.toronto.edu Sun Feb 25 13:33:46 1996 From: hinton at cs.toronto.edu (Geoffrey Hinton) Date: Sun, 25 Feb 1996 13:33:46 -0500 Subject: yet more on shift invariance Message-ID: <96Feb25.133354edt.525@neuron.ai.toronto.edu> The argument that Jerry Feldman gave for the difficulty of learning shift invariance illustrates a nice point about learning. He talked about learning to recognize a SINGLE shape in different locations. I think he is probably right that this is impossible without built in prejudices. But the implication was that if it was true for ONE shape then it would be true for a bunch of shapes. This is where the argument breaks down. Its like the point being made by the life-time learning people, except that here the separate tasks are not presented one after another but jumbled together. Geoff From juergen at idsia.ch Mon Feb 26 03:00:00 1996 From: juergen at idsia.ch (Juergen Schmidhuber) Date: Mon, 26 Feb 96 09:00:00 +0100 Subject: neural text compression Message-ID: <9602260800.AA24691@fava.idsia.ch> Now available online: SEQUENTIAL NEURAL TEXT COMPRESSION (9 pages, 68 K) IEEE Transactions on Neural Networks, 7(1):142-146, 1996 Juergen Schmidhuber, IDSIA Stefan Heil, TUM http://www.idsia.ch/~juergen Abstract: Neural nets may be promising tools for loss-free data compression. We combine predictive neural nets and statistical coding techniques to compress text files. We apply our methods to short newspaper articles and obtain compression ratios exceeding those of widely used Lempel-Ziv algorithms (the basis of UNIX functions `compress' and `gzip'). The main disadvantage of our methods is: on conventional machines they are about three orders of magnitude slower than standard methods. To obtain a copy, cut and paste this: netscape ftp://ftp.idsia.ch/pub/juergen/textcompression.ps.gz ------------ P.S.: Have you got a question on recent work on "learning to learn" and "incremental self-improvement"? Stewart Wilson asked me to place the corresponding paper "Environment-independent reinforcement acceleration" in his NetQ web site. Now it is sitting there and waiting for a friendly question or two (questions may be anonymous): netscape http://netq.rowland.org Juergen Schmidhuber, IDSIA From giles at research.nj.nec.com Mon Feb 26 13:01:25 1996 From: giles at research.nj.nec.com (Lee Giles) Date: Mon, 26 Feb 96 13:01:25 EST Subject: Shift Invariance Message-ID: <9602261801.AA26373@alta> We and others [1, 2, 3, 4] showed that invariances, actually affine transformations, could directly be encoded into feedforward higher-order (sometimes called polynomial, sigma-pi, gated, ...) neural nets such that these networks are invariant to shift, scale, and rotation of individual patterns. As mentioned previously, similar invariant encodings can be had for associative memories in autonomous recurrent networks. Interestingly, this idea of encoding geometric invariances into neural networks is an old one [5]. [1] C.L. Giles, T. Maxwell,``Learning, Invariance, and Generalization in High-Order Neural Networks'', Applied Optics, 26(23), p 4972, 1987. Reprinted in ``Artificial Neural Networks: Concepts and Theory,'' eds. P. Mehra and B. W. Wah, IEEE Computer Society Press, Los Alamitos, CA. 1992. [2] C.L. Giles, R.D. Griffin, T. Maxwell,``Encoding Geometric Invariances in Higher-Order Neural Networks'', Neural Information Processing Systems, ed. D.Z. Anderson, Am. Inst. of Physics, N.Y., N.Y., p 301-309, 1988. [3] S.J. Perantonis, P.J.G. Lisboa, ``Translation, Rotation, and Scale Invariant Pattern Recognition by Higher-Order Neural Networks and Moment Classifiers'', IEEE Transactions on Neural Networks'', 3(2), p 241, 1992. [4] L. Spirkovska, M.B. Reid,``Higher-Order Neural Networks Applied to 2D and 3D Object Recognition'', Machine Learning, 15(2), p. 169-200, 1994. [5] W. Pitts, W.S. McCulloch, ``How We Know Universals: The Perception of Auditory and Visual Forms'', Bulletin of Mathematical Biophysics, vol 9, p. 127, 1947. Bibtex entries for the above can be found in: ftp://external.nj.nec.com/pub/giles/papers/high-order.bib -- C. Lee Giles / Computer Sciences / NEC Research Institute / 4 Independence Way / Princeton, NJ 08540, USA / 609-951-2642 / Fax 2482 www.neci.nj.nec.com/homepages/giles.html == From kim.plunkett at psy.ox.ac.uk Mon Feb 26 09:47:50 1996 From: kim.plunkett at psy.ox.ac.uk (Kim Plunkett) Date: Mon, 26 Feb 1996 14:47:50 +0000 Subject: LCP.txt Message-ID: <9602261447.AA50643@mac17.psych.ox.ac.uk> REMINDER: Manuscript submissions are invited for inclusion in a Special Issue of Language and Cognitive Processes on Connectionist Approaches to Language Development. It is anticipated that most of the papers in the special issue will describe previously unpublished work on some aspect of language development (first or second language learning in either normal or disordered populations) that incorporates a neural network modelling component. However, theoretical papers discussing the general enterprise of connectionist modelling within the domain of language development are also welcome. The deadline for submissions is 1st April 1996. Manuscripts should be sent to the guest editor for this special issue: Kim Plunkett Department of Experimental Psychology Oxford University South Parks Road Oxford, OX1 3UD UK email: plunkett at psy.ox.ac.uk FAX: 1865-310447 All manuscripts will be submitted to the usual Language and Cognitive Processes peer review process. From perso at DI.Unipi.IT Mon Feb 26 06:36:04 1996 From: perso at DI.Unipi.IT (Alessandro Sperduti) Date: Mon, 26 Feb 1996 12:36:04 +0100 (MET) Subject: ICML'96 W/S on EC&ML Message-ID: <199602261136.MAA11154@neuron.di.unipi.it> ICML'96 Workshop on EVOLUTIONARY COMPUTING AND MACHINE LEARNING to be held in Bari, Italy, July 2-3, 1996, at the International Conference on Machine Learning. http://zen.btc.uwe.ac.uk/evol/cfp.html In the last decade, research concentrating on the interaction between evolutionary computing and machine learning has developed from the study and use of genetic algorithms and reinforcement learning in rule based systems (i.e. classifier systems) to a variety of learning systems such as neural networks, fuzzy systems and hybrid symbolic/evolutionary systems. Many kinds of learning process are now being integrated with evolutionary algorithms, e.g. supervised, unsupervised, reinforcement, on/off-line and incremental. The aim of this workshop is to bring together people involved and interested in this field to share common theory and practice, and to represent the state of the art. Submissions are invited on topics related to: machine learning using evolutionary algorithms, the artificial evolution of machine learning systems, systems exploring the interaction between evolution and learning, systems integrating evolutionary and machine learning algorithms and on applications that make use of both machine learning and evolutionary algorithms. Contributions that argue a position, give an overview, give a review, or report recent work are all encouraged. Copies of extended abstracts or full papers no longer than 15 pages should be sent (by April 23 1996) to: Terry Fogarty Faculty of Computer Studies and Mathematics University of the West of England Frenchay Phone: (+44) 117 965 6261 Bristol BS16 1QY Fax: (+44) 117 975 0416 UK Email: tcf at btc.uwe.ac.uk or: Gilles Venturini Ecole d'Ingenieurs en Informatique pour l'Industrie Universite de Tours, 64, Avenue Jean Portalis, Phone: (+33)-47-36-14-33 Technopole Boite No 4, Fax: (+33)-47-36-14-22 37913 Tours Cedex 9 Email: venturi at lri.fr FRANCE venturini at univ-tours.fr Accepted papers will constitute the workshop notes and will be refereed by the program committee for inclusion in the post-workshop proceedings in the light of scientific progress made at the workshop. Program committee: F. Baiardi, University of Pisa, Italy. H. Bersini, Universite Libre de Bruxelles, Belgium. L.B. Booker, MITRE Corporation, USA. D. Cliff, University of Sussex, UK. M. Colombetti, Politecnico di Milano, Italy. K. De Jong, George Mason University, USA. M. Dorigo, Universite Libre de Bruxelles, Belgium. T.C. Fogarty, University of the West of England, UK. A. Giordana, University of Torino, Italy. J.G. Grefenstette, Naval Research Laboratory, USA. J.A. Meyer, Ecole Normale Superieure, France. M. Patel, University of Newcastle, UK. M. Schoenauer, Ecole Polytechnique, France. R.E. Smith, University of Alabama, USA. G. Venturini, University of Tours, France. S.W. Wilson, Rowland Institute for Science, USA. Important Dates: April 23: Extended abstracts and papers due May 14: Notification of acceptance June 4: Camera-ready copy for workshop notes due July 2-3: Workshop Prof. Fabrizio Baiardi Dip. di Informatica, Universita di Pisa C. Italia 40, 56123 Pisa, Italy ph: +39/50/887262 email: baiardi at di.unipi.it From stefan.kremer at crc.doc.ca Mon Feb 26 14:35:00 1996 From: stefan.kremer at crc.doc.ca (Dr. Stefan C. Kremer) Date: Mon, 26 Feb 1996 14:35:00 -0500 Subject: shift invariance and recurrent networks Message-ID: <2.2.32.19960226193500.00694f68@digame.dgcd.doc.ca> At 08:12 96-02-21 -0800, Jerry Feldman wrote: > The one dimensional case of shift invariance can be handled by treating >each string as a sequence and learning a finite-state acceptor. But the >methods that work for this are not local or biologically plausible and >don't extend to two dimensions. Recently, many recurrent connectionist networks have been applied to the problem of grammatical induction (i.e. inducing a grammar, or equivalently a finite state acceptor for a given set of example strings) [see, for example: Giles (1990)]. These types of networks are capable of learning many types of regular grammars (e.g. (0)*(101)(0)*). Learning of context-free grammars by connectionist networks has also been studied elsewhere [Das (1993)]. The resulting trained networks work only on the basis of local (both spatially and temporally) interactions among neighbouring processing elements. There are a variety of learning algorithms for these networks. Some like backpropagation through time [Rumelhart, 1986] are spatially local, but temporally global, some like real-time recurrent learning [Williams, 1989] are temporally local and spatially global, and some are both temporally and spatially local like Elman's truncated gradient descent [Elman, 1990] and various locally recurrent networks [Tsoi, 1994]. Don't these types of networks can handle shift invariance problems using local processing? (I'd agree that they're not biologically plausible... ;) ). > The unlearnability of shift invarince is not a problem in practice because >people use preprocessing, weight sharing or other techniques to get shift >invariance where it is known to be needed. However, it does pose a problem for >the brain and for theories that are overly dependent on learning. I'm not sure I understand this last part. Are you saying that "preprocessing" and "weight sharing" can handle shift invariance problems because they are a type of non-local processing? -Stefan P.S. Here's the refs: @INPROCEEDINGS{giles90p, AUTHOR = "C.L. Giles and G.Z. Sun and H.H. Chen and Y.C. Lee and D. Chen", TITLE = "Higher Order Recurrent Networks & Grammatical Inference", BOOKTITLE = "Advances in Neural Information Processing Systems~2", YEAR = "1990", EDITOR = "D.S. Touretzky", PUBLISHER = "Morgan Kaufmann Publishers", ADDRESS = "San Mateo, CA", PAGES = "380-387"} @INPROCEEDINGS{das93p, AUTHOR = "S. Das and C.L. Giles and G.Z. Sun ", TITLE = "Using Prior Knowledge in a NNPDA to Learn Context-Free Languages", BOOKTITLE = "Advances in Neural Information Processing Systems 5", PUBLISHER = "Morgan Kaufmann Publishers", EDITOR = "S.J. Hanson and J.D. Cowan and C.L. Giles", PAGES = "65--72", ADDRESS = "San Mateo, CA" YEAR = "1993"} @BOOK{rumelhart86b1, EDITOR = "J. L. McClelland, D.E. Rumelhart and the P.D.P. Group (Eds.)", AUTHOR = "D. Rumberlhart, G. Hinton, R. Williams", TITLE = "Learning Internal Representation by Error Propagation", VOLUME = "1: Foundations", BOOKTITLE = "Parallel Distributed Processing: Explorations in the Microstructure of Cognition", YEAR = "1986", PUBLISHER = "MIT Press", ADDRESS = "Cambridge, MA"} @ARTICLE{williams89j1, AUTHOR = "R.J. Williams and D. Zipser", TITLE = "A Learning Algorithm for Continually Running Fully Recurrent Neural Networks", JOURNAL = "Neural Computation", YEAR = "1989", VOLUME = "1", NUMBER = "2", PAGES = "270-280"} @ARTICLE{elman90j, AUTHOR = "J.L. Elman", TITLE = "Finding Structure in Time", JOURNAL = "Cognitive Science", YEAR = "1990", VOLUME = "14", PAGES = "179-211"} @ARTICLE{tsoi94j, AUTHOR = "A.C. Tsoi and A. Back", TITLE = "Locally Recurrent Globally Feedforward Networks, A Critical Review of Architectures", JOURNAL = "IEEE Transactions on Neural Networks", VOLUME = "5", NUMBER = "2", PAGES = "229-239", YEAR = "1994"} -- Dr. Stefan C. Kremer, Neural Network Research Scientist, Communications Research Centre, 3701 Carling Avenue, P.O. Box 11490, Station H Ottawa, Ontario K2H 8S2 # Tel: (613)990-8175 Fax: (613)990-8369 E-mail: Stefan.Kremer at crc.doc.ca # WWW: http://digame.dgcd.doc.ca/~kremer/ From jamie at atlas.ex.ac.uk Mon Feb 26 15:35:05 1996 From: jamie at atlas.ex.ac.uk (jamie@atlas.ex.ac.uk) Date: Mon, 26 Feb 96 20:35:05 GMT Subject: shift invariance Message-ID: <1708.9602262035@sirius.dcs.exeter.ac.uk> Jerry Feldman writes: > Shift invariance is the ability of a neural system to recognize a pattern >independent of where appears on the retina. It is generally understood that >this property can not be learned by neural network methods, I agree with Jerry that connectionist networks cannot actually learn shift invariance. A connectionist network can exhibit shift invariant behavior by either being exhaustively trained on a set of patterns that happens to have this property, or by having shift invariance somehow wired into the network before the learning occurs. However, neither of these situation constitutes "learning shift invariance". On the other hand, we are still left with the problem of explaining shift invariant behavior. Some of the responses so far imply training on all patterns in all positions (exhaustive training). I don't find this approach interesting, since it doesn't address the basic issue of generalization ability. Thus the question seems to be how shift invariance can be wired in while still using a learning rule that is local, biologically plausible, etc. Geoff Hinton writes: >shift invariance can be learned by backpropagation. It was one of the >problems that I tried when fiddling about with backprop in the mid 80's. Geoff Hinton's experiment clearly does not train the network exhaustively on all patterns in all positions, so (by the above argument) I have to claim that shift invariance is wired in. The network does not use weight sharing, which would be the most direct way of wiring in shift invariance. However, it does appear to use "error sharing". As I understood Geoff's description, the weights between the position-dependent and position-independent hidden layers are not modified by learning. Each position-independent feature detector is connected to all its associated position-dependent feature detectors with links of the same weight (in particular, they are ORed). Using backprop, this has the effect of distributing the same error signal to each of these position-dependent feature detectors. Thus they all tend to converge to the same feature. In this way, fixing the weights between the two hidden layers to equal values makes the feature detectors learned in one position tend to generalize to other positions. I suspect "tend to" may be an important caveat here, but in essence the equivalence of all positions has been wired in. This equivalence is simply shift invariance. On the other hand, the learning rule is still local, so in that sense it does seem to meet Jerry's challenge. Jerry Feldman also writes: > The one dimensional case of shift invariance can be handled by treating >each string as a sequence and learning a finite-state acceptor. But the >methods that work for this are not local or biologically plausible and >don't extend to two dimensions. I have to disagree with this dismissal of recurrent networks for handling shift invariance. I'm not in a position to judge biological plausibility, but I would take issue with the claim that methods of training recurrent networks are nonlocal and can't be generalized to higher dimensions. These learning methods are to some extent temporally nonlocal, but some degree of temporal nonlocality is necessary for any computation that extends over time. The important thing is that they are just as spatially local as the feedforward methods they are based on. Jerry's own definition of locality is spatial locality: >A "local" learning rule is one that updates the input weights of a unit as a >function of the unit's own activity and some performance measure for the >network on the training example. Now finally I get to my primary gripe. Contrary to Jerry's claim, learning methods for recurrent networks can be generalized to more than one dimension. The issues for two dimensions are entirely the same as those for one. All that is necessary to extend recurrence to two dimensions is units that pulse periodically. In engineering terms, a single network is time-multiplexed across one dimension while being sequenced across the other. Conceptually, learning can be done by unfolding the network over one time dimension, then unfolding the result over the other time dimension, then using a feedforward method. The idea of using time to represent two different dimensions has in fact already been proposed. At an abstract level, this dual use of the time dimension is the core idea behind temporal synchrony variable binding (TSVB) (Shastri and Ajjanagadde, 1993). Recurrent networks use the time dimension to represent position in the input sequence (or computation sequence). TSVB also uses the time dimension to represent variables. Because the same network is being used at every point in the input sequence, recurrent networks inherently generalize things learned in one input sequence position to other input sequence positions. In this way shift invariance is "wired in". Exactly analogously, because the same network is being used for every variable, TSVB networks inherently generalize things learned using one variable to other variables. I argue in (Henderson, submitted) that this results in a network that exhibits systematicity. Replacing the labels "sequence position" and "variable" with the labels "horizontal position" and "vertical position" does not change this basic ability to generalize across both dimensions. Work on applying learning to TSVB networks is being done by both Shastri and myself. (Note that this description of TSVB is at a very abstract level. Issues of biological plausibility are addressed in (Shastri and Ajjanagadde, 1993) and the papers cited there.) Shastri, L. and Ajjanagadde, V. (1993). From simple associations to systematic reasoning: A connectionist representation of rules, variables, and dynamic bindings using temporal synchrony. Behavioral and Brain Sciences, 16:417--451. Henderson, J. (submitted). A connectionist architecture with inherent systematicity. Submitted to the Eighteenth Annual Conference of the Cognitive Science Society. - Jamie ------------------------------ Dr James Henderson Department of Computer Science University of Exeter Exeter EX4 4PT, U.K. ------------------------------ From johnd at saturn.sdsu.edu Mon Feb 26 16:35:10 1996 From: johnd at saturn.sdsu.edu (John Donald) Date: Mon, 26 Feb 1996 13:35:10 -0800 (PST) Subject: shift invariance Message-ID: <199602262135.NAA15297@saturn.sdsu.edu> Abu-Mostafa's "learning from hints" is a very simple approach to learning neural net representations of functions that satisfy global constraints such as shift invariance. Cf Scientific American April 1995 (!) and the references therein, eg. "Learning from hints", Yaser Abu-Mostafa, J. of Complexity 10 (165-178), 1994. His idea is to add to the training examples additional invented examples that represent the global properties. He claims significant speed-up, eg in training nets to learn an even (global property) shift invariant (global) function. From wimw at mbfys.kun.nl Tue Feb 27 06:27:31 1996 From: wimw at mbfys.kun.nl (Wim Wiegerinck) Date: Tue, 27 Feb 1996 12:27:31 +0100 (MET) Subject: Paper Available Message-ID: <199602271127.MAA16094@septimius.mbfys.kun.nl> A non-text attachment was scrubbed... Name: not available Type: text Size: 2250 bytes Desc: not available Url : https://mailman.srv.cs.cmu.edu/mailman/private/connectionists/attachments/00000000/696d56a4/attachment-0001.ksh From edelman at wisdom.weizmann.ac.il Tue Feb 27 08:19:18 1996 From: edelman at wisdom.weizmann.ac.il (Edelman Shimon) Date: Tue, 27 Feb 1996 13:19:18 GMT Subject: shift invariance In-Reply-To: (message from Shlomo Geva on Mon, 26 Feb 1996 10:05:33 +1000 (EST)) Message-ID: <199602271319.NAA05288@lachesis.wisdom.weizmann.ac.il> > Date: Mon, 26 Feb 1996 10:05:33 +1000 (EST) > From: Shlomo Geva > > Regarding shift invariance: > > [some stuff omitted] > > Now I'd like to make a conjecture. > It appears to make sense to assume that this difficulty is inherent > to the shift invariance requirement itself. If this is so > then unless you have an additional constraint imposed on objects - > they cannot be allowed to be identical under the invariant feature > extraction transformation you wish to employ - > then you cannot solve the problem. In other words, one needs a guarantee that > all permissible objects are uniquely transformed by the procedure. > It seems to follow that > no general procedure, that does not take into account the nature of the objects > for which the procedure is intended, can exist. > > I am wondering if anyone could clarify whether this is a valid argument. > > Shlomo Geva A number of people (see the refs below) have proved in the past that no universal invariants with respect to viewpoint exist for objects represented as point sets in 3D. The proofs hinged on the possibility of two different objects having the same 2D projection. Offhand, it seems that a similar argument could be used to prove Shlomo's conjecture. -Shimon Dr. Shimon Edelman, Applied Math. & Computer Science Weizmann Institute of Science, Rehovot 76100, Israel The Web: http://eris.wisdom.weizmann.ac.il/~edelman fax: (+972) 8 344122 tel: 8 342856 sec: 8 343545 @inproceedings{MosesUllman92, author="Y. Moses and S. Ullman", title="Limitations of non model-based recognition schemes", booktitle="Proc. 2nd European Conf. on Computer Vision, Lecture Notes in Computer Science", volume="588", pages="820-828", editor="G. Sandini", publisher="Springer Verlag", addredd="Berlin", year="1992" } @article{BurWeiRis93, author="J.B. Burns and R. Weiss and E. Riseman", title="View variation of point-set and line segment features", journal=pami, volume="15", pages = "51-68", year = 1993 } From wiskott at salk.edu Tue Feb 27 20:14:15 1996 From: wiskott at salk.edu (Laurenz Wiskott) Date: Tue, 27 Feb 1996 17:14:15 -0800 Subject: shift invariance Message-ID: <199602280114.RAA21840@bayes.salk.edu> Dear connectionists, here is an attempt to put the arguments so far into order (and to to add a bit). I have put it into the form of a list of statements, which are, of course, subjective. You can skip the indented comments in the first reading. --------------------------------------------------------------------------- 1) With respect to shift invariance, there are two types of artificial neural nets (ANNs): a) ANNs with a build in concept of spatial order (e.g. neocognitron and other weight sharing ANNs, neural shifter circuits, dynamic link matching), let me call these ANNs structured. b) ANNs without any build in concept of spatial order (e.g. fully connected back-propagation), let me call these ANNs isotropic. (This distinction is important. For instance, Jerry Feldman's statement referred to isotropic ANNs while Rolf W"urtz' disagreement was based on structured ANNs.) 2) Structured ANNs can DO shift invariant pattern discrimination, but they do not LEARN it. (It is important to note that structured ANNs for shift invariant pattern recognition usually do NOT LEARN the shift invariance. It is already build in (I'd be glad to see a counterexample). What they learn is pattern discrimination, under the constraint that, whatever they do, it is shift invariant.) 3) The isotropic ANNs can learn shift invariant pattern recognition, given that during training the patterns are presented at ALL locations. This is not surprising and not what we are asking for. (This is what Christopher Lee pointed out:>>... for a small "test problem"-like space, if given an appropriate number of nodes a network could simply "memorize" all the configurations of an object at all locations. Clearly, this isn't what one would normally considering "learning" shift invariance.<<) 4) What we are asking for is generalization. I see two types of generalization: a) generalization of pattern recognition from one part of the input plain to another. b) generalization of shift invariance from one pattern to another. (4a example: training on patterns A {101} and B {011} in the left half-plane, i.e. {101000, 010100, 011000, 001100}, and testing on patterns A and B in the right half-plane, e.g. {000101, 000011}. 4b example: training on some patterns {111, 011, 010} in all possible locations, i.e. {111000, 011100, 001110, 000111, 011000, ..., 000010}, and on pattern A {101} in the left half-plane, i.e {101000, 010100}, and testing on pattern A in the right half-plane, e.g. {000101}. This is again an important distinction. For instance, Jerry Feldman's statement referred to generalization 4a and Geoffrey Hinton's disagreement referred to generalization 4b.) 5) Generalization 4a seems to be impossible for an isotropic ANN. (This was illustrated by Jerry Feldman and, more elaborately, by Georg Dorffner.) 6) Generalization 4b is possible. (This has been demonstrated by the model of Geoffrey Hinton.) 7) Models which generalize according to 4b usually loose discriminative power, because patterns with the same set of features but in different spatial order get confused. (This has been pointed out by Shlomo Geva. This also holds for some structured ANNs, such as the neocognitron and other weight sharing ANNs, but does not hold for the neural shifter circuits and dynamic link matching. The loss of discriminative power can be avoided by using sufficiently complex features, which has its own drawbacks.) --------------------------------------------------------------------------- Best regards, Laurenz Wiskott. =========================================================================== .---. .---. Dr. L a u r e n z W i s k o t t | |. S A L K .| | Computational Neurobiology Laboratory | ||. C N L .|| | The Salk Institute for Biological Studies | ||| L W ||| | mail: PO Box 85800, San Diego, CA 92186-5800 | |||=======||| | street, parcels: 10010 North Torrey Pines Road `---''' " ```---' La Jolla, CA 92037 phone: +1 (619) 453-4100 ext 1463; fax: +1 (619) 587-0417; email: wiskott at salk.edu; WWW: http://www.cnl.salk.edu/~wiskott/ =========================================================================== From sontag at control.rutgers.edu Wed Feb 28 11:01:03 1996 From: sontag at control.rutgers.edu (Eduardo Sontag) Date: Wed, 28 Feb 1996 11:01:03 -0500 Subject: TR available - classification of points in general position Message-ID: <199602281601.LAA19838@control.rutgers.edu> SHATTERING ALL SETS OF k POINTS IN GENERAL POSITION REQUIRES (k-1)/2 PARAMETERS Rutgers Center for Systems and Control (SYCON) Report 96-01 Eduardo D. Sontag Department of Mathematics, Rutgers University The generalization capabilities of neural networks are often quantified in terms of the maximal number of possible binary classifications that could be obtained, by means of weight assignments, on any given set of input patterns. The Vapnik-Chervonenkis (VC) dimension is the size of the largest set of inputs that can be shattered (i.e, arbitrary binary labeling is possible). Recent results show that the VC dimension grows at least as fast as the square n**2 of the number of adjustable weights n in the net, and this number might grow as fast as n**4. These results are quite pessimistic, since they imply that the number of samples required for reliable generalization, in the sense of PAC learning, is very high. On the other hand, it is conceivable that those sets of input patterns which can be shattered are all in some sense ``special'' and that if we ask instead, as done in the classical literature in pattern recognition, for the shattering of all sets in ``general position,'' then an upper bound of O(n) might hold. This paper shows a linear upper bound for arbitrary sigmoidal (as well as threshold) neural nets, proving that in that sense the classical results can be recovered in a strong sense (up to a factor of two). Specifically, for classes of concepts defined by certain classes of analytic functions depending on n parameters, it is shown that there are nonempty open sets of samples of length 2n+2 which cannot be shattered. ============================================================================ The paper is available starting from Eduardo Sontag's WWW HomePage at URL: http://www.math.rutgers.edu/~sontag/ (follow links to FTP archive, file generic-vc.ps.gz) or directly via anonymous FTP: ftp math.rutgers.edu login: anonymous cd pub/sontag bin get generic-vc.ps.gz Once file is retrieved, use gunzip to uncompress and then print as postscript. ============================================================================ Comments welcome. From andre at icmsc.sc.usp.br Wed Feb 28 14:52:21 1996 From: andre at icmsc.sc.usp.br ( Andre Carlos P. de Leon F. de Carvalho ) Date: Wed, 28 Feb 96 14:52:21 EST Subject: II Workshop on Cybernetic Vision Message-ID: <9602281752.AA01880@xavante> ====================================================== First Call for Contributions II Workshop on Cybernetic Vision Instituto de Fisica e Quimica de Sao Carlos Universidade de Sao Paulo Sao Carlos, SP, Brazil 9-11 December 1996 ====================================================== As stated in his classical book *Cybernetics*, Norbert Wiener believed that the most interesting and exciting possibilities for original research were to be found in the interface between the major scientific areas. In a tribute to Wiener's visionary approach, the term CYBERNETIC VISION has been proposed to express those research activities lying in the natural/artificial vision interface. It is believed that not only more powerful and versatile artificial visual systems can be obtained through the incorporation of biological insights, but also that our understanding of the natural visual systems can benefit from advances in artificial vision research, thus sustaining a positive feedback. The I WORKSHOP ON CYBERNETIC VISION took place at the Brazilian town of Sao Carlos, SP, in 1994 and attracted the attention of many researchers from the most diverse areas. The second issue of this meeting is aimed at providing another opportunity for scientific interchange as well as the beginning of new collaborations between the related communities. Quality papers related to any of the areas below are welcomed. Prospective authors should send (through FAX, conventional mail, or e-mail) an abstract of approximately 500 words no later than 15th April 1996 to: Prof Dr Luciano da Fontoura Costa Cybernetic Vision Research Group IFSC-USP Av Dr Carlos Botelho, 1465 Caixa Postal 369 Sao Carlos, SP 13560-970 Brazil FAX: +55 162 71 3616 Electronic-mail submission of the abstracts, to be sent to the address below, is strongly encouraged. Luciano at olive.ifqsc.sc.usp.br Upon acceptance of the proposed abstracts, the authors will be asked to prepare the full manuscript (full paper or communication), for further assessment. Accepted contributions will be included in the Workshop Proceedings. The abstracts of the accepted papers will be eventually incorporated into a WWW page. Although contributions in the Portuguese language are welcomed, preference will be given to manuscript in the English language. Subjected to the author's consent, accepted papers in the English language may also be considered for publication in international journals in some of the areas covered. The organizing committee will do its best towards providing a limited number of grants. Areas covered include, but are by no means limited to: - Active Vision - Anatomy and Histology - Eletrophysiology - Ethology - Fuzzy Models - Image Analysis and Computer Vision - Medical Imaging - Modeling and Simulation of Biological Visual Systems - Neural Networks (natural and artificial) - Neurogenesys - Neuromorphometry - Neuroscience - Optical Computing - Psychophysics - Robotics - Scientific Visualization - Vertebrate and Invertebrate Vision - Vision and Consciousness ====================================================== Important Dates: * A 500-word abstract by April 15 * Feedback to author on abstract by May 17 * Three copies of the complete version of the paper by July 26 * Notification of accepted papers by September 6 * WORKSHOP December 9-11 ====================================================== Organizing Committee: Luciano da F. Costa, CVRG, IFSC-USP (Coordinator) Sergio Fukusima, USP-Rib. Preto Roland Koberle, CVRG, IFSC-USP Rafael Linden, UFRJ Valentin O. Roda, CVRG, IFSC-USP Jan F. W. Slaets, IFSC-USP ====================================================== Program Committee (preliminary): Arnaldo Albuquerque, UFMG Junior Barrera, IME-USP Paulo E. Cruvinel, EMBRAPA-CNPDIA Antonio Francisco, INPE Annie F. Frere, EESC-USP Sergio Fukusima, USP-Rib. Preto Andre P. de Leon, ICMSC-USP Rafael Linden, UFRJ Roberto de A. Lotufo, DCA-UNICAMP Ricardo Machado, IBM Joaquim P. Brasil Neto, UnB Nelson D. D. Mascarenhas, UFSCar Valdir F. Pessoa, UnB Anna H. R. C. Rillo, PCS-EPUSP ======================================================= From jfeldman at ICSI.Berkeley.EDU Wed Feb 28 13:00:39 1996 From: jfeldman at ICSI.Berkeley.EDU (Jerry Feldman) Date: Wed, 28 Feb 1996 10:00:39 -0800 Subject: shift invariance Message-ID: <9602281000.ZM15421@ICSI.Berkeley.edu> There seem to be three separate threads arising from my cryptic post and it might be useful to separate them. 1) The capabilities of spatial feedforward nets and backprop(ffbp) Everyone seems to now agree that conventional feedforward nets and backprop (ffbp) will not learn the simple 0*1010* languages of my posting. Of course any formal technique has limitations; the interesting point is that shift invariance is a basic property of apparent biological significance. Geoff Hinton's series of messages asserts that the world and experience are (could be?) structured such that ffbp will learn shift invariance in practice because patterns overlap and are dense enough in space. My inference is that Geoff would like to extend this claim (the world makes ffbp work) to everything of biological importance. Results along these lines would be remarkable indeed. 2) Understanding how the visual system achieves shift invariance. This thread has been non-argumentative. The problem of invariances and constancies in the visual system remains central in visual science. I can't think of any useful message-sized summary, but this is an area where connectionist models should play a crucial role in expressing and testing theories. But, as several people have pointed out, we can't expect much from tabula rasa learning. 3) Shift invariance in time and recurrent networks. I threw in some (even more cryptic) comments on this anticipating that some readers would morph the original task into this form. The 0*1010* problem is an easy one for FSA induction and many simple techniques might work for this. But consider a task that is only slightly more general, and much more natural. Suppose the task is to learn any FSL from the class b*pb* where b and p are fixed for each case and might overlap. Any learning technique that just tried to predict (the probability of) successors will fail because there are three distinct regimes and the learning algorithm needs to learn this. I don't have a way to characterize all recurrent net learning algorithms to show that they can't do this and it will be interesting to see if one can. There are a variety on non-connectionist FSA induction methods that can effectively learn such languages, but they all depend on some overall measure of simplicity of the machine and its fit to the data - and are thus non-local. The remark about not extending to two dimensions referred to the fact that we have no formal grammar for two dimensional patterns (although several proposals for one) and, a fortiori, no algorithm for learning same. One can, as Jamie Henderson suggests, try to linearize two-dimensional problems. But no one has done this successfully for shift (rotation, scale, etc.) invariance and it doesn't seem to me a promising approach to these issues. Jerry F. From pjs at aig.jpl.nasa.gov Wed Feb 28 14:18:41 1996 From: pjs at aig.jpl.nasa.gov (Padhraic J. Smyth) Date: Wed, 28 Feb 1996 11:18:41 -0800 (PST) Subject: TR on HMMs and graphical models Message-ID: <199602281918.LAA03228@amorgos.jpl.nasa.gov> The following technical report is available online at: ftp://aig.jpl.nasa.gov/pub/smyth/papers/TR-96-03.ps.Z PROBABILISTIC INDEPENDENCE NETWORKS FOR HIDDEN MARKOV PROBABILITY MODELS Padhraic Smyth [a], David Heckerman [b], and Michael Jordan [c] [a] Jet Propulsion Laboratory and Department of Information and Computer Science, UCI [b] Microsoft Research [c] Department of Brain and Cognitive Sciences, MIT Abstract Graphical techniques for modeling the dependencies of random variables have been explored in a variety of different areas including statistics, statistical physics, artificial intelligence, speech recognition, image processing, and genetics. Formalisms for manipulating these models have been developed relatively independently in these research communities. In this paper we explore hidden Markov models (HMMs) and related structures within the general framework of probabilistic independence networks (PINs). The paper contains a self-contained review of the basic principles of PINs. It is shown that the well-known forward-backward (F-B) and Viterbi algorithms for HMMs are special cases of more general inference algorithms for arbitrary PINs. Furthermore, the existence of inference and estimation algorithms for more general graphical models provides a set of analysis tools for HMM practitioners who wish to explore a richer class of HMM structures. Examples of relatively complex models to handle sensor fusion and coarticulation in speech recognition are introduced and treated within the graphical model framework to illustrate the advantages of the general approach. This TR is available as Microsoft Research Technical Report TR-96-03, Microsoft Research, Redmond, WA. and as AI Lab Memo AIM-1565, Massachusetts Institute of Technology, Cambridge, MA. From clee at it.wustl.edu Wed Feb 28 16:33:05 1996 From: clee at it.wustl.edu (Christopher Lee) Date: Wed, 28 Feb 1996 15:33:05 -0600 Subject: paper available-Nonlinear Hebbian learning Message-ID: <199602282133.PAA01001@it> Announcing the availability of a paper that may be of relevance to those who have been following the recent discussion of shift-invariance. Key words: Hebbian learning, disparity, nonlinear systems, random-dot stereograms. --------------------------------------------------------------------- A nonlinear Hebbian network that learns to detect disparity in random-dot stereograms. C.W. Lee and B.A. Olshausen An intrinsic limitation of linear, Hebbian networks is that they are capable of learning only from the linear pairwise correlations within an input stream. In order to explore what higher forms of structure could be learned with a nonlinear Hebbian network, we have constructed a model network containing a simple form of nonlinearity and we have applied the network to the problem of learning to detect the disparities present in random-dot stereograms. The network consists of three layers, with nonlinear, sigmoidal activation functions in the second layer units. The nonlinearities allow the second layer to transform the pixel-based representation in the input into a new representation based on coupled pairs of left-right inputs. The third layer of the network then clusters patterns occurring on the second layer outputs according to their disparity via a standard competitive learning rule. Analysis of the network dynamics shows that the second-layer units' nonlinearities interact with the Hebbian learning rule to expand the region over which pairs of left-right inputs are stable. The learning rule is neurobiologically inspired and plausible, and the model may shed light on how the nervous system learns to use coincidence detection in general. (To appear in Neural Computation 8:3) This paper is available via World Wide Web at: http://v1.wustl.edu/chris/chris.html Hard copies are available upon request from clee at v1.wustl.edu, or write to: Chris Lee Campus Box 8108 Washington University 660 S. Euclid Ave St. Louis, MO 63110. From giles at research.nj.nec.com Thu Feb 29 10:03:15 1996 From: giles at research.nj.nec.com (Lee Giles) Date: Thu, 29 Feb 96 10:03:15 EST Subject: shift invariance Message-ID: <9602291503.AA29190@alta> We and others [1, 2, 3, 4] showed that invariances, actually affine transformations, could directly be encoded into feedforward higher-order (sometimes called polynomial, sigma-pi, gated, ...) neural nets such that these networks are invariant to shift, scale, and rotation of individual patterns. As mentioned previously, similar invariant encodings can be had for associative memories in autonomous recurrent networks. Interestingly, this idea of encoding geometric invariances into neural networks is an old one [5]. [1] C.L. Giles, T. Maxwell, ``Learning, Invariance, and Generalization in High-Order Neural Networks'', Applied Optics, 26(23), p 4972, 1987. Reprinted in ``Artificial Neural Networks: Concepts and Theory,'' eds. P. Mehra and B. W. Wah, IEEE Computer Society Press, Los Alamitos, CA. 1992. [2] C.L. Giles, R.D. Griffin, T. Maxwell,``Encoding Geometric Invariances in Higher-Order Neural Networks'', Neural Information Processing Systems, Eds. D.Z. Anderson, Am. Inst. of Physics, N.Y., N.Y., p 301-309, 1988. [3] S.J. Perantonis, P.J.G. Lisboa, ``Translation, Rotation, and Scale Invariant Pattern Recognition by Higher-Order Neural Networks and Moment Classifiers'', IEEE Transactions on Neural Networks, 3(2), p 241, 1992. [4] L. Spirkovska, M.B. Reid,``Higher-Order Neural Networks Applied to 2D and 3D Object Recognition'', Machine Learning, 15(2), p. 169-200, 1994. [5] W. Pitts, W.S. McCulloch, ``How We Know Universals: The Perception of Auditory and Visual Forms'', Bulletin of Mathematical Biophysics, vol 9, p. 127, 1947. A bibtex entry for the above references can be found in: ftp://external.nj.nec.com/pub/giles/papers/high-order.bib -- C. Lee Giles / Computer Sciences / NEC Research Institute / 4 Independence Way / Princeton, NJ 08540, USA / 609-951-2642 / Fax 2482 www.neci.nj.nec.com/homepages/giles.html ==  From tp-temp at ai.mit.edu Thu Feb 29 20:37:36 1996 From: tp-temp at ai.mit.edu (Tomaso Poggio) Date: Thu, 29 Feb 96 20:37:36 EST Subject: Shift Invariance In-Reply-To: Lee Giles's message of Mon, 26 Feb 96 13:01:25 EST <9602261801.AA26373@alta> Message-ID: <9603010137.AA00462@corpus-callosum.ai.mit.edu> A footnote to Lee Giles msg. Polynomial networks, analog perceptrons, Kolmogorov theorem and invariances were described in an old paper with Werner Reichardt. ``On the representation of multi-input systems: computational properties of polynomial algorithms,'' (T. Poggio and W. Reichardt). {\it Biol. Cyber.}, {\bf 37}, 167-186, 1980.