From Connectionists-Request at CS.CMU.EDU Mon Jul 1 00:05:25 1991 From: Connectionists-Request at CS.CMU.EDU (Connectionists-Request@CS.CMU.EDU) Date: Mon, 01 Jul 91 00:05:25 EDT Subject: Bi-monthly Reminder Message-ID: <14317.678341125@B.GP.CS.CMU.EDU> *** DO NOT FORWARD TO ANY OTHER LISTS *** This is an automatically posted bi-monthly reminder about how the CONNECTIONISTS list works and how to access various online resources. CONNECTIONISTS is not an edited forum like the Neuron Digest, or a free-for-all newsgroup like comp.ai.neural-nets. It's somewhere in between, relying on the self-restraint of its subscribers. Membership in CONNECTIONISTS is restricted to persons actively involved in neural net research. The following posting guidelines are designed to reduce the amount of irrelevant messages sent to the list. Before you post, please remember that this list is distributed to over a thousand busy people who don't want their time wasted on trivia. Also, many subscribers pay cash for each kbyte; they shouldn't be forced to pay for junk mail. Happy hacking. -- Dave Touretzky & Scott Crowder --------------------------------------------------------------------- What to post to CONNECTIONISTS ------------------------------ - The list is primarily intended to support the discussion of technical issues relating to neural computation. - We encourage people to post the abstracts of their latest papers and tech reports. - Conferences and workshops may be announced on this list AT MOST twice: once to send out a call for papers, and once to remind non-authors about the registration deadline. A flood of repetitive announcements about the same conference is not welcome here. - Requests for ADDITIONAL references. This has been a particularly sensitive subject lately. Please try to (a) demonstrate that you have already pursued the quick, obvious routes to finding the information you desire, and (b) give people something back in return for bothering them. The easiest way to do both these things is to FIRST do the library work to find the basic references, then POST these as part of your query. Here's an example: WRONG WAY: "Can someone please mail me all references to cascade correlation?" RIGHT WAY: "I'm looking for references to work on cascade correlation. I've already read Fahlman's paper in NIPS 2, his NIPS 3 abstract, and found the code in the nn-bench archive. Is anyone aware of additional work with this algorithm? I'll summarize and post results to the list." - Announcements of job openings related to neural computation. - Short reviews of new text books related to neural computation. To send mail to everyone on the list, address it to Connectionists at CS.CMU.EDU ------------------------------------------------------------------- What NOT to post to CONNECTIONISTS: ----------------------------------- - Requests for addition to the list, change of address and other administrative matters should be sent to: "Connectionists-Request at cs.cmu.edu" (note the exact spelling: many "connectionists", one "request"). If you mention our mailing list to someone who may apply to be added to it, please make sure they use the above and NOT "Connectionists at cs.cmu.edu". - Requests for e-mail addresses of people who are believed to subscribe to CONNECTIONISTS should be sent to postmaster at appropriate-site. If the site address is unknown, send your request to Connectionists-Request at cs.cmu.edu and we'll do our best to help. A phone call to the appropriate institution may sometimes be simpler and faster. - Note that in many mail programs a reply to a message is automatically "CC"-ed to all the addresses on the "To" and "CC" lines of the original message. If the mailer you use has this property, please make sure your personal response (request for a Tech Report etc.) is NOT broadcast over the net. - Do NOT tell a friend about Connectionists at cs.cmu.edu. Tell him or her only about Connectionists-Request at cs.cmu.edu. This will save your friend from public embarrassment if she/he tries to subscribe. - Limericks should not be posted here. ------------------------------------------------------------------------------- The CONNECTIONISTS Archive: --------------------------- All e-mail messages sent to "Connectionists at cs.cmu.edu" starting 27-Feb-88 are now available for public perusal. A separate file exists for each month. The files' names are: arch.yymm where yymm stand for the obvious thing. Thus the earliest available data are in the file: arch.8802 Files ending with .Z are compressed using the standard unix compress program. To browse through these files (as well as through other files, see below) you must FTP them to your local machine. ------------------------------------------------------------------------------- How to FTP Files from the CONNECTIONISTS Archive ------------------------------------------------ 1. Open an FTP connection to host B.GP.CS.CMU.EDU (Internet address 128.2.242.8). 2. Login as user anonymous with password your username. 3. 'cd' directly to one of the following directories: /usr/connect/connectionists/archives /usr/connect/connectionists/bibliographies 4. The archives and bibliographies directories are the ONLY ones you can access. You can't even find out whether any other directories exist. If you are using the 'cd' command you must cd DIRECTLY into one of these two directories. Access will be denied to any others, including their parent directory. 5. The archives subdirectory contains back issues of the mailing list. Some bibliographies are in the bibliographies subdirectory. Problems? - contact us at "Connectionists-Request at cs.cmu.edu". ------------------------------------------------------------------------------- How to FTP Files from the Neuroprose Archive -------------------------------------------- Anonymous FTP on cheops.cis.ohio-state.edu (128.146.8.62) pub/neuroprose directory This directory contains technical reports as a public service to the connectionist and neural network scientific community. Researchers may place electronic versions of their preprints or articles in this directory, announce availability, and other interested researchers can rapidly retrieve and print the postscripts. This saves copying, postage and handling, by having the interested reader supply the paper. To place a file, put it in the Inbox subdirectory, and send mail to pollack at cis.ohio-state.edu. Within a couple of days, I will move and protect it, and suggest a different name if necessary. Current naming convention is author.title.filetype[.Z] where title is enough to discriminate among the files of the same author. The filetype is usually "ps" for postscript, our desired universal printing format, but may be tex, which requires more local software than a spooler. Very large files (e.g. over 200k) must be squashed (with either a sigmoid function :) or the standard unix "compress" utility, which results in the .Z affix. To place or retrieve .Z files, make sure to issue the FTP command "BINARY" before transfering files. After retrieval, call the standard unix "uncompress" utility, which removes the .Z affix. An example of placing a file is attached as an appendix, and a shell script called Getps in the directory can perform the necessary retrival operations. For further questions contact: Jordan Pollack Email: pollack at cis.ohio-state.edu Here is an example of naming and placing a file: gvax> cp i-was-right.txt.ps rosenblatt.reborn.ps gvax> compress rosenblatt.reborn.ps gvax> ftp cheops.cis.ohio-state.edu Connected to cheops.cis.ohio-state.edu. 220 cheops.cis.ohio-state.edu FTP server ready. Name: anonymous 331 Guest login ok, send ident as password. Password:neuron 230 Guest login ok, access restrictions apply. ftp> binary 200 Type set to I. ftp> cd pub/neuroprose/Inbox 250 CWD command successful. ftp> put rosenblatt.reborn.ps.Z 200 PORT command successful. 150 Opening BINARY mode data connection for rosenblatt.reborn.ps.Z 226 Transfer complete. 100000 bytes sent in 3.14159 seconds ftp> quit 221 Goodbye. gvax> mail pollack at cis.ohio-state.edu Subject: file in Inbox. Jordan, I just placed the file rosenblatt.reborn.ps.Z in the Inbox. The INDEX sentence is "Boastful statements by the deceased leader of the neurocomputing field." Please let me know when it is ready to announce to Connectionists at cmu. BTW, I enjoyed reading your review of the new edition of Perceptrons! Frank ------------------------------------------------------------------------ How to FTP Files from the NN-Bench Collection --------------------------------------------- 1. Create an FTP connection from wherever you are to machine "pt.cs.cmu.edu" (128.2.254.155). 2. Log in as user "anonymous" with password your username. 3. Change remote directory to "/afs/cs/project/connect/bench". Any subdirectories of this one should also be accessible. Parent directories should not be. 4. At this point FTP should be able to get a listing of files in this directory and fetch the ones you want. Problems? - contact us at "nn-bench-request at cs.cmu.edu". From schmidhu at kiss.informatik.tu-muenchen.de Mon Jul 1 08:04:48 1991 From: schmidhu at kiss.informatik.tu-muenchen.de (Juergen Schmidhuber) Date: Mon, 1 Jul 91 14:04:48 +0200 Subject: change of address Message-ID: <9107011204.AA03458@kiss.informatik.tu-muenchen.de> Old address until July 4, 1991: Juergen Schmidhuber Institut fuer Informatik, Technische Universitaet Muenchen Arcisstr. 21 8000 Muenchen 2 GERMANY fax: Germany (49) Munich (89) 2105 8207 email: schmidhu at informatik.tu-muenchen.de New address after July 4, 1991: Juergen Schmidhuber Department of Computer Science University of Colorado Campus Box 430 Boulder, CO 80309, USA fax: (303) 492 2844 email: yirgan at neuron.colorado.edu From haussler at saturn.ucsc.edu Mon Jul 1 18:57:37 1991 From: haussler at saturn.ucsc.edu (David Haussler) Date: Mon, 1 Jul 91 15:57:37 -0700 Subject: don't forget to register for COLT '91 Message-ID: <9107012257.AA12399@saturn.ucsc.edu> Don't forget to register for the Workshop on Computational Learning Theory Monday, August 5 through Wednesday, August 7, 1991 University of California, Santa Cruz, California Conference information, program and room registration forms can be obtained by anonymous FTP. Connect to midgard.ucsc.edu and look in the directory pub/colt. Alternatively, send E-mail to "colt at cis.ucsc.edu" for instructions on obtaining the forms by electronic mail. Early registration discounts are ending NOW. If you have questions regarding registration or accommodations, contact: Jean McKnight, COLT '91, Dept. of Computer Science, UCSC, Santa Cruz, CA 95064. Her emergency phone number is (408) 459-2303, but she prefers E-mail to jean at cs.ucsc.edu or facsimile at (408) 429-0146. Proceedings from the workshop will be published by Morgan Kaufmann. -David From dwunsch at atc.boeing.com Mon Jul 1 21:15:00 1991 From: dwunsch at atc.boeing.com (Don Wunsch) Date: Mon, 1 Jul 91 18:15:00 PDT Subject: Various IJCNN-91-Seattle info. Message-ID: <9107020115.AA22035@atc.boeing.com> IJCNN-91-Seattle is almost upon us! Come to Seattle July 8-12 for the largest neural network, hybrid, and fuzzy systems conference in the world. Researchers, scientists, engineers, consultants, applications specialists and students from a variety of disciplines will present invited and contributed papers, tutorials, panel discussions, exhibits and demos. Exhibitors will also present the latest in neural networks, including neurocomputers, VLSI neural networks, software systems and applications. Conference registration is $395 for IEEE or INNS members, $495 for non-members, and $95 for students. Tutorials are $295 for everyone except students, who pay $85. One-day registration is $125 for members or $175 for non-members. The tutorial fee includes all the tutorials you choose to attend, not just one. Also, students are full conference participants, receiving a proceedings and admission to all events, including social events. To register by VISA or MasterCard, call (206) 543-2310, or FAX a registration form (containing your name, title, affiliation, full address, country, full daytime phone number, FAX number, e-mail address, and registration for the conference or tutorials, with total fees and credit card number with name as it appears on card) to (206) 685-9359. Finally, for those of you who will be attending the conference, you have an opportunity to attend a tour of the world's largest manufacturing facility. Read on for further details and reservation information: Boeing will be offering a limited number of free tours of its manufacturing plant to attendees of the International Joint Conference on Neural Networks, on a first-come, first-served basis. Families may also be accomodated, but only if space is available. See the 747 and 767 being built, in the largest volume building in the world. We will also be providing transportation leaving he conference site one hour before the tour begins. The tour lasts approximately one hour. Tour times are Monday through Friday at 10:30 AM and 3:00 PM. Therefore, be prepared to leave the conference site at 9:30 AM or 2:00 PM for your chosen tour. Please indicate your preferred tour times and send a fax of this form with your name, number of persons on the tour, and hotel (if known). Preferred time:_________________ Alternate choice:___________________ Second alternate:_____________________________________________________ Name (please print): _________________________________________________ Number of persons: __________ Hotel:___________________________ Please do not reply to this address. Instead, please send a FAX to Jean Norris at (206) 865-2957. If you have any questions, call her at (206) 865-5616. Alternatively, you may e-mail the form or any questions to David Newman at: dnewman at caissa.boeing.com. If you will not be staying at a hotel, please put down a number where you can be reached. There will be a tour coordination booth available at the time of registration, where further information is available, and tickets may be picked up. The best time to schedule your tour will probably be Monday, before things start heating up. This tour is actually a very hot tourist attraction around here, because of the universal fascination with flight. So it will be well worth your while--don't miss it! We look forward to seeing you in Seattle! Don Wunsch Local Arrangements Chair IJCNN-91-Seattle From magnuson at mcc.com Tue Jul 2 10:26:38 1991 From: magnuson at mcc.com (Tim Magnuson) Date: Tue, 2 Jul 91 09:26:38 CDT Subject: Job Announcement: MCC Message-ID: <9107021426.AA09964@brush.aca.mcc.com> MCC, a leading U.S. cooperative research organization in Austin, Texas, currently seeks two individuals: a project manager and a research scientist for the MCC Neural Network Project. MCC's neural network research takes both a theoretical and a practical form. The theoretical work includes the development and analysis of neural network algorithms and architectures. The practical work takes the form of large-scale, real-world application projects. The specific focus of the MCC Neural Network Project is handwritten character recognition. Current research projects are focused on concurrent segmentation and recognition of handwritten symbols, including numeric digits, upper and lower case alpha characters, punctuation characters, and cursive handwriting. The requirements for the research scientist position include a Ph.D. in a discipline related to neural networks, a proven track record for leading edge research in neural networks, and experience in handwritten character recognition using neural networks. The requirements for the research manager are a M.S. or Ph.D. in a discipline related to neural networks, strong interpersonal and organizational skills, and a working knowledge of neural networks. Management experience in handwritten character recognition projects is a definite plus. MCC is well positioned as a conduit between university research and industrial application due to strong contacts on both sides. In addition to an excellent compensation/benefits program, MCC offers an outstanding corporate environment and the opportunity to conduct ground breaking research. MCC will be interviewing candidates at IJCNN in Seattle on July 11th and 12th. Interviews will be by appointment only. Interested parties should mail their credentials, including salary history and requirements to: Roger Malek Manager, Human Resources MCC 3500 West Balcones Center Drive Austin, Texas 78759 or FAX them to Roger Malek at 512-338-3888 or E-mail to Malek at mcc.com From Connectionists-Request at CS.CMU.EDU Tue Jul 2 11:16:14 1991 From: Connectionists-Request at CS.CMU.EDU (Connectionists-Request@CS.CMU.EDU) Date: Tue, 02 Jul 91 11:16:14 EDT Subject: IJCNN request Message-ID: <4175.678467774@B.GP.CS.CMU.EDU> > IJCNN-91-Seattle is almost upon us! Come to Seattle July 8-12 > for the largest neural network, hybrid, and fuzzy systems > conference in the world. Researchers, scientists, engineers, > consultants, applications specialists and students from a variety > of disciplines will present invited and contributed papers, tutorials, > panel discussions, exhibits and demos. Exhibitors will also present > the latest in neural networks, including neurocomputers, VLSI neural > networks, software systems and applications. We typically get a flood of new member requests after each IJCNN. Many of them get broadcast to all 900+ members of CONNECTIONISTS. Since I am the only one that can add anybody to the list, we can save *everybody* some time by remembering the following. Do NOT tell your new friend about Connectionists at cs.cmu.edu. Tell him or her only about Connectionists-Request at cs.cmu.edu. This will save your friend from public embarrassment if she/he tries to subscribe. It will save *me* some time if you remember to tell your new friend to include their full name and a 1 sentence description of their current work in their request message. I send the standard restricted membership note to *everyone* whose request does not contain these two items. Sometimes people get their feelings hurt that I don't recognize them as the author of the most-talked-about new paper. The truth is that I don't even read the name if there is no work description. -Scott Crowder Connectionists-Request at cs.cmu.edu (ARPAnet) From dwunsch at atc.boeing.com Tue Jul 2 13:37:08 1991 From: dwunsch at atc.boeing.com (Don Wunsch) Date: Tue, 2 Jul 91 10:37:08 PDT Subject: Boeing plant tours for IJCNN-91-Seattle Message-ID: <9107021737.AA27543@atc.boeing.com> In a recent note, I wrote: > IJCNN-91-Seattle is almost upon us! Come > ... > Boeing will be offering a limited number of free tours > ... > first-served basis. Families may also be accomodated, but > only if space is available. See the 747 and 767 being built, > ... Forgot to mention--the minimum age for the Boeing plant tour is twelve years old. Hope to see you there! Don From fritzke at immd2.informatik.uni-erlangen.de Wed Jul 3 09:22:20 1991 From: fritzke at immd2.informatik.uni-erlangen.de (B. Fritzke) Date: Wed, 3 Jul 91 9:22:20 MET DST Subject: TSP paper available Message-ID: <9107030723.AA06096@faui28.informatik.uni-erlangen.de> Hi connectionists, I just placed a paper in the Neuroprose Archive, which has been submitted to IJCNN-91 Singapore. The filename is: fritzke.linear_tsp.ps.Z And here's the abstract: FLEXMAP -- A Neural Network For The Traveling Salesman Problem With Linear Time And Space Complexity Bernd FRITZKE and Peter WILKE We present a self-organizing ''neural'' network for the traveling salesman problem. It is partly based on the model of Kohonen. Our approach differs from former work in this direction as no ring structure with a fixed number of elements is used. Instead a small initial structure is enlarged during a distribution pro- cess. This allows us to replace the central search step, which normally needs time O(n), by a local procedure that needs time O(1). Since the total number of search steps we have to perform is O(n) the runtime of our model scales linear with problem size. This is better than every known neural or conventional algorithm. The path lengths of the generated solutions are less than 9 per- cent longer than the optimum solutions of solved problems from the literature. The described network is based on: > Fritzke, B., "Let it grow - self-organizing feature maps with > problem dependent cell structure," Proc. of ICANN-91, Helsinki, > 1991, pp. 403-408. (see the previously placed file fritzke.cell_structures.ps.Z) Furthermore related work will be presented next week in Seattle: > Fritzke, B., "Unsupervised clustering with growing cell struc- > tures," to appear in: Proc. of IJCNN-91, Seattle, 1991. (see the previously placed file fritzke.clustering.ps.Z and the Poster No. W19 in Seattle) See you in Seattle, Bernd Bernd Fritzke ----------> e-mail: fritzke at immd2.informatik.uni-erlangen.de University of Erlangen, CS IMMD II, Martensstr. 3, 8520 Erlangen (Germany) From R09614 at BBRBFU01.BITNET Wed Jul 3 11:19:37 1991 From: R09614 at BBRBFU01.BITNET (R09614) Date: Wed, 3 Jul 91 17:19:37 +0200 Subject: European Neural Network Society Message-ID: PRESS RELEASE During ICANN91, the International Conference on Artificial Neural Networks, held in Helsinki, June 24-28, 1991, ENNS, the European Neural Network Society has been created. Its main objectives are to sponsor furture ICANN conferences and organize summer schools in the area, and to develop a quarterly newsletter and electronic bulletin board that will keep members informed of developments in the field of artificial and biological neural networks. It is expected that Special Interest Groups will be formed in the future under the umbrella of the society. The new society welcomes membership worldwide from those interested in Neural Network research. One of the benefits will be de reduction of fees for all future activities of the society. The current officiers of the ENNS are: President Tuevo Kohonen Helsinki University of Technology Espoo, Finland Vice Presidents Igor Aleksander Imperial College London London, U.K. Rolf Eckmiller Heinrich Heine University Dusseldorf, F.R.G. John Taylor King's College London London, U.K. Secretary Agnessa Babloyantz Universite Libre de Bruxelles Brussels, Belgium Treasurer Rodney Cotterill Technical University of Denmark Lyngby, Denmark For further information, please contact A. Babloyantz at: Universite Libre de Bruxelles Phone: 32 - 2 - 650 55 40 CP 231 - Campus Plaine, Fax: 32 - 2 - 650 57 67 Boulevard du Triomphe, E-mail: adestex @ bbrbfu60.bitnet B-1050 Bruxelles, BELGIUM From WEYMAERE at lem.rug.ac.be Wed Jul 3 16:09:00 1991 From: WEYMAERE at lem.rug.ac.be (WEYMAERE@lem.rug.ac.be) Date: Wed, 3 Jul 91 16:09 N Subject: New Paper: A Fast and Robust Learning Algorithm for Feedforward NN Message-ID: <01G7QU7KJYXS0000SV@BGERUG51.BITNET> The following paper has appeared in "Neural Networks", Vol. 4, No 3 (1991), pp 361-369: -------------------------------------------------------------------------------- A Fast and Robust Learning Algorithm for Feedforward Neural Networks Nico WEYMAERE and Jean-Pierre MARTENS Laboratorium voor Elektronika en Meettechniek Rijksuniversiteit Gent Gent, Belgium ABSTRACT The back-propagation algorithm caused a tremendous break-through in the application of multilayer perceptrons. However, it has some important drawbacks: long training times and sensitivity to the presence of local minima. Another problem is the network topology: the exact number of units in a particular hidden layer, as well as the number of hidden layers need to be known in advance. A lot of time is often spent in finding the optimal topology. In this paper, we consider multilayer networks with one hidden layer of Gaussian units and an output layer of conventional units. We show that for this kind of networks, it is possible to perform a fast dimensionality analysis, by analyzing only a small fraction of the input patterns. Moreover, as a result of this approach, it is possible to initialize the weights of the network before starting the back-propagation training. Several classification problems are taken as examples. -------------------------------------------------------------------------------- Unfortunately, there is not an electronic version of this paper. Reprint requests should be sent to : Weymaere Nico Laboratorium voor Elektronika en Meettechniek St. Pietersnieuwstraat 41 - B9000 Gent From R09614 at BBRBFU01.BITNET Fri Jul 5 12:51:12 1991 From: R09614 at BBRBFU01.BITNET (R09614) Date: Fri, 5 Jul 91 18:51:12 +0200 Subject: EUROPEAN NEURAL NETWORK SOCIETY Message-ID: <4FA5C21E6080385F@BITNET.CC.CMU.EDU> EUROPEAN NEURAL NETWORK SOCIETY Due to connection problems, some E-mails did not arrive to the address given in the first announcement. Please send requests or comments for ENNS at the following address: R09614 at BBRBFU01.bitnet If you had already sent a message, please re-send a copy to this address. Some files have been lost. Thank you From wray at ptolemy.arc.nasa.gov Fri Jul 5 14:19:46 1991 From: wray at ptolemy.arc.nasa.gov (Wray Buntine) Date: Fri, 5 Jul 91 11:19:46 PDT Subject: two new papers on back-prop available from neuroprose Message-ID: <9107051819.AA04256@ptolemy.arc.nasa.gov> The following two reports are currently under journal review and have been made available on the "/pub/neuroprose" archive. Those unable to access this should send requests to the address below. Both papers are intended as a guide for the "theoretically-aware practitioner/algorithm-designer intent on building a better algorithm". Wray Buntine NASA Ames Research Center phone: (415) 604 3389 Mail Stop 244-17 Moffett Field, CA, 94035 email: wray at ptolemy.arc.nasa.gov ---------------- Bayesian Back-Propagation by Wray L. Buntine and Andreas S. Weigend available as /pub/neuroprose/buntine.bayes1.ps.Z (pages 1-17) /pub/neuroprose/buntine.bayes2.ps.Z (pages 1-34) Connectionist feed-forward networks, trained with back-propagation, can be used both for non-linear regression and for (discrete one-of-$C$) classification, depending on the form of training. This paper works through approximate Bayesian methods to both these problems. Methods are presented for various statistical components of back-propagation: choosing the appropriate cost function and regularizer (interpreted as a prior), eliminating extra weights, estimating the uncertainty of the remaining weights, predicting for new patterns (``out-of-sample''), estimating the uncertainty in the choice of this prediction (``error bars''), estimating the generalization error, comparing different network structures, and adjustments for missing values in the training patterns. These techniques refine and extend some popular heuristic techniques suggested in the literature, and in most cases require at most a small additional factor in computation during back-propagation, or computation once back-propagation has finished. The paper begins with a comparative discussion of Bayesian and related frameworks for the training problem. Contents: 1. Introduction 2. On Bayesian methods 3. Multi-Layer networks 4. Probabilistic neural networks 4.1. Logistic networks 4.2. Cluster networks 4.3. Regression networks 5. Probabilistic analysis 5.1. The network likelihood function 5.2. The sample likelihood 5.3. Prior probability of the weights 5.4. Posterior analysis 6. Analyzing weights 6.1. Cost functions 6.2. Weight evaluation 6.3. Minimum encoding methods 7. Applications to network training 7.1. Weight variance and elimination 7.2. Prediction and generalization error 7.3. Adjustments for missing values 8. Conclusion ----------------------- Calculating Second Derivatives on Feed-Forward Networks by Wray L. Buntine and Andreas S. Weigend available as /pub/neuroprose/buntine.second.ps.Z Recent techniques for training connectionist feed-forward networks require the calculation of second derivatives to calculate error bars for weights and network outputs, and to eliminate weights, etc. This note describes some exact algorithms for calculating second derivatives. They require at the worst case approximately $2K$ back/forward-propagation cycles where $K$ is the number of nodes in the network. For networks with two-hidden layers or less, computation can be much quicker. Three previous approximations, ignoring some components of the second derivative, numerical differentiation, and scoring, are also reviewed and compared. From STCS8013 at IRUCCVAX.UCC.IE Tue Jul 9 07:09:00 1991 From: STCS8013 at IRUCCVAX.UCC.IE (STCS8013@IRUCCVAX.UCC.IE) Date: Tue, 9 Jul 91 11:09 GMT Subject: Please inlcude info in CONNECTIONIST Message-ID: <5B509D275E800064@BITNET.CC.CMU.EDU> Call for Attendance Fourth Irish Conference on Artificial Intelligence and Cognitive Science (AICS'91) 19 - 20 September, 1991 University College, Cork, Ireland For Details and Registration Forms, contact the Conference Chair: Humphrey Sorensen, Computer Science Department, University College, Cork, Ireland. email: aics91 at iruccvax.ucc.ie From CHELLA at IPACRES.BITNET Tue Jul 9 14:56:00 1991 From: CHELLA at IPACRES.BITNET (CHELLA@IPACRES.BITNET) Date: Tue, 9 Jul 91 18:56 GMT Subject: No subject Message-ID: <4A58EAFC681F21021F@IPACRES.BITNET> I should like to subscribe the mailing list. Antonio Chella From 7923570 at TWNCTU01.BITNET Wed Jul 10 14:08:00 1991 From: 7923570 at TWNCTU01.BITNET (7923570) Date: Wed, 10 Jul 91 14:08 U Subject: No subject Message-ID: hi: Could you provide me the E-mail address about the neural net discussing list ? From lacher at lambda.cs.fsu.edu Wed Jul 10 15:12:29 1991 From: lacher at lambda.cs.fsu.edu (Chris Lacher) Date: Wed, 10 Jul 91 15:12:29 -0400 Subject: Distributed v Local Message-ID: <9107101912.AA01866@lambda.cs.fsu.edu> Remark related to the distributed/local representation conversation. We've been studying expert networks lately: discrete-time computational networks with symbolic-level node functionality. The digraph structure comes from a rule base in an expert system, and the synaptic and node functionality mimics the inferential dynamics of the ES. The nodes represent things like AND, NOT, NOR and EVIDENCE ACCUMULATION. (We announced a preprint to connectionists on this topic in January 91.) We have some of these things that actually reason like expert systems and can learn from data. Clearly these nets use "local" representation in the sense that single components of the architecture have identifiable meaning outside the net. (Although, the identifiable meaning is that of a process that makes sense at a cognitive level rather than an object or attribute as in the more common interpretation of local representation.) These symbolic-level processors can be replaced with relatively small networks of sub-symbolic processors that do the common "output:= squash(weighted sum of inputs)" kind of processing. One can imagine either feed-forward or recurrent subnets that accomplish this. The expert networks tend to be sparsely connected, O(n), while the subnets will generally be highly connected, O(n^2). Wildly speculate the existence of a system containing N symbolic processors with O(N) interconnectivity, each processor actually consisting of K sub-symbolic processors with O(K^2) intraconnectivity, and assume there are overall 10^10 sub-symbolic processors and 10^13 connections (about the numbers for the cerebral cortex). Then we have NK = 10^10 and NK^2 = 10^13 which yields N = 10^7 and K = 10^3. That is, an organization of about 10^7 sparsely connected subnetworks each being a highly intraconnected network of around 1000 sub-symbolic processors. Aren't the columnar structures in the CC highly connected subnets of about a thousand neurons? --- Chris Lacher (My coworkers include colleague Susan Hruska, recent PhD Dave Kuncicky, and a number of current graduate students. They should not be held responsible for this note, however. Susan and Dave are in Seattle at IJCNN. I'm here in Tallahassee trying to survive dog days.) From russ at oceanus.mitre.org Mon Jul 15 11:42:06 1991 From: russ at oceanus.mitre.org (Russell Leighton) Date: Mon, 15 Jul 91 11:42:06 EDT Subject: Narendra's stability proof Message-ID: <9107151542.AA11885@oceanus.mitre.org> At IJCNN `91 Narendra spoke of a paper where he has proven stability for a control system using backpropagation neural networks. Does anyone know where this was published? Thanks. Russ INTERNET: russ at dash.mitre.org Russell Leighton MITRE Signal Processing Lab 7525 Colshire Dr. McLean, Va. 22102 USA From russ at oceanus.mitre.org Mon Jul 15 13:39:41 1991 From: russ at oceanus.mitre.org (Russell Leighton) Date: Mon, 15 Jul 91 13:39:41 EDT Subject: Paper announcement Message-ID: <9107151739.AA12618@oceanus.mitre.org> The following paper is available in the neuroprose library (leighton.ar-backprop.ps.Z). The Autoregressive Backpropagation Algorithm {To appear in the Proceedings of the International Joint Conference on Neural Networks, 1991} Russell R. Leighton and Bartley C. Conrath The MITRE Corporation 7525 Colshire Drive, McLean, VA 22102 This paper describes an extension to error backpropagation that allows the nodes in a neural network to encode state information in an autoregressive ``memory.'' This neural model gives such networks the ability to learn to recognize sequences and context-sensitive patterns. Building upon the work of Wieland concerning nodes with a single feedback connection, this paper generalizes the method to $n$ feedback connections and addresses stability issues. The learning algorithm is derived, and a few applications are presented. To get the paper: 1. ftp 128.146.8.62 2. cd pub/neuroprose 3. binary 4. get leighton.ar-backprop.ps.Z 5. quit 6. uncompress leighton.ar-backprop.ps.Z 7. lpr leighton.ar-backprop.ps Russ INTERNET: russ at dash.mitre.org Russell Leighton MITRE Signal Processing Lab 7525 Colshire Dr. McLean, Va. 22102 USA From sontag at control.rutgers.edu Mon Jul 15 14:48:18 1991 From: sontag at control.rutgers.edu (sontag@control.rutgers.edu) Date: Mon, 15 Jul 91 14:48:18 EDT Subject: Narendra's stability proof Message-ID: <9107151848.AA15176@control.rutgers.edu> From: Russell Leighton At IJCNN `91 Narendra spoke of a paper where he has proven stability for a control system using backpropagation neural networks. Does anyone know where this was published? Thanks. Russ At the American Automatic Control Conference, three weeks ago in Boston, there were a few papers dealing with adaptive control using neural nets. Among them: TA1: 8:30-9:00 Intelligent Control Using Neural Networks Narendra, K., Yale University Mukhopadhyay, S., Yale University TP1: 17:30-18:00 Regulation of Nonlinear Dynamical Systems Using Neural Networks Narendra, K., Yale University Levine, A., Yale University FA1: 11:15-11:45 Gradient Methods for Learning in Dynamical System Containing Neural Networks Narendra, K., Yale University Parthasarathy, K., Yale University 12:15-12:45 Stability and Convergence Issues in Neural Network Control Slotine, J., Massachusetts Institute of Technology As far as I recall, all results on stability dealt with RADIAL-BASIS types of networks, assuming FIXED centers, so the estimation problem is a LINEAR one. The paper of Slotine has a nice technique for estimating weights at the lower level, using spectral information on the training data (I guess in the same spirit that others would use clustering). Before the conference, there was a one-day course, organized by Narendra, which covered neural net approaches to control; he had a writeup prepared for that, which might cover the stability results (I don't know, nor do I know how you can get a copy). The email addresses for Slotine and Narendra are as follows: jjs at athena.mit.edu (Jean-Jacques Slotine, Mech Engr, MIT) narendra at bart.eng.yale.edu (Narendra, Engineering, Yale) -eduardo PS: My paper in the same proceedings, WP9: "Feedback Stabilization Using Two-Hidden-Layer Nets", covered the results on why *TWO* hidden layers are needed for control (and some other) problems, rather than one. (A tech report was posted late last year to neuroprose, covering the contents of this paper.) From stolcke at ICSI.Berkeley.EDU Mon Jul 15 15:00:20 1991 From: stolcke at ICSI.Berkeley.EDU (stolcke@ICSI.Berkeley.EDU) Date: Mon, 15 Jul 91 13:00:20 MDT Subject: new cluster available Message-ID: <9107152000.AA05004@icsib30.Berkeley.EDU> Dear Connectionists: After several months of testing I'm releasing a slightly revised version my enhanced cluster utility. A major memory allocation glitch was fixed and support for System 5 curses pads was added. I should note that for viewing the graph output of cluster the original version of the xgraph program is not enough because it cannot handle labeled datapoints. An enhanced version that works well with cluster can be ftped at the same location as cluster (see below). Andreas HOW TO GET CLUSTER cluster is available via anonymous ftp from icsi-ftp.berkeley.edu (128.32.201.55). To get it use FTP as follows: % ftp icsi-ftp.berkeley.edu Connected to icsic.Berkeley.EDU. 220 icsi-ftp (icsic) FTP server (Version 5.60 local) ready. Name (icsic.Berkeley.EDU:stolcke): anonymous Password (icsic.Berkeley.EDU:anonymous): 331 Guest login ok, send ident as password. 230 Guest login Ok, access restrictions apply. ftp> cd pub/ai 250 CWD command successful. ftp> binary 200 Type set to I. ftp> get cluster-2.2.tar.Z 200 PORT command successful. 150 Opening BINARY mode data connection for cluster-2.2.tar.Z (15531 bytes). 226 Transfer complete. 15531 bytes received in 0.08 seconds (1.9e+02 Kbytes/s) ftp> quit 221 Goodbye. HOW TO BUILD CLUSTER Unpack in an empty directory using % zcat cluster-2.2.tar.Z | tar xf - Read the README and especially the man page (cluster.man) for information. Check the Makefile for any compile time flags that might need adjustment. Then compile with % make After making the appropriate adjustments in the Makefile you can % make install From T.Rickards at cs.ucl.ac.uk Tue Jul 16 07:10:05 1991 From: T.Rickards at cs.ucl.ac.uk (T.Rickards@cs.ucl.ac.uk) Date: Tue, 16 Jul 91 12:10:05 +0100 Subject: subscription Message-ID: I would like to subscribe to Connectionists.( I am working for a Neural Network Technology Transfer Club based at University College London.) Tessa Rickards T.Rickards at cs.ucl.ac.uk From rosauer at fzi.uka.de Tue Jul 16 18:47:52 1991 From: rosauer at fzi.uka.de (Bernd Rosauer) Date: Tue, 16 Jul 91 22:47:52 GMT Subject: SUMMARY: GA&NN Message-ID: Some weeks ago I posted a request concerning the combination of genetic algorithms and neural networks. In the following you will find a summary of the references I received. This summary is preliminary and the references are not completely reviewed. Maybe, I will post an annotated one at the end of this year when I have got all the relevant proceedings of this year. I would like to make some general comments in advance. First of all, two summaries have already been published which cover the stuff until 1990: Rudnick, Mike. "A Bibliography of the Intersection of Genetic Search and Artificial Neural Networks." Technical Report CS/E 90-001, Department of Computer Science and Engineering, Oregon Graduate Institute, January 1990. Weiss, Gerhard. "Combining Neural and Evolutionary Learning: Aspects and Approaches." Report FKI-132-90, Institut fuer Informatik, Technische Universitaet Muenchen, May 1990. As one of my thrustworthy informants told me the proceedings of ICGA'91 and NIPS'91 (will) contain tons of stuff on that topic. Finally, there is a mailing list on "neuro-evolution". Because of the administrator did not yet answer my request I do not know whether this list is still active. Anyway, try for further information. Now, here is the summary. Many thanks to everyone who responded. Feel free to send me further references. Bernd -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Ackley, D. H., and M. S. Littman. "Learning from natural selection in an artificial environment." Proceedings of the International Joint Conference on Neural Networks Washington, D.C., January 1990. Ackley, D. H., and M. L. Littman. "Interactions between learning and evolution." Artificial Life 2. Ed. Chris Langton. New York: Addison-Wesley, in press. Belew, R. K. "Evolution, learning and culture: computational metaphors for adaptive search." Complex Systems 4.1 (1990): 11-49. Belew, R. K., J. McInerney and N. N. Schraudolph. "Evolving networks: Using the Genetic Algorithm with connectionist learning." Proc. 2nd Artificial Life Conference. New York: Addison-Wesley, in press. Belew, R. K., J. McInerney and N. Schraudolph. "Evolving Networks: Using the Genetic Algorithm with Connectionist Learning." Technical Report CS90-174, University of California at San Diego, 1990. Hinton, G. E., and S. J. Nowlan S.J. "How Learning Guides Evolution." Complex System 1 (1987): 495-502. Ichikawa, Y. "Evolution of Neural Networks and Applications to Motion Control." Proc. of IEEE Int. Work. on Intelligent Motion Control, Vol.1, 1990. Keesing, Ron, and David Stork. N.t. NIPS-3, 1990. Kitano, Hiroaki. "Empirical Study on the Speed of Convergence of Neural Network Training using Genetic Algorithms." Proceedings of AAAI-90. Kitano, Hiroaki. "Designing Neural Networks with Genetic Algorithms using Graph Generation System." Complex System 4.4 (1990). Kouchi, Masahiro, Hiroaki Inayoshi and Tsutomu Hoshino. "Optimization of Neural-Net Structure by Genetic Algorithm with Diploidy and Geographical Isolation Model." Inst. of Engineering Mechanics, Univ. Tsukuba, Ibaraki 305, Japan. Menczer, F., and D. Parisi. "`Sexual' reproduction in neural networks." Technical Report PCIA-90-06, Institute of Psychology, C.N.R., Rome, 1990. Menczer, F., and D. Parisi. "Evidence of hyperplanes in the genetic learning of neural networks." Technical Report PCIA-91-08, Institute of Psychology, C.N.R., Rome, 1991. Miglino, O., and D. Parisi. "Evolutionary stable and unstable strategies in neural networks." Technical Report PCIA-91-09, Institute of Psychology, C.N.R., Rome, 1991. Mjolsness, Eric, David H. Sharp and Bradley K. Alpert. "Scaling, Machine Learning, and Genetic Neural Nets." Advances in Applied Mathematics 10 (1989): 137-163. Montana, David J., and Lawrence Davis. "Training Feedforward Neural Networks using Genetic Algorithms." Proceedings of the 11th Intern. Joint Conference on Artificial Intelligence, 1989, pp. 762-767. Muehlenbein, H., and J. Kindermann. "The Dynamics of Evolution and Learning - Towards Genetic Neural Networks." Connectionism in Perspective. Ed. R. Pfeifer et. al. Elsevier, 1989. pp. 173-197. Nolfi, S., J. Elman and D. Parisi. "Learning and Evolution in Neural Networks." CRL Technical Report 9019, University of California at San Diego, 1990. Nolfi, S., and D. Parisi. "Auto-teaching: neural networks that develop their own teaching input." Technical Report PCIA-91-03, Institute of Psychology, C.N.R., Rome, 1991. Parisi, D., F. Cecconi and S. Nolfi. "Econets: Neural Networks that Learn in an Environment." Network 1 (1990): 149-168. Parisi, D., S. Nolfi, and F. Cecconi. "Learning, Behavior, and Evolution." Technical Report PCIA-91-14, Institute of Psychology, C.N.R., Rome, 1991. Radcliff, Nick. "Genetic Neural Networks on MIMD Computers." Ph.D. Thesis, University of Edinburgh. Radcliff, Nick. "Equivalence Class Analysis of Genetic Algorithms." Complex Systems, in press. Radcliff, Nick. "Forma Analysis and Random Respectful Recombination." Proceedings of ICGA91, in press. Todd, P. M. and G. F. Miller, G.F. "Exploring adaptive agency II: simulating the evolution of associative learning." From Animals to Animats. Eds. J. A. Meyer and S. W. Wilson. Cambridge, MA: MIT, 1991. From shen at iro.umontreal.ca Wed Jul 17 19:29:20 1991 From: shen at iro.umontreal.ca (Yu Shen) Date: Wed, 17 Jul 91 19:29:20 EDT Subject: network that ranks Message-ID: <9107172329.AA09591@kovic.IRO.UMontreal.CA> Suppose a unit in the network stands for an option being compared with others, and given the pairwise prferences, such as option A is better than option B, etc. I would like the responses of the units to be proportional to their ranks in the preference relationship. I tried the Interactive Activation and Competition network in PDP book, IAC. It seems to work, by coding the weights among the units as follows: if A is better than B, then the weight from B to A is 1, and the weight from A to B is -1. And the initial state of the units are taken to be constant, 0.5 (the rest value). I tested up to cases of 4 units. For A>B>C>D, the response of the same ranks is produced (unit A is the highest, unit b is the next, etc.). But I'm wondering if it work for all the cases. Any pointer for better connectionist solution? Thank you, Yu Shen From NEZA at ALF.LET.UVA.NL Thu Jul 18 14:33:00 1991 From: NEZA at ALF.LET.UVA.NL (Neza van der Leeuw) Date: Thu, 18 Jul 91 14:33 MET Subject: Simple pictures, tough problems. Message-ID: <635B127EA0803C46@BITNET.CC.CMU.EDU> Hello, I am currently working on the "grounding problem", in the sense that I try to derive meaning of language from pictures coupled with sentences (my own version of the Berkeley ICSI L0 project: Miniature Language Acquisition; A touchstone for cognitive science. TR-90-009). Right now I am working with a self-organising neural net (kind of Kohonen), having bitmaps and strings as input. These bitmaps represent circles, squares and triangles in a bitwise manner. The problem is that I want the architecture to generalize. This means that it should group small and large circles, rather than circles of size X with squa- res of size nearly X. In the current implementation, coding is done by repre- senting each line of the picture by a random vector. But when doing this, the net generalizes not on second-order properties like form, but on first-order properties (roughly: where the black spots in the picture reside). Thus, circles are seen similar to squares when their sizes "match". One could avoid this problem by choosing a representation in language-like propositions, but this seems to me to be the "solving the problem by doing it yourself" approach. Some represenational mechanism should provide the answer to my problem, but I want it to be "picture-like" instead of "language-like". Otherwise I bounce back into my favourite "grounding problem" again. Does anyone know of any work that has been done in this field? Any hints, refe- rences or other useful messages are very welcome. As I am more a computer lin- guist than a grafical scientist my knowledge of the whole thing is rather limited. If some useful stuff comes out, I will write a nice report and send it to the list, if others are interested in the matter as well. Many thanks, and don't get overworked during these "holiday months" (whatever that may be). Neza van der Leeuw Dept. of Computational Linguistics Faculty of Arts University of Amsterdam Spuistraat 134 1012 VB Amsterdam The Netherlands From slehar at park.bu.edu Thu Jul 18 12:22:35 1991 From: slehar at park.bu.edu (Steve Lehar) Date: Thu, 18 Jul 91 12:22:35 -0400 Subject: Simple pictures, tough problems. Message-ID: <9107181622.AA28726@park.bu.edu> > The problem is that I want the architecture to generalize. This means > that it should group small and large circles, rather than circles of > size X with squares of size nearly X. > ... > > the net generalizes not on second-order properties like form, but on > first-order properties (roughly: where the black spots in the picture > reside). Thus, circles are seen similar to squares when their sizes > "match". Maybe the solution is to take a cue from nature. How does the brain represent visual information? Hubel and Wiesel [1] found simple cells which respond to the very simplest visual primitives- oriented edges or moving edges. They also found complex cells, which generalized the simple cell responses "spatially", but not "featurally". This is an important point, and is, I believe, the key to understanding how the brain solves the generalization problem. Say you have a simple cell that fires in response to a vertical edge in a very specific location. A complex cell might fire for a vertical edge in a much larger range of locations (spatial generalization), but this is not due to the complex cell having a coarser representation of the world, because the complex cell will not fire in response to a cruder fuzzier edge, it is every bit as specific about the sharpness of that vertical edge as was the simple cell- i.e. we have NO featural generalization, just spatial. When you move up to complex and hyper complex cells, you get cells that respond to even more specialized features, such as end-stop detectors that fire for a vertical edge in a large region but only if it terminates, not if it goes straight through, and corner detectors which fire for two end-stop detectors, one vertical and one horizontal. Notice the trend- as we become more general spatially, we become more specific featurally. This is what I call the spatial/featural hierarchy, and one can posit that at the pinnacle of the hierarchy would be found very specific detectors that respond, for example, to your grandmother's face, wherever it may appear in the visual field. This is the basic idea behind the Neocognitron [2], although I believe that that model is lacking in one important element, that being resonant feedback between the levels of the hierarchy, which Grossberg [3] shows is so important to maintain a consistancy between different levels of the representation. I discuss a resonant spatial/featural hierarchy and how it may be implemented in [4] and [5]. Now you might argue that the construction of such a hierarchy would be very expensive in both space and time (memory and computation) especially if it is implemented as I propose, with resonant feedback between all the layers of the hierarchy. My response would be that the problem of vision is by no means trivial, and that until we come up with a better solution, we cannot presume to do better than nature, and if nature deems it necessary to create such a hierarchy, then I strongly suspect that that hierarchy is an essential prerequisite for featural generalization. [1] Hubel & Wiesel RECEPTIVE FIELDS AND FUNCTIONAL ARCHITECTURE IN TWO NONSTRIATE VISUAL AREAS OF THE CAT(1965) Journal of Neurophysiology 28 229-289 [2] Fukushima & Miyake NEOCOGNITRON: A NEW ALGORITHM FOR PATTERN RECOGNITION TOLERANT OF DEFORMATIONS AND SHIFTS IN POSITION.(1982) Pattern Recognition 15, 6 455-469 [3] Grossberg, Stephen & Mingolla, Ennio. NEURAL DYNAMICS OF PERCEPTUAL GROUPING: TEXTURES, BOUNDARIES AND EMERGENT SEGMENTATIONS Perception & Psychophysics (1985), 38 (2), 141-171. [4] Lehar S., Worth A. MULTI RESONANT BOUNDARY CONTOUR SYSTEM, Boston University, Center for Adaptive Systems technical report CAS/CNS-TR-91-017. To get a copy, write to... Boston University Center for Adaptive Systems 111 Cummington Street, Second Floor Boston, MA 02215 (617) 353-7857,7858 [5] Lehar S., Worth A. MULTIPLE RESONANT BOUNDARY CONTOUR SYSTEM. In: PROGRESS IN NEURAL NETWORKS volume 3 (Ed. by Ablex Publishing Corp.) In print. (i.e. not available yet) From ross at psych.psy.uq.oz.au Fri Jul 19 03:31:49 1991 From: ross at psych.psy.uq.oz.au (Ross Gayler) Date: Fri, 19 Jul 1991 17:31:49 +1000 Subject: simple pictures, tough problems (language grounding) Message-ID: <9107190731.AA10594@psych.psy.uq.oz.au> Neza van der Leeuw (Computational Linguistics, University of Amsterdam) writes: >I am currently working on the "grounding problem", in the sense that I try to >derive meaning of language from pictures coupled with sentences ... > >The problem is that I want the architecture to generalize. This means that it >should group small and large circles, rather than circles of size X with squa- >res of size nearly X. ... > >One could avoid this problem by choosing a representation in language-like >propositions, but this seems to me to be the "solving the problem by doing it >yourself" approach. Some represenational mechanism should provide the answer >to my problem, but I want it to be "picture-like" instead of "language-like". >Otherwise I bounce back into my favourite "grounding problem" again. I think you will have to do one of two things: either hand-craft an environment representation that is biased towards generalisation along the directions that you think are 'natural'; or provide the system with some way of manipulating the environment so that some properties of the environment are invariant under manipulation. It may help to think of the problem in terms of learning a first language and a second language. In learning a second language, the concepts are (mostly) already present - so only the language has to be learned. In learning a first language, the concepts are learned more or less at the same time as the language (without wanting to get into an argument about the Whorf hypothesis). I think that learning concepts by pure observation (as your system has to) is generally impossible. What is there in the input to suggest that 'circle-ness' or 'square-ness' is a better basis for generalisation than pixel overlap? Imagine, that a civilisation living on the surface of a neutron star has just made contact with us by sending a subtitled videotape of life on their world. The environmental scenes would make no sense to us - we probably would not even be able to segment the scene into objects. So how could we learn the language if we can't 'understand' the picture? The human visual system has certain generalisation biases (say, based on edge detectors etc), but I think a stronger requirement is to be able to manipulate the environment. By grabbing an object and moving it backwards and forwards under our control, we can learn that the shape remains constant while the apparent size varies. I would appreciate any references to more formal arguments for the necessity (or otherwise) of being able to manipulate the environment (or perceptual apparatus) in order to learn 'natural' concepts. Ross Gayler ross at psych.psy.uq.oz.au From pjh at compsci.stirling.ac.uk Fri Jul 19 10:52:57 1991 From: pjh at compsci.stirling.ac.uk (Peter J.B. Hancock) Date: 19 Jul 91 10:52:57 BST (Fri) Subject: Simple pictures, tough problems. Message-ID: <9107191052.AA11204@uk.ac.stir.cs.nevis> A comment on Steve Lehar's comment: just how hierarchical is the visual system? I'm told (sorry, I don't know a reference and my source is away on holiday) of evidence that some complex cells respond {\em before} simple cells. I understand there is some debate about the simple/complex dichotomy anyway, but such a result does challenge the traditional story of simple cells feeding into complex ones. More remarkably, recent work from Dave Perrett's group at St. Andrew's is showing that face-sensitive cells in monkeys not only respond within 90mS (yes, ninety milliseconds) of stimulus presentation, but are highly stimulus-specific at that time (I don't know if this is yet published). This does not leave very much time for any pretty hierachies and feedback loops. It implies that recognition of such highly familiar objects is extremely feed-forward and that any model that {\em always} requires many cycles to settle down is wrong. This begs the question of what all the feedback loops in the brain are doing. It may be that not all stimuli are as privileged as faces and that more processing is required for less familiar things. It may be to do with the initial learning, rather like Grossberg's ART which may cycle while learning, but is one-shot thereafter. It may be to do with tuning early systems for given visual tasks. It certainly needs more research... Peter Hancock Centre for Cognitive and Computational Neuroscience University of Stirling. From harnad at Princeton.EDU Fri Jul 19 10:20:55 1991 From: harnad at Princeton.EDU (Stevan Harnad) Date: Fri, 19 Jul 91 10:20:55 EDT Subject: simple pictures, tough problems (language grounding) Message-ID: <9107191420.AA03936@clarity.Princeton.EDU> Ross Gayler wrote: > I would appreciate any references to more formal arguments for the > necessity (or otherwise) of being able to manipulate the environment > (or perceptual apparatus) in order to learn 'natural' concepts. Try: (1) Harnad, S. (1990) The Symbol Grounding Problem. Physica D 42: 335-346. (2) Harnad, S., Hanson, S.J. & Lubin, J. (1991) Categorical Perception and the Evolution of Supervised Learning in Neural Nets. Presented at American Association for Artificial Intelligence Symposium on Symbol Grounding: Problems and Practice. Stanford University, March 1991. (3) Harnad, S. (1992) Connecting Object to Symbol in Modeling Cognition. In: A. Clarke and R. Lutz (Eds) "Connectionism in Context" Springer Verlag (forthcoming) These articles are retrievable by anonymous ftp from directory pub/harnad on princeton.edu (IP: 128.112.128.1) in binary mode as the compressed files: (1) harnad90.sgproblem.Z (2) harnad91.cpnets.Z (3) harnad92.symbol.object.Z From yang at judy.cs.iastate.edu Fri Jul 19 12:46:13 1991 From: yang at judy.cs.iastate.edu (Jihoon Yang) Date: Fri, 19 Jul 91 11:46:13 CDT Subject: tech report (fwd) Message-ID: <9107191646.AA06647@judy.cs.iastate.edu> > ------------------------------------------------------------------------ > > The following tech report is now available as a compressed postscript > file "yang.cascor.ps.Z" through anonymous ftp from the neurprose archive > (directory pub/neuroprose on cheops.cis.ohio-state.edu - Thanks to > Jordan Pollack of Ohio State University). > > Experiments with the Cascade-Correlation Algorithm > Technical report # 91-16 (July 1991) > Jihoon Yang & Vasant Honavar > Department of Computer Science > Iowa State University > > ----------------------------------------------------------------------- > Jihoon Yang > yang at judy.cs.iastate.edu > From steck at spock.wsu.ukans.edu Fri Jul 19 12:51:05 1991 From: steck at spock.wsu.ukans.edu (jim steck (ME) Date: Fri, 19 Jul 91 11:51:05 -0500 Subject: Stability proofs for recurrent networks Message-ID: <9107191651.AA13577@spock.wsu.UKans.EDU> I am currently looking at the stability issue of syncronous fully recurrent neural networks. I am aware of literature (mainly for Hopfield networks) where the stability issue is adressed using Lyapunov "energy" functions, but have not seen any publication of other types of approaches. I would appreciate E-mail regarding articles where this problem is discussed. Jim Steck steck at spock.wsu.ukans.edu From PF103 at phx.cam.AC.UK Fri Jul 19 20:39:37 1991 From: PF103 at phx.cam.AC.UK (Peter Foldiak) Date: Fri, 19 Jul 91 20:39:37 BST Subject: 'natural' concepts Message-ID: Ross Gayler writes: > I think that learning concepts by pure observation (as your system has to) is > generally impossible. What is there in the input to suggest that 'circle-ness' > or 'square-ness' is a better basis for generalisation than pixel overlap? It may not be easy, but I don't think it is generally impossible. Barlow's redundancy reduction principle, for instance, would say that features that result in lower statistical redundancy are better. (Redundancy here is not first-order (bit probabilities) but pairwise and higher-order redundancy.) Peter Foldiak From gary at cs.UCSD.EDU Fri Jul 19 16:00:29 1991 From: gary at cs.UCSD.EDU (Gary Cottrell) Date: Fri, 19 Jul 91 13:00:29 PDT Subject: talking networks Message-ID: <9107192000.AA19599@desi.ucsd.edu> Dear Neza, I couldn't reply to your address, so I'm sending this to the net. Here's a few references. Also, you should be aware of Georg Dorffner's work at the University of Vienna. Some of the work is overlapping, but I give you all of these in hopes you can find one of them. I will send them to you anyway. gary cottrell 619-534-6640 Sec'y: 619-534-5288 FAX: 619-534-7029 Computer Science and Engineering C-014 UCSD, La Jolla, Ca. 92093 gary at cs.ucsd.edu (INTERNET) {ucbvax,decvax,akgua,dcdwest}!sdcsvax!gary (USENET) gcottrell at ucsd.edu (BITNET) Cottrell, G., Bartell, B. & C. Haupt (1990) Gounding meaning in perception. In \fIProceedings of the 14th German Workshop on AI\fP, H. Marburger (Ed.), pp. 307-321. Berlin: Springer Verlag. Cottrell, G. (1990). Extracting features from faces using compression networks. In Touretzky, D.S., Elman, J.L., Sejnowski, T.J. and Hinton G.E. (Eds.) \fIProceedings of the 1990 Connectionist Models Summer School\fP. San Mateo: Morgan Kaufmann. Bartell, B. & Cottrell, G.W. (1991) A model of symbol grounding in a temporal environment. In \fIProceedings of the International Joint Conference on Neural Networks\fP, Seattle, Washington, June, 1991. From ITGT500 at INDYCMS.BITNET Fri Jul 19 18:59:47 1991 From: ITGT500 at INDYCMS.BITNET (Bo Xu) Date: Fri, 19 Jul 91 17:59:47 EST Subject: Requests Message-ID: <85672B7B60800064@BITNET.CC.CMU.EDU> Besides the slow rate of convergence, local minima, etc., another charge against the feedforward backpropagation neural network is that it is non-biological. I have already got some references on this problem given by G.E.Hinton, F.Crick, A.G.Barto, P.Mazzoni, and R.A.Andersen, etc.. I want to collect more references on this topic. Could someone send me more references which are not covered above? Thanks a lot in advance. Bo Xu Indiana University ITGT500 at INDYCMS.BITNET From ross at psych.psy.uq.oz.au Sat Jul 20 10:46:34 1991 From: ross at psych.psy.uq.oz.au (Ross Gayler) Date: Sun, 21 Jul 1991 00:46:34 +1000 Subject: 'natural' concepts (symbol grounding) Message-ID: <9107201446.AA29959@psych.psy.uq.oz.au> Peter Foldiak writes: >[Learning concepts by pure observation] may not be easy, >but I don't think it is generally impossible. >Barlow's redundancy reduction principle, for instance, would say that >features that result in lower statistical redundancy are better. >(Redundancy here is not first-order (bit probabilities) but pairwise >and higher-order redundancy.) OK, let me expand on my position a little. I realise that connectionist systems can learn to categorise inputs and can generalise on this task. I'm not entirely convinced that this warrants being called 'concept learning' (but I don't think I can justify this belief). More importantly, most systems have had a considearble amount of effort put into the architecture and training set to ensure that they categorise and generalise as expected. My point is that models of the input data can be compared in terms of redundancy - but having a measure of goodness of fit does not directly help construct an optimal network. If the input can be simply characterised in terms of a higher order redundancy, it is very unlikely that a net with no representational biases, starting from input at the pixel level, will discover anything close to the optimal model inside a pragmatically bounded time. This is what I meant when I said that learning from observation was generally impossible. The point I wanted to make was that I think learning at a pragmatically useful rate in a complex environment requires the learner to be able to manipulate the environment. Research methodologists distinguish between experimental and non-experimental methods. Non-experimental methods rely on correlation of observations - so that patterns can be described but causation cannot be ascribed. Experimental methods involve assigning manipulations to the environment in a randomised way - so that much stronger statements can be made about the workings of the world (assuming your assumptions hold :-). Being able to manipulate the environment allows the experimenter to drag information out of the environment at a higher rate if the experimenter is in relative ignorance. Cosmologists use non-experimental methods - BUT they have to apply a lot of prior knowledge that has been validated in other ways. The other point I want to make is that typical connectionist categorisers do not have a model of the input environment as an external reality. Assume that the input signals come from a sensor that imposes some kind of gross distortion. The net learns to categorise the distorted signal as it stands with no separate models of the environment and the distortion. In order to develop a concept that is closer to the human notion of some external reality, the system has to be able to factor its model into an environmental part and a perceptual part. This can't be done by pure observation, it needs some ability to interact with the environment, even if only by panning the camera across the scene. From slehar at park.bu.edu Sat Jul 20 15:01:05 1991 From: slehar at park.bu.edu (Steve Lehar) Date: Sat, 20 Jul 91 15:01:05 -0400 Subject: Simple pictures, tough problems. In-Reply-To: connectionists@c.cs.cmu.edu's message of 19 Jul 91 18:12:48 GM Message-ID: <9107201901.AA26511@park.bu.edu> "Peter J.B. Hancock" argues... > just how hierarchical is the visual system? I'm told of evidence that > some complex cells respond BEFORE simple cells. The fact that a complex cell fires BEFORE a simple cell does not preclude the possibility that the signal was provided originally by the simple cells. What we are talking about here is a resonant process- like a bow running over a violin string. The bow produces random vibration on the string, but the string only responds to one frequency component of that vibration. The resultant tone does not "start" at the bow and spread outward towards the bridge and neck, but rather, a resonant vibration emerges from the entire system simultaneously. In the same way, simple cells of all orientations are constantly firing to greater or lesser degree to any visual input, but the complex cells (by simultaneous cooperative and competitive interactions) will resonate to a larger global pattern hidden in all the confusion of the simple cells. The fact that the complex cell fires first simply reflects the fact that the visual system puts more faith in a global harmony of coherent edges than in a cacophany of individual local responses, and indeed the local edges may not even be perceived until the global coherence is established. This same kind of interaction would dictate that even highe level patterns, as detected by hypercomplex or higher order cells, would register before the complex cells, so that although the input signal arrives bottom-up, recognition resonance is established top-down, thus ensuring global consistancy of the entire scene. According to the Boundary Contour System (BCS) model, local competitive and cooperative interactions occur between the lowest level detectors to enhance those that are compatible with global segmentations while suppressing those that are incompatible. If the lowest level interactions could settle into stable configurations before the global ones could exert their influence, the perception would be dominated by local, not global consistancies, like a damped violin string which produces a scratchy random noise when bowed. Peter Hancock continues... > face-sensitive cells in monkeys not only respond within 90mS ... of > stimulus presentation, but are highly stimulus-specific at that time > ... This does not leave very much time for any pretty hierachies and > feedback loops. It implies that recognition of such highly familiar > objects is extremely feed-forward and that any model that ALWAYS > requires many cycles to settle down is wrong. Well, it depends on what kind of cycles we are talking about. If you mean iterations of a sequential algorithm then I would have to agree, and current implementations of the BCS and MRBCS are of necessity performed iteratively on sequential binary machines. But the visual architecture that they presume to emulate is a parallel analog resonant system, like a violin string (but of course much more complex) so that it does not take any number of "cycles" as such for a resonance to set in. Also, in considering recognition time, top-down expectation plays a very large role- it takes considerably longer to recognise an unexpected or out-of-context pattern than an expected one. From pjh at compsci.stirling.ac.uk Mon Jul 22 10:59:41 1991 From: pjh at compsci.stirling.ac.uk (Peter J.B. Hancock) Date: 22 Jul 91 10:59:41 BST (Mon) Subject: Simple pictures, tough problems. Message-ID: <9107221059.AA14184@uk.ac.stir.cs.nevis> I think Steve Lehar and Ken Laws are making essentially the same, valid point: that there is time for quite a lot of very local analogue processing, for instance within a hypercolumn, even within a 90mS overall response time. Thus indeed a cell may actually fire before those that are driving it, provided the activation is transferred without the need for action potentials. The real time constraint comes when you want to start sending messages between cortical areas, which requires action potentials with their associated delays. There are huge numbers of fibres going top-down: they are obviously doing something, but within 90mS I don't believe they can be contributing much to the immediate recognition process, though they certainly might have something to do with priming. In the case of the monkey face-recognition work, it's unclear at the moment to what extent the experimental setup may have primed them to expect faces, but they were being shown all sorts of other things as well. Peter Hancock From roeschei at tumult.informatik.tu-muenchen.de Mon Jul 22 09:13:26 1991 From: roeschei at tumult.informatik.tu-muenchen.de (Martin Roescheisen) Date: Mon, 22 Jul 91 15:13:26 +0200 Subject: No subject Message-ID: <9107220913.AA00858@tumult.informatik.tu-muenchen.de> Technical Report available INCORPORATING PRIOR KNOWLEDGE IN PARSIMONIOUS NETWORKS OF LOCALLY-TUNED UNITS Keywords: Gaussian Units (Moody, Poggio), higher-dimensionality, rolling mills, use of prior knowledge. Reimar Hofmann Munich Technical University Martin R\"oscheisen Munich Technical University Volker Tresp Siemens AG Corporate Research and Development Abstract: Utilizing Bayes decision theory, we develop a theoretical foundation for a localized network architecture that requires few centers to be allocated and can therefore be employed in problems which because of their high input dimensionality could not yet be tackled by such networks. We show how quantitative {\it a priori} knowledge can be readily incorporated by choosing a specific training regime. The network was employed as a neural controller for a hot line rolling mill and achieved in this application one to two orders of magnitude higher accuracy than optimally-tuned standard algorithms such as sigmoidal backpropagation and performed significantly better than a state-of-the-art analytic model. _________________ Hardcopies of the full paper can be obtained by sending e-mail to hofmannr at lan.informatik.tu-muenchen.dbp.de From jose at tractatus.siemens.com Mon Jul 22 08:45:59 1991 From: jose at tractatus.siemens.com (Steve Hanson) Date: Mon, 22 Jul 1991 08:45:59 -0400 (EDT) Subject: Requests In-Reply-To: <85672B7B60800064@BITNET.CC.CMU.EDU> References: <85672B7B60800064@BITNET.CC.CMU.EDU> Message-ID: "Biological plausibility" is somewhat of a throwaway, given the massively documented understanding we have of learning in brain circuits (he said sarcastically). There are many seemingly incompatible brain circuits for implementing classical conditioning, depending on the beast, the stimuli, and the context and maybe just depending on the whims of the local circuitry development that day. Certainly, the way Backprop is typically implemented in "C" code, may be biologically implausible... ie information doesn't tend to flow back down axons... (at least not at a fast enough rate). I am not sure that slow rate of convergence is such a good argument.. given you can do things to speed up learning in feedforward nets... also given the rate people really seem to learn at... In any case, there are now published counter-examples of people showing how back-prop could be seen as biologically plausible, which I suspect given our present state knowledge is a much more useful enterprise. to wit: G. Tesauro, Neural Models of Classical Conditioning: A theoretical Perspective, Hanson & Olson, (Eds.) MIT Press, Bradford, 1990. D. Zipser, in Gluck & Rumelhart, Neuroscience and Connectionism, LEA, 1990. There may be others.... --Steve Stephen J. Hanson Learning & Knowledge Acquisition Group SIEMENS Research 755 College Rd. East Princeton, NJ 08540 From schyns at clouseau.cog.brown.edu Mon Jul 22 08:53:32 1991 From: schyns at clouseau.cog.brown.edu (Phillipe Schyns) Date: Mon, 22 Jul 91 08:53:32 -0400 Subject: the symbol grounding problem Message-ID: <9107221253.AA17341@clouseau.cog.brown.edu> Here is the abstract of a paper that will appear in Cognitive Science. Although its main topic is conceptual development and lexical acquisition, it presents a solution to the symbol grounding problem which is compatible with Harnard's proposal. Schyns, P. G. (in press). A neural network model of conceptual development, Cognitive Science. Abstract Previous neural network models of concept learning were mainly implemented with supervised learning schemes. However, studies of human conceptual memory have shown that concepts may be learned without a teacher who provides the category name to associate with exemplars. A modular neural network architecture that realizes concept acquisition through two functionally distinct operations, categorizing and naming, is proposed as an alternative. An unsupervised algorithm realizes the categorizing module by constructing representations of categories compatible with Prototype Theory. The naming module associates category names to the output of the categorizing module in a supervised mode. In such modular architecture, the interface between the modules can be conceived of as an "informational relay" that encodes, constrains and propagates important information. Five experiments were conducted to analyze the relationships between internal conceptual codes and simple conceptual and lexical development. The first wo experiments show a prototype effect and illustrate some basic characteristics of the system. The third experiment presents a bottom-up model of the narrowing down of children's early lexical categories that honors Mutual Exclusivity. The fourth experiment introduces top-down constraints on conceptual coding. The fifth experiment exhibits how hierarchical relationships between concepts are learned by the architecture while it also demonstrates how a spectrum of conceptual expertise may gradually emerge as a consequence of experiencing more with certain categories than with others. Part of this work appeared in the Proceedings of the 1990 Connectionist Models Summer School. Schyns, P. G. (1990). A modular neural network model of the acquisition of category names in children. Proceedings of the 1990 Connectionist Models Summer School, 228-235, Morgan Kaufmann, CA. Philippe =========================================================================== Philippe G. Schyns Dpt. of Cognitive and Linguistic Sciences Box 1978 Brown University Providence, RI, 02912 From ray at espresso.boeing.com Tue Jul 23 11:25:24 1991 From: ray at espresso.boeing.com (Ray Allis 5-3583) Date: Tue, 23 Jul 91 08:25:24 PDT Subject: Simple pictures, tough problems. Message-ID: <9107231525.AA15897@espresso.bcs.eca> > From: Ken Laws > Subject: Re: Simple pictures, tough problems. > To: connectionists at RI.CMU.EDU > > This does not leave very much time for any pretty > > hierachies and feedback loops. > > Feedback loops are not necessarily slow. Analog computers can be > much faster than digital ones for many tasks, and I think we're > beginning to understand neural bundles in analog terms. > > -- Ken Laws > ------- > I sincerely hope so. It's long past time to do so. Ray Allis From gary at cs.UCSD.EDU Mon Jul 22 18:42:37 1991 From: gary at cs.UCSD.EDU (Gary Cottrell) Date: Mon, 22 Jul 91 15:42:37 PDT Subject: Stability proofs for recurrent networks Message-ID: <9107222242.AA22701@desi.ucsd.edu> Hal White has shown convergence conditions for learning in recurrent nets. Try writing him for reprints. He is: Hal White Dept. of Economics UCSD La Jolla, CA 92093 gary From georgiou at rex.cs.tulane.edu Tue Jul 23 19:58:05 1991 From: georgiou at rex.cs.tulane.edu (George Georgiou) Date: Tue, 23 Jul 91 18:58:05 -0500 Subject: CALL FOR PAPERS (INFORMATION SCIENCES) Message-ID: <9107232358.AA11866@rex.cs.tulane.edu> Call for Papers Special Issue of INFORMATION SCIENCES Topic: Neural Networks and Artificial Intelligence The motivation for this Special Issue, which will be published in 1992, is to evaluate the advances in the field of neural networks in perspective. Critics have charged that engineering neural computing has turned out to be no more than a new technique of clustering data. From berg at cs.albany.edu Tue Jul 23 18:08:59 1991 From: berg at cs.albany.edu (George Berg) Date: Tue, 23 Jul 91 18:08:59 EDT Subject: TR available: learning phrase structure Message-ID: <9107232209.AA11241@odin.albany.edu> The following paper is available: ------------------------------------------------------------------------------- Learning Recursive Phrase structure: Combining the Strengths of PDP and X-Bar Syntax George Berg Department of Computer Science Department of Linguistics and Cognitive Science State University of New York at Albany ABSTRACT In this paper we show how a connectionist model, the XERIC Parser, can be trained to build a representation of the syntactic structure of sentences. One of the strengths of this model is that it avoids placing a priori restrictions on the length of sentences or the depth of phrase structure nesting. The XERIC architecture uses X-Bar grammar, an "unrolled" virtual architecture reminiscent of Rumelhart and McClelland's back-propagation through time, recurrent networks and reduced descriptions similar to Pollack's RAAM. Representations of words are presented one at a time, and the parser incrementally builds a representation of the sentence structure. Along the way it does lexical and number/person disambiguation. The limits on the current model's performance are consistent with the difficulty of encoding information (especially lexical information) as the length and complexity of the sentence increases. This paper is to be presented at the IJCAI-91 Workshop on Natural Language Learning, and is also available as SUNY Albany Computer Science Department technical report TR 91-5. =============================================================================== This paper is available three ways. Please DO NOT write me for a copy (among other reasons, because I'll be out of town most of the rest of the Summer). I will, however, be happy to answer questions and otherwise discuss the paper. ---- First Way: anonymous ftp via the neuroprose archive: The file is available via anonymous ftp from cheops.cis.ohio-state.edu as the file berg.phrase_structure.ps.Z in the pub/neuroprose directory. It is a compressed postscript file. Below is the log of a typical ftp session to retrieve the file: yourprompt> ftp cheops.cis.ohio-state.edu Connected to cheops.cis.ohio-state.edu. 220 cheops.cis.ohio-state.edu FTP server (Version 5.49 Tue May 9 14:01:04 EDT 19 89) ready. Name (cheops.cis.ohio-state.edu:you): anonymous 331 Guest login ok, send ident as password. Password: 230 Guest login ok, access restrictions apply. ftp> cd pub/neuroprose 250 CWD command successful. ftp> binary 200 Type set to I. ftp> get berg.phrase_structure.ps.Z 200 PORT command successful. 150 Opening BINARY mode data connection for berg.phrase_structure.ps.Z (107077 b ytes). 226 Transfer complete. local: berg.phrase_structure.ps.Z remote: berg.phrase_structure.ps.Z 107077 bytes received in 7.3 seconds (14 Kbytes/s) ftp> quit 221 Goodbye. yourprompt> Then uncompress the file and print it in your local fashion for printing postscript. ---- Second Way: anonymous ftp via SUNY Albany The file is available via anonymous ftp from ftp.cs.albany.edu (128.204.2.32) as the file tr91-5.ps.Z in the pub directory. It is a compressed postscript file. Below is the log of a typical ftp session to retrieve the file: yourprompt> ftp ftp.cs.albany.edu Connected to karp.albany.edu. 220 karp.albany.edu FTP server (SunOS 4.1) ready. Name (ftp.cs.albany.edu:you): anonymous 331 Guest login ok, send ident as password. Password: 230 Guest login ok, access restrictions apply. ftp> cd pub 250 CWD command successful. ftp> binary 200 Type set to I. ftp> get tr91-5.ps.Z 200 PORT command successful. 150 Binary data connection for tr91-5.ps.Z (128.204.2.36,2116) (107077 bytes). 226 Binary Transfer complete. local: tr91-5.ps.Z remote: tr91-5.ps.Z 107077 bytes received in 1 seconds (1e+02 Kbytes/s) ftp> quit 221 Goodbye. yourprompt> Then uncompress the file and print it in your local fashion for printing postscript. --- Third Way: SUNY Albany Computer Science Department Technical Reports Secretary. A copy of the paper may be requested by writing: Technical Reports Secretary Computer Science Department, LI-67A State University of New York at Albany Albany, New York 12222 USA and requesting a copy of Technical Report TR 91-5 ("Learning Recursive Phrase structure: Combining the Strengths of PDP and X-Bar Syntax" by George Berg). As I do not wish to make an enemy of the technical reports secretary, please only request a copy if you are unable to get one by ftp. ------------------------------------------------------------------------------- | George Berg | Computer Science Dept. | If you want wit in 15 words | | berg at cs.albany.edu | SUNY at Albany, LI 67A | or less, go check Bartlett's | | (518) 442 4267 | Albany, NY 12222 USA | quotations -- I'm busy. | ------------------------------------------------------------------------------- From stevep at cs.uq.oz.au Wed Jul 24 04:41:43 1991 From: stevep at cs.uq.oz.au (stevep@cs.uq.oz.au) Date: Wed, 24 Jul 91 18:41:43 +1000 Subject: NetTools - a package of tools for NN analysis Message-ID: <9107240841.AA03888@client> NetTools is a package of analysis tools and a tech. report demonstrating two of these techniques. Analysis Tools for Neural Networks. by Simon Dennis and Steven Phillips Abstract - A large volume of neural net research in the 1980's involved applying backpropagation to difficult and generally poorly understood tasks. Success was sometimes measured on the ability of the network to replicate the required mapping. The difficulty with this approach, which is essentially a black box analysis, is that we are left with little additional understanding of the problem or the way in which the neural net has solved it. Techniques which can look inside the black box are required. This report focuses on two statistical analysis techniques (Principal Components Analysis and Canonical Discriminant Analysis) as tools for analysing and interpreting network behaviour in the hidden unit layers. Net Tools The following package contains three tools for network analysis: gea - Group Error Analysis pca - Principal Components Analysis cda - Canonical Discriminants Analysis TOOL DESCRIPTIONS Group Error Analysis (gea) Gea counts errors. It takes an output file and a target file and optionally a groups file. Each line in the output file is an output vector and the lines in the targets file are the corresponding correct values. If all values in the output file are within criterion of the those in the target file then the pattern is considered correct. Note that this is a more stringent measure of correctness than the total sum of squares. In particular it requires the outputs to be either high or low rather than taking some average intermediate value. If a groups file is provided then gea will separate the error count into the groups provided. Principal Components Analysis (pca) Principle components analysis takes a set of points in a high dimensional space and determines the major components of variation. The principal components are labeled 0-(n-1) where n is the dimensionality of the space (i.e. the number of hidden units). The original points can be projected onto these vectors. The result is a low dimensional plot which has hopefully extracted the important information from the high dimensional space. Canonical Discriminants Analysis (cda) Canonical discriminant analysis takes a set of grouped points in a high dimensional space and determines the components such that points within a group form tight clusters. These points are called the canonical variates and are labeled 0-(n-1) where n is the dimensionality of the space (i.e. the number of hidden units). The original points can be projected on to these vectors. The result is a low dimensional plot which has clustered the points belonging to each group. TECHNICAL REPORT Reference: Simon Dennis and Steven Phillips. Analysis Tools for Neural Networks. Technical Report 207, Department of Computer, University of Queensland, Queensland, 4072 Australia May, 1991 NetTools.ps is a technical report which demonstrates the results which can be obtained from pca and cda. It outlines the advantages of each and points out some interpretive pitfalls which should be avoided. TUTORIAL The directory tute contains a tutorial designed at the University of Queensland by Janet Wiles and Simon Dennis to introduce students to network analysis. It uses the iris data first published by Fisher in 1936. The backpropagation simulator is tlearn developed at UCSD by Jeffery Elman and colleagues. In addition the tutorial uses the hierarchical clustering program, cluster, which was written by Yoshiro Miyata and modified by Andreas Stolcke. These tools can be obtained as follows $ ftp crl.ucsd.edu Connected to crl.ucsd.edu. 220 crl FTP server (SunOS 4.1) ready. Name (crl.ucsd.edu:mav): anonymous 331 Guest login ok, send ident as password. Password: 230 Guest login ok, access restrictions apply. ftp> cd pub/neuralnets 250 CWD command successful. ftp> bin 200 Type set to I. ftp> get NetTools.tar.Z 200 PORT command successful. 150 Binary data connection for NetTools.tar.Z (130.102.64.15,1240) (185900 bytes). 226 Binary Transfer complete. local: NetTools.tar.Z remote: NetTools.tar.Z 185900 bytes received in 1.9e+02 seconds (0.97 Kbytes/s) ftp> quit 221 Goodbye. $ zcat NetTools.tar.Z | tar -xf - Shalom Simon and Steven ------------------------------------------------------------------------------- Simon Dennis Address: Department of Computer Science Email: mav at cs.uq.oz.au University of Queensland QLD 4072 Australia ------------------------------------------------------------------------------- From T.Rickards at cs.ucl.ac.uk Wed Jul 24 07:43:23 1991 From: T.Rickards at cs.ucl.ac.uk (T.Rickards@cs.ucl.ac.uk) Date: Wed, 24 Jul 91 12:43:23 +0100 Subject: V and V database Message-ID: Dear Connectionists, I am compiling a database of references in the area of `Validation and Verification of Neural Nets'. This is part of a Government sponsored Technology Transfer activity in the UK through the LINNET Neural Network Club. I would appreciate your contributions, and can make the reference list available to the connectionists net. Thanks in advance, Tessa Rickards LINNET INFORMATION DESK T.Rickards at uk.ac.ucl.cs From COBOLEWI at macc.wisc.edu Wed Jul 24 22:40:00 1991 From: COBOLEWI at macc.wisc.edu (Alan B. Cobo-Lewis) Date: Wed, 24 Jul 91 21:40 CDT Subject: simple pictures, tough problems Message-ID: <21072421404519@vms.macc.wisc.edu> Peter J. B. Hancock argues... > evidence that some complex cells respond {\em before} > simple cells. I understand there is some debate about the > simple/complex dichotomy anyway, but such a result does challenge the > traditional story of simple cells feeding into complex ones. Steve Lehar responds... > The fact that a complex cell fires BEFORE a simple cell does not > preclude the possibility that the signal was provided originally by > the simple cells. The nice hierarchical classification of simple, complex, and hypercomplex cells has been assaulted for two reasons: the hypercomplex category is questionable, and the hierarchy is questionable. Hubel and Wiesel (1965) added the hypercomplex classification to describe cells that otherwise seemed like complex cells, but were also end-stopped. Since then, end-stopping has been reported in both simple and complex cells (Schiller et al., 1976; Gilbert, 1977; Kato et al., 1978). The conclusion that end-stopped cells represent a distinct population, rather than there being a continuous distribution of amount of end-stopping has been challenged (Schiller et al., 1976), though Kato et al. (1978) report justification for the use of the discrete classifications "hypercomplex (simple family)" and "hypercomplex (complex family)". Whatever the outcome of that argument, the singular discrete classification "hypercomplex" is typically abandoned today. Hubel and Wiesel (1962, 1965) proposed that one level's input consists of purely excitatory connections from the immediately inferior level (simple -(+)-> complex -(+)-> hypercomplex). This arrangement cannot account for certain features of the cells' receptive fields, but to what extent their proposal for the wiring must be modified is unclear (Rose, 1979). There is evidence that the processing by simple and complex cells take place at least partially in parallel. For one thing, input to the striate cortex feeds into complex cells as well as simple cells. Direct monosynaptic input from the lateral geniculate nucleus to complex cells has been reported (Hoffman & Stone, 1971; Stone, 1972; Bullier & Henry, 1979a, b, c; Henry et al., 1979). For another thing, output from the striate cortex must include projections from simple as well as complex cells. After all, we certainly have absolute phase specificity in our visual perception, though complex cells lack such specificity. Steve Lehar continues... > current implementations of the BCS and MRBCS are of necessity > performed iteratively on sequential binary machines. But the visual > architecture that they presume to emulate is a parallel analog > resonant system, like a violn string (but of course much more > complex) so that it does not take any number of "cycles" as such for a > resonance to set in. Even a parallel system evolves through time. We can treat the vibration of a violin string as proceding in discrete time if our sampling rate is high enough (for bandlimited behavior). Each moment of this discrete time constitutes an iteration. To the extent that the time constant of a biological neural network's behavior is finite, we _do_ have to worry about how many iterations (how much time) it takes for the system to arrive at a solution. References Bullier, J., & Henry, G. H. (1979a). Ordinal position of neurons in cat striate cortex. JOURNAL OF NEUROPHYSIOLOGY, 42, 1251-1263. Bullier, J., & Henry, G. H. (1979b). Neural path taken by afferent streams in striate cortex of the cat. JOURNAL OF NEUROPHYSIOLOGY, 42, 1264-1270. Bullier, J., & Henry, G. H. (1979c). Laminar distribution of first-order neurons and afferent terminals in cat striate cortex. JOURNAL OF NEUROPHYSIOLOGY, 42, 1271-1281. Gilbert, C. D. (1977). Laminar differences in receptive field properties of cells in cat primary visual cortex. JOURNAL OF PHYSIOLOGY, 268, 391-421. Henry, G. H., Harvey, A. R., & Lund, J. S. (1979). The afferent connections and laminar distribution of cells in the cat striate cortex. JOURNAL OF COMPARATIVE NEUROLOGY, 187, 725-744. Hoffman, K.-P., & Stone, J. (1971). Conduction velocity of afferents to cat visual cortex: a correlation with cortical receptive field properties. BRAIN RESEARCH, 32, 460-466. Hubel, D. H., & Wiesel, T. N. (1962). Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. JOURNAL OF PHYSIOLOGY, 160, 106-154. Hubel, D. H., & Wiesel, T. N. (1965). Receptive fields and functional architecture in two nonstriate visual areas (18 and 19) of the cat. JOURNAL OF NEUROPHYSIOLOGY, 28, 229-289. Kato, H., Bishop, P. O., & Orban, G. A. (1978). Hypercomplex and simple/complex cell classification in cat striate cortex. JOURNAL OF NEUROPHYSIOLOGY, 42, 1071-1095. Rose, D. (1979). Mechanisms underlying the receptive field properties of neurons in cat visual cortex. VISION RESEARCH, 19, 533-544. Schiller, P. H., Finlay, B. L., & Volman, S. F. (1976). Quantitative studies of single-cell properties in monkey striate cortex. I. Spatiotemporal organization of receptive fields. JOURNAL OF NEUROPHYSIOLOGY, 39, 1288-1319. Stone, J. (1972). Morphology and physiology of the geniculocortical synapse in the cat: The question of parallel input to the striate cortex. INVESTIGATIVE OPTHAMOLOGY, 11, 338-346. From slehar at park.bu.edu Thu Jul 25 10:52:03 1991 From: slehar at park.bu.edu (Steve Lehar) Date: Thu, 25 Jul 91 10:52:03 -0400 Subject: simple pictures, tough problems In-Reply-To: "Alan B. Cobo-Lewis"'s message of Wed, 24 Jul 91 21:40 CDT <21072421404519@vms.macc.wisc.edu> Message-ID: <9107251452.AA13816@park.bu.edu> > The nice hierarchical classification of simple, complex, and > hypercomplex cells has been assaulted for two reasons: the > hypercomplex category is questionable, and the hierarchy is > questionable. We observe in the visual cortex many different cells, some respond to very simple features, others to more complex features, others to hypercomplex- Ahem! Excuse me- Even more complex features and so on upwards through the temporal lobe to cells that respond to very complex specific stimuli. Are you suggesting that the most complex cells do not take their input from the intermediate level cells, but compute their response directly from the raw input from lateral geniculate? I find this most unlikely! What would be the purpose of all those intermediate level representations if not for the use of the higher level cells? And why do we find ascending complexity of representation in a continuous spatial progression? Why would we not find very high level cells mixed in with simple cells at V1 if they compute their responses independantly to the intermediate levels in V2, V3...? > There is evidence that the processing by simple and complex cells take > place at least partially in parallel. So is your complaint that the visual hierarchy is not a PURE hierarchy because certain connections jump from low levels to very high levels bypassing intermediate levels? You will find no argument from me on that matter. I would assume however that most high level cells would also take input from intermediate levels, so we have a mixed hierarchy with lots of connections everywhere, but a hierarchy nevertheless! The spatial arrangement of cortical regions alone strongly suggests that to be the case. My argument about the spatial/featural transform at each stage holds for a mixed hierarchy as it does for a pure hierarchy. The evidence for this seems overwealming- that the higher the cell is in the hierarchy, generally the larger is the region in the visual field to which it will respond. This is what I mean by the spatial/featural hierarchy, that every stage in the hierarchy increases featural specificity while decreasing spatial specificity. And I maintain that it is that aspect of the hierarchical structure which lends the property of spatial generality which is so hard to achieve in conventional recognition algorithms. > Even a parallel system evolves through time. We can treat the > vibration of a violin string as proceding in discrete time if our > sampling rate is high enough (for bandlimited behavior). Each moment > of this discrete time constitutes an iteration. To the extent that > the time constant of a biological neural network's behavior is finite, > we _do_ have to worry about how many iterations (how much time) it > takes for the system to arrive at a solution. I did not mean to suggest that resonance can be established instantaneously, of course it requires a finite time. I merely meant to say that in the case of resonance, a causal order (bowing causes string to resonate) does not necessarily imply a temporal order (first bowing then resonance) but that the resonance can emerge essentially simultaneous to the bowing, even though the bowing is the cause of the resonance. In the same way, I suggest that the fact that a higher level cell fires before the lower level cell, does not necessarily imply that it is therefore causally independant. Of course by firing I mean firing above ambient noise level, and I would assume that there is some a signal being sent from the lower level cell to the higher one, albeit a weak, noisy and incoherent signal, and that the higher level cell responds to and accentuates any global coherency that it detects in the cacophany of noisy inputs that it receives from many lower cells. The BCS model suggests that the output of the lower level cell is greatly boosted and enhanced when it receives top-down confirmation, or suppressed if it receives top-down disconfirmation, thus a global pattern detected higher up is reflected in the pattern of firing in the lowest levels of the hierarchy. It is this stronger, resonant firing of the low level cell that occurs AFTER the higher cell response, the initial firing might be lost in the noise. This arrangement seems eminantly plausible to me, accounting for a large body of psychophysical data including the ease with which local objects are recognized when they are consistant with the global picture, and conversely, the longer time required to recognize objects that are inconsistant with the global scene. It is clear that the global context plays a large role in local recognition, although of course the global context itself must be built up out of local pieces. How else can one account for these phenomena besides a simultaneous resonant matching between low and high level recognition? From R14502%BBRBFU01.BITNET at CUNYVM.CUNY.EDU Thu Jul 25 13:13:57 1991 From: R14502%BBRBFU01.BITNET at CUNYVM.CUNY.EDU (R14502%BBRBFU01.BITNET@CUNYVM.CUNY.EDU) Date: 25 Jul 91 17:14:57 +01 Subject: No subject Message-ID: Bruxelles, le 24 July 1991 Concerns : Information about ENNS. Thanks to all of your who have shown interest in ENNS. Your name will be on a list and you will get available information in due time. Here are answers to the questions you all asked. - To join, please send a check of 50 ECUs (European community currency) or its equivalent in any currency to : Prof. John Taylor King's College London Dept. of Mathematics University of London Stand, London WC2R 2LS England -The special interest groups are not yet formed and will materialize during ICANN 1992 in Brighton Professor Igor Aleksander is one of the organizers. His address is I. Aleksander Imperial College of Science and Technology Dept. of Computing 180 Queen's Gate London SW7 2B2 U.K. The question of bylaws will be settled by 1992 A. Babloyantz Secretary ENNS From BOYD at RHODES.BITNET Fri Jul 26 07:58:00 1991 From: BOYD at RHODES.BITNET (BOYD) Date: Fri, 26 Jul 91 05:58 CST Subject: unsubscribe Message-ID: <9EDDC4C9C0A00063@BITNET.CC.CMU.EDU> Please delete me from this list. From koch at CitIago.Bitnet Fri Jul 26 23:19:03 1991 From: koch at CitIago.Bitnet (Christof Koch) Date: Fri, 26 Jul 91 20:19:03 PDT Subject: Time to converge Message-ID: <910726201758.2040d436@Iago.Caltech.Edu> Re. the 90 msec response time for face cells in the temporal lobe. Most of that time is due to retinal elements. Given an average neuronal time-constant of cortical cells of 10 msec, this does not leave any time to iterate at all, given support to the idea that for the class of highly-overlearned patterns, such as faces, letters, etc. the brain essentially acts like a look-up table and does not compute in any real sense of the worl. This fits with Poggio's RBF approach or with Bartlet Mel's sigma-pi neurons. The almost infinite class of objects which we see only a few times in our life is much more interesting to investigate. Howvever, since we can respond to these objects with say, approx. 200 msec, we don;t have time for a lot of iterations, whether their digital cycles or analog time-constants. This is one reason the original Marr-Poggio cooperative stereo algoprith was so interesting, since it converged in 7-10 cycles. Finally, there exists no good physiological experiment (with the exception of loss of length inhibition in LGN; see Silito in Nature, 1989) showing that any functional property goes away after inactivation of a higher area. This is rather embarassing, given, for instance, the fact that at least 10-20 times more fibers project from layer VI in area 17 to LGN than from LGN to area 17. Christof P.S. That is no evidence for fast communication not involving action potentials in the brain. The distances are too big and all relevant biophysical mechanism except solitons too slow... C. From jose at tractatus.siemens.com Sat Jul 27 11:07:59 1991 From: jose at tractatus.siemens.com (Steve Hanson) Date: Sat, 27 Jul 1991 11:07:59 -0400 (EDT) Subject: Time to converge In-Reply-To: <910726201758.2040d436@Iago.Caltech.Edu> References: <910726201758.2040d436@Iago.Caltech.Edu> Message-ID: >entially acts like a look-up table and does not compute in any real >sense of the worl. This fits with Poggio's RBF approach or with >Bartlet Mel's sigma-pi neurons. Wouldn't this fit better with any feed-forward net...? and assuming lots of memorys systematically related a linear-logistic type net (more global)... Steve Stephen J. Hanson Learning & Knowledge Acquisition Group SIEMENS Research 755 College Rd. East Princeton, NJ 08540 From mclennan at cs.utk.edu Mon Jul 29 12:07:06 1991 From: mclennan at cs.utk.edu (mclennan@cs.utk.edu) Date: Mon, 29 Jul 91 12:07:06 -0400 Subject: convergence times Message-ID: <9107291607.AA28355@duncan.cs.utk.edu> The scope for iterative algorithms in the brain is much greater if they take place in the dendritic net via graded interactions. Since a chemical synapse has about a 0.5 msec. delay, 10 iterations might occur in 10 msec. As long as there are no action potentials in the loop, iteration can go pretty fast (in neural terms). For several decades now Gordon Shepherd has been stressing the importance of computation in the dendritic net. Bruce MacLennan Department of Computer Science 107 Ayres Hall The University of Tennessee Knoxville, TN 37996-1301 (615)974-0994/5067 maclennan at cs.utk.edu From terry at jeeves.UCSD.EDU Mon Jul 29 18:33:20 1991 From: terry at jeeves.UCSD.EDU (Terry Sejnowski) Date: Mon, 29 Jul 91 15:33:20 PDT Subject: convergence times Message-ID: <9107292233.AA03423@jeeves.UCSD.EDU> There is very little evidence for dendrodendritic interactions in cerebral cortex, although there is a lot in the olfactory bulb, thalamus, and retina. Thus, even short range cortical interactions must use action potentials. Since there is a dendritic delay of around 10 ms because of electrotonic conduction, the minimal cycle time is 10 ms. Terry ----- From koch at CitIago.Bitnet Mon Jul 29 19:17:59 1991 From: koch at CitIago.Bitnet (Christof Koch) Date: Mon, 29 Jul 91 16:17:59 PDT Subject: convergence times In-Reply-To: Your message <9107291607.AA28355@duncan.cs.utk.edu> dated 29-Jul-1991 Message-ID: <910729161754.2040995b@Iago.Caltech.Edu> The problem with graded interaction in the brain, mediated through dendro- dendritic synapses, is that for the most part they don't seem to exist in cortex proper. In the retina, in the thalamus, in the brain stem etc. but not in cortex. Yes, the hippocampus has some, in particular in young animals, but its certainly not a wide-spread phenomena. This leaves action-potentials or much slower active or passive transport mechanisms. Christof From gbugmann at nsis86.cl.nec.co.jp Mon Jul 29 20:07:50 1991 From: gbugmann at nsis86.cl.nec.co.jp (Masahide.Nomura) Date: Tue, 30 Jul 91 09:07:50+0900 Subject: time Message-ID: <9107300007.AA23502@nsis86.cl.nec.co.jp> C. Koch said: "The brain essentially acts like a look-up table and does not compute in any real sense of the worl. This fits with Poggio's RBF approach or with Bartlet Mel's sigma-pi neurons." Steve J. Hanson asked: "Wouldn't this fit better with any feed-forward net...?" In fact, both feedforward nets and Poggio's technique can be used to realize multidimensional mappings. The difference lies in the robustness. While a feedforward net mapping can be badly degraded by the loss of a neuron, the mapping is only locally degraded with Poggio's technique. Guido Bugmann Fundamental Research Laboratory NEC Corporation 34 Miyukigaoka Tsukuba, Ibaraki 305 Japan A A A A A A A A A A A A A A A From nelsonde%avlab.dnet at wrdc.af.mil Tue Jul 30 08:35:41 1991 From: nelsonde%avlab.dnet at wrdc.af.mil (nelsonde%avlab.dnet@wrdc.af.mil) Date: Tue, 30 Jul 91 08:35:41 EDT Subject: Taxonomy of Neural Network Optimality Message-ID: <9107301235.AA07942@wrdc.af.mil> I N T E R O F F I C E M E M O R A N D U M Date: 30-Jul-1991 08:25am EST From: Dale E. Nelson NELSONDE Dept: AAAT-1 Tel No: 57646 TO: Remote Addressee ( _LABDDN::"CONNECTIONISTS at CS.CMU.EDU" ) Subject: Taxonomy of Neural Network Optimality We are working on a taxonomy of parameters which may be used to determine if one network is better than another. To this end the following list has been developed. I would be interested in any comments or references to this kind of listing. PERFORMANCE: 1. Most accurate on training set. 2. Most accurate on test set. 3. Best at generalization. 4. Performance independent of starting weights. 5. Performance independent of training exemplar order. TRAINING: 6. Trains in fewest epochs. 7. Trains in fewest Floating Point/Integer Operations. 8. Trains in least clock time. 9. Trains in fewest exemplars. 10. Uses least memory. TOPOLOGY: 11. Has fewest layers. 12. Has fewest nodes. 13. Has fewest interconnects. 14. Distributed representation (fault tolerant) I know that there is no explaination of what each of these mean, which means that they are open to some interpretation. I would appreciate any comments about additions to this list. Dale E. Nelson nelsonde%avlab.dnet at wrdc.af.mil From erikf at sans.bion.kth.se Tue Jul 30 08:45:35 1991 From: erikf at sans.bion.kth.se (Erik Fransen) Date: Tue, 30 Jul 91 14:45:35 +0200 Subject: Time to converge References: <910726201758.2040d436@Iago.Caltech.Edu> Message-ID: <9107301245.AA00658@cerebellum> We have recently done some simulations of recurrent networks with realistic, spiking pyramidal cells (Hodgkin-Huxley type eq., multi-compartment model (1)) as units. Our results show that relaxation times are rather short (2). In case of a complete and undistorted pattern as stimuli, response time was around 25 ms. With incomplete or distorted or mixed patterns as stimuli, response time was around 50 ms. This has been done with a small network of 50 cells. Axonal plus synaptic delay times were 1 ms. Currently we are working on a much larger network. Thus, relaxation in recurrent cortical circuits seems compatible with a stimulus-response time of 90 ms. In a sequence of processing stages (retina, LGN, V1 ...) the first "leading wave" would take about 15 ms per stage. Priming could lower this time considerably. Actual computation in each stage will take place in parallel, but in cases of "familiar" inputs later responses will not differ much from the first. So, with "familiar" inputs the response will look like a pure feed-forward operation. Only with more complex inputs relaxation will modify the initial response. Our feeling is that in 200 ms a lot of multi-stage relaxations can take place... Erik Fransen Anders Lansner SANS Dept. of Numerical Analysis and Computing Sci. Royal Inst. of Technology, Stockholm (1) Ekeberg, Wallen, Lansner, Traven, Brodin, Grillner (1991), A Computer Based Model for Realistic Simulations of Neural Networks, (to appear in Biol. Cybernetics) (2) Lansner & Fransen (1991), Modeling Hebbian Cell Assemblies Comprised of Cortical Neurons, (Submitted) From petsche at learning.siemens.com Tue Jul 30 09:24:44 1991 From: petsche at learning.siemens.com (Thomas Petsche) Date: Tue, 30 Jul 91 09:24:44 EDT Subject: time In-Reply-To: <9107300007.AA23502@nsis86.cl.nec.co.jp> Message-ID: <9107301324.AA02004@learning.siemens.com.siemens.com> Masahide.Nomura wrote: >In fact, both feedforward nets and Poggio's technique can be used to >realize multidimensional mappings. The difference lies in the robustness. >While a feedforward net mapping can be badly degraded by the loss of a >neuron, the mapping is only locally degraded with Poggio's technique. But this is not a property of a feedforward network (or RBF's for that matter). For FF nets, the lack of predictable and dependable fault tolerance is a property of `vanilla' back prop. For backprop with weight decay, we can accurately predict that the network will be completely INtolerant to any faults. OTOH, it is quite possible to obtain a fault tolerant feedforward network by (1) designing a fault tolerant generic network which can then be trained [1] or (2) modifying backprop to encourage fault tolerant representations [as yet unpublished work by other researchers]. [1] @article{petsche-dickinson-1990, author = {T. Petsche and B.W. Dickinson}, journal = {IEEE Transactions on Neural Networks}, month = jun, number = {2}, pages = {154--166}, title = {Trellis Codes, Receptive Fields, and Fault-Tolerant, Self-Repairing Neural Networks}, volume = {1}, year = {1990}, note={Errata for eqn 1 available from first author.} } From lyle at ai.mit.edu Tue Jul 30 11:43:20 1991 From: lyle at ai.mit.edu (Lyle J. Borg-Graham) Date: Tue, 30 Jul 91 11:43:20 EDT Subject: convergence times In-Reply-To: mclennan@cs.utk.edu's message of Mon, 29 Jul 91 12:07:06 -0400 <9107291607.AA28355@duncan.cs.utk.edu> Message-ID: <9107301543.AA22780@peduncle> might occur in 10 msec. As long as there are no action potentials in the loop, iteration can go pretty fast (in neural terms). For several decades now Gordon Shepherd has been stressing the importance of computation in the dendritic net. It seems pretty likely that intradendritic interactions (electrical or chemical) are important; at least the biophysical substrate is rich enough, and especially so after adding in the morphometrics. Finding the smoking gun (as opposed to simulations which show plausibility) is probably imminent. But iterations, strictly speaking, require a discrete *loop*, and the speed of such a pathway is of course limited by the slowest element in the chain, not by the fastest (e.g. some 500us *component* of a synaptic event). I think that iterative mechanisms constitute one class of interactions, while (continous) feedback (which can be very fast) mechanisms are another. From Scott.Fahlman at SEF-PMAX.SLISP.CS.CMU.EDU Tue Jul 30 12:33:03 1991 From: Scott.Fahlman at SEF-PMAX.SLISP.CS.CMU.EDU (Scott.Fahlman@SEF-PMAX.SLISP.CS.CMU.EDU) Date: Tue, 30 Jul 91 12:33:03 EDT Subject: Taxonomy of Neural Network Optimality In-Reply-To: Your message of Tue, 30 Jul 91 08:35:41 -0400. <9107301235.AA07942@wrdc.af.mil> Message-ID: PERFORMANCE: 1. Most accurate on training set. 2. Most accurate on test set. 3. Best at generalization. What does this mean if not the same as 2? Also, "most accurate" might mean number of cases wrong or something like sum-squared error, depending on the problem. 4. Performance independent of starting weights. 5. Performance independent of training exemplar order. TRAINING: 6. Trains in fewest epochs. Some problems and algorithms just don't fit into epochs. Probably better to use "pattern presentations", but some algorithms don't even fit into that. 7. Trains in fewest Floating Point/Integer Operations. 8. Trains in least clock time. Machine-dependent, of course, so it says very little about the algorithm. 9. Trains in fewest exemplars. 10. Uses least memory. TOPOLOGY: 11. Has fewest layers. "Layers" may be ill-defined. Maybe look instead at the longest path from input to output. 12. Has fewest nodes. 13. Has fewest interconnects. 14. Distributed representation (fault tolerant) A few others: 15. How hard it is to partition/parallelize the algorithm? 16. How many parameters must the user adjust, and how critical are the adjustments? 17. Related to 16: What chance of immediate success on a new problem? 18. Range of problems covered: Discrete vs. analog inputs and outputs Can it handle time series? Can it handle noisy data? (i.e misclassifying a few training points leads to better generalization) 19. {Debugged, supported, portable, free} implementation available? 20. If not, how hard it the algorithm to implement? 21. Biologically plausible? 22. How does it scale with problem size? -- Scott From nmg at skivs.ski.org Tue Jul 30 13:10:38 1991 From: nmg at skivs.ski.org (nmg@skivs.ski.org) Date: 30 Jul 91 10:10:38 PDT (Tue) Subject: convergence times Message-ID: <9107301010.AA25081@skivs.ski.org> Two cautionary notes on Sejnowski's latest message: 1) I presume that the 10 msec estimate for cycle time comes from typical values of membrane time constant. However, this estimate might be substantially off under physiological conditions. Under these conditions, cell membranes might be constantly bathed in neurotransmitters, which by increasing membrane conductance, reduce the cell's time constants. Moreover, voltage-dependent ionic channels in dendritic trees might considerably reduce electrotonic conductance times. Hence, it would not be surprising if the contribution of electrotonic condution to "cycle time" is significantly smaller than 1 msec. 2) It is true that conventional dendrodendritic synapses appear to be essentially inexistent in the neocortex. But one must remember that this statement relates to vesicle-containing synapses identified with electron-microscopy methods. In the retina, non-vesicular synapses have been observed. Although, once again, the retina and the cortex have several differences, it seems that it would be hard to rule out the existence of such non-vesicular synapses in the cortex based on available data. And if one may speculate, then why not to have this type of synapse being dendrodendritic. Anyway, my point is that it might be too soon to state that action potentials mediate all cortical local synaptic circuits. Norberto From sloman%meme at Forsythe.Stanford.EDU Tue Jul 30 13:32:26 1991 From: sloman%meme at Forsythe.Stanford.EDU (sloman%meme@Forsythe.Stanford.EDU) Date: Tue, 30 Jul 91 10:32:26 PDT Subject: manuscript available: Feature-based induction Message-ID: <9107301732.AA08753@meme> A compressed postscript version of the following paper has been placed in the pub/neuroprose directory for anonymous ftp from cheops.cis.ohio-state.edu. The paper concerns a very simple connectionist model (n inputs, one output, and delta-rule learning) of people's willingness to affirm a property of one natural-kind category given confirmation of the property in other categories. The paper has been submitted for publication. Feature-Based Induction Steven A. Sloman Dept. of Psychology University of Michigan e-mail: sloman at psych.stanford.edu Abstract A connectionist model of argument strength is proposed that applies to categorical arguments involving natural categories and predicates about which subjects have few prior beliefs. An example is *robins have sesamoid bones, therefore falcons have sesamoid bones*. The model is based on the hypotheses that argument strength (i) increases with the overlap between features of the combined premise categories and features of the conclusion category; and (ii) decreases with the amount of prior knowledge about the conclusion category. The model assumes a two-stage process. First, premises are encoded by connecting the features of premise categories to the predicate. Second, conclusions are tested by examining the degree of activation of the predicate upon presentation of the features of the conclusion category. The model accounts for 13 qualitative phenomena and shows close quantitative fits to several sets of argument-strength ratings. From SCHOLTES at ALF.LET.UVA.NL Tue Jul 30 21:10:00 1991 From: SCHOLTES at ALF.LET.UVA.NL (SCHOLTES) Date: Tue, 30 Jul 91 21:10 MET Subject: Neural Nets in Information Retrieval Message-ID: <0892B405AAA00063@BITNET.CC.CMU.EDU> Recently, we started some research in the possible applications of Neural Nets in Information Retrieval. At this moment we are trying to compile a list of literature references and names of other people interested in this subject. Any information would be greatly appreciated. I will put the final list on the network. Thanks, Jan Scholtes ******************************************************************************* Jan C. Scholtes University of Amsterdam Faculty of Arts Department of Computational Linguistics Dufaystraat 1 1075 GR AMSTERDAM The Netherlands Tel: +31 20 6794273 Fax: +31 20 6710793 Email: scholtes at alf.let.uva.nl ******************************************************************************* From BOYD at RHODES.BITNET Wed Jul 31 07:57:00 1991 From: BOYD at RHODES.BITNET (BOYD) Date: Wed, 31 Jul 91 05:57 CST Subject: UNSUBSCRIBE Message-ID: <8C8E36FD0AA00063@BITNET.CC.CMU.EDU> UNSUBSCRIBE From BOYD at RHODES.BITNET Wed Jul 31 07:58:00 1991 From: BOYD at RHODES.BITNET (BOYD) Date: Wed, 31 Jul 91 05:58 CST Subject: SIGNOFF Message-ID: <8CBC76486AA00063@BITNET.CC.CMU.EDU> SIGNOFF From kamil at apple.com Wed Jul 31 10:01:10 1991 From: kamil at apple.com (Kamil A. Grajski) Date: Wed, 31 Jul 91 07:01:10 -0700 Subject: Biblio Request - NN Implementations Message-ID: <9107311401.AA15358@apple.com> Hi folks, I am interested in receiving bibliographic information concerning the implementation of connectionist architectures on existing machines. Beyond basic back-prop benchmarks, I'm interested in the braod range of machine architectures which have been explored, as well as the full range of connectionist networks. I'd like to focus more on machines than custom chips, etc., for the moment. I'd especially like to hear from our European and Pacific colleagues. An organized bibliography will be posted to the board. Thanks in advance, Kamil P.S. If you happen to have a reprint you'd like to share, I'm at: Kamil A. Grajski Apple Computer Inc. 20525 Mariani Avenue Mail Stop 76-7E Cupertino, CA 95014 kamil at apple.com (408) 974-1313 From issnnet at park.bu.edu Wed Jul 31 14:04:39 1991 From: issnnet at park.bu.edu (issnnet@park.bu.edu) Date: Wed, 31 Jul 91 14:04:39 -0400 Subject: comp.org.issnnet PASSES Message-ID: <9107311804.AA29359@copley.bu.edu> VOTE RESULT: comp.org.issnnet PASSES (188 YES / 22 NO) ------------------------------------------------------ VOTING PERIOD ENDED JULY 25, 1991 GROUP NAME: comp.org.issnnet STATUS: unmoderated CHARTER: The newsgroup shall serve as a medium for discussions pertaining to the International Student Society for Neural Networks (ISSNNet), Inc., and to its activities and programs as they pertain to the role of students in the field of neural networks. Details were posted in the REQUEST FOR DISCUSSION, and can be requested from . In accord with USENET guidelines on creation of new newsgroups, the proposed group passes because YES votes exceed NO votes by more than 100 votes, and more than 2/3 of all votes were in favor of creation. A complete list of YES and NO voters can be found on the newsgroups comp.ai.neural-nets and news.announce.newgroups, or may be requested from issnnet at park.bu.edu. Any discrepancies in this list should be brought to the immediate attention of issnnet at park.bu.edu. The new group will be created after a 5 day waiting period (scheduled on August 4) unless serious objections are raised over these results. ------------------------------------------------------------------------ >> Many thanks to all the voters who supported creation of this new << >> group. See you on comp.org.issnnet in a few days! << ------------------------------------------------------------------------ ISSNNet, Inc. is a non-profit corporation in the Commonwealth of Massachusetts. NOTE -- NEW SURFACE ADDRESS: ISSNNet, Inc. P.O. Box 15661 Boston, MA 02215 USA Contact person: Paolo Gaudiano, ISSNNet Vice President Department of Cognitive and Neural Systems Boston University 111 Cummington Street Boston, MA 02215 Phone: (617) 353-6181 353-9482 e-mail: gaudiano at park.bu.edu (INTERNET) From Connectionists-Request at CS.CMU.EDU Mon Jul 1 00:05:25 1991 From: Connectionists-Request at CS.CMU.EDU (Connectionists-Request@CS.CMU.EDU) Date: Mon, 01 Jul 91 00:05:25 EDT Subject: Bi-monthly Reminder Message-ID: <14317.678341125@B.GP.CS.CMU.EDU> *** DO NOT FORWARD TO ANY OTHER LISTS *** This is an automatically posted bi-monthly reminder about how the CONNECTIONISTS list works and how to access various online resources. CONNECTIONISTS is not an edited forum like the Neuron Digest, or a free-for-all newsgroup like comp.ai.neural-nets. It's somewhere in between, relying on the self-restraint of its subscribers. Membership in CONNECTIONISTS is restricted to persons actively involved in neural net research. The following posting guidelines are designed to reduce the amount of irrelevant messages sent to the list. Before you post, please remember that this list is distributed to over a thousand busy people who don't want their time wasted on trivia. Also, many subscribers pay cash for each kbyte; they shouldn't be forced to pay for junk mail. Happy hacking. -- Dave Touretzky & Scott Crowder --------------------------------------------------------------------- What to post to CONNECTIONISTS ------------------------------ - The list is primarily intended to support the discussion of technical issues relating to neural computation. - We encourage people to post the abstracts of their latest papers and tech reports. - Conferences and workshops may be announced on this list AT MOST twice: once to send out a call for papers, and once to remind non-authors about the registration deadline. A flood of repetitive announcements about the same conference is not welcome here. - Requests for ADDITIONAL references. This has been a particularly sensitive subject lately. Please try to (a) demonstrate that you have already pursued the quick, obvious routes to finding the information you desire, and (b) give people something back in return for bothering them. The easiest way to do both these things is to FIRST do the library work to find the basic references, then POST these as part of your query. Here's an example: WRONG WAY: "Can someone please mail me all references to cascade correlation?" RIGHT WAY: "I'm looking for references to work on cascade correlation. I've already read Fahlman's paper in NIPS 2, his NIPS 3 abstract, and found the code in the nn-bench archive. Is anyone aware of additional work with this algorithm? I'll summarize and post results to the list." - Announcements of job openings related to neural computation. - Short reviews of new text books related to neural computation. To send mail to everyone on the list, address it to Connectionists at CS.CMU.EDU ------------------------------------------------------------------- What NOT to post to CONNECTIONISTS: ----------------------------------- - Requests for addition to the list, change of address and other administrative matters should be sent to: "Connectionists-Request at cs.cmu.edu" (note the exact spelling: many "connectionists", one "request"). If you mention our mailing list to someone who may apply to be added to it, please make sure they use the above and NOT "Connectionists at cs.cmu.edu". - Requests for e-mail addresses of people who are believed to subscribe to CONNECTIONISTS should be sent to postmaster at appropriate-site. If the site address is unknown, send your request to Connectionists-Request at cs.cmu.edu and we'll do our best to help. A phone call to the appropriate institution may sometimes be simpler and faster. - Note that in many mail programs a reply to a message is automatically "CC"-ed to all the addresses on the "To" and "CC" lines of the original message. If the mailer you use has this property, please make sure your personal response (request for a Tech Report etc.) is NOT broadcast over the net. - Do NOT tell a friend about Connectionists at cs.cmu.edu. Tell him or her only about Connectionists-Request at cs.cmu.edu. This will save your friend from public embarrassment if she/he tries to subscribe. - Limericks should not be posted here. ------------------------------------------------------------------------------- The CONNECTIONISTS Archive: --------------------------- All e-mail messages sent to "Connectionists at cs.cmu.edu" starting 27-Feb-88 are now available for public perusal. A separate file exists for each month. The files' names are: arch.yymm where yymm stand for the obvious thing. Thus the earliest available data are in the file: arch.8802 Files ending with .Z are compressed using the standard unix compress program. To browse through these files (as well as through other files, see below) you must FTP them to your local machine. ------------------------------------------------------------------------------- How to FTP Files from the CONNECTIONISTS Archive ------------------------------------------------ 1. Open an FTP connection to host B.GP.CS.CMU.EDU (Internet address 128.2.242.8). 2. Login as user anonymous with password your username. 3. 'cd' directly to one of the following directories: /usr/connect/connectionists/archives /usr/connect/connectionists/bibliographies 4. The archives and bibliographies directories are the ONLY ones you can access. You can't even find out whether any other directories exist. If you are using the 'cd' command you must cd DIRECTLY into one of these two directories. Access will be denied to any others, including their parent directory. 5. The archives subdirectory contains back issues of the mailing list. Some bibliographies are in the bibliographies subdirectory. Problems? - contact us at "Connectionists-Request at cs.cmu.edu". ------------------------------------------------------------------------------- How to FTP Files from the Neuroprose Archive -------------------------------------------- Anonymous FTP on cheops.cis.ohio-state.edu (128.146.8.62) pub/neuroprose directory This directory contains technical reports as a public service to the connectionist and neural network scientific community. Researchers may place electronic versions of their preprints or articles in this directory, announce availability, and other interested researchers can rapidly retrieve and print the postscripts. This saves copying, postage and handling, by having the interested reader supply the paper. To place a file, put it in the Inbox subdirectory, and send mail to pollack at cis.ohio-state.edu. Within a couple of days, I will move and protect it, and suggest a different name if necessary. Current naming convention is author.title.filetype[.Z] where title is enough to discriminate among the files of the same author. The filetype is usually "ps" for postscript, our desired universal printing format, but may be tex, which requires more local software than a spooler. Very large files (e.g. over 200k) must be squashed (with either a sigmoid function :) or the standard unix "compress" utility, which results in the .Z affix. To place or retrieve .Z files, make sure to issue the FTP command "BINARY" before transfering files. After retrieval, call the standard unix "uncompress" utility, which removes the .Z affix. An example of placing a file is attached as an appendix, and a shell script called Getps in the directory can perform the necessary retrival operations. For further questions contact: Jordan Pollack Email: pollack at cis.ohio-state.edu Here is an example of naming and placing a file: gvax> cp i-was-right.txt.ps rosenblatt.reborn.ps gvax> compress rosenblatt.reborn.ps gvax> ftp cheops.cis.ohio-state.edu Connected to cheops.cis.ohio-state.edu. 220 cheops.cis.ohio-state.edu FTP server ready. Name: anonymous 331 Guest login ok, send ident as password. Password:neuron 230 Guest login ok, access restrictions apply. ftp> binary 200 Type set to I. ftp> cd pub/neuroprose/Inbox 250 CWD command successful. ftp> put rosenblatt.reborn.ps.Z 200 PORT command successful. 150 Opening BINARY mode data connection for rosenblatt.reborn.ps.Z 226 Transfer complete. 100000 bytes sent in 3.14159 seconds ftp> quit 221 Goodbye. gvax> mail pollack at cis.ohio-state.edu Subject: file in Inbox. Jordan, I just placed the file rosenblatt.reborn.ps.Z in the Inbox. The INDEX sentence is "Boastful statements by the deceased leader of the neurocomputing field." Please let me know when it is ready to announce to Connectionists at cmu. BTW, I enjoyed reading your review of the new edition of Perceptrons! Frank ------------------------------------------------------------------------ How to FTP Files from the NN-Bench Collection --------------------------------------------- 1. Create an FTP connection from wherever you are to machine "pt.cs.cmu.edu" (128.2.254.155). 2. Log in as user "anonymous" with password your username. 3. Change remote directory to "/afs/cs/project/connect/bench". Any subdirectories of this one should also be accessible. Parent directories should not be. 4. At this point FTP should be able to get a listing of files in this directory and fetch the ones you want. Problems? - contact us at "nn-bench-request at cs.cmu.edu". From schmidhu at kiss.informatik.tu-muenchen.de Mon Jul 1 08:04:48 1991 From: schmidhu at kiss.informatik.tu-muenchen.de (Juergen Schmidhuber) Date: Mon, 1 Jul 91 14:04:48 +0200 Subject: change of address Message-ID: <9107011204.AA03458@kiss.informatik.tu-muenchen.de> Old address until July 4, 1991: Juergen Schmidhuber Institut fuer Informatik, Technische Universitaet Muenchen Arcisstr. 21 8000 Muenchen 2 GERMANY fax: Germany (49) Munich (89) 2105 8207 email: schmidhu at informatik.tu-muenchen.de New address after July 4, 1991: Juergen Schmidhuber Department of Computer Science University of Colorado Campus Box 430 Boulder, CO 80309, USA fax: (303) 492 2844 email: yirgan at neuron.colorado.edu From haussler at saturn.ucsc.edu Mon Jul 1 18:57:37 1991 From: haussler at saturn.ucsc.edu (David Haussler) Date: Mon, 1 Jul 91 15:57:37 -0700 Subject: don't forget to register for COLT '91 Message-ID: <9107012257.AA12399@saturn.ucsc.edu> Don't forget to register for the Workshop on Computational Learning Theory Monday, August 5 through Wednesday, August 7, 1991 University of California, Santa Cruz, California Conference information, program and room registration forms can be obtained by anonymous FTP. Connect to midgard.ucsc.edu and look in the directory pub/colt. Alternatively, send E-mail to "colt at cis.ucsc.edu" for instructions on obtaining the forms by electronic mail. Early registration discounts are ending NOW. If you have questions regarding registration or accommodations, contact: Jean McKnight, COLT '91, Dept. of Computer Science, UCSC, Santa Cruz, CA 95064. Her emergency phone number is (408) 459-2303, but she prefers E-mail to jean at cs.ucsc.edu or facsimile at (408) 429-0146. Proceedings from the workshop will be published by Morgan Kaufmann. -David From dwunsch at atc.boeing.com Mon Jul 1 21:15:00 1991 From: dwunsch at atc.boeing.com (Don Wunsch) Date: Mon, 1 Jul 91 18:15:00 PDT Subject: Various IJCNN-91-Seattle info. Message-ID: <9107020115.AA22035@atc.boeing.com> IJCNN-91-Seattle is almost upon us! Come to Seattle July 8-12 for the largest neural network, hybrid, and fuzzy systems conference in the world. Researchers, scientists, engineers, consultants, applications specialists and students from a variety of disciplines will present invited and contributed papers, tutorials, panel discussions, exhibits and demos. Exhibitors will also present the latest in neural networks, including neurocomputers, VLSI neural networks, software systems and applications. Conference registration is $395 for IEEE or INNS members, $495 for non-members, and $95 for students. Tutorials are $295 for everyone except students, who pay $85. One-day registration is $125 for members or $175 for non-members. The tutorial fee includes all the tutorials you choose to attend, not just one. Also, students are full conference participants, receiving a proceedings and admission to all events, including social events. To register by VISA or MasterCard, call (206) 543-2310, or FAX a registration form (containing your name, title, affiliation, full address, country, full daytime phone number, FAX number, e-mail address, and registration for the conference or tutorials, with total fees and credit card number with name as it appears on card) to (206) 685-9359. Finally, for those of you who will be attending the conference, you have an opportunity to attend a tour of the world's largest manufacturing facility. Read on for further details and reservation information: Boeing will be offering a limited number of free tours of its manufacturing plant to attendees of the International Joint Conference on Neural Networks, on a first-come, first-served basis. Families may also be accomodated, but only if space is available. See the 747 and 767 being built, in the largest volume building in the world. We will also be providing transportation leaving he conference site one hour before the tour begins. The tour lasts approximately one hour. Tour times are Monday through Friday at 10:30 AM and 3:00 PM. Therefore, be prepared to leave the conference site at 9:30 AM or 2:00 PM for your chosen tour. Please indicate your preferred tour times and send a fax of this form with your name, number of persons on the tour, and hotel (if known). Preferred time:_________________ Alternate choice:___________________ Second alternate:_____________________________________________________ Name (please print): _________________________________________________ Number of persons: __________ Hotel:___________________________ Please do not reply to this address. Instead, please send a FAX to Jean Norris at (206) 865-2957. If you have any questions, call her at (206) 865-5616. Alternatively, you may e-mail the form or any questions to David Newman at: dnewman at caissa.boeing.com. If you will not be staying at a hotel, please put down a number where you can be reached. There will be a tour coordination booth available at the time of registration, where further information is available, and tickets may be picked up. The best time to schedule your tour will probably be Monday, before things start heating up. This tour is actually a very hot tourist attraction around here, because of the universal fascination with flight. So it will be well worth your while--don't miss it! We look forward to seeing you in Seattle! Don Wunsch Local Arrangements Chair IJCNN-91-Seattle From magnuson at mcc.com Tue Jul 2 10:26:38 1991 From: magnuson at mcc.com (Tim Magnuson) Date: Tue, 2 Jul 91 09:26:38 CDT Subject: Job Announcement: MCC Message-ID: <9107021426.AA09964@brush.aca.mcc.com> MCC, a leading U.S. cooperative research organization in Austin, Texas, currently seeks two individuals: a project manager and a research scientist for the MCC Neural Network Project. MCC's neural network research takes both a theoretical and a practical form. The theoretical work includes the development and analysis of neural network algorithms and architectures. The practical work takes the form of large-scale, real-world application projects. The specific focus of the MCC Neural Network Project is handwritten character recognition. Current research projects are focused on concurrent segmentation and recognition of handwritten symbols, including numeric digits, upper and lower case alpha characters, punctuation characters, and cursive handwriting. The requirements for the research scientist position include a Ph.D. in a discipline related to neural networks, a proven track record for leading edge research in neural networks, and experience in handwritten character recognition using neural networks. The requirements for the research manager are a M.S. or Ph.D. in a discipline related to neural networks, strong interpersonal and organizational skills, and a working knowledge of neural networks. Management experience in handwritten character recognition projects is a definite plus. MCC is well positioned as a conduit between university research and industrial application due to strong contacts on both sides. In addition to an excellent compensation/benefits program, MCC offers an outstanding corporate environment and the opportunity to conduct ground breaking research. MCC will be interviewing candidates at IJCNN in Seattle on July 11th and 12th. Interviews will be by appointment only. Interested parties should mail their credentials, including salary history and requirements to: Roger Malek Manager, Human Resources MCC 3500 West Balcones Center Drive Austin, Texas 78759 or FAX them to Roger Malek at 512-338-3888 or E-mail to Malek at mcc.com From Connectionists-Request at CS.CMU.EDU Tue Jul 2 11:16:14 1991 From: Connectionists-Request at CS.CMU.EDU (Connectionists-Request@CS.CMU.EDU) Date: Tue, 02 Jul 91 11:16:14 EDT Subject: IJCNN request Message-ID: <4175.678467774@B.GP.CS.CMU.EDU> > IJCNN-91-Seattle is almost upon us! Come to Seattle July 8-12 > for the largest neural network, hybrid, and fuzzy systems > conference in the world. Researchers, scientists, engineers, > consultants, applications specialists and students from a variety > of disciplines will present invited and contributed papers, tutorials, > panel discussions, exhibits and demos. Exhibitors will also present > the latest in neural networks, including neurocomputers, VLSI neural > networks, software systems and applications. We typically get a flood of new member requests after each IJCNN. Many of them get broadcast to all 900+ members of CONNECTIONISTS. Since I am the only one that can add anybody to the list, we can save *everybody* some time by remembering the following. Do NOT tell your new friend about Connectionists at cs.cmu.edu. Tell him or her only about Connectionists-Request at cs.cmu.edu. This will save your friend from public embarrassment if she/he tries to subscribe. It will save *me* some time if you remember to tell your new friend to include their full name and a 1 sentence description of their current work in their request message. I send the standard restricted membership note to *everyone* whose request does not contain these two items. Sometimes people get their feelings hurt that I don't recognize them as the author of the most-talked-about new paper. The truth is that I don't even read the name if there is no work description. -Scott Crowder Connectionists-Request at cs.cmu.edu (ARPAnet) From dwunsch at atc.boeing.com Tue Jul 2 13:37:08 1991 From: dwunsch at atc.boeing.com (Don Wunsch) Date: Tue, 2 Jul 91 10:37:08 PDT Subject: Boeing plant tours for IJCNN-91-Seattle Message-ID: <9107021737.AA27543@atc.boeing.com> In a recent note, I wrote: > IJCNN-91-Seattle is almost upon us! Come > ... > Boeing will be offering a limited number of free tours > ... > first-served basis. Families may also be accomodated, but > only if space is available. See the 747 and 767 being built, > ... Forgot to mention--the minimum age for the Boeing plant tour is twelve years old. Hope to see you there! Don From fritzke at immd2.informatik.uni-erlangen.de Wed Jul 3 09:22:20 1991 From: fritzke at immd2.informatik.uni-erlangen.de (B. Fritzke) Date: Wed, 3 Jul 91 9:22:20 MET DST Subject: TSP paper available Message-ID: <9107030723.AA06096@faui28.informatik.uni-erlangen.de> Hi connectionists, I just placed a paper in the Neuroprose Archive, which has been submitted to IJCNN-91 Singapore. The filename is: fritzke.linear_tsp.ps.Z And here's the abstract: FLEXMAP -- A Neural Network For The Traveling Salesman Problem With Linear Time And Space Complexity Bernd FRITZKE and Peter WILKE We present a self-organizing ''neural'' network for the traveling salesman problem. It is partly based on the model of Kohonen. Our approach differs from former work in this direction as no ring structure with a fixed number of elements is used. Instead a small initial structure is enlarged during a distribution pro- cess. This allows us to replace the central search step, which normally needs time O(n), by a local procedure that needs time O(1). Since the total number of search steps we have to perform is O(n) the runtime of our model scales linear with problem size. This is better than every known neural or conventional algorithm. The path lengths of the generated solutions are less than 9 per- cent longer than the optimum solutions of solved problems from the literature. The described network is based on: > Fritzke, B., "Let it grow - self-organizing feature maps with > problem dependent cell structure," Proc. of ICANN-91, Helsinki, > 1991, pp. 403-408. (see the previously placed file fritzke.cell_structures.ps.Z) Furthermore related work will be presented next week in Seattle: > Fritzke, B., "Unsupervised clustering with growing cell struc- > tures," to appear in: Proc. of IJCNN-91, Seattle, 1991. (see the previously placed file fritzke.clustering.ps.Z and the Poster No. W19 in Seattle) See you in Seattle, Bernd Bernd Fritzke ----------> e-mail: fritzke at immd2.informatik.uni-erlangen.de University of Erlangen, CS IMMD II, Martensstr. 3, 8520 Erlangen (Germany) From R09614 at BBRBFU01.BITNET Wed Jul 3 11:19:37 1991 From: R09614 at BBRBFU01.BITNET (R09614) Date: Wed, 3 Jul 91 17:19:37 +0200 Subject: European Neural Network Society Message-ID: PRESS RELEASE During ICANN91, the International Conference on Artificial Neural Networks, held in Helsinki, June 24-28, 1991, ENNS, the European Neural Network Society has been created. Its main objectives are to sponsor furture ICANN conferences and organize summer schools in the area, and to develop a quarterly newsletter and electronic bulletin board that will keep members informed of developments in the field of artificial and biological neural networks. It is expected that Special Interest Groups will be formed in the future under the umbrella of the society. The new society welcomes membership worldwide from those interested in Neural Network research. One of the benefits will be de reduction of fees for all future activities of the society. The current officiers of the ENNS are: President Tuevo Kohonen Helsinki University of Technology Espoo, Finland Vice Presidents Igor Aleksander Imperial College London London, U.K. Rolf Eckmiller Heinrich Heine University Dusseldorf, F.R.G. John Taylor King's College London London, U.K. Secretary Agnessa Babloyantz Universite Libre de Bruxelles Brussels, Belgium Treasurer Rodney Cotterill Technical University of Denmark Lyngby, Denmark For further information, please contact A. Babloyantz at: Universite Libre de Bruxelles Phone: 32 - 2 - 650 55 40 CP 231 - Campus Plaine, Fax: 32 - 2 - 650 57 67 Boulevard du Triomphe, E-mail: adestex @ bbrbfu60.bitnet B-1050 Bruxelles, BELGIUM From WEYMAERE at lem.rug.ac.be Wed Jul 3 16:09:00 1991 From: WEYMAERE at lem.rug.ac.be (WEYMAERE@lem.rug.ac.be) Date: Wed, 3 Jul 91 16:09 N Subject: New Paper: A Fast and Robust Learning Algorithm for Feedforward NN Message-ID: <01G7QU7KJYXS0000SV@BGERUG51.BITNET> The following paper has appeared in "Neural Networks", Vol. 4, No 3 (1991), pp 361-369: -------------------------------------------------------------------------------- A Fast and Robust Learning Algorithm for Feedforward Neural Networks Nico WEYMAERE and Jean-Pierre MARTENS Laboratorium voor Elektronika en Meettechniek Rijksuniversiteit Gent Gent, Belgium ABSTRACT The back-propagation algorithm caused a tremendous break-through in the application of multilayer perceptrons. However, it has some important drawbacks: long training times and sensitivity to the presence of local minima. Another problem is the network topology: the exact number of units in a particular hidden layer, as well as the number of hidden layers need to be known in advance. A lot of time is often spent in finding the optimal topology. In this paper, we consider multilayer networks with one hidden layer of Gaussian units and an output layer of conventional units. We show that for this kind of networks, it is possible to perform a fast dimensionality analysis, by analyzing only a small fraction of the input patterns. Moreover, as a result of this approach, it is possible to initialize the weights of the network before starting the back-propagation training. Several classification problems are taken as examples. -------------------------------------------------------------------------------- Unfortunately, there is not an electronic version of this paper. Reprint requests should be sent to : Weymaere Nico Laboratorium voor Elektronika en Meettechniek St. Pietersnieuwstraat 41 - B9000 Gent From R09614 at BBRBFU01.BITNET Fri Jul 5 12:51:12 1991 From: R09614 at BBRBFU01.BITNET (R09614) Date: Fri, 5 Jul 91 18:51:12 +0200 Subject: EUROPEAN NEURAL NETWORK SOCIETY Message-ID: <4FA5C21E6080385F@BITNET.CC.CMU.EDU> EUROPEAN NEURAL NETWORK SOCIETY Due to connection problems, some E-mails did not arrive to the address given in the first announcement. Please send requests or comments for ENNS at the following address: R09614 at BBRBFU01.bitnet If you had already sent a message, please re-send a copy to this address. Some files have been lost. Thank you From wray at ptolemy.arc.nasa.gov Fri Jul 5 14:19:46 1991 From: wray at ptolemy.arc.nasa.gov (Wray Buntine) Date: Fri, 5 Jul 91 11:19:46 PDT Subject: two new papers on back-prop available from neuroprose Message-ID: <9107051819.AA04256@ptolemy.arc.nasa.gov> The following two reports are currently under journal review and have been made available on the "/pub/neuroprose" archive. Those unable to access this should send requests to the address below. Both papers are intended as a guide for the "theoretically-aware practitioner/algorithm-designer intent on building a better algorithm". Wray Buntine NASA Ames Research Center phone: (415) 604 3389 Mail Stop 244-17 Moffett Field, CA, 94035 email: wray at ptolemy.arc.nasa.gov ---------------- Bayesian Back-Propagation by Wray L. Buntine and Andreas S. Weigend available as /pub/neuroprose/buntine.bayes1.ps.Z (pages 1-17) /pub/neuroprose/buntine.bayes2.ps.Z (pages 1-34) Connectionist feed-forward networks, trained with back-propagation, can be used both for non-linear regression and for (discrete one-of-$C$) classification, depending on the form of training. This paper works through approximate Bayesian methods to both these problems. Methods are presented for various statistical components of back-propagation: choosing the appropriate cost function and regularizer (interpreted as a prior), eliminating extra weights, estimating the uncertainty of the remaining weights, predicting for new patterns (``out-of-sample''), estimating the uncertainty in the choice of this prediction (``error bars''), estimating the generalization error, comparing different network structures, and adjustments for missing values in the training patterns. These techniques refine and extend some popular heuristic techniques suggested in the literature, and in most cases require at most a small additional factor in computation during back-propagation, or computation once back-propagation has finished. The paper begins with a comparative discussion of Bayesian and related frameworks for the training problem. Contents: 1. Introduction 2. On Bayesian methods 3. Multi-Layer networks 4. Probabilistic neural networks 4.1. Logistic networks 4.2. Cluster networks 4.3. Regression networks 5. Probabilistic analysis 5.1. The network likelihood function 5.2. The sample likelihood 5.3. Prior probability of the weights 5.4. Posterior analysis 6. Analyzing weights 6.1. Cost functions 6.2. Weight evaluation 6.3. Minimum encoding methods 7. Applications to network training 7.1. Weight variance and elimination 7.2. Prediction and generalization error 7.3. Adjustments for missing values 8. Conclusion ----------------------- Calculating Second Derivatives on Feed-Forward Networks by Wray L. Buntine and Andreas S. Weigend available as /pub/neuroprose/buntine.second.ps.Z Recent techniques for training connectionist feed-forward networks require the calculation of second derivatives to calculate error bars for weights and network outputs, and to eliminate weights, etc. This note describes some exact algorithms for calculating second derivatives. They require at the worst case approximately $2K$ back/forward-propagation cycles where $K$ is the number of nodes in the network. For networks with two-hidden layers or less, computation can be much quicker. Three previous approximations, ignoring some components of the second derivative, numerical differentiation, and scoring, are also reviewed and compared. From STCS8013 at IRUCCVAX.UCC.IE Tue Jul 9 07:09:00 1991 From: STCS8013 at IRUCCVAX.UCC.IE (STCS8013@IRUCCVAX.UCC.IE) Date: Tue, 9 Jul 91 11:09 GMT Subject: Please inlcude info in CONNECTIONIST Message-ID: <5B509D275E800064@BITNET.CC.CMU.EDU> Call for Attendance Fourth Irish Conference on Artificial Intelligence and Cognitive Science (AICS'91) 19 - 20 September, 1991 University College, Cork, Ireland For Details and Registration Forms, contact the Conference Chair: Humphrey Sorensen, Computer Science Department, University College, Cork, Ireland. email: aics91 at iruccvax.ucc.ie From CHELLA at IPACRES.BITNET Tue Jul 9 14:56:00 1991 From: CHELLA at IPACRES.BITNET (CHELLA@IPACRES.BITNET) Date: Tue, 9 Jul 91 18:56 GMT Subject: No subject Message-ID: <4A58EAFC681F21021F@IPACRES.BITNET> I should like to subscribe the mailing list. Antonio Chella From 7923570 at TWNCTU01.BITNET Wed Jul 10 14:08:00 1991 From: 7923570 at TWNCTU01.BITNET (7923570) Date: Wed, 10 Jul 91 14:08 U Subject: No subject Message-ID: hi: Could you provide me the E-mail address about the neural net discussing list ? From lacher at lambda.cs.fsu.edu Wed Jul 10 15:12:29 1991 From: lacher at lambda.cs.fsu.edu (Chris Lacher) Date: Wed, 10 Jul 91 15:12:29 -0400 Subject: Distributed v Local Message-ID: <9107101912.AA01866@lambda.cs.fsu.edu> Remark related to the distributed/local representation conversation. We've been studying expert networks lately: discrete-time computational networks with symbolic-level node functionality. The digraph structure comes from a rule base in an expert system, and the synaptic and node functionality mimics the inferential dynamics of the ES. The nodes represent things like AND, NOT, NOR and EVIDENCE ACCUMULATION. (We announced a preprint to connectionists on this topic in January 91.) We have some of these things that actually reason like expert systems and can learn from data. Clearly these nets use "local" representation in the sense that single components of the architecture have identifiable meaning outside the net. (Although, the identifiable meaning is that of a process that makes sense at a cognitive level rather than an object or attribute as in the more common interpretation of local representation.) These symbolic-level processors can be replaced with relatively small networks of sub-symbolic processors that do the common "output:= squash(weighted sum of inputs)" kind of processing. One can imagine either feed-forward or recurrent subnets that accomplish this. The expert networks tend to be sparsely connected, O(n), while the subnets will generally be highly connected, O(n^2). Wildly speculate the existence of a system containing N symbolic processors with O(N) interconnectivity, each processor actually consisting of K sub-symbolic processors with O(K^2) intraconnectivity, and assume there are overall 10^10 sub-symbolic processors and 10^13 connections (about the numbers for the cerebral cortex). Then we have NK = 10^10 and NK^2 = 10^13 which yields N = 10^7 and K = 10^3. That is, an organization of about 10^7 sparsely connected subnetworks each being a highly intraconnected network of around 1000 sub-symbolic processors. Aren't the columnar structures in the CC highly connected subnets of about a thousand neurons? --- Chris Lacher (My coworkers include colleague Susan Hruska, recent PhD Dave Kuncicky, and a number of current graduate students. They should not be held responsible for this note, however. Susan and Dave are in Seattle at IJCNN. I'm here in Tallahassee trying to survive dog days.) From russ at oceanus.mitre.org Mon Jul 15 11:42:06 1991 From: russ at oceanus.mitre.org (Russell Leighton) Date: Mon, 15 Jul 91 11:42:06 EDT Subject: Narendra's stability proof Message-ID: <9107151542.AA11885@oceanus.mitre.org> At IJCNN `91 Narendra spoke of a paper where he has proven stability for a control system using backpropagation neural networks. Does anyone know where this was published? Thanks. Russ INTERNET: russ at dash.mitre.org Russell Leighton MITRE Signal Processing Lab 7525 Colshire Dr. McLean, Va. 22102 USA From russ at oceanus.mitre.org Mon Jul 15 13:39:41 1991 From: russ at oceanus.mitre.org (Russell Leighton) Date: Mon, 15 Jul 91 13:39:41 EDT Subject: Paper announcement Message-ID: <9107151739.AA12618@oceanus.mitre.org> The following paper is available in the neuroprose library (leighton.ar-backprop.ps.Z). The Autoregressive Backpropagation Algorithm {To appear in the Proceedings of the International Joint Conference on Neural Networks, 1991} Russell R. Leighton and Bartley C. Conrath The MITRE Corporation 7525 Colshire Drive, McLean, VA 22102 This paper describes an extension to error backpropagation that allows the nodes in a neural network to encode state information in an autoregressive ``memory.'' This neural model gives such networks the ability to learn to recognize sequences and context-sensitive patterns. Building upon the work of Wieland concerning nodes with a single feedback connection, this paper generalizes the method to $n$ feedback connections and addresses stability issues. The learning algorithm is derived, and a few applications are presented. To get the paper: 1. ftp 128.146.8.62 2. cd pub/neuroprose 3. binary 4. get leighton.ar-backprop.ps.Z 5. quit 6. uncompress leighton.ar-backprop.ps.Z 7. lpr leighton.ar-backprop.ps Russ INTERNET: russ at dash.mitre.org Russell Leighton MITRE Signal Processing Lab 7525 Colshire Dr. McLean, Va. 22102 USA From sontag at control.rutgers.edu Mon Jul 15 14:48:18 1991 From: sontag at control.rutgers.edu (sontag@control.rutgers.edu) Date: Mon, 15 Jul 91 14:48:18 EDT Subject: Narendra's stability proof Message-ID: <9107151848.AA15176@control.rutgers.edu> From: Russell Leighton At IJCNN `91 Narendra spoke of a paper where he has proven stability for a control system using backpropagation neural networks. Does anyone know where this was published? Thanks. Russ At the American Automatic Control Conference, three weeks ago in Boston, there were a few papers dealing with adaptive control using neural nets. Among them: TA1: 8:30-9:00 Intelligent Control Using Neural Networks Narendra, K., Yale University Mukhopadhyay, S., Yale University TP1: 17:30-18:00 Regulation of Nonlinear Dynamical Systems Using Neural Networks Narendra, K., Yale University Levine, A., Yale University FA1: 11:15-11:45 Gradient Methods for Learning in Dynamical System Containing Neural Networks Narendra, K., Yale University Parthasarathy, K., Yale University 12:15-12:45 Stability and Convergence Issues in Neural Network Control Slotine, J., Massachusetts Institute of Technology As far as I recall, all results on stability dealt with RADIAL-BASIS types of networks, assuming FIXED centers, so the estimation problem is a LINEAR one. The paper of Slotine has a nice technique for estimating weights at the lower level, using spectral information on the training data (I guess in the same spirit that others would use clustering). Before the conference, there was a one-day course, organized by Narendra, which covered neural net approaches to control; he had a writeup prepared for that, which might cover the stability results (I don't know, nor do I know how you can get a copy). The email addresses for Slotine and Narendra are as follows: jjs at athena.mit.edu (Jean-Jacques Slotine, Mech Engr, MIT) narendra at bart.eng.yale.edu (Narendra, Engineering, Yale) -eduardo PS: My paper in the same proceedings, WP9: "Feedback Stabilization Using Two-Hidden-Layer Nets", covered the results on why *TWO* hidden layers are needed for control (and some other) problems, rather than one. (A tech report was posted late last year to neuroprose, covering the contents of this paper.) From stolcke at ICSI.Berkeley.EDU Mon Jul 15 15:00:20 1991 From: stolcke at ICSI.Berkeley.EDU (stolcke@ICSI.Berkeley.EDU) Date: Mon, 15 Jul 91 13:00:20 MDT Subject: new cluster available Message-ID: <9107152000.AA05004@icsib30.Berkeley.EDU> Dear Connectionists: After several months of testing I'm releasing a slightly revised version my enhanced cluster utility. A major memory allocation glitch was fixed and support for System 5 curses pads was added. I should note that for viewing the graph output of cluster the original version of the xgraph program is not enough because it cannot handle labeled datapoints. An enhanced version that works well with cluster can be ftped at the same location as cluster (see below). Andreas HOW TO GET CLUSTER cluster is available via anonymous ftp from icsi-ftp.berkeley.edu (128.32.201.55). To get it use FTP as follows: % ftp icsi-ftp.berkeley.edu Connected to icsic.Berkeley.EDU. 220 icsi-ftp (icsic) FTP server (Version 5.60 local) ready. Name (icsic.Berkeley.EDU:stolcke): anonymous Password (icsic.Berkeley.EDU:anonymous): 331 Guest login ok, send ident as password. 230 Guest login Ok, access restrictions apply. ftp> cd pub/ai 250 CWD command successful. ftp> binary 200 Type set to I. ftp> get cluster-2.2.tar.Z 200 PORT command successful. 150 Opening BINARY mode data connection for cluster-2.2.tar.Z (15531 bytes). 226 Transfer complete. 15531 bytes received in 0.08 seconds (1.9e+02 Kbytes/s) ftp> quit 221 Goodbye. HOW TO BUILD CLUSTER Unpack in an empty directory using % zcat cluster-2.2.tar.Z | tar xf - Read the README and especially the man page (cluster.man) for information. Check the Makefile for any compile time flags that might need adjustment. Then compile with % make After making the appropriate adjustments in the Makefile you can % make install From T.Rickards at cs.ucl.ac.uk Tue Jul 16 07:10:05 1991 From: T.Rickards at cs.ucl.ac.uk (T.Rickards@cs.ucl.ac.uk) Date: Tue, 16 Jul 91 12:10:05 +0100 Subject: subscription Message-ID: I would like to subscribe to Connectionists.( I am working for a Neural Network Technology Transfer Club based at University College London.) Tessa Rickards T.Rickards at cs.ucl.ac.uk From rosauer at fzi.uka.de Tue Jul 16 18:47:52 1991 From: rosauer at fzi.uka.de (Bernd Rosauer) Date: Tue, 16 Jul 91 22:47:52 GMT Subject: SUMMARY: GA&NN Message-ID: Some weeks ago I posted a request concerning the combination of genetic algorithms and neural networks. In the following you will find a summary of the references I received. This summary is preliminary and the references are not completely reviewed. Maybe, I will post an annotated one at the end of this year when I have got all the relevant proceedings of this year. I would like to make some general comments in advance. First of all, two summaries have already been published which cover the stuff until 1990: Rudnick, Mike. "A Bibliography of the Intersection of Genetic Search and Artificial Neural Networks." Technical Report CS/E 90-001, Department of Computer Science and Engineering, Oregon Graduate Institute, January 1990. Weiss, Gerhard. "Combining Neural and Evolutionary Learning: Aspects and Approaches." Report FKI-132-90, Institut fuer Informatik, Technische Universitaet Muenchen, May 1990. As one of my thrustworthy informants told me the proceedings of ICGA'91 and NIPS'91 (will) contain tons of stuff on that topic. Finally, there is a mailing list on "neuro-evolution". Because of the administrator did not yet answer my request I do not know whether this list is still active. Anyway, try for further information. Now, here is the summary. Many thanks to everyone who responded. Feel free to send me further references. Bernd -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Ackley, D. H., and M. S. Littman. "Learning from natural selection in an artificial environment." Proceedings of the International Joint Conference on Neural Networks Washington, D.C., January 1990. Ackley, D. H., and M. L. Littman. "Interactions between learning and evolution." Artificial Life 2. Ed. Chris Langton. New York: Addison-Wesley, in press. Belew, R. K. "Evolution, learning and culture: computational metaphors for adaptive search." Complex Systems 4.1 (1990): 11-49. Belew, R. K., J. McInerney and N. N. Schraudolph. "Evolving networks: Using the Genetic Algorithm with connectionist learning." Proc. 2nd Artificial Life Conference. New York: Addison-Wesley, in press. Belew, R. K., J. McInerney and N. Schraudolph. "Evolving Networks: Using the Genetic Algorithm with Connectionist Learning." Technical Report CS90-174, University of California at San Diego, 1990. Hinton, G. E., and S. J. Nowlan S.J. "How Learning Guides Evolution." Complex System 1 (1987): 495-502. Ichikawa, Y. "Evolution of Neural Networks and Applications to Motion Control." Proc. of IEEE Int. Work. on Intelligent Motion Control, Vol.1, 1990. Keesing, Ron, and David Stork. N.t. NIPS-3, 1990. Kitano, Hiroaki. "Empirical Study on the Speed of Convergence of Neural Network Training using Genetic Algorithms." Proceedings of AAAI-90. Kitano, Hiroaki. "Designing Neural Networks with Genetic Algorithms using Graph Generation System." Complex System 4.4 (1990). Kouchi, Masahiro, Hiroaki Inayoshi and Tsutomu Hoshino. "Optimization of Neural-Net Structure by Genetic Algorithm with Diploidy and Geographical Isolation Model." Inst. of Engineering Mechanics, Univ. Tsukuba, Ibaraki 305, Japan. Menczer, F., and D. Parisi. "`Sexual' reproduction in neural networks." Technical Report PCIA-90-06, Institute of Psychology, C.N.R., Rome, 1990. Menczer, F., and D. Parisi. "Evidence of hyperplanes in the genetic learning of neural networks." Technical Report PCIA-91-08, Institute of Psychology, C.N.R., Rome, 1991. Miglino, O., and D. Parisi. "Evolutionary stable and unstable strategies in neural networks." Technical Report PCIA-91-09, Institute of Psychology, C.N.R., Rome, 1991. Mjolsness, Eric, David H. Sharp and Bradley K. Alpert. "Scaling, Machine Learning, and Genetic Neural Nets." Advances in Applied Mathematics 10 (1989): 137-163. Montana, David J., and Lawrence Davis. "Training Feedforward Neural Networks using Genetic Algorithms." Proceedings of the 11th Intern. Joint Conference on Artificial Intelligence, 1989, pp. 762-767. Muehlenbein, H., and J. Kindermann. "The Dynamics of Evolution and Learning - Towards Genetic Neural Networks." Connectionism in Perspective. Ed. R. Pfeifer et. al. Elsevier, 1989. pp. 173-197. Nolfi, S., J. Elman and D. Parisi. "Learning and Evolution in Neural Networks." CRL Technical Report 9019, University of California at San Diego, 1990. Nolfi, S., and D. Parisi. "Auto-teaching: neural networks that develop their own teaching input." Technical Report PCIA-91-03, Institute of Psychology, C.N.R., Rome, 1991. Parisi, D., F. Cecconi and S. Nolfi. "Econets: Neural Networks that Learn in an Environment." Network 1 (1990): 149-168. Parisi, D., S. Nolfi, and F. Cecconi. "Learning, Behavior, and Evolution." Technical Report PCIA-91-14, Institute of Psychology, C.N.R., Rome, 1991. Radcliff, Nick. "Genetic Neural Networks on MIMD Computers." Ph.D. Thesis, University of Edinburgh. Radcliff, Nick. "Equivalence Class Analysis of Genetic Algorithms." Complex Systems, in press. Radcliff, Nick. "Forma Analysis and Random Respectful Recombination." Proceedings of ICGA91, in press. Todd, P. M. and G. F. Miller, G.F. "Exploring adaptive agency II: simulating the evolution of associative learning." From Animals to Animats. Eds. J. A. Meyer and S. W. Wilson. Cambridge, MA: MIT, 1991. From shen at iro.umontreal.ca Wed Jul 17 19:29:20 1991 From: shen at iro.umontreal.ca (Yu Shen) Date: Wed, 17 Jul 91 19:29:20 EDT Subject: network that ranks Message-ID: <9107172329.AA09591@kovic.IRO.UMontreal.CA> Suppose a unit in the network stands for an option being compared with others, and given the pairwise prferences, such as option A is better than option B, etc. I would like the responses of the units to be proportional to their ranks in the preference relationship. I tried the Interactive Activation and Competition network in PDP book, IAC. It seems to work, by coding the weights among the units as follows: if A is better than B, then the weight from B to A is 1, and the weight from A to B is -1. And the initial state of the units are taken to be constant, 0.5 (the rest value). I tested up to cases of 4 units. For A>B>C>D, the response of the same ranks is produced (unit A is the highest, unit b is the next, etc.). But I'm wondering if it work for all the cases. Any pointer for better connectionist solution? Thank you, Yu Shen From NEZA at ALF.LET.UVA.NL Thu Jul 18 14:33:00 1991 From: NEZA at ALF.LET.UVA.NL (Neza van der Leeuw) Date: Thu, 18 Jul 91 14:33 MET Subject: Simple pictures, tough problems. Message-ID: <635B127EA0803C46@BITNET.CC.CMU.EDU> Hello, I am currently working on the "grounding problem", in the sense that I try to derive meaning of language from pictures coupled with sentences (my own version of the Berkeley ICSI L0 project: Miniature Language Acquisition; A touchstone for cognitive science. TR-90-009). Right now I am working with a self-organising neural net (kind of Kohonen), having bitmaps and strings as input. These bitmaps represent circles, squares and triangles in a bitwise manner. The problem is that I want the architecture to generalize. This means that it should group small and large circles, rather than circles of size X with squa- res of size nearly X. In the current implementation, coding is done by repre- senting each line of the picture by a random vector. But when doing this, the net generalizes not on second-order properties like form, but on first-order properties (roughly: where the black spots in the picture reside). Thus, circles are seen similar to squares when their sizes "match". One could avoid this problem by choosing a representation in language-like propositions, but this seems to me to be the "solving the problem by doing it yourself" approach. Some represenational mechanism should provide the answer to my problem, but I want it to be "picture-like" instead of "language-like". Otherwise I bounce back into my favourite "grounding problem" again. Does anyone know of any work that has been done in this field? Any hints, refe- rences or other useful messages are very welcome. As I am more a computer lin- guist than a grafical scientist my knowledge of the whole thing is rather limited. If some useful stuff comes out, I will write a nice report and send it to the list, if others are interested in the matter as well. Many thanks, and don't get overworked during these "holiday months" (whatever that may be). Neza van der Leeuw Dept. of Computational Linguistics Faculty of Arts University of Amsterdam Spuistraat 134 1012 VB Amsterdam The Netherlands From slehar at park.bu.edu Thu Jul 18 12:22:35 1991 From: slehar at park.bu.edu (Steve Lehar) Date: Thu, 18 Jul 91 12:22:35 -0400 Subject: Simple pictures, tough problems. Message-ID: <9107181622.AA28726@park.bu.edu> > The problem is that I want the architecture to generalize. This means > that it should group small and large circles, rather than circles of > size X with squares of size nearly X. > ... > > the net generalizes not on second-order properties like form, but on > first-order properties (roughly: where the black spots in the picture > reside). Thus, circles are seen similar to squares when their sizes > "match". Maybe the solution is to take a cue from nature. How does the brain represent visual information? Hubel and Wiesel [1] found simple cells which respond to the very simplest visual primitives- oriented edges or moving edges. They also found complex cells, which generalized the simple cell responses "spatially", but not "featurally". This is an important point, and is, I believe, the key to understanding how the brain solves the generalization problem. Say you have a simple cell that fires in response to a vertical edge in a very specific location. A complex cell might fire for a vertical edge in a much larger range of locations (spatial generalization), but this is not due to the complex cell having a coarser representation of the world, because the complex cell will not fire in response to a cruder fuzzier edge, it is every bit as specific about the sharpness of that vertical edge as was the simple cell- i.e. we have NO featural generalization, just spatial. When you move up to complex and hyper complex cells, you get cells that respond to even more specialized features, such as end-stop detectors that fire for a vertical edge in a large region but only if it terminates, not if it goes straight through, and corner detectors which fire for two end-stop detectors, one vertical and one horizontal. Notice the trend- as we become more general spatially, we become more specific featurally. This is what I call the spatial/featural hierarchy, and one can posit that at the pinnacle of the hierarchy would be found very specific detectors that respond, for example, to your grandmother's face, wherever it may appear in the visual field. This is the basic idea behind the Neocognitron [2], although I believe that that model is lacking in one important element, that being resonant feedback between the levels of the hierarchy, which Grossberg [3] shows is so important to maintain a consistancy between different levels of the representation. I discuss a resonant spatial/featural hierarchy and how it may be implemented in [4] and [5]. Now you might argue that the construction of such a hierarchy would be very expensive in both space and time (memory and computation) especially if it is implemented as I propose, with resonant feedback between all the layers of the hierarchy. My response would be that the problem of vision is by no means trivial, and that until we come up with a better solution, we cannot presume to do better than nature, and if nature deems it necessary to create such a hierarchy, then I strongly suspect that that hierarchy is an essential prerequisite for featural generalization. [1] Hubel & Wiesel RECEPTIVE FIELDS AND FUNCTIONAL ARCHITECTURE IN TWO NONSTRIATE VISUAL AREAS OF THE CAT(1965) Journal of Neurophysiology 28 229-289 [2] Fukushima & Miyake NEOCOGNITRON: A NEW ALGORITHM FOR PATTERN RECOGNITION TOLERANT OF DEFORMATIONS AND SHIFTS IN POSITION.(1982) Pattern Recognition 15, 6 455-469 [3] Grossberg, Stephen & Mingolla, Ennio. NEURAL DYNAMICS OF PERCEPTUAL GROUPING: TEXTURES, BOUNDARIES AND EMERGENT SEGMENTATIONS Perception & Psychophysics (1985), 38 (2), 141-171. [4] Lehar S., Worth A. MULTI RESONANT BOUNDARY CONTOUR SYSTEM, Boston University, Center for Adaptive Systems technical report CAS/CNS-TR-91-017. To get a copy, write to... Boston University Center for Adaptive Systems 111 Cummington Street, Second Floor Boston, MA 02215 (617) 353-7857,7858 [5] Lehar S., Worth A. MULTIPLE RESONANT BOUNDARY CONTOUR SYSTEM. In: PROGRESS IN NEURAL NETWORKS volume 3 (Ed. by Ablex Publishing Corp.) In print. (i.e. not available yet) From ross at psych.psy.uq.oz.au Fri Jul 19 03:31:49 1991 From: ross at psych.psy.uq.oz.au (Ross Gayler) Date: Fri, 19 Jul 1991 17:31:49 +1000 Subject: simple pictures, tough problems (language grounding) Message-ID: <9107190731.AA10594@psych.psy.uq.oz.au> Neza van der Leeuw (Computational Linguistics, University of Amsterdam) writes: >I am currently working on the "grounding problem", in the sense that I try to >derive meaning of language from pictures coupled with sentences ... > >The problem is that I want the architecture to generalize. This means that it >should group small and large circles, rather than circles of size X with squa- >res of size nearly X. ... > >One could avoid this problem by choosing a representation in language-like >propositions, but this seems to me to be the "solving the problem by doing it >yourself" approach. Some represenational mechanism should provide the answer >to my problem, but I want it to be "picture-like" instead of "language-like". >Otherwise I bounce back into my favourite "grounding problem" again. I think you will have to do one of two things: either hand-craft an environment representation that is biased towards generalisation along the directions that you think are 'natural'; or provide the system with some way of manipulating the environment so that some properties of the environment are invariant under manipulation. It may help to think of the problem in terms of learning a first language and a second language. In learning a second language, the concepts are (mostly) already present - so only the language has to be learned. In learning a first language, the concepts are learned more or less at the same time as the language (without wanting to get into an argument about the Whorf hypothesis). I think that learning concepts by pure observation (as your system has to) is generally impossible. What is there in the input to suggest that 'circle-ness' or 'square-ness' is a better basis for generalisation than pixel overlap? Imagine, that a civilisation living on the surface of a neutron star has just made contact with us by sending a subtitled videotape of life on their world. The environmental scenes would make no sense to us - we probably would not even be able to segment the scene into objects. So how could we learn the language if we can't 'understand' the picture? The human visual system has certain generalisation biases (say, based on edge detectors etc), but I think a stronger requirement is to be able to manipulate the environment. By grabbing an object and moving it backwards and forwards under our control, we can learn that the shape remains constant while the apparent size varies. I would appreciate any references to more formal arguments for the necessity (or otherwise) of being able to manipulate the environment (or perceptual apparatus) in order to learn 'natural' concepts. Ross Gayler ross at psych.psy.uq.oz.au From pjh at compsci.stirling.ac.uk Fri Jul 19 10:52:57 1991 From: pjh at compsci.stirling.ac.uk (Peter J.B. Hancock) Date: 19 Jul 91 10:52:57 BST (Fri) Subject: Simple pictures, tough problems. Message-ID: <9107191052.AA11204@uk.ac.stir.cs.nevis> A comment on Steve Lehar's comment: just how hierarchical is the visual system? I'm told (sorry, I don't know a reference and my source is away on holiday) of evidence that some complex cells respond {\em before} simple cells. I understand there is some debate about the simple/complex dichotomy anyway, but such a result does challenge the traditional story of simple cells feeding into complex ones. More remarkably, recent work from Dave Perrett's group at St. Andrew's is showing that face-sensitive cells in monkeys not only respond within 90mS (yes, ninety milliseconds) of stimulus presentation, but are highly stimulus-specific at that time (I don't know if this is yet published). This does not leave very much time for any pretty hierachies and feedback loops. It implies that recognition of such highly familiar objects is extremely feed-forward and that any model that {\em always} requires many cycles to settle down is wrong. This begs the question of what all the feedback loops in the brain are doing. It may be that not all stimuli are as privileged as faces and that more processing is required for less familiar things. It may be to do with the initial learning, rather like Grossberg's ART which may cycle while learning, but is one-shot thereafter. It may be to do with tuning early systems for given visual tasks. It certainly needs more research... Peter Hancock Centre for Cognitive and Computational Neuroscience University of Stirling. From harnad at Princeton.EDU Fri Jul 19 10:20:55 1991 From: harnad at Princeton.EDU (Stevan Harnad) Date: Fri, 19 Jul 91 10:20:55 EDT Subject: simple pictures, tough problems (language grounding) Message-ID: <9107191420.AA03936@clarity.Princeton.EDU> Ross Gayler wrote: > I would appreciate any references to more formal arguments for the > necessity (or otherwise) of being able to manipulate the environment > (or perceptual apparatus) in order to learn 'natural' concepts. Try: (1) Harnad, S. (1990) The Symbol Grounding Problem. Physica D 42: 335-346. (2) Harnad, S., Hanson, S.J. & Lubin, J. (1991) Categorical Perception and the Evolution of Supervised Learning in Neural Nets. Presented at American Association for Artificial Intelligence Symposium on Symbol Grounding: Problems and Practice. Stanford University, March 1991. (3) Harnad, S. (1992) Connecting Object to Symbol in Modeling Cognition. In: A. Clarke and R. Lutz (Eds) "Connectionism in Context" Springer Verlag (forthcoming) These articles are retrievable by anonymous ftp from directory pub/harnad on princeton.edu (IP: 128.112.128.1) in binary mode as the compressed files: (1) harnad90.sgproblem.Z (2) harnad91.cpnets.Z (3) harnad92.symbol.object.Z From yang at judy.cs.iastate.edu Fri Jul 19 12:46:13 1991 From: yang at judy.cs.iastate.edu (Jihoon Yang) Date: Fri, 19 Jul 91 11:46:13 CDT Subject: tech report (fwd) Message-ID: <9107191646.AA06647@judy.cs.iastate.edu> > ------------------------------------------------------------------------ > > The following tech report is now available as a compressed postscript > file "yang.cascor.ps.Z" through anonymous ftp from the neurprose archive > (directory pub/neuroprose on cheops.cis.ohio-state.edu - Thanks to > Jordan Pollack of Ohio State University). > > Experiments with the Cascade-Correlation Algorithm > Technical report # 91-16 (July 1991) > Jihoon Yang & Vasant Honavar > Department of Computer Science > Iowa State University > > ----------------------------------------------------------------------- > Jihoon Yang > yang at judy.cs.iastate.edu > From steck at spock.wsu.ukans.edu Fri Jul 19 12:51:05 1991 From: steck at spock.wsu.ukans.edu (jim steck (ME) Date: Fri, 19 Jul 91 11:51:05 -0500 Subject: Stability proofs for recurrent networks Message-ID: <9107191651.AA13577@spock.wsu.UKans.EDU> I am currently looking at the stability issue of syncronous fully recurrent neural networks. I am aware of literature (mainly for Hopfield networks) where the stability issue is adressed using Lyapunov "energy" functions, but have not seen any publication of other types of approaches. I would appreciate E-mail regarding articles where this problem is discussed. Jim Steck steck at spock.wsu.ukans.edu From PF103 at phx.cam.AC.UK Fri Jul 19 20:39:37 1991 From: PF103 at phx.cam.AC.UK (Peter Foldiak) Date: Fri, 19 Jul 91 20:39:37 BST Subject: 'natural' concepts Message-ID: Ross Gayler writes: > I think that learning concepts by pure observation (as your system has to) is > generally impossible. What is there in the input to suggest that 'circle-ness' > or 'square-ness' is a better basis for generalisation than pixel overlap? It may not be easy, but I don't think it is generally impossible. Barlow's redundancy reduction principle, for instance, would say that features that result in lower statistical redundancy are better. (Redundancy here is not first-order (bit probabilities) but pairwise and higher-order redundancy.) Peter Foldiak From gary at cs.UCSD.EDU Fri Jul 19 16:00:29 1991 From: gary at cs.UCSD.EDU (Gary Cottrell) Date: Fri, 19 Jul 91 13:00:29 PDT Subject: talking networks Message-ID: <9107192000.AA19599@desi.ucsd.edu> Dear Neza, I couldn't reply to your address, so I'm sending this to the net. Here's a few references. Also, you should be aware of Georg Dorffner's work at the University of Vienna. Some of the work is overlapping, but I give you all of these in hopes you can find one of them. I will send them to you anyway. gary cottrell 619-534-6640 Sec'y: 619-534-5288 FAX: 619-534-7029 Computer Science and Engineering C-014 UCSD, La Jolla, Ca. 92093 gary at cs.ucsd.edu (INTERNET) {ucbvax,decvax,akgua,dcdwest}!sdcsvax!gary (USENET) gcottrell at ucsd.edu (BITNET) Cottrell, G., Bartell, B. & C. Haupt (1990) Gounding meaning in perception. In \fIProceedings of the 14th German Workshop on AI\fP, H. Marburger (Ed.), pp. 307-321. Berlin: Springer Verlag. Cottrell, G. (1990). Extracting features from faces using compression networks. In Touretzky, D.S., Elman, J.L., Sejnowski, T.J. and Hinton G.E. (Eds.) \fIProceedings of the 1990 Connectionist Models Summer School\fP. San Mateo: Morgan Kaufmann. Bartell, B. & Cottrell, G.W. (1991) A model of symbol grounding in a temporal environment. In \fIProceedings of the International Joint Conference on Neural Networks\fP, Seattle, Washington, June, 1991. From ITGT500 at INDYCMS.BITNET Fri Jul 19 18:59:47 1991 From: ITGT500 at INDYCMS.BITNET (Bo Xu) Date: Fri, 19 Jul 91 17:59:47 EST Subject: Requests Message-ID: <85672B7B60800064@BITNET.CC.CMU.EDU> Besides the slow rate of convergence, local minima, etc., another charge against the feedforward backpropagation neural network is that it is non-biological. I have already got some references on this problem given by G.E.Hinton, F.Crick, A.G.Barto, P.Mazzoni, and R.A.Andersen, etc.. I want to collect more references on this topic. Could someone send me more references which are not covered above? Thanks a lot in advance. Bo Xu Indiana University ITGT500 at INDYCMS.BITNET From ross at psych.psy.uq.oz.au Sat Jul 20 10:46:34 1991 From: ross at psych.psy.uq.oz.au (Ross Gayler) Date: Sun, 21 Jul 1991 00:46:34 +1000 Subject: 'natural' concepts (symbol grounding) Message-ID: <9107201446.AA29959@psych.psy.uq.oz.au> Peter Foldiak writes: >[Learning concepts by pure observation] may not be easy, >but I don't think it is generally impossible. >Barlow's redundancy reduction principle, for instance, would say that >features that result in lower statistical redundancy are better. >(Redundancy here is not first-order (bit probabilities) but pairwise >and higher-order redundancy.) OK, let me expand on my position a little. I realise that connectionist systems can learn to categorise inputs and can generalise on this task. I'm not entirely convinced that this warrants being called 'concept learning' (but I don't think I can justify this belief). More importantly, most systems have had a considearble amount of effort put into the architecture and training set to ensure that they categorise and generalise as expected. My point is that models of the input data can be compared in terms of redundancy - but having a measure of goodness of fit does not directly help construct an optimal network. If the input can be simply characterised in terms of a higher order redundancy, it is very unlikely that a net with no representational biases, starting from input at the pixel level, will discover anything close to the optimal model inside a pragmatically bounded time. This is what I meant when I said that learning from observation was generally impossible. The point I wanted to make was that I think learning at a pragmatically useful rate in a complex environment requires the learner to be able to manipulate the environment. Research methodologists distinguish between experimental and non-experimental methods. Non-experimental methods rely on correlation of observations - so that patterns can be described but causation cannot be ascribed. Experimental methods involve assigning manipulations to the environment in a randomised way - so that much stronger statements can be made about the workings of the world (assuming your assumptions hold :-). Being able to manipulate the environment allows the experimenter to drag information out of the environment at a higher rate if the experimenter is in relative ignorance. Cosmologists use non-experimental methods - BUT they have to apply a lot of prior knowledge that has been validated in other ways. The other point I want to make is that typical connectionist categorisers do not have a model of the input environment as an external reality. Assume that the input signals come from a sensor that imposes some kind of gross distortion. The net learns to categorise the distorted signal as it stands with no separate models of the environment and the distortion. In order to develop a concept that is closer to the human notion of some external reality, the system has to be able to factor its model into an environmental part and a perceptual part. This can't be done by pure observation, it needs some ability to interact with the environment, even if only by panning the camera across the scene. From slehar at park.bu.edu Sat Jul 20 15:01:05 1991 From: slehar at park.bu.edu (Steve Lehar) Date: Sat, 20 Jul 91 15:01:05 -0400 Subject: Simple pictures, tough problems. In-Reply-To: connectionists@c.cs.cmu.edu's message of 19 Jul 91 18:12:48 GM Message-ID: <9107201901.AA26511@park.bu.edu> "Peter J.B. Hancock" argues... > just how hierarchical is the visual system? I'm told of evidence that > some complex cells respond BEFORE simple cells. The fact that a complex cell fires BEFORE a simple cell does not preclude the possibility that the signal was provided originally by the simple cells. What we are talking about here is a resonant process- like a bow running over a violin string. The bow produces random vibration on the string, but the string only responds to one frequency component of that vibration. The resultant tone does not "start" at the bow and spread outward towards the bridge and neck, but rather, a resonant vibration emerges from the entire system simultaneously. In the same way, simple cells of all orientations are constantly firing to greater or lesser degree to any visual input, but the complex cells (by simultaneous cooperative and competitive interactions) will resonate to a larger global pattern hidden in all the confusion of the simple cells. The fact that the complex cell fires first simply reflects the fact that the visual system puts more faith in a global harmony of coherent edges than in a cacophany of individual local responses, and indeed the local edges may not even be perceived until the global coherence is established. This same kind of interaction would dictate that even highe level patterns, as detected by hypercomplex or higher order cells, would register before the complex cells, so that although the input signal arrives bottom-up, recognition resonance is established top-down, thus ensuring global consistancy of the entire scene. According to the Boundary Contour System (BCS) model, local competitive and cooperative interactions occur between the lowest level detectors to enhance those that are compatible with global segmentations while suppressing those that are incompatible. If the lowest level interactions could settle into stable configurations before the global ones could exert their influence, the perception would be dominated by local, not global consistancies, like a damped violin string which produces a scratchy random noise when bowed. Peter Hancock continues... > face-sensitive cells in monkeys not only respond within 90mS ... of > stimulus presentation, but are highly stimulus-specific at that time > ... This does not leave very much time for any pretty hierachies and > feedback loops. It implies that recognition of such highly familiar > objects is extremely feed-forward and that any model that ALWAYS > requires many cycles to settle down is wrong. Well, it depends on what kind of cycles we are talking about. If you mean iterations of a sequential algorithm then I would have to agree, and current implementations of the BCS and MRBCS are of necessity performed iteratively on sequential binary machines. But the visual architecture that they presume to emulate is a parallel analog resonant system, like a violin string (but of course much more complex) so that it does not take any number of "cycles" as such for a resonance to set in. Also, in considering recognition time, top-down expectation plays a very large role- it takes considerably longer to recognise an unexpected or out-of-context pattern than an expected one. From pjh at compsci.stirling.ac.uk Mon Jul 22 10:59:41 1991 From: pjh at compsci.stirling.ac.uk (Peter J.B. Hancock) Date: 22 Jul 91 10:59:41 BST (Mon) Subject: Simple pictures, tough problems. Message-ID: <9107221059.AA14184@uk.ac.stir.cs.nevis> I think Steve Lehar and Ken Laws are making essentially the same, valid point: that there is time for quite a lot of very local analogue processing, for instance within a hypercolumn, even within a 90mS overall response time. Thus indeed a cell may actually fire before those that are driving it, provided the activation is transferred without the need for action potentials. The real time constraint comes when you want to start sending messages between cortical areas, which requires action potentials with their associated delays. There are huge numbers of fibres going top-down: they are obviously doing something, but within 90mS I don't believe they can be contributing much to the immediate recognition process, though they certainly might have something to do with priming. In the case of the monkey face-recognition work, it's unclear at the moment to what extent the experimental setup may have primed them to expect faces, but they were being shown all sorts of other things as well. Peter Hancock From roeschei at tumult.informatik.tu-muenchen.de Mon Jul 22 09:13:26 1991 From: roeschei at tumult.informatik.tu-muenchen.de (Martin Roescheisen) Date: Mon, 22 Jul 91 15:13:26 +0200 Subject: No subject Message-ID: <9107220913.AA00858@tumult.informatik.tu-muenchen.de> Technical Report available INCORPORATING PRIOR KNOWLEDGE IN PARSIMONIOUS NETWORKS OF LOCALLY-TUNED UNITS Keywords: Gaussian Units (Moody, Poggio), higher-dimensionality, rolling mills, use of prior knowledge. Reimar Hofmann Munich Technical University Martin R\"oscheisen Munich Technical University Volker Tresp Siemens AG Corporate Research and Development Abstract: Utilizing Bayes decision theory, we develop a theoretical foundation for a localized network architecture that requires few centers to be allocated and can therefore be employed in problems which because of their high input dimensionality could not yet be tackled by such networks. We show how quantitative {\it a priori} knowledge can be readily incorporated by choosing a specific training regime. The network was employed as a neural controller for a hot line rolling mill and achieved in this application one to two orders of magnitude higher accuracy than optimally-tuned standard algorithms such as sigmoidal backpropagation and performed significantly better than a state-of-the-art analytic model. _________________ Hardcopies of the full paper can be obtained by sending e-mail to hofmannr at lan.informatik.tu-muenchen.dbp.de From jose at tractatus.siemens.com Mon Jul 22 08:45:59 1991 From: jose at tractatus.siemens.com (Steve Hanson) Date: Mon, 22 Jul 1991 08:45:59 -0400 (EDT) Subject: Requests In-Reply-To: <85672B7B60800064@BITNET.CC.CMU.EDU> References: <85672B7B60800064@BITNET.CC.CMU.EDU> Message-ID: "Biological plausibility" is somewhat of a throwaway, given the massively documented understanding we have of learning in brain circuits (he said sarcastically). There are many seemingly incompatible brain circuits for implementing classical conditioning, depending on the beast, the stimuli, and the context and maybe just depending on the whims of the local circuitry development that day. Certainly, the way Backprop is typically implemented in "C" code, may be biologically implausible... ie information doesn't tend to flow back down axons... (at least not at a fast enough rate). I am not sure that slow rate of convergence is such a good argument.. given you can do things to speed up learning in feedforward nets... also given the rate people really seem to learn at... In any case, there are now published counter-examples of people showing how back-prop could be seen as biologically plausible, which I suspect given our present state knowledge is a much more useful enterprise. to wit: G. Tesauro, Neural Models of Classical Conditioning: A theoretical Perspective, Hanson & Olson, (Eds.) MIT Press, Bradford, 1990. D. Zipser, in Gluck & Rumelhart, Neuroscience and Connectionism, LEA, 1990. There may be others.... --Steve Stephen J. Hanson Learning & Knowledge Acquisition Group SIEMENS Research 755 College Rd. East Princeton, NJ 08540 From schyns at clouseau.cog.brown.edu Mon Jul 22 08:53:32 1991 From: schyns at clouseau.cog.brown.edu (Phillipe Schyns) Date: Mon, 22 Jul 91 08:53:32 -0400 Subject: the symbol grounding problem Message-ID: <9107221253.AA17341@clouseau.cog.brown.edu> Here is the abstract of a paper that will appear in Cognitive Science. Although its main topic is conceptual development and lexical acquisition, it presents a solution to the symbol grounding problem which is compatible with Harnard's proposal. Schyns, P. G. (in press). A neural network model of conceptual development, Cognitive Science. Abstract Previous neural network models of concept learning were mainly implemented with supervised learning schemes. However, studies of human conceptual memory have shown that concepts may be learned without a teacher who provides the category name to associate with exemplars. A modular neural network architecture that realizes concept acquisition through two functionally distinct operations, categorizing and naming, is proposed as an alternative. An unsupervised algorithm realizes the categorizing module by constructing representations of categories compatible with Prototype Theory. The naming module associates category names to the output of the categorizing module in a supervised mode. In such modular architecture, the interface between the modules can be conceived of as an "informational relay" that encodes, constrains and propagates important information. Five experiments were conducted to analyze the relationships between internal conceptual codes and simple conceptual and lexical development. The first wo experiments show a prototype effect and illustrate some basic characteristics of the system. The third experiment presents a bottom-up model of the narrowing down of children's early lexical categories that honors Mutual Exclusivity. The fourth experiment introduces top-down constraints on conceptual coding. The fifth experiment exhibits how hierarchical relationships between concepts are learned by the architecture while it also demonstrates how a spectrum of conceptual expertise may gradually emerge as a consequence of experiencing more with certain categories than with others. Part of this work appeared in the Proceedings of the 1990 Connectionist Models Summer School. Schyns, P. G. (1990). A modular neural network model of the acquisition of category names in children. Proceedings of the 1990 Connectionist Models Summer School, 228-235, Morgan Kaufmann, CA. Philippe =========================================================================== Philippe G. Schyns Dpt. of Cognitive and Linguistic Sciences Box 1978 Brown University Providence, RI, 02912 From ray at espresso.boeing.com Tue Jul 23 11:25:24 1991 From: ray at espresso.boeing.com (Ray Allis 5-3583) Date: Tue, 23 Jul 91 08:25:24 PDT Subject: Simple pictures, tough problems. Message-ID: <9107231525.AA15897@espresso.bcs.eca> > From: Ken Laws > Subject: Re: Simple pictures, tough problems. > To: connectionists at RI.CMU.EDU > > This does not leave very much time for any pretty > > hierachies and feedback loops. > > Feedback loops are not necessarily slow. Analog computers can be > much faster than digital ones for many tasks, and I think we're > beginning to understand neural bundles in analog terms. > > -- Ken Laws > ------- > I sincerely hope so. It's long past time to do so. Ray Allis From gary at cs.UCSD.EDU Mon Jul 22 18:42:37 1991 From: gary at cs.UCSD.EDU (Gary Cottrell) Date: Mon, 22 Jul 91 15:42:37 PDT Subject: Stability proofs for recurrent networks Message-ID: <9107222242.AA22701@desi.ucsd.edu> Hal White has shown convergence conditions for learning in recurrent nets. Try writing him for reprints. He is: Hal White Dept. of Economics UCSD La Jolla, CA 92093 gary From georgiou at rex.cs.tulane.edu Tue Jul 23 19:58:05 1991 From: georgiou at rex.cs.tulane.edu (George Georgiou) Date: Tue, 23 Jul 91 18:58:05 -0500 Subject: CALL FOR PAPERS (INFORMATION SCIENCES) Message-ID: <9107232358.AA11866@rex.cs.tulane.edu> Call for Papers Special Issue of INFORMATION SCIENCES Topic: Neural Networks and Artificial Intelligence The motivation for this Special Issue, which will be published in 1992, is to evaluate the advances in the field of neural networks in perspective. Critics have charged that engineering neural computing has turned out to be no more than a new technique of clustering data. From berg at cs.albany.edu Tue Jul 23 18:08:59 1991 From: berg at cs.albany.edu (George Berg) Date: Tue, 23 Jul 91 18:08:59 EDT Subject: TR available: learning phrase structure Message-ID: <9107232209.AA11241@odin.albany.edu> The following paper is available: ------------------------------------------------------------------------------- Learning Recursive Phrase structure: Combining the Strengths of PDP and X-Bar Syntax George Berg Department of Computer Science Department of Linguistics and Cognitive Science State University of New York at Albany ABSTRACT In this paper we show how a connectionist model, the XERIC Parser, can be trained to build a representation of the syntactic structure of sentences. One of the strengths of this model is that it avoids placing a priori restrictions on the length of sentences or the depth of phrase structure nesting. The XERIC architecture uses X-Bar grammar, an "unrolled" virtual architecture reminiscent of Rumelhart and McClelland's back-propagation through time, recurrent networks and reduced descriptions similar to Pollack's RAAM. Representations of words are presented one at a time, and the parser incrementally builds a representation of the sentence structure. Along the way it does lexical and number/person disambiguation. The limits on the current model's performance are consistent with the difficulty of encoding information (especially lexical information) as the length and complexity of the sentence increases. This paper is to be presented at the IJCAI-91 Workshop on Natural Language Learning, and is also available as SUNY Albany Computer Science Department technical report TR 91-5. =============================================================================== This paper is available three ways. Please DO NOT write me for a copy (among other reasons, because I'll be out of town most of the rest of the Summer). I will, however, be happy to answer questions and otherwise discuss the paper. ---- First Way: anonymous ftp via the neuroprose archive: The file is available via anonymous ftp from cheops.cis.ohio-state.edu as the file berg.phrase_structure.ps.Z in the pub/neuroprose directory. It is a compressed postscript file. Below is the log of a typical ftp session to retrieve the file: yourprompt> ftp cheops.cis.ohio-state.edu Connected to cheops.cis.ohio-state.edu. 220 cheops.cis.ohio-state.edu FTP server (Version 5.49 Tue May 9 14:01:04 EDT 19 89) ready. Name (cheops.cis.ohio-state.edu:you): anonymous 331 Guest login ok, send ident as password. Password: 230 Guest login ok, access restrictions apply. ftp> cd pub/neuroprose 250 CWD command successful. ftp> binary 200 Type set to I. ftp> get berg.phrase_structure.ps.Z 200 PORT command successful. 150 Opening BINARY mode data connection for berg.phrase_structure.ps.Z (107077 b ytes). 226 Transfer complete. local: berg.phrase_structure.ps.Z remote: berg.phrase_structure.ps.Z 107077 bytes received in 7.3 seconds (14 Kbytes/s) ftp> quit 221 Goodbye. yourprompt> Then uncompress the file and print it in your local fashion for printing postscript. ---- Second Way: anonymous ftp via SUNY Albany The file is available via anonymous ftp from ftp.cs.albany.edu (128.204.2.32) as the file tr91-5.ps.Z in the pub directory. It is a compressed postscript file. Below is the log of a typical ftp session to retrieve the file: yourprompt> ftp ftp.cs.albany.edu Connected to karp.albany.edu. 220 karp.albany.edu FTP server (SunOS 4.1) ready. Name (ftp.cs.albany.edu:you): anonymous 331 Guest login ok, send ident as password. Password: 230 Guest login ok, access restrictions apply. ftp> cd pub 250 CWD command successful. ftp> binary 200 Type set to I. ftp> get tr91-5.ps.Z 200 PORT command successful. 150 Binary data connection for tr91-5.ps.Z (128.204.2.36,2116) (107077 bytes). 226 Binary Transfer complete. local: tr91-5.ps.Z remote: tr91-5.ps.Z 107077 bytes received in 1 seconds (1e+02 Kbytes/s) ftp> quit 221 Goodbye. yourprompt> Then uncompress the file and print it in your local fashion for printing postscript. --- Third Way: SUNY Albany Computer Science Department Technical Reports Secretary. A copy of the paper may be requested by writing: Technical Reports Secretary Computer Science Department, LI-67A State University of New York at Albany Albany, New York 12222 USA and requesting a copy of Technical Report TR 91-5 ("Learning Recursive Phrase structure: Combining the Strengths of PDP and X-Bar Syntax" by George Berg). As I do not wish to make an enemy of the technical reports secretary, please only request a copy if you are unable to get one by ftp. ------------------------------------------------------------------------------- | George Berg | Computer Science Dept. | If you want wit in 15 words | | berg at cs.albany.edu | SUNY at Albany, LI 67A | or less, go check Bartlett's | | (518) 442 4267 | Albany, NY 12222 USA | quotations -- I'm busy. | ------------------------------------------------------------------------------- From stevep at cs.uq.oz.au Wed Jul 24 04:41:43 1991 From: stevep at cs.uq.oz.au (stevep@cs.uq.oz.au) Date: Wed, 24 Jul 91 18:41:43 +1000 Subject: NetTools - a package of tools for NN analysis Message-ID: <9107240841.AA03888@client> NetTools is a package of analysis tools and a tech. report demonstrating two of these techniques. Analysis Tools for Neural Networks. by Simon Dennis and Steven Phillips Abstract - A large volume of neural net research in the 1980's involved applying backpropagation to difficult and generally poorly understood tasks. Success was sometimes measured on the ability of the network to replicate the required mapping. The difficulty with this approach, which is essentially a black box analysis, is that we are left with little additional understanding of the problem or the way in which the neural net has solved it. Techniques which can look inside the black box are required. This report focuses on two statistical analysis techniques (Principal Components Analysis and Canonical Discriminant Analysis) as tools for analysing and interpreting network behaviour in the hidden unit layers. Net Tools The following package contains three tools for network analysis: gea - Group Error Analysis pca - Principal Components Analysis cda - Canonical Discriminants Analysis TOOL DESCRIPTIONS Group Error Analysis (gea) Gea counts errors. It takes an output file and a target file and optionally a groups file. Each line in the output file is an output vector and the lines in the targets file are the corresponding correct values. If all values in the output file are within criterion of the those in the target file then the pattern is considered correct. Note that this is a more stringent measure of correctness than the total sum of squares. In particular it requires the outputs to be either high or low rather than taking some average intermediate value. If a groups file is provided then gea will separate the error count into the groups provided. Principal Components Analysis (pca) Principle components analysis takes a set of points in a high dimensional space and determines the major components of variation. The principal components are labeled 0-(n-1) where n is the dimensionality of the space (i.e. the number of hidden units). The original points can be projected onto these vectors. The result is a low dimensional plot which has hopefully extracted the important information from the high dimensional space. Canonical Discriminants Analysis (cda) Canonical discriminant analysis takes a set of grouped points in a high dimensional space and determines the components such that points within a group form tight clusters. These points are called the canonical variates and are labeled 0-(n-1) where n is the dimensionality of the space (i.e. the number of hidden units). The original points can be projected on to these vectors. The result is a low dimensional plot which has clustered the points belonging to each group. TECHNICAL REPORT Reference: Simon Dennis and Steven Phillips. Analysis Tools for Neural Networks. Technical Report 207, Department of Computer, University of Queensland, Queensland, 4072 Australia May, 1991 NetTools.ps is a technical report which demonstrates the results which can be obtained from pca and cda. It outlines the advantages of each and points out some interpretive pitfalls which should be avoided. TUTORIAL The directory tute contains a tutorial designed at the University of Queensland by Janet Wiles and Simon Dennis to introduce students to network analysis. It uses the iris data first published by Fisher in 1936. The backpropagation simulator is tlearn developed at UCSD by Jeffery Elman and colleagues. In addition the tutorial uses the hierarchical clustering program, cluster, which was written by Yoshiro Miyata and modified by Andreas Stolcke. These tools can be obtained as follows $ ftp crl.ucsd.edu Connected to crl.ucsd.edu. 220 crl FTP server (SunOS 4.1) ready. Name (crl.ucsd.edu:mav): anonymous 331 Guest login ok, send ident as password. Password: 230 Guest login ok, access restrictions apply. ftp> cd pub/neuralnets 250 CWD command successful. ftp> bin 200 Type set to I. ftp> get NetTools.tar.Z 200 PORT command successful. 150 Binary data connection for NetTools.tar.Z (130.102.64.15,1240) (185900 bytes). 226 Binary Transfer complete. local: NetTools.tar.Z remote: NetTools.tar.Z 185900 bytes received in 1.9e+02 seconds (0.97 Kbytes/s) ftp> quit 221 Goodbye. $ zcat NetTools.tar.Z | tar -xf - Shalom Simon and Steven ------------------------------------------------------------------------------- Simon Dennis Address: Department of Computer Science Email: mav at cs.uq.oz.au University of Queensland QLD 4072 Australia ------------------------------------------------------------------------------- From T.Rickards at cs.ucl.ac.uk Wed Jul 24 07:43:23 1991 From: T.Rickards at cs.ucl.ac.uk (T.Rickards@cs.ucl.ac.uk) Date: Wed, 24 Jul 91 12:43:23 +0100 Subject: V and V database Message-ID: Dear Connectionists, I am compiling a database of references in the area of `Validation and Verification of Neural Nets'. This is part of a Government sponsored Technology Transfer activity in the UK through the LINNET Neural Network Club. I would appreciate your contributions, and can make the reference list available to the connectionists net. Thanks in advance, Tessa Rickards LINNET INFORMATION DESK T.Rickards at uk.ac.ucl.cs From COBOLEWI at macc.wisc.edu Wed Jul 24 22:40:00 1991 From: COBOLEWI at macc.wisc.edu (Alan B. Cobo-Lewis) Date: Wed, 24 Jul 91 21:40 CDT Subject: simple pictures, tough problems Message-ID: <21072421404519@vms.macc.wisc.edu> Peter J. B. Hancock argues... > evidence that some complex cells respond {\em before} > simple cells. I understand there is some debate about the > simple/complex dichotomy anyway, but such a result does challenge the > traditional story of simple cells feeding into complex ones. Steve Lehar responds... > The fact that a complex cell fires BEFORE a simple cell does not > preclude the possibility that the signal was provided originally by > the simple cells. The nice hierarchical classification of simple, complex, and hypercomplex cells has been assaulted for two reasons: the hypercomplex category is questionable, and the hierarchy is questionable. Hubel and Wiesel (1965) added the hypercomplex classification to describe cells that otherwise seemed like complex cells, but were also end-stopped. Since then, end-stopping has been reported in both simple and complex cells (Schiller et al., 1976; Gilbert, 1977; Kato et al., 1978). The conclusion that end-stopped cells represent a distinct population, rather than there being a continuous distribution of amount of end-stopping has been challenged (Schiller et al., 1976), though Kato et al. (1978) report justification for the use of the discrete classifications "hypercomplex (simple family)" and "hypercomplex (complex family)". Whatever the outcome of that argument, the singular discrete classification "hypercomplex" is typically abandoned today. Hubel and Wiesel (1962, 1965) proposed that one level's input consists of purely excitatory connections from the immediately inferior level (simple -(+)-> complex -(+)-> hypercomplex). This arrangement cannot account for certain features of the cells' receptive fields, but to what extent their proposal for the wiring must be modified is unclear (Rose, 1979). There is evidence that the processing by simple and complex cells take place at least partially in parallel. For one thing, input to the striate cortex feeds into complex cells as well as simple cells. Direct monosynaptic input from the lateral geniculate nucleus to complex cells has been reported (Hoffman & Stone, 1971; Stone, 1972; Bullier & Henry, 1979a, b, c; Henry et al., 1979). For another thing, output from the striate cortex must include projections from simple as well as complex cells. After all, we certainly have absolute phase specificity in our visual perception, though complex cells lack such specificity. Steve Lehar continues... > current implementations of the BCS and MRBCS are of necessity > performed iteratively on sequential binary machines. But the visual > architecture that they presume to emulate is a parallel analog > resonant system, like a violn string (but of course much more > complex) so that it does not take any number of "cycles" as such for a > resonance to set in. Even a parallel system evolves through time. We can treat the vibration of a violin string as proceding in discrete time if our sampling rate is high enough (for bandlimited behavior). Each moment of this discrete time constitutes an iteration. To the extent that the time constant of a biological neural network's behavior is finite, we _do_ have to worry about how many iterations (how much time) it takes for the system to arrive at a solution. References Bullier, J., & Henry, G. H. (1979a). Ordinal position of neurons in cat striate cortex. JOURNAL OF NEUROPHYSIOLOGY, 42, 1251-1263. Bullier, J., & Henry, G. H. (1979b). Neural path taken by afferent streams in striate cortex of the cat. JOURNAL OF NEUROPHYSIOLOGY, 42, 1264-1270. Bullier, J., & Henry, G. H. (1979c). Laminar distribution of first-order neurons and afferent terminals in cat striate cortex. JOURNAL OF NEUROPHYSIOLOGY, 42, 1271-1281. Gilbert, C. D. (1977). Laminar differences in receptive field properties of cells in cat primary visual cortex. JOURNAL OF PHYSIOLOGY, 268, 391-421. Henry, G. H., Harvey, A. R., & Lund, J. S. (1979). The afferent connections and laminar distribution of cells in the cat striate cortex. JOURNAL OF COMPARATIVE NEUROLOGY, 187, 725-744. Hoffman, K.-P., & Stone, J. (1971). Conduction velocity of afferents to cat visual cortex: a correlation with cortical receptive field properties. BRAIN RESEARCH, 32, 460-466. Hubel, D. H., & Wiesel, T. N. (1962). Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. JOURNAL OF PHYSIOLOGY, 160, 106-154. Hubel, D. H., & Wiesel, T. N. (1965). Receptive fields and functional architecture in two nonstriate visual areas (18 and 19) of the cat. JOURNAL OF NEUROPHYSIOLOGY, 28, 229-289. Kato, H., Bishop, P. O., & Orban, G. A. (1978). Hypercomplex and simple/complex cell classification in cat striate cortex. JOURNAL OF NEUROPHYSIOLOGY, 42, 1071-1095. Rose, D. (1979). Mechanisms underlying the receptive field properties of neurons in cat visual cortex. VISION RESEARCH, 19, 533-544. Schiller, P. H., Finlay, B. L., & Volman, S. F. (1976). Quantitative studies of single-cell properties in monkey striate cortex. I. Spatiotemporal organization of receptive fields. JOURNAL OF NEUROPHYSIOLOGY, 39, 1288-1319. Stone, J. (1972). Morphology and physiology of the geniculocortical synapse in the cat: The question of parallel input to the striate cortex. INVESTIGATIVE OPTHAMOLOGY, 11, 338-346. From slehar at park.bu.edu Thu Jul 25 10:52:03 1991 From: slehar at park.bu.edu (Steve Lehar) Date: Thu, 25 Jul 91 10:52:03 -0400 Subject: simple pictures, tough problems In-Reply-To: "Alan B. Cobo-Lewis"'s message of Wed, 24 Jul 91 21:40 CDT <21072421404519@vms.macc.wisc.edu> Message-ID: <9107251452.AA13816@park.bu.edu> > The nice hierarchical classification of simple, complex, and > hypercomplex cells has been assaulted for two reasons: the > hypercomplex category is questionable, and the hierarchy is > questionable. We observe in the visual cortex many different cells, some respond to very simple features, others to more complex features, others to hypercomplex- Ahem! Excuse me- Even more complex features and so on upwards through the temporal lobe to cells that respond to very complex specific stimuli. Are you suggesting that the most complex cells do not take their input from the intermediate level cells, but compute their response directly from the raw input from lateral geniculate? I find this most unlikely! What would be the purpose of all those intermediate level representations if not for the use of the higher level cells? And why do we find ascending complexity of representation in a continuous spatial progression? Why would we not find very high level cells mixed in with simple cells at V1 if they compute their responses independantly to the intermediate levels in V2, V3...? > There is evidence that the processing by simple and complex cells take > place at least partially in parallel. So is your complaint that the visual hierarchy is not a PURE hierarchy because certain connections jump from low levels to very high levels bypassing intermediate levels? You will find no argument from me on that matter. I would assume however that most high level cells would also take input from intermediate levels, so we have a mixed hierarchy with lots of connections everywhere, but a hierarchy nevertheless! The spatial arrangement of cortical regions alone strongly suggests that to be the case. My argument about the spatial/featural transform at each stage holds for a mixed hierarchy as it does for a pure hierarchy. The evidence for this seems overwealming- that the higher the cell is in the hierarchy, generally the larger is the region in the visual field to which it will respond. This is what I mean by the spatial/featural hierarchy, that every stage in the hierarchy increases featural specificity while decreasing spatial specificity. And I maintain that it is that aspect of the hierarchical structure which lends the property of spatial generality which is so hard to achieve in conventional recognition algorithms. > Even a parallel system evolves through time. We can treat the > vibration of a violin string as proceding in discrete time if our > sampling rate is high enough (for bandlimited behavior). Each moment > of this discrete time constitutes an iteration. To the extent that > the time constant of a biological neural network's behavior is finite, > we _do_ have to worry about how many iterations (how much time) it > takes for the system to arrive at a solution. I did not mean to suggest that resonance can be established instantaneously, of course it requires a finite time. I merely meant to say that in the case of resonance, a causal order (bowing causes string to resonate) does not necessarily imply a temporal order (first bowing then resonance) but that the resonance can emerge essentially simultaneous to the bowing, even though the bowing is the cause of the resonance. In the same way, I suggest that the fact that a higher level cell fires before the lower level cell, does not necessarily imply that it is therefore causally independant. Of course by firing I mean firing above ambient noise level, and I would assume that there is some a signal being sent from the lower level cell to the higher one, albeit a weak, noisy and incoherent signal, and that the higher level cell responds to and accentuates any global coherency that it detects in the cacophany of noisy inputs that it receives from many lower cells. The BCS model suggests that the output of the lower level cell is greatly boosted and enhanced when it receives top-down confirmation, or suppressed if it receives top-down disconfirmation, thus a global pattern detected higher up is reflected in the pattern of firing in the lowest levels of the hierarchy. It is this stronger, resonant firing of the low level cell that occurs AFTER the higher cell response, the initial firing might be lost in the noise. This arrangement seems eminantly plausible to me, accounting for a large body of psychophysical data including the ease with which local objects are recognized when they are consistant with the global picture, and conversely, the longer time required to recognize objects that are inconsistant with the global scene. It is clear that the global context plays a large role in local recognition, although of course the global context itself must be built up out of local pieces. How else can one account for these phenomena besides a simultaneous resonant matching between low and high level recognition? From R14502%BBRBFU01.BITNET at CUNYVM.CUNY.EDU Thu Jul 25 13:13:57 1991 From: R14502%BBRBFU01.BITNET at CUNYVM.CUNY.EDU (R14502%BBRBFU01.BITNET@CUNYVM.CUNY.EDU) Date: 25 Jul 91 17:14:57 +01 Subject: No subject Message-ID: Bruxelles, le 24 July 1991 Concerns : Information about ENNS. Thanks to all of your who have shown interest in ENNS. Your name will be on a list and you will get available information in due time. Here are answers to the questions you all asked. - To join, please send a check of 50 ECUs (European community currency) or its equivalent in any currency to : Prof. John Taylor King's College London Dept. of Mathematics University of London Stand, London WC2R 2LS England -The special interest groups are not yet formed and will materialize during ICANN 1992 in Brighton Professor Igor Aleksander is one of the organizers. His address is I. Aleksander Imperial College of Science and Technology Dept. of Computing 180 Queen's Gate London SW7 2B2 U.K. The question of bylaws will be settled by 1992 A. Babloyantz Secretary ENNS From BOYD at RHODES.BITNET Fri Jul 26 07:58:00 1991 From: BOYD at RHODES.BITNET (BOYD) Date: Fri, 26 Jul 91 05:58 CST Subject: unsubscribe Message-ID: <9EDDC4C9C0A00063@BITNET.CC.CMU.EDU> Please delete me from this list. From koch at CitIago.Bitnet Fri Jul 26 23:19:03 1991 From: koch at CitIago.Bitnet (Christof Koch) Date: Fri, 26 Jul 91 20:19:03 PDT Subject: Time to converge Message-ID: <910726201758.2040d436@Iago.Caltech.Edu> Re. the 90 msec response time for face cells in the temporal lobe. Most of that time is due to retinal elements. Given an average neuronal time-constant of cortical cells of 10 msec, this does not leave any time to iterate at all, given support to the idea that for the class of highly-overlearned patterns, such as faces, letters, etc. the brain essentially acts like a look-up table and does not compute in any real sense of the worl. This fits with Poggio's RBF approach or with Bartlet Mel's sigma-pi neurons. The almost infinite class of objects which we see only a few times in our life is much more interesting to investigate. Howvever, since we can respond to these objects with say, approx. 200 msec, we don;t have time for a lot of iterations, whether their digital cycles or analog time-constants. This is one reason the original Marr-Poggio cooperative stereo algoprith was so interesting, since it converged in 7-10 cycles. Finally, there exists no good physiological experiment (with the exception of loss of length inhibition in LGN; see Silito in Nature, 1989) showing that any functional property goes away after inactivation of a higher area. This is rather embarassing, given, for instance, the fact that at least 10-20 times more fibers project from layer VI in area 17 to LGN than from LGN to area 17. Christof P.S. That is no evidence for fast communication not involving action potentials in the brain. The distances are too big and all relevant biophysical mechanism except solitons too slow... C. From jose at tractatus.siemens.com Sat Jul 27 11:07:59 1991 From: jose at tractatus.siemens.com (Steve Hanson) Date: Sat, 27 Jul 1991 11:07:59 -0400 (EDT) Subject: Time to converge In-Reply-To: <910726201758.2040d436@Iago.Caltech.Edu> References: <910726201758.2040d436@Iago.Caltech.Edu> Message-ID: >entially acts like a look-up table and does not compute in any real >sense of the worl. This fits with Poggio's RBF approach or with >Bartlet Mel's sigma-pi neurons. Wouldn't this fit better with any feed-forward net...? and assuming lots of memorys systematically related a linear-logistic type net (more global)... Steve Stephen J. Hanson Learning & Knowledge Acquisition Group SIEMENS Research 755 College Rd. East Princeton, NJ 08540 From mclennan at cs.utk.edu Mon Jul 29 12:07:06 1991 From: mclennan at cs.utk.edu (mclennan@cs.utk.edu) Date: Mon, 29 Jul 91 12:07:06 -0400 Subject: convergence times Message-ID: <9107291607.AA28355@duncan.cs.utk.edu> The scope for iterative algorithms in the brain is much greater if they take place in the dendritic net via graded interactions. Since a chemical synapse has about a 0.5 msec. delay, 10 iterations might occur in 10 msec. As long as there are no action potentials in the loop, iteration can go pretty fast (in neural terms). For several decades now Gordon Shepherd has been stressing the importance of computation in the dendritic net. Bruce MacLennan Department of Computer Science 107 Ayres Hall The University of Tennessee Knoxville, TN 37996-1301 (615)974-0994/5067 maclennan at cs.utk.edu From terry at jeeves.UCSD.EDU Mon Jul 29 18:33:20 1991 From: terry at jeeves.UCSD.EDU (Terry Sejnowski) Date: Mon, 29 Jul 91 15:33:20 PDT Subject: convergence times Message-ID: <9107292233.AA03423@jeeves.UCSD.EDU> There is very little evidence for dendrodendritic interactions in cerebral cortex, although there is a lot in the olfactory bulb, thalamus, and retina. Thus, even short range cortical interactions must use action potentials. Since there is a dendritic delay of around 10 ms because of electrotonic conduction, the minimal cycle time is 10 ms. Terry ----- From koch at CitIago.Bitnet Mon Jul 29 19:17:59 1991 From: koch at CitIago.Bitnet (Christof Koch) Date: Mon, 29 Jul 91 16:17:59 PDT Subject: convergence times In-Reply-To: Your message <9107291607.AA28355@duncan.cs.utk.edu> dated 29-Jul-1991 Message-ID: <910729161754.2040995b@Iago.Caltech.Edu> The problem with graded interaction in the brain, mediated through dendro- dendritic synapses, is that for the most part they don't seem to exist in cortex proper. In the retina, in the thalamus, in the brain stem etc. but not in cortex. Yes, the hippocampus has some, in particular in young animals, but its certainly not a wide-spread phenomena. This leaves action-potentials or much slower active or passive transport mechanisms. Christof From gbugmann at nsis86.cl.nec.co.jp Mon Jul 29 20:07:50 1991 From: gbugmann at nsis86.cl.nec.co.jp (Masahide.Nomura) Date: Tue, 30 Jul 91 09:07:50+0900 Subject: time Message-ID: <9107300007.AA23502@nsis86.cl.nec.co.jp> C. Koch said: "The brain essentially acts like a look-up table and does not compute in any real sense of the worl. This fits with Poggio's RBF approach or with Bartlet Mel's sigma-pi neurons." Steve J. Hanson asked: "Wouldn't this fit better with any feed-forward net...?" In fact, both feedforward nets and Poggio's technique can be used to realize multidimensional mappings. The difference lies in the robustness. While a feedforward net mapping can be badly degraded by the loss of a neuron, the mapping is only locally degraded with Poggio's technique. Guido Bugmann Fundamental Research Laboratory NEC Corporation 34 Miyukigaoka Tsukuba, Ibaraki 305 Japan A A A A A A A A A A A A A A A From nelsonde%avlab.dnet at wrdc.af.mil Tue Jul 30 08:35:41 1991 From: nelsonde%avlab.dnet at wrdc.af.mil (nelsonde%avlab.dnet@wrdc.af.mil) Date: Tue, 30 Jul 91 08:35:41 EDT Subject: Taxonomy of Neural Network Optimality Message-ID: <9107301235.AA07942@wrdc.af.mil> I N T E R O F F I C E M E M O R A N D U M Date: 30-Jul-1991 08:25am EST From: Dale E. Nelson NELSONDE Dept: AAAT-1 Tel No: 57646 TO: Remote Addressee ( _LABDDN::"CONNECTIONISTS at CS.CMU.EDU" ) Subject: Taxonomy of Neural Network Optimality We are working on a taxonomy of parameters which may be used to determine if one network is better than another. To this end the following list has been developed. I would be interested in any comments or references to this kind of listing. PERFORMANCE: 1. Most accurate on training set. 2. Most accurate on test set. 3. Best at generalization. 4. Performance independent of starting weights. 5. Performance independent of training exemplar order. TRAINING: 6. Trains in fewest epochs. 7. Trains in fewest Floating Point/Integer Operations. 8. Trains in least clock time. 9. Trains in fewest exemplars. 10. Uses least memory. TOPOLOGY: 11. Has fewest layers. 12. Has fewest nodes. 13. Has fewest interconnects. 14. Distributed representation (fault tolerant) I know that there is no explaination of what each of these mean, which means that they are open to some interpretation. I would appreciate any comments about additions to this list. Dale E. Nelson nelsonde%avlab.dnet at wrdc.af.mil From erikf at sans.bion.kth.se Tue Jul 30 08:45:35 1991 From: erikf at sans.bion.kth.se (Erik Fransen) Date: Tue, 30 Jul 91 14:45:35 +0200 Subject: Time to converge References: <910726201758.2040d436@Iago.Caltech.Edu> Message-ID: <9107301245.AA00658@cerebellum> We have recently done some simulations of recurrent networks with realistic, spiking pyramidal cells (Hodgkin-Huxley type eq., multi-compartment model (1)) as units. Our results show that relaxation times are rather short (2). In case of a complete and undistorted pattern as stimuli, response time was around 25 ms. With incomplete or distorted or mixed patterns as stimuli, response time was around 50 ms. This has been done with a small network of 50 cells. Axonal plus synaptic delay times were 1 ms. Currently we are working on a much larger network. Thus, relaxation in recurrent cortical circuits seems compatible with a stimulus-response time of 90 ms. In a sequence of processing stages (retina, LGN, V1 ...) the first "leading wave" would take about 15 ms per stage. Priming could lower this time considerably. Actual computation in each stage will take place in parallel, but in cases of "familiar" inputs later responses will not differ much from the first. So, with "familiar" inputs the response will look like a pure feed-forward operation. Only with more complex inputs relaxation will modify the initial response. Our feeling is that in 200 ms a lot of multi-stage relaxations can take place... Erik Fransen Anders Lansner SANS Dept. of Numerical Analysis and Computing Sci. Royal Inst. of Technology, Stockholm (1) Ekeberg, Wallen, Lansner, Traven, Brodin, Grillner (1991), A Computer Based Model for Realistic Simulations of Neural Networks, (to appear in Biol. Cybernetics) (2) Lansner & Fransen (1991), Modeling Hebbian Cell Assemblies Comprised of Cortical Neurons, (Submitted) From petsche at learning.siemens.com Tue Jul 30 09:24:44 1991 From: petsche at learning.siemens.com (Thomas Petsche) Date: Tue, 30 Jul 91 09:24:44 EDT Subject: time In-Reply-To: <9107300007.AA23502@nsis86.cl.nec.co.jp> Message-ID: <9107301324.AA02004@learning.siemens.com.siemens.com> Masahide.Nomura wrote: >In fact, both feedforward nets and Poggio's technique can be used to >realize multidimensional mappings. The difference lies in the robustness. >While a feedforward net mapping can be badly degraded by the loss of a >neuron, the mapping is only locally degraded with Poggio's technique. But this is not a property of a feedforward network (or RBF's for that matter). For FF nets, the lack of predictable and dependable fault tolerance is a property of `vanilla' back prop. For backprop with weight decay, we can accurately predict that the network will be completely INtolerant to any faults. OTOH, it is quite possible to obtain a fault tolerant feedforward network by (1) designing a fault tolerant generic network which can then be trained [1] or (2) modifying backprop to encourage fault tolerant representations [as yet unpublished work by other researchers]. [1] @article{petsche-dickinson-1990, author = {T. Petsche and B.W. Dickinson}, journal = {IEEE Transactions on Neural Networks}, month = jun, number = {2}, pages = {154--166}, title = {Trellis Codes, Receptive Fields, and Fault-Tolerant, Self-Repairing Neural Networks}, volume = {1}, year = {1990}, note={Errata for eqn 1 available from first author.} } From lyle at ai.mit.edu Tue Jul 30 11:43:20 1991 From: lyle at ai.mit.edu (Lyle J. Borg-Graham) Date: Tue, 30 Jul 91 11:43:20 EDT Subject: convergence times In-Reply-To: mclennan@cs.utk.edu's message of Mon, 29 Jul 91 12:07:06 -0400 <9107291607.AA28355@duncan.cs.utk.edu> Message-ID: <9107301543.AA22780@peduncle> might occur in 10 msec. As long as there are no action potentials in the loop, iteration can go pretty fast (in neural terms). For several decades now Gordon Shepherd has been stressing the importance of computation in the dendritic net. It seems pretty likely that intradendritic interactions (electrical or chemical) are important; at least the biophysical substrate is rich enough, and especially so after adding in the morphometrics. Finding the smoking gun (as opposed to simulations which show plausibility) is probably imminent. But iterations, strictly speaking, require a discrete *loop*, and the speed of such a pathway is of course limited by the slowest element in the chain, not by the fastest (e.g. some 500us *component* of a synaptic event). I think that iterative mechanisms constitute one class of interactions, while (continous) feedback (which can be very fast) mechanisms are another. From Scott.Fahlman at SEF-PMAX.SLISP.CS.CMU.EDU Tue Jul 30 12:33:03 1991 From: Scott.Fahlman at SEF-PMAX.SLISP.CS.CMU.EDU (Scott.Fahlman@SEF-PMAX.SLISP.CS.CMU.EDU) Date: Tue, 30 Jul 91 12:33:03 EDT Subject: Taxonomy of Neural Network Optimality In-Reply-To: Your message of Tue, 30 Jul 91 08:35:41 -0400. <9107301235.AA07942@wrdc.af.mil> Message-ID: PERFORMANCE: 1. Most accurate on training set. 2. Most accurate on test set. 3. Best at generalization. What does this mean if not the same as 2? Also, "most accurate" might mean number of cases wrong or something like sum-squared error, depending on the problem. 4. Performance independent of starting weights. 5. Performance independent of training exemplar order. TRAINING: 6. Trains in fewest epochs. Some problems and algorithms just don't fit into epochs. Probably better to use "pattern presentations", but some algorithms don't even fit into that. 7. Trains in fewest Floating Point/Integer Operations. 8. Trains in least clock time. Machine-dependent, of course, so it says very little about the algorithm. 9. Trains in fewest exemplars. 10. Uses least memory. TOPOLOGY: 11. Has fewest layers. "Layers" may be ill-defined. Maybe look instead at the longest path from input to output. 12. Has fewest nodes. 13. Has fewest interconnects. 14. Distributed representation (fault tolerant) A few others: 15. How hard it is to partition/parallelize the algorithm? 16. How many parameters must the user adjust, and how critical are the adjustments? 17. Related to 16: What chance of immediate success on a new problem? 18. Range of problems covered: Discrete vs. analog inputs and outputs Can it handle time series? Can it handle noisy data? (i.e misclassifying a few training points leads to better generalization) 19. {Debugged, supported, portable, free} implementation available? 20. If not, how hard it the algorithm to implement? 21. Biologically plausible? 22. How does it scale with problem size? -- Scott From nmg at skivs.ski.org Tue Jul 30 13:10:38 1991 From: nmg at skivs.ski.org (nmg@skivs.ski.org) Date: 30 Jul 91 10:10:38 PDT (Tue) Subject: convergence times Message-ID: <9107301010.AA25081@skivs.ski.org> Two cautionary notes on Sejnowski's latest message: 1) I presume that the 10 msec estimate for cycle time comes from typical values of membrane time constant. However, this estimate might be substantially off under physiological conditions. Under these conditions, cell membranes might be constantly bathed in neurotransmitters, which by increasing membrane conductance, reduce the cell's time constants. Moreover, voltage-dependent ionic channels in dendritic trees might considerably reduce electrotonic conductance times. Hence, it would not be surprising if the contribution of electrotonic condution to "cycle time" is significantly smaller than 1 msec. 2) It is true that conventional dendrodendritic synapses appear to be essentially inexistent in the neocortex. But one must remember that this statement relates to vesicle-containing synapses identified with electron-microscopy methods. In the retina, non-vesicular synapses have been observed. Although, once again, the retina and the cortex have several differences, it seems that it would be hard to rule out the existence of such non-vesicular synapses in the cortex based on available data. And if one may speculate, then why not to have this type of synapse being dendrodendritic. Anyway, my point is that it might be too soon to state that action potentials mediate all cortical local synaptic circuits. Norberto From sloman%meme at Forsythe.Stanford.EDU Tue Jul 30 13:32:26 1991 From: sloman%meme at Forsythe.Stanford.EDU (sloman%meme@Forsythe.Stanford.EDU) Date: Tue, 30 Jul 91 10:32:26 PDT Subject: manuscript available: Feature-based induction Message-ID: <9107301732.AA08753@meme> A compressed postscript version of the following paper has been placed in the pub/neuroprose directory for anonymous ftp from cheops.cis.ohio-state.edu. The paper concerns a very simple connectionist model (n inputs, one output, and delta-rule learning) of people's willingness to affirm a property of one natural-kind category given confirmation of the property in other categories. The paper has been submitted for publication. Feature-Based Induction Steven A. Sloman Dept. of Psychology University of Michigan e-mail: sloman at psych.stanford.edu Abstract A connectionist model of argument strength is proposed that applies to categorical arguments involving natural categories and predicates about which subjects have few prior beliefs. An example is *robins have sesamoid bones, therefore falcons have sesamoid bones*. The model is based on the hypotheses that argument strength (i) increases with the overlap between features of the combined premise categories and features of the conclusion category; and (ii) decreases with the amount of prior knowledge about the conclusion category. The model assumes a two-stage process. First, premises are encoded by connecting the features of premise categories to the predicate. Second, conclusions are tested by examining the degree of activation of the predicate upon presentation of the features of the conclusion category. The model accounts for 13 qualitative phenomena and shows close quantitative fits to several sets of argument-strength ratings. From SCHOLTES at ALF.LET.UVA.NL Tue Jul 30 21:10:00 1991 From: SCHOLTES at ALF.LET.UVA.NL (SCHOLTES) Date: Tue, 30 Jul 91 21:10 MET Subject: Neural Nets in Information Retrieval Message-ID: <0892B405AAA00063@BITNET.CC.CMU.EDU> Recently, we started some research in the possible applications of Neural Nets in Information Retrieval. At this moment we are trying to compile a list of literature references and names of other people interested in this subject. Any information would be greatly appreciated. I will put the final list on the network. Thanks, Jan Scholtes ******************************************************************************* Jan C. Scholtes University of Amsterdam Faculty of Arts Department of Computational Linguistics Dufaystraat 1 1075 GR AMSTERDAM The Netherlands Tel: +31 20 6794273 Fax: +31 20 6710793 Email: scholtes at alf.let.uva.nl ******************************************************************************* From BOYD at RHODES.BITNET Wed Jul 31 07:57:00 1991 From: BOYD at RHODES.BITNET (BOYD) Date: Wed, 31 Jul 91 05:57 CST Subject: UNSUBSCRIBE Message-ID: <8C8E36FD0AA00063@BITNET.CC.CMU.EDU> UNSUBSCRIBE From BOYD at RHODES.BITNET Wed Jul 31 07:58:00 1991 From: BOYD at RHODES.BITNET (BOYD) Date: Wed, 31 Jul 91 05:58 CST Subject: SIGNOFF Message-ID: <8CBC76486AA00063@BITNET.CC.CMU.EDU> SIGNOFF From kamil at apple.com Wed Jul 31 10:01:10 1991 From: kamil at apple.com (Kamil A. Grajski) Date: Wed, 31 Jul 91 07:01:10 -0700 Subject: Biblio Request - NN Implementations Message-ID: <9107311401.AA15358@apple.com> Hi folks, I am interested in receiving bibliographic information concerning the implementation of connectionist architectures on existing machines. Beyond basic back-prop benchmarks, I'm interested in the braod range of machine architectures which have been explored, as well as the full range of connectionist networks. I'd like to focus more on machines than custom chips, etc., for the moment. I'd especially like to hear from our European and Pacific colleagues. An organized bibliography will be posted to the board. Thanks in advance, Kamil P.S. If you happen to have a reprint you'd like to share, I'm at: Kamil A. Grajski Apple Computer Inc. 20525 Mariani Avenue Mail Stop 76-7E Cupertino, CA 95014 kamil at apple.com (408) 974-1313 From issnnet at park.bu.edu Wed Jul 31 14:04:39 1991 From: issnnet at park.bu.edu (issnnet@park.bu.edu) Date: Wed, 31 Jul 91 14:04:39 -0400 Subject: comp.org.issnnet PASSES Message-ID: <9107311804.AA29359@copley.bu.edu> VOTE RESULT: comp.org.issnnet PASSES (188 YES / 22 NO) ------------------------------------------------------ VOTING PERIOD ENDED JULY 25, 1991 GROUP NAME: comp.org.issnnet STATUS: unmoderated CHARTER: The newsgroup shall serve as a medium for discussions pertaining to the International Student Society for Neural Networks (ISSNNet), Inc., and to its activities and programs as they pertain to the role of students in the field of neural networks. Details were posted in the REQUEST FOR DISCUSSION, and can be requested from . In accord with USENET guidelines on creation of new newsgroups, the proposed group passes because YES votes exceed NO votes by more than 100 votes, and more than 2/3 of all votes were in favor of creation. A complete list of YES and NO voters can be found on the newsgroups comp.ai.neural-nets and news.announce.newgroups, or may be requested from issnnet at park.bu.edu. Any discrepancies in this list should be brought to the immediate attention of issnnet at park.bu.edu. The new group will be created after a 5 day waiting period (scheduled on August 4) unless serious objections are raised over these results. ------------------------------------------------------------------------ >> Many thanks to all the voters who supported creation of this new << >> group. See you on comp.org.issnnet in a few days! << ------------------------------------------------------------------------ ISSNNet, Inc. is a non-profit corporation in the Commonwealth of Massachusetts. NOTE -- NEW SURFACE ADDRESS: ISSNNet, Inc. P.O. Box 15661 Boston, MA 02215 USA Contact person: Paolo Gaudiano, ISSNNet Vice President Department of Cognitive and Neural Systems Boston University 111 Cummington Street Boston, MA 02215 Phone: (617) 353-6181 353-9482 e-mail: gaudiano at park.bu.edu (INTERNET)