<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Dear Juergen,<br>
<br>
Congratulations on the draft and 600+ references! Thank you very
much for asking for reference. Cresceptron generated heated debate
then (e.g., Takeo Kanade's comments). Some people commented that
Cresceptron started the learning for computer vision from cluttered
scenes. Of course, it had many problems then. To save your time, I
cut and paste the major characterization of Cresceptron from my web:<br>
<br>
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
<font color="#000000">1991 (IJCNN 1992) - 1997 (IJCV): <a
href="http://www.cse.msu.edu/%7Eweng/research/cresceptron.html">Cresceptron</a>.
</font>
<dl>
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
<dt><font color="#000000"> It appeared to be the first deep
learning network that adapts its connection structure. </font>
</dt>
<dt><font color="#000000">- It appeared to be the first visual
learning program for both detecting and recognizing general
objects from cluttered complex natural background. </font> </dt>
<dt><font color="#000000">- It also did segmentation, but in
another separate top-down segmentation phase while the network
did not do recognition. </font> </dt>
<dt><font color="#000000">- The number of neural planes
dynamically and incrementally grew from interactive
experience, but the number of layers (15 in the experiments)
was determined by the image size.</font> </dt>
<dt><font color="#000000">- All the internal network learning was
fully automatic --- there was no need for manual intervention
once the learning (development) had started. </font> </dt>
<dt><font color="#000000">- It required pre-segmentation for
teaching: A human outlined the object contours for supervised
learning. This avoided learning background.</font> </dt>
<dt><font color="#000000">- Its internal features were
automatically grouped through the last-layer motor supervision
(class labels) but learnings of internal features were all
unsupervised.</font> </dt>
<dt><font color="#000000">- It uses local match-and-maximization
paired-layer architecture which corresponds to logic-AND and
logic-OR in multivalue logic (Tommy Poggio used a term HMAX
later).</font> </dt>
<dt><font color="#000000">- The intrinsic convolution mechanism of
the network provided both shift invariance and distortion
tolerance. (Later WWNs are better in learning location as one
of concepts.)</font> </dt>
<dt><font color="#000000">- </font>It is a cascade network:
features in a layer are learned from features of the previous
layer, but not earlier. (This cascade restriction was overcome
by later WWNs.) </dt>
<dt><font color="#000000">- Inspired by Neocognitron (K. Fukushima
1975) which was for recognition of individual characters in a
uniform background.</font> </dt>
</dl>
If you are so kind to cite it, I guess that probably it belongs to
your section 5.9 1991-: Deep Hierarchy of Recurrent NNs.<br>
If it does not fit your article, please accept my apology for
wasting your time.<br>
<br>
Just my 2 cents of worth. :)<br>
<br>
Best regards,<br>
<br>
-John<br>
<br>
<div class="moz-cite-prefix">On 4/17/14 11:40 AM, Schmidhuber
Juergen wrote:<br>
</div>
<blockquote cite="mid:DDD42B8B-7F0F-4D8E-93E1-CE348B81BB0D@idsia.ch"
type="cite">
<pre wrap="">Dear connectionists,
here the preliminary draft of an invited Deep Learning overview:
<a class="moz-txt-link-freetext" href="http://www.idsia.ch/~juergen/DeepLearning17April2014.pdf">http://www.idsia.ch/~juergen/DeepLearning17April2014.pdf</a>
Abstract. In recent years, deep neural networks (including recurrent ones) have won numerous contests in pattern recognition and machine learning. This historical survey compactly summarises relevant work, much of it from the previous millennium. Shallow and deep learners are distinguished by the depth of their credit assignment paths, which are chains of possibly learnable, causal links between actions and effects. I review deep supervised learning (also recapitulating the history of backpropagation), unsupervised learning, reinforcement learning & evolutionary computation, and indirect search for short programs encoding deep and large networks.
The draft mostly consists of references (about 600 entries so far). Many important citations are still missing though. As a machine learning researcher, I am obsessed with credit assignment. In case you know of references to add or correct, please send brief explanations and bibtex entries to <a class="moz-txt-link-abbreviated" href="mailto:juergen@idsia.ch">juergen@idsia.ch</a> (NOT to the entire list), preferably together with URL links to PDFs for verification. Please also do not hesitate to send me additional corrections / improvements / suggestions / Deep Learning success stories with feedforward and recurrent neural networks. I'll post a revised version later.
Thanks a lot!
Juergen Schmidhuber
<a class="moz-txt-link-freetext" href="http://www.idsia.ch/~juergen/">http://www.idsia.ch/~juergen/</a>
<a class="moz-txt-link-freetext" href="http://www.idsia.ch/~juergen/whatsnew.html">http://www.idsia.ch/~juergen/whatsnew.html</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
--
Juyang (John) Weng, Professor
Department of Computer Science and Engineering
MSU Cognitive Science Program and MSU Neuroscience Program
428 S Shaw Ln Rm 3115
Michigan State University
East Lansing, MI 48824 USA
Tel: 517-353-4388
Fax: 517-432-1061
Email: <a class="moz-txt-link-abbreviated" href="mailto:weng@cse.msu.edu">weng@cse.msu.edu</a>
URL: <a class="moz-txt-link-freetext" href="http://www.cse.msu.edu/~weng/">http://www.cse.msu.edu/~weng/</a>
----------------------------------------------
</pre>
</body>
</html>