<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<div class="moz-cite-prefix">M&P wrote:</div>
<div class="moz-cite-prefix">
<div class="moz-cite-prefix">
<div class="moz-cite-prefix">" <span style="color: rgb(82, 89,
96); font-family: -apple-system, BlinkMacSystemFont,
"Segoe UI", "Liberation Sans",
sans-serif; font-size: 15px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-weight: 400; letter-spacing: normal; orphans: 2;
text-align: left; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255,
255, 255); text-decoration-thickness: initial;
text-decoration-style: initial; text-decoration-color:
initial; display: inline !important; float: none;">The
perceptron has shown itself worthy of study despite (and
even because of!) its severe limitations, It has many
features to attract attention: its linearity; its intriguing
learning theorem; its clear paradigmatic simplicity as a
kind of parallel computation.<span> </span></span><span
style="margin: 0px; padding: 0px; border: 0px none;
font-style: normal; font-variant-ligatures: normal;
font-variant-caps: normal; font-variant-numeric: inherit;
font-variant-east-asian: inherit; font-stretch: inherit;
line-height: inherit; vertical-align: baseline; box-sizing:
inherit; color: rgb(82, 89, 96); letter-spacing: normal;
text-align: left; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255,
255, 255); text-decoration-thickness: initial;
text-decoration-style: initial; text-decoration-color:
initial;">There is no reason to suppose that any of these
virtues carry over to the many-layered version.</span><span
style="font-weight: normal;"><strong style="margin: 0px;
padding: 0px; border: 0px none; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-variant-numeric: inherit; font-variant-east-asian:
inherit; font-stretch: inherit; line-height: inherit;
vertical-align: baseline; box-sizing: inherit; color:
rgb(82, 89, 96); letter-spacing: normal; text-align: left;
text-indent: 0px; text-transform: none; white-space:
normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;
background-color: rgb(255, 255, 255);
text-decoration-thickness: initial; text-decoration-style:
initial; text-decoration-color: initial;"></strong></span><strong
style="margin: 0px; padding: 0px; border: 0px; font-style:
normal; font-variant-ligatures: normal; font-variant-caps:
normal; font-variant-numeric: inherit;
font-variant-east-asian: inherit; font-weight: bold;
font-stretch: inherit; line-height: inherit; font-family:
-apple-system, BlinkMacSystemFont, "Segoe UI",
"Liberation Sans", sans-serif; font-size: 15px;
vertical-align: baseline; box-sizing: inherit; color:
rgb(82, 89, 96); letter-spacing: normal; orphans: 2;
text-align: left; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255,
255, 255); text-decoration-thickness: initial;
text-decoration-style: initial; text-decoration-color:
initial;"> Nevertheless, we consider it to be an important
research problem to elucidate (or reject) our intuitive
judgment that the extension is sterile. Perhaps some
powerful convergence theorem will be discovered, or some
profound reason for the failure to produce an interesting
"learning theorem" for the multilayered machine will be
found." (pp231-232)</strong></div>
<div class="moz-cite-prefix"><strong style="margin: 0px;
padding: 0px; border: 0px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-variant-numeric: inherit; font-variant-east-asian:
inherit; font-weight: bold; font-stretch: inherit;
line-height: inherit; font-family: -apple-system,
BlinkMacSystemFont, "Segoe UI", "Liberation
Sans", sans-serif; font-size: 15px; vertical-align:
baseline; box-sizing: inherit; color: rgb(82, 89, 96);
letter-spacing: normal; orphans: 2; text-align: left;
text-indent: 0px; text-transform: none; white-space: normal;
widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255);
text-decoration-thickness: initial; text-decoration-style:
initial; text-decoration-color: initial;"><br>
</strong></div>
<div class="moz-cite-prefix"><span style="margin: 0px; padding:
0px; border: 0px none; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-variant-numeric: inherit; font-variant-east-asian:
inherit; font-stretch: inherit; line-height: inherit;
vertical-align: baseline; box-sizing: inherit; color:
rgb(82, 89, 96); letter-spacing: normal; text-align: left;
text-indent: 0px; text-transform: none; white-space: normal;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
background-color: rgb(255, 255, 255);
text-decoration-thickness: initial; text-decoration-style:
initial; text-decoration-color: initial;">The claim that
they killed NNs in the 1970s is indeed an exaggeration
propagated by the PDP group for obvious reasons. There was
plenty of NN research in the 1970s--Amari, Grossberg, etc.
Not surprisingly, the same is true of the so-called "second
neural network winter". <br>
</span><strong style="margin: 0px; padding: 0px; border: 0px
none; font-style: normal; font-variant-ligatures: normal;
font-variant-caps: normal; font-variant-numeric: inherit;
font-variant-east-asian: inherit; font-stretch: inherit;
line-height: inherit; vertical-align: baseline; box-sizing:
inherit; color: rgb(82, 89, 96); letter-spacing: normal;
text-align: left; text-indent: 0px; text-transform: none;
white-space: normal; word-spacing: 0px;
-webkit-text-stroke-width: 0px; background-color: rgb(255,
255, 255); text-decoration-thickness: initial;
text-decoration-style: initial; text-decoration-color:
initial;"></strong><strong style="margin: 0px; padding: 0px;
border: 0px; font-style: normal; font-variant-ligatures:
normal; font-variant-caps: normal; font-variant-numeric:
inherit; font-variant-east-asian: inherit; font-weight:
bold; font-stretch: inherit; line-height: inherit;
font-family: -apple-system, BlinkMacSystemFont, "Segoe
UI", "Liberation Sans", sans-serif;
font-size: 15px; vertical-align: baseline; box-sizing:
inherit; color: rgb(82, 89, 96); letter-spacing: normal;
orphans: 2; text-align: left; text-indent: 0px;
text-transform: none; white-space: normal; widows: 2;
word-spacing: 0px; -webkit-text-stroke-width: 0px;
background-color: rgb(255, 255, 255);
text-decoration-thickness: initial; text-decoration-style:
initial; text-decoration-color: initial;"> </strong></div>
<div class="moz-cite-prefix"><strong style="margin: 0px;
padding: 0px; border: 0px; font-style: normal;
font-variant-ligatures: normal; font-variant-caps: normal;
font-variant-numeric: inherit; font-variant-east-asian:
inherit; font-weight: bold; font-stretch: inherit;
line-height: inherit; font-family: -apple-system,
BlinkMacSystemFont, "Segoe UI", "Liberation
Sans", sans-serif; font-size: 15px; vertical-align:
baseline; box-sizing: inherit; color: rgb(82, 89, 96);
letter-spacing: normal; orphans: 2; text-align: left;
text-indent: 0px; text-transform: none; white-space: normal;
widows: 2; word-spacing: 0px; -webkit-text-stroke-width:
0px; background-color: rgb(255, 255, 255);
text-decoration-thickness: initial; text-decoration-style:
initial; text-decoration-color: initial;"><br>
</strong></div>
</div>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">On 2/1/2022 3:33 AM, Schmidhuber
Juergen wrote:<br>
</div>
<blockquote type="cite"
cite="mid:E505976E-16DB-4EDF-A94B-F975ADB2FF1B@supsi.ch">
<pre class="moz-quote-pre" wrap="">Thanks, Barak! Indeed, I should have said in the email msg that _others_ interpreted the book of Minsky & Papert [M69] in this way. My report explicitly mentions Terry [S20] who wrote in 2020:
"The great expectations in the press (Fig. 3) were dashed by Minsky and Papert (7), who showed in their book Perceptrons that a perceptron can only represent categories that are linearly separable in weight space. Although at the end of their book Minsky and Papert considered the prospect of generalizing single- to multiple-layer perceptrons, one layer feeding into the next, they doubted there would ever be a way to train these more powerful multilayer perceptrons. Unfortunately, many took this doubt to be definitive, and the field was abandoned until a new generation of neural network researchers took a fresh look at the problem in the 1980s.”
However, as mentioned above, the 1969 book [M69] addressed a "problem" of Gauss & Legendre's shallow learning (~1800) [DL1-2] that had already been solved 4 years prior by Ivakhnenko & Lapa's popular deep learning method [DEEP1-2][DL2] (and then also in 1967 by Amari's SGD for MLPs [GD1-2]). Deep learning research was not abandoned in the 1970s. It was alive and kicking, especially outside of the Anglosphere. [DEEP2][GD1-3][CNN1][DL1-2]
See Sec. II and Sec. XIII of the report: <a class="moz-txt-link-freetext" href="https://people.idsia.ch/~juergen/scientific-integrity-turing-award-deep-learning.html">https://people.idsia.ch/~juergen/scientific-integrity-turing-award-deep-learning.html</a>
Cheers,
Jürgen
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">On 1 Feb 2022, at 14:17, Barak A. Pearlmutter <a class="moz-txt-link-rfc2396E" href="mailto:barak@pearlmutter.net"><barak@pearlmutter.net></a> wrote:
Jürgen,
It's fantastic that you're helping expose people to some important bits of scientific literature.
But...
</pre>
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">Minsky & Papert [M69] made some people think that Rosenblatt [R58-62] had only linear NNs plus threshold functions
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">
If you actually read Minsk and Papert's "Perceptrons" book, this is not a misconception it encourages. It defines a "k-th order perceptron" as a linear threshold unit preceded by an arbitrary set of fixed nonlinearities with fan-in k. (A linear threshold unit with binary inputs would, in this terminology, be a 1st-order perceptron.) All their theorems are for k>1. For instance, they prove that a k-th order perceptron cannot do (k+1)-bit parity, which in the special case of k=1 simplifies to the trivial observation that a simple linear threshold unit cannot do xor.
<perceptrons-book-cover-1.jpg> <perceptron-diagram-1.jpg>
This is why you're not supposed to directly cite things you have not actually read: it's too easy to misconstrue them based on inaccurate summaries transmitted over a series of biased noisy compressive channels.
Cheers,
--Barak.
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">
</pre>
</blockquote>
<p><br>
</p>
<pre class="moz-signature" cols="72">--
Pierre Baldi, Ph.D.
Distinguished Professor, Department of Computer Science
Director, Institute for Genomics and Bioinformatics
Associate Director, Center for Machine Learning and Intelligent Systems
University of California, Irvine
Irvine, CA 92697-3435
(949) 824-5809
(949) 824-9813 [FAX]
Assistant: Janet Ko <a class="moz-txt-link-abbreviated" href="mailto:jko@uci.edu">jko@uci.edu</a></pre>
</body>
</html>