<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#ecca99">
<p><br>
</p>
<div class="moz-cite-prefix">On 2/1/22 3:19 PM, Asim Roy wrote:<br>
</div>
<blockquote type="cite"
cite="mid:BYAPR06MB40697273BC2FA2A8BEC5B3909B269@BYAPR06MB4069.namprd06.prod.outlook.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style>@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}@font-face
{font-family:"Segoe UI";
panose-1:2 11 5 2 4 2 4 2 2 3;}p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
font-size:10.0pt;
font-family:"Courier New";}span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri",sans-serif;}.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}div.WordSection1
{page:WordSection1;}</style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoPlainText">I had some communication with Marvin
Minsky and Jerry Fodor years ago. Here are two quotes from
them:<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<br>
</div>
</blockquote>
<blockquote type="cite"
cite="mid:BYAPR06MB40697273BC2FA2A8BEC5B3909B269@BYAPR06MB4069.namprd06.prod.outlook.com">
<div class="WordSection1">
<p class="MsoPlainText"><o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoNormal">2. Jerry Fodor: “<span
style="background:yellow;mso-highlight:yellow">Arguing with
Connectionists is like arguing with zombies; both are
dead, but neither has noticed it</span>.”</p>
</div>
</blockquote>
<p>I used to argue with Jerry at our Pizza lunchs, and miss him as
he had great counter arguments to almost anything.... but he
would have hated the DL results with language... GPT-xs --would
have liked to know what his counters would have been.. <br>
</p>
<p>but no, he didn't think all connectionists were zombies.. <br>
</p>
<p>Sorry Asim.</p>
<p><br>
</p>
<p>Steve<br>
</p>
<blockquote type="cite"
cite="mid:BYAPR06MB40697273BC2FA2A8BEC5B3909B269@BYAPR06MB4069.namprd06.prod.outlook.com">
<div class="WordSection1">
<p class="MsoNormal"><o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoNormal">Asim Roy<o:p></o:p></p>
<p class="MsoNormal">Professor, Information Systems<o:p></o:p></p>
<p class="MsoNormal">Arizona State University<o:p></o:p></p>
<p class="MsoNormal"><a
href="https://lifeboat.com/ex/bios.asim.roy"
moz-do-not-send="true">Lifeboat Foundation Bios: Professor
Asim Roy</a><o:p></o:p></p>
<p class="MsoNormal"><a
href="https://isearch.asu.edu/profile/9973"
moz-do-not-send="true">Asim Roy | iSearch (asu.edu)</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1
1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Connectionists
<a class="moz-txt-link-rfc2396E" href="mailto:connectionists-bounces@mailman.srv.cs.cmu.edu"><connectionists-bounces@mailman.srv.cs.cmu.edu></a>
<b>On Behalf Of </b>Baldi,Pierre<br>
<b>Sent:</b> Tuesday, February 1, 2022 7:41 AM<br>
<b>To:</b> Schmidhuber Juergen <a class="moz-txt-link-rfc2396E" href="mailto:juergen@idsia.ch"><juergen@idsia.ch></a>;
<a class="moz-txt-link-abbreviated" href="mailto:connectionists@cs.cmu.edu">connectionists@cs.cmu.edu</a><br>
<b>Subject:</b> Re: Connectionists: Scientific Integrity,
the 2021 Turing Lecture, etc.<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">M&P wrote:<o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal">" <span
style="font-size:11.5pt;font-family:"Segoe
UI",sans-serif">
The perceptron has shown itself worthy of study
despite (and even because of!) its severe limitations,
It has many features to attract attention: its
linearity; its intriguing learning theorem; its clear
paradigmatic simplicity as a kind of parallel
computation. </span><span
style="color:#525960;border:none windowtext
1.0pt;padding:0in">There is no reason to suppose that
any of these virtues carry over to the many-layered
version.</span><strong><span
style="font-size:11.5pt;font-family:"Segoe
UI",sans-serif;color:#525960;border:none
windowtext 1.0pt;padding:0in"> Nevertheless, we
consider it to be an important research problem to
elucidate (or reject) our intuitive judgment that
the extension is sterile. Perhaps some powerful
convergence theorem will be discovered, or some
profound reason for the failure to produce an
interesting "learning theorem" for the multilayered
machine will be found." (pp231-232)</span></strong><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span
style="color:#525960;border:none windowtext
1.0pt;padding:0in;background:white">The claim that
they killed NNs in the 1970s is indeed an exaggeration
propagated by the PDP group for obvious reasons. There
was plenty of NN research in the 1970s--Amari,
Grossberg, etc. Not surprisingly, the same is true of
the so-called "second neural network winter".
<br>
<br>
</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">On 2/1/2022 3:33 AM, Schmidhuber Juergen
wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<pre>Thanks, Barak! Indeed, I should have said in the email msg that _others_ interpreted the book of Minsky & Papert [M69] in this way. My report explicitly mentions Terry [S20] who wrote in 2020:<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>"The great expectations in the press (Fig. 3) were dashed by Minsky and Papert (7), who showed in their book Perceptrons that a perceptron can only represent categories that are linearly separable in weight space. Although at the end of their book Minsky and Papert considered the prospect of generalizing single- to multiple-layer perceptrons, one layer feeding into the next, they doubted there would ever be a way to train these more powerful multilayer perceptrons. Unfortunately, many took this doubt to be definitive, and the field was abandoned until a new generation of neural network researchers took a fresh look at the problem in the 1980s.” <o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>However, as mentioned above, the 1969 book [M69] addressed a "problem" of Gauss & Legendre's shallow learning (~1800) [DL1-2] that had already been solved 4 years prior by Ivakhnenko & Lapa's popular deep learning method [DEEP1-2][DL2] (and then also in 1967 by Amari's SGD for MLPs [GD1-2]). Deep learning research was not abandoned in the 1970s. It was alive and kicking, especially outside of the Anglosphere. [DEEP2][GD1-3][CNN1][DL1-2]<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>See Sec. II and Sec. XIII of the report: <a href="https://urldefense.com/v3/__https:/people.idsia.ch/*juergen/scientific-integrity-turing-award-deep-learning.html__;fg!!IKRxdwAv5BmarQ!Nm1TJYh6q6xcsPIanBhciK1xmToTGDvqGz7eWfVGZ5nK8q4Xc0Y2BIejmbYcDho$" moz-do-not-send="true">https://people.idsia.ch/~juergen/scientific-integrity-turing-award-deep-learning.html</a> <o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Cheers,<o:p></o:p></pre>
<pre>Jürgen<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<pre>On 1 Feb 2022, at 14:17, Barak A. Pearlmutter <a href="mailto:barak@pearlmutter.net" moz-do-not-send="true"><barak@pearlmutter.net></a> wrote:<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Jürgen,<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>It's fantastic that you're helping expose people to some important bits of scientific literature.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>But...<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<pre>Minsky & Papert [M69] made some people think that Rosenblatt [R58-62] had only linear NNs plus threshold functions<o:p></o:p></pre>
</blockquote>
<pre><o:p> </o:p></pre>
<pre>If you actually read Minsk and Papert's "Perceptrons" book, this is not a misconception it encourages. It defines a "k-th order perceptron" as a linear threshold unit preceded by an arbitrary set of fixed nonlinearities with fan-in k. (A linear threshold unit with binary inputs would, in this terminology, be a 1st-order perceptron.) All their theorems are for k>1. For instance, they prove that a k-th order perceptron cannot do (k+1)-bit parity, which in the special case of k=1 simplifies to the trivial observation that a simple linear threshold unit cannot do xor.<o:p></o:p></pre>
<pre><perceptrons-book-cover-1.jpg> <perceptron-diagram-1.jpg><o:p></o:p></pre>
<pre>This is why you're not supposed to directly cite things you have not actually read: it's too easy to misconstrue them based on inaccurate summaries transmitted over a series of biased noisy compressive channels.<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Cheers,<o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>--Barak.<o:p></o:p></pre>
</blockquote>
<pre><o:p> </o:p></pre>
<pre><o:p> </o:p></pre>
</blockquote>
<p><o:p> </o:p></p>
<pre>-- <o:p></o:p></pre>
<pre>Pierre Baldi, Ph.D.<o:p></o:p></pre>
<pre>Distinguished Professor, Department of Computer Science<o:p></o:p></pre>
<pre>Director, Institute for Genomics and Bioinformatics<o:p></o:p></pre>
<pre>Associate Director, Center for Machine Learning and Intelligent Systems<o:p></o:p></pre>
<pre>University of California, Irvine<o:p></o:p></pre>
<pre>Irvine, CA 92697-3435<o:p></o:p></pre>
<pre>(949) 824-5809<o:p></o:p></pre>
<pre>(949) 824-9813 [FAX]<o:p></o:p></pre>
<pre>Assistant: Janet Ko <a href="mailto:jko@uci.edu" moz-do-not-send="true">jko@uci.edu</a><o:p></o:p></pre>
</div>
</blockquote>
<div class="moz-signature">-- <br>
<img src="cid:part6.E850C7E7.A930632D@rubic.rutgers.edu"
border="0"></div>
</body>
</html>