Connectionists: Scientific Integrity, the 2021 Turing Lecture, etc.

[Asim Roy]

I had some communication with Marvin Minsky and Jerry Fodor years ago. Here are two quotes from them:



1. Marvin Minsky: "Don’t pay any attention to the critics. Don’t even ignore them.” -By Sam Goldwyn


2. Jerry Fodor: “Arguing with Connectionists is like arguing with zombies; both are dead, but  neither has noticed it.”


Asim Roy
Professor, Information Systems
Arizona State University
Lifeboat Foundation Bios: Professor Asim Roy<https://lifeboat.com/ex/bios.asim.roy>
Asim Roy | iSearch (asu.edu)<https://isearch.asu.edu/profile/9973>


From: Connectionists <connectionists-bounces at mailman.srv.cs.cmu.edu> On Behalf Of Baldi,Pierre
Sent: Tuesday, February 1, 2022 7:41 AM
To: Schmidhuber Juergen <juergen at idsia.ch>; connectionists at cs.cmu.edu
Subject: Re: Connectionists: Scientific Integrity, the 2021 Turing Lecture, etc.

M&P wrote:
" The perceptron has shown itself worthy of study despite (and even because of!) its severe limitations, It has many features to attract attention: its linearity; its intriguing learning theorem; its clear paradigmatic simplicity as a kind of parallel computation. There is no reason to suppose that any of these virtues carry over to the many-layered version. Nevertheless, we consider it to be an important research problem to elucidate (or reject) our intuitive judgment that the extension is sterile. Perhaps some powerful convergence theorem will be discovered, or some profound reason for the failure to produce an interesting "learning theorem" for the multilayered machine will be found." (pp231-232)

The claim that they killed NNs in the 1970s is indeed an exaggeration propagated by the PDP group for obvious reasons. There was plenty of NN research in the 1970s--Amari, Grossberg, etc.  Not surprisingly, the same is true of the so-called "second neural network winter".




On 2/1/2022 3:33 AM, Schmidhuber Juergen wrote:

Thanks, Barak! Indeed, I should have said in the email msg that _others_ interpreted the book of Minsky & Papert [M69] in this way. My report explicitly mentions Terry [S20] who wrote in 2020:



"The great expectations in the press (Fig. 3) were dashed by Minsky and Papert (7), who showed in their book Perceptrons that a perceptron can only represent categories that are linearly separable in weight space. Although at the end of their book Minsky and Papert considered the prospect of generalizing single- to multiple-layer perceptrons, one layer feeding into the next, they doubted there would ever be a way to train these more powerful multilayer perceptrons. Unfortunately, many took this doubt to be definitive, and the field was abandoned until a new generation of neural network researchers took a fresh look at the problem in the 1980s.”



However, as mentioned above, the 1969 book [M69] addressed a "problem" of Gauss & Legendre's shallow learning (~1800) [DL1-2] that had already been solved 4 years prior by Ivakhnenko & Lapa's popular deep learning method [DEEP1-2][DL2] (and then also in 1967 by Amari's SGD for MLPs [GD1-2]). Deep learning research was not abandoned in the 1970s. It was alive and kicking, especially outside of the Anglosphere. [DEEP2][GD1-3][CNN1][DL1-2]



See Sec. II and Sec. XIII of the report: https://people.idsia.ch/~juergen/scientific-integrity-turing-award-deep-learning.html<https://urldefense.com/v3/__https:/people.idsia.ch/*juergen/scientific-integrity-turing-award-deep-learning.html__;fg!!IKRxdwAv5BmarQ!Nm1TJYh6q6xcsPIanBhciK1xmToTGDvqGz7eWfVGZ5nK8q4Xc0Y2BIejmbYcDho$>



Cheers,

Jürgen









On 1 Feb 2022, at 14:17, Barak A. Pearlmutter <barak at pearlmutter.net><mailto:barak at pearlmutter.net> wrote:



Jürgen,



It's fantastic that you're helping expose people to some important bits of scientific literature.



But...



Minsky & Papert [M69] made some people think that Rosenblatt [R58-62] had only linear NNs plus threshold functions



If you actually read Minsk and Papert's "Perceptrons" book, this is not a misconception it encourages. It defines a "k-th order perceptron" as a linear threshold unit preceded by an arbitrary set of fixed nonlinearities with fan-in k. (A linear threshold unit with binary inputs would, in this terminology, be a 1st-order perceptron.) All their theorems are for k>1. For instance, they prove that a k-th order perceptron cannot do (k+1)-bit parity, which in the special case of k=1 simplifies to the trivial observation that a simple linear threshold unit cannot do xor.

<perceptrons-book-cover-1.jpg> <perceptron-diagram-1.jpg>

This is why you're not supposed to directly cite things you have not actually read: it's too easy to misconstrue them based on inaccurate summaries transmitted over a series of biased noisy compressive channels.



Cheers,



--Barak.







--

Pierre Baldi, Ph.D.

Distinguished Professor, Department of Computer Science

Director, Institute for Genomics and Bioinformatics

Associate Director, Center for Machine Learning and Intelligent Systems

University of California, Irvine

Irvine, CA 92697-3435

(949) 824-5809

(949) 824-9813 [FAX]

Assistant: Janet Ko  jko at uci.edu<mailto:jko at uci.edu>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/connectionists/attachments/20220201/cb4ab245/attachment.html>