Connectionists: Scientific Integrity, the 2021 Turing Lecture, etc.

[Stephen José Hanson]

On 2/1/22 3:19 PM, Asim Roy wrote:
>
> I had some communication with Marvin Minsky and Jerry Fodor years ago. 
> Here are two quotes from them:
>
>
> 2. Jerry Fodor: “Arguing with Connectionists is like arguing with 
> zombies; both are dead, but  neither has noticed it.”
>
I used to argue with Jerry at our Pizza lunchs, and miss him as he had 
great counter arguments  to almost anything....  but he would have hated 
the DL results with language...  GPT-xs --would have liked to know what 
his counters would have been..

but no, he didn't think all connectionists were zombies..

Sorry Asim.


Steve

> Asim Roy
>
> Professor, Information Systems
>
> Arizona State University
>
> Lifeboat Foundation Bios: Professor Asim Roy 
> <https://lifeboat.com/ex/bios.asim.roy>
>
> Asim Roy | iSearch (asu.edu) <https://isearch.asu.edu/profile/9973>
>
> *From:* Connectionists <connectionists-bounces at mailman.srv.cs.cmu.edu> 
> *On Behalf Of *Baldi,Pierre
> *Sent:* Tuesday, February 1, 2022 7:41 AM
> *To:* Schmidhuber Juergen <juergen at idsia.ch>; connectionists at cs.cmu.edu
> *Subject:* Re: Connectionists: Scientific Integrity, the 2021 Turing 
> Lecture, etc.
>
> M&P wrote:
>
> " The perceptron has shown itself worthy of study despite (and even 
> because of!) its severe limitations, It has many features to attract 
> attention: its linearity; its intriguing learning theorem; its clear 
> paradigmatic simplicity as a kind of parallel computation. There is no 
> reason to suppose that any of these virtues carry over to the 
> many-layered version.*Nevertheless, we consider it to be an important 
> research problem to elucidate (or reject) our intuitive judgment that 
> the extension is sterile. Perhaps some powerful convergence theorem 
> will be discovered, or some profound reason for the failure to produce 
> an interesting "learning theorem" for the multilayered machine will be 
> found." (pp231-232)*
>
> The claim that they killed NNs in the 1970s is indeed an exaggeration 
> propagated by the PDP group for obvious reasons. There was plenty of 
> NN research in the 1970s--Amari, Grossberg, etc.  Not surprisingly, 
> the same is true of the so-called "second neural network winter".
>
> On 2/1/2022 3:33 AM, Schmidhuber Juergen wrote:
>
>     Thanks, Barak! Indeed, I should have said in the email msg that _others_ interpreted the book of Minsky & Papert [M69] in this way. My report explicitly mentions Terry [S20] who wrote in 2020:
>
>     "The great expectations in the press (Fig. 3) were dashed by Minsky and Papert (7), who showed in their book Perceptrons that a perceptron can only represent categories that are linearly separable in weight space. Although at the end of their book Minsky and Papert considered the prospect of generalizing single- to multiple-layer perceptrons, one layer feeding into the next, they doubted there would ever be a way to train these more powerful multilayer perceptrons. Unfortunately, many took this doubt to be definitive, and the field was abandoned until a new generation of neural network researchers took a fresh look at the problem in the 1980s.”
>
>     However, as mentioned above, the 1969 book [M69] addressed a "problem" of Gauss & Legendre's shallow learning (~1800) [DL1-2] that had already been solved 4 years prior by Ivakhnenko & Lapa's popular deep learning method [DEEP1-2][DL2] (and then also in 1967 by Amari's SGD for MLPs [GD1-2]). Deep learning research was not abandoned in the 1970s. It was alive and kicking, especially outside of the Anglosphere. [DEEP2][GD1-3][CNN1][DL1-2]
>
>     See Sec. II and Sec. XIII of the report:https://people.idsia.ch/~juergen/scientific-integrity-turing-award-deep-learning.html  <https://urldefense.com/v3/__https:/people.idsia.ch/*juergen/scientific-integrity-turing-award-deep-learning.html__;fg!!IKRxdwAv5BmarQ!Nm1TJYh6q6xcsPIanBhciK1xmToTGDvqGz7eWfVGZ5nK8q4Xc0Y2BIejmbYcDho$>  
>
>     Cheers,
>
>     Jürgen
>
>         On 1 Feb 2022, at 14:17, Barak A. Pearlmutter<barak at pearlmutter.net>  <mailto:barak at pearlmutter.net>  wrote:
>
>         Jürgen,
>
>         It's fantastic that you're helping expose people to some important bits of scientific literature.
>
>         But...
>
>             Minsky & Papert [M69] made some people think that Rosenblatt [R58-62] had only linear NNs plus threshold functions
>
>         If you actually read Minsk and Papert's "Perceptrons" book, this is not a misconception it encourages. It defines a "k-th order perceptron" as a linear threshold unit preceded by an arbitrary set of fixed nonlinearities with fan-in k. (A linear threshold unit with binary inputs would, in this terminology, be a 1st-order perceptron.) All their theorems are for k>1. For instance, they prove that a k-th order perceptron cannot do (k+1)-bit parity, which in the special case of k=1 simplifies to the trivial observation that a simple linear threshold unit cannot do xor.
>
>         <perceptrons-book-cover-1.jpg> <perceptron-diagram-1.jpg>
>
>         This is why you're not supposed to directly cite things you have not actually read: it's too easy to misconstrue them based on inaccurate summaries transmitted over a series of biased noisy compressive channels.
>
>         Cheers,
>
>         --Barak.
>
> -- 
> Pierre Baldi, Ph.D.
> Distinguished Professor, Department of Computer Science
> Director, Institute for Genomics and Bioinformatics
> Associate Director, Center for Machine Learning and Intelligent Systems
> University of California, Irvine
> Irvine, CA 92697-3435
> (949) 824-5809
> (949) 824-9813 [FAX]
> Assistant: Janet Kojko at uci.edu  <mailto:jko at uci.edu>
-- 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/connectionists/attachments/20220201/3fb69417/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.png
Type: image/png
Size: 19957 bytes
Desc: not available
URL: <http://mailman.srv.cs.cmu.edu/pipermail/connectionists/attachments/20220201/3fb69417/attachment.png>