Connectionists: Scientific Integrity, the 2021 Turing Lecture, etc.

[Baldi,Pierre]

M&P wrote:
" The perceptron has shown itself worthy of study despite (and even 
because of!) its severe limitations, It has many features to attract 
attention: its linearity; its intriguing learning theorem; its clear 
paradigmatic simplicity as a kind of parallel computation.There is no 
reason to suppose that any of these virtues carry over to the 
many-layered version.***Nevertheless, we consider it to be an important 
research problem to elucidate (or reject) our intuitive judgment that 
the extension is sterile. Perhaps some powerful convergence theorem will 
be discovered, or some profound reason for the failure to produce an 
interesting "learning theorem" for the multilayered machine will be 
found." (pp231-232)*
*
*
The claim that they killed NNs in the 1970s is indeed an exaggeration 
propagated by the PDP group for obvious reasons. There was plenty of NN 
research in the 1970s--Amari, Grossberg, etc. Not surprisingly, the same 
is true of the so-called "second neural network winter".
****
*
*


On 2/1/2022 3:33 AM, Schmidhuber Juergen wrote:
> Thanks, Barak! Indeed, I should have said in the email msg that _others_ interpreted the book of Minsky & Papert [M69] in this way. My report explicitly mentions Terry [S20] who wrote in 2020:
>
> "The great expectations in the press (Fig. 3) were dashed by Minsky and Papert (7), who showed in their book Perceptrons that a perceptron can only represent categories that are linearly separable in weight space. Although at the end of their book Minsky and Papert considered the prospect of generalizing single- to multiple-layer perceptrons, one layer feeding into the next, they doubted there would ever be a way to train these more powerful multilayer perceptrons. Unfortunately, many took this doubt to be definitive, and the field was abandoned until a new generation of neural network researchers took a fresh look at the problem in the 1980s.”
>
> However, as mentioned above, the 1969 book [M69] addressed a "problem" of Gauss & Legendre's shallow learning (~1800) [DL1-2] that had already been solved 4 years prior by Ivakhnenko & Lapa's popular deep learning method [DEEP1-2][DL2] (and then also in 1967 by Amari's SGD for MLPs [GD1-2]). Deep learning research was not abandoned in the 1970s. It was alive and kicking, especially outside of the Anglosphere. [DEEP2][GD1-3][CNN1][DL1-2]
>
> See Sec. II and Sec. XIII of the report:https://people.idsia.ch/~juergen/scientific-integrity-turing-award-deep-learning.html  
>
> Cheers,
> Jürgen
>
>
>
>
>> On 1 Feb 2022, at 14:17, Barak A. Pearlmutter<barak at pearlmutter.net>  wrote:
>>
>> Jürgen,
>>
>> It's fantastic that you're helping expose people to some important bits of scientific literature.
>>
>> But...
>>
>>> Minsky & Papert [M69] made some people think that Rosenblatt [R58-62] had only linear NNs plus threshold functions
>> If you actually read Minsk and Papert's "Perceptrons" book, this is not a misconception it encourages. It defines a "k-th order perceptron" as a linear threshold unit preceded by an arbitrary set of fixed nonlinearities with fan-in k. (A linear threshold unit with binary inputs would, in this terminology, be a 1st-order perceptron.) All their theorems are for k>1. For instance, they prove that a k-th order perceptron cannot do (k+1)-bit parity, which in the special case of k=1 simplifies to the trivial observation that a simple linear threshold unit cannot do xor.
>> <perceptrons-book-cover-1.jpg> <perceptron-diagram-1.jpg>
>> This is why you're not supposed to directly cite things you have not actually read: it's too easy to misconstrue them based on inaccurate summaries transmitted over a series of biased noisy compressive channels.
>>
>> Cheers,
>>
>> --Barak.
>

-- 
Pierre Baldi, Ph.D.
Distinguished Professor, Department of Computer Science
Director, Institute for Genomics and Bioinformatics
Associate Director, Center for Machine Learning and Intelligent Systems
University of California, Irvine
Irvine, CA 92697-3435
(949) 824-5809
(949) 824-9813 [FAX]
Assistant: Janet Kojko at uci.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/connectionists/attachments/20220201/3d5f05d6/attachment-0001.html>