Connectionists: Annotated History of Modern AI and Deep Learning

[Schmidhuber Juergen]

PS: the weirdest thing is that later Minsky & Papert published a famous book (1969) [M69] that cited neither Amari’s SGD-based deep learning (1967-68) nor the original layer-by-layer deep learning (1965) by Ivakhnenko & Lapa [DEEP1-2][DL2]. 

Minsky & Papert's book [M69] showed that shallow NNs without hidden layers are very limited. Duh! That’s exactly why people like Ivakhnenko & Lapa and Amari had earlier overcome this problem through _deep_ learning with many learning layers. 

Minsky & Papert apparently were unaware of this. Unfortunately, even later they failed to correct their book [T22]. 

Much later, others took this as an opportunity to promulgate a rather self-serving revisionist history of deep learning [S20][DL3][DL3a][T22] that simply ignored pre-Minsky deep learning.

However, as Elvis Presley put it, "Truth is like the sun. You can shut it out for a time, but it ain't goin' away.” [T22]

Juergen



> On 26. Jan 2023, at 16:29, Schmidhuber Juergen <juergen at idsia.ch> wrote:
> 
> And in 1967-68, the same Shun-Ichi Amari trained multilayer perceptrons (MLPs) with many layers by stochastic gradient descent (SGD) in end-to-end fashion. See Sec. 7 of the survey: https://people.idsia.ch/~juergen/deep-learning-history.html#2nddl
> 
> Amari's implementation [GD2,GD2a] (with his student Saito) learned internal representations in a five layer MLP with two modifiable layers, which was trained to classify non-linearily separable pattern classes. 
> 
> Back then compute was billions of times more expensive than today.   
> 
> To my knowledge, this was the first implementation of learning internal representations through SGD-based deep learning. 
> 
> If anyone knows of an earlier one then please let me know :)
> 
> Jürgen 
> 
> 
>> On 25. Jan 2023, at 16:44, Schmidhuber Juergen <juergen at idsia.ch> wrote:
>> 
>> Some are not aware of this historic tidbit in Sec. 4 of the survey: half a century ago, Shun-Ichi Amari published a learning recurrent neural network (1972) which was later called the Hopfield network.
>> 
>> https://people.idsia.ch/~juergen/deep-learning-history.html#rnn
>> 
>> Jürgen
>> 
>> 
>> 
>> 
>>> On 13. Jan 2023, at 11:13, Schmidhuber Juergen <juergen at idsia.ch> wrote:
>>> 
>>> Machine learning is the science of credit assignment. My new survey credits the pioneers of deep learning and modern AI (supplementing my award-winning 2015 survey): 
>>> 
>>> https://arxiv.org/abs/2212.11279
>>> 
>>> https://people.idsia.ch/~juergen/deep-learning-history.html
>>> 
>>> This was already reviewed by several deep learning pioneers and other experts. Nevertheless, let me know under juergen at idsia.ch if you can spot any remaining error or have suggestions for improvements.
>>> 
>>> Happy New Year!
>>> 
>>> Jürgen
>>>