Connectionists: Annotated History of Modern AI and Deep Learning

[Andrzej Wichert]

Dear Jurgen,

At this time internet did not exist and many discoveries were done in parallel without people knowing what other dis. 
There was also some research published in Russian language which it seems is lost. But the truth is that we only see what
defines us now our time. Many researchers think that DL is a breakthrough; as people thought before about symbolical AI…
Quite sure in some years there will be a new wave when some thing becomes in.
"The definition of artificial intelligence leads to the paradox of a discipline whose principal purpose is its own definition.” from my book Principles of Quantum Artificial Intelligence...
Like Serge Gainsbourg sang:
Jusqu'à neuf, c'est OK, tu es "in"
Après quoi, tu es KO, tu es "out"
And when something is out, we do not see it (symbolical AI) any more...

Best,

Andreas

--------------------------------------------------------------------------------------------------
Prof. Auxiliar Andreas Wichert   

http://web.tecnico.ulisboa.pt/andreas.wichert/
-
https://www.amazon.com/author/andreaswichert

Instituto Superior Técnico - Universidade de Lisboa
Campus IST-Taguspark 
Avenida Professor Cavaco Silva                 Phone: +351  214233231
2744-016 Porto Salvo, Portugal

> On 3 Feb 2023, at 02:41, Schmidhuber Juergen <juergen at idsia.ch> wrote:
> 
> PS: the weirdest thing is that later Minsky & Papert published a famous book (1969) [M69] that cited neither Amari’s SGD-based deep learning (1967-68) nor the original layer-by-layer deep learning (1965) by Ivakhnenko & Lapa [DEEP1-2][DL2]. 
> 
> Minsky & Papert's book [M69] showed that shallow NNs without hidden layers are very limited. Duh! That’s exactly why people like Ivakhnenko & Lapa and Amari had earlier overcome this problem through _deep_ learning with many learning layers. 
> 
> Minsky & Papert apparently were unaware of this. Unfortunately, even later they failed to correct their book [T22]. 
> 
> Much later, others took this as an opportunity to promulgate a rather self-serving revisionist history of deep learning [S20][DL3][DL3a][T22] that simply ignored pre-Minsky deep learning.
> 
> However, as Elvis Presley put it, "Truth is like the sun. You can shut it out for a time, but it ain't goin' away.” [T22]
> 
> Juergen
> 
> 
> 
>> On 26. Jan 2023, at 16:29, Schmidhuber Juergen <juergen at idsia.ch> wrote:
>> 
>> And in 1967-68, the same Shun-Ichi Amari trained multilayer perceptrons (MLPs) with many layers by stochastic gradient descent (SGD) in end-to-end fashion. See Sec. 7 of the survey: https://people.idsia.ch/~juergen/deep-learning-history.html#2nddl
>> 
>> Amari's implementation [GD2,GD2a] (with his student Saito) learned internal representations in a five layer MLP with two modifiable layers, which was trained to classify non-linearily separable pattern classes. 
>> 
>> Back then compute was billions of times more expensive than today.   
>> 
>> To my knowledge, this was the first implementation of learning internal representations through SGD-based deep learning. 
>> 
>> If anyone knows of an earlier one then please let me know :)
>> 
>> Jürgen 
>> 
>> 
>>> On 25. Jan 2023, at 16:44, Schmidhuber Juergen <juergen at idsia.ch> wrote:
>>> 
>>> Some are not aware of this historic tidbit in Sec. 4 of the survey: half a century ago, Shun-Ichi Amari published a learning recurrent neural network (1972) which was later called the Hopfield network.
>>> 
>>> https://people.idsia.ch/~juergen/deep-learning-history.html#rnn
>>> 
>>> Jürgen
>>> 
>>> 
>>> 
>>> 
>>>> On 13. Jan 2023, at 11:13, Schmidhuber Juergen <juergen at idsia.ch> wrote:
>>>> 
>>>> Machine learning is the science of credit assignment. My new survey credits the pioneers of deep learning and modern AI (supplementing my award-winning 2015 survey): 
>>>> 
>>>> https://arxiv.org/abs/2212.11279
>>>> 
>>>> https://people.idsia.ch/~juergen/deep-learning-history.html
>>>> 
>>>> This was already reviewed by several deep learning pioneers and other experts. Nevertheless, let me know under juergen at idsia.ch if you can spot any remaining error or have suggestions for improvements.
>>>> 
>>>> Happy New Year!
>>>> 
>>>> Jürgen
>>>> 
> 
>