Connectionists: Annotated History of Modern AI and Deep Learning

[Schmidhuber Juergen]

Dear Andrzej, thanks, but come on, the report cites lots of “symbolic” AI from theorem proving (e.g., Zuse 1948) to later surveys of expert systems and “traditional" AI. Note that Sec. 18 and Sec. 19 go back even much further in time (not even speaking of Sec. 20). The survey also explains why AI histories written in the 1980s/2000s/2020s differ. Here again the table of contents:

Sec. 1: Introduction
Sec. 2: 1676: The Chain Rule For Backward Credit Assignment
Sec. 3: Circa 1800: First Neural Net (NN) / Linear Regression / Shallow Learning
Sec. 4: 1920-1925: First Recurrent NN (RNN) Architecture. ~1972: First Learning RNNs
Sec. 5: 1958: Multilayer Feedforward NN (without Deep Learning)
Sec. 6: 1965: First Deep Learning
Sec. 7: 1967-68: Deep Learning by Stochastic Gradient Descent 
Sec. 8: 1970: Backpropagation. 1982: For NNs. 1960: Precursor. 
Sec. 9: 1979: First Deep Convolutional NN (1969: Rectified Linear Units) 
Sec. 10: 1980s-90s: Graph NNs / Stochastic Delta Rule (Dropout) / More RNNs / Etc
Sec. 11: Feb 1990: Generative Adversarial Networks / Artificial Curiosity / NN Online Planners
Sec. 12: April 1990: NNs Learn to Generate Subgoals / Work on Command 
Sec. 13: March 1991: NNs Learn to Program NNs. Transformers with Linearized Self-Attention
Sec. 14: April 1991: Deep Learning by Self-Supervised Pre-Training. Distilling NNs
Sec. 15: June 1991: Fundamental Deep Learning Problem: Vanishing/Exploding Gradients
Sec. 16: June 1991: Roots of Long Short-Term Memory / Highway Nets / ResNets
Sec. 17: 1980s-: NNs for Learning to Act Without a Teacher 
Sec. 18: It's the Hardware, Stupid!
Sec. 19: But Don't Neglect the Theory of AI (Since 1931) and Computer Science
Sec. 20: The Broader Historic Context from Big Bang to Far Future
Sec. 21: Acknowledgments
Sec. 22: 555+ Partially Annotated References (many more in the award-winning survey [DL1])

Tweet: https://twitter.com/SchmidhuberAI/status/1606333832956973060?cxt=HHwWiMC8gYiH7MosAAAA

Jürgen





> On 13. Jan 2023, at 14:40, Andrzej Wichert <andreas.wichert at tecnico.ulisboa.pt> wrote:
> 
> Dear Juergen,
> 
> You make the same mistake at it was done in the earlier 1970. You identify deep learning with modern AI, the paper should be called instead "Annotated History of Deep Learning”
> 
> Otherwise, you ignore symbolical AI, like search, production systems, knowledge representation, search, planning etc., as if is not part of AI anymore (suggested by your title). 
> 
> Best,
> 
> Andreas
> 
> --------------------------------------------------------------------------------------------------
> Prof. Auxiliar Andreas Wichert   
> 
> http://web.tecnico.ulisboa.pt/andreas.wichert/
> -
> https://www.amazon.com/author/andreaswichert
> 
> Instituto Superior Técnico - Universidade de Lisboa
> Campus IST-Taguspark 
> Avenida Professor Cavaco Silva                 Phone: +351  214233231
> 2744-016 Porto Salvo, Portugal
> 
>> On 13 Jan 2023, at 08:13, Schmidhuber Juergen <juergen at idsia.ch> wrote:
>> 
>> Machine learning is the science of credit assignment. My new survey credits the pioneers of deep learning and modern AI (supplementing my award-winning 2015 survey): 
>> 
>> https://arxiv.org/abs/2212.11279
>> 
>> https://people.idsia.ch/~juergen/deep-learning-history.html
>> 
>> This was already reviewed by several deep learning pioneers and other experts. Nevertheless, let me know under juergen at idsia.ch if you can spot any remaining error or have suggestions for improvements.
>> 
>> Happy New Year!
>> 
>> Jürgen
>> 
>> 
>> 
>