Connectionists: Annotated History of Modern AI and Deep Learning

[Gary Marcus]

Dear Juergen,

You have made a good case that the history of deep learning is often misrepresented. But, by parity of reasoning, a few pointers to a tiny fraction of the work done in symbolic AI does not in any way make this a thorough and balanced exercise with respect to the field as a whole.

I am 100% with Andrzej Wichert, in thinking that vast areas of AI such as planning, reasoning, natural language understanding, robotics and knowledge representation are treated very superficially here. A few pointers to theorem proving and the like does not solve that. 

Your essay is a fine if opinionated history of deep learning, with a special emphasis on your own work, but of somewhat limited value beyond a few terse references in explicating other approaches to AI. This would be ok if the title and aspiration didn’t aim for as a whole; if you really want the paper to reflect the field as a whole, and the ambitions of the title, you have more work to do. 

My own hunch is that in a decade, maybe much sooner, a major emphasis of the field will be on neurosymbolic integration. Your own startup is heading in that direction, and the commericial desire to make LLMs reliable and truthful will also push in that direction. 
Historians looking back on this paper will see too little about that roots of that trend documented here.

Gary 

> On Jan 14, 2023, at 12:42 AM, Schmidhuber Juergen <juergen at idsia.ch> wrote:
> 
> Dear Andrzej, thanks, but come on, the report cites lots of “symbolic” AI from theorem proving (e.g., Zuse 1948) to later surveys of expert systems and “traditional" AI. Note that Sec. 18 and Sec. 19 go back even much further in time (not even speaking of Sec. 20). The survey also explains why AI histories written in the 1980s/2000s/2020s differ. Here again the table of contents:
> 
> Sec. 1: Introduction
> Sec. 2: 1676: The Chain Rule For Backward Credit Assignment
> Sec. 3: Circa 1800: First Neural Net (NN) / Linear Regression / Shallow Learning
> Sec. 4: 1920-1925: First Recurrent NN (RNN) Architecture. ~1972: First Learning RNNs
> Sec. 5: 1958: Multilayer Feedforward NN (without Deep Learning)
> Sec. 6: 1965: First Deep Learning
> Sec. 7: 1967-68: Deep Learning by Stochastic Gradient Descent 
> Sec. 8: 1970: Backpropagation. 1982: For NNs. 1960: Precursor. 
> Sec. 9: 1979: First Deep Convolutional NN (1969: Rectified Linear Units) 
> Sec. 10: 1980s-90s: Graph NNs / Stochastic Delta Rule (Dropout) / More RNNs / Etc
> Sec. 11: Feb 1990: Generative Adversarial Networks / Artificial Curiosity / NN Online Planners
> Sec. 12: April 1990: NNs Learn to Generate Subgoals / Work on Command 
> Sec. 13: March 1991: NNs Learn to Program NNs. Transformers with Linearized Self-Attention
> Sec. 14: April 1991: Deep Learning by Self-Supervised Pre-Training. Distilling NNs
> Sec. 15: June 1991: Fundamental Deep Learning Problem: Vanishing/Exploding Gradients
> Sec. 16: June 1991: Roots of Long Short-Term Memory / Highway Nets / ResNets
> Sec. 17: 1980s-: NNs for Learning to Act Without a Teacher 
> Sec. 18: It's the Hardware, Stupid!
> Sec. 19: But Don't Neglect the Theory of AI (Since 1931) and Computer Science
> Sec. 20: The Broader Historic Context from Big Bang to Far Future
> Sec. 21: Acknowledgments
> Sec. 22: 555+ Partially Annotated References (many more in the award-winning survey [DL1])
> 
> Tweet: https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_SchmidhuberAI_status_1606333832956973060-3Fcxt-3DHHwWiMC8gYiH7MosAAAA&d=DwIDaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=wQR1NePCSj6dOGDD0r6B5Kn1fcNaTMg7tARe7TdEDqQ&m=oGn-OID5YOewbgo3j_HjFjI3I2N3hx-w0hoIfLR_JJsn8q5UZDYAl5HOHPY-87N5&s=nWCXLKazOjmixYrJVR0CMlR12PasGbAd8bsS6VZ10bk&e= 
> 
> Jürgen
> 
> 
> 
> 
> 
>> On 13. Jan 2023, at 14:40, Andrzej Wichert <andreas.wichert at tecnico.ulisboa.pt> wrote:
>> Dear Juergen,
>> You make the same mistake at it was done in the earlier 1970. You identify deep learning with modern AI, the paper should be called instead "Annotated History of Deep Learning”
>> Otherwise, you ignore symbolical AI, like search, production systems, knowledge representation, search, planning etc., as if is not part of AI anymore (suggested by your title).
>> Best,
>> Andreas
>> --------------------------------------------------------------------------------------------------
>> Prof. Auxiliar Andreas Wichert   
>> https://urldefense.proofpoint.com/v2/url?u=http-3A__web.tecnico.ulisboa.pt_andreas.wichert_&d=DwIDaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=wQR1NePCSj6dOGDD0r6B5Kn1fcNaTMg7tARe7TdEDqQ&m=oGn-OID5YOewbgo3j_HjFjI3I2N3hx-w0hoIfLR_JJsn8q5UZDYAl5HOHPY-87N5&s=h5Zy9Hk2IoWPt7me1mLhcYHEuJ55mmNOAppZKcivxAk&e=
>> -
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.amazon.com_author_andreaswichert&d=DwIDaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=wQR1NePCSj6dOGDD0r6B5Kn1fcNaTMg7tARe7TdEDqQ&m=oGn-OID5YOewbgo3j_HjFjI3I2N3hx-w0hoIfLR_JJsn8q5UZDYAl5HOHPY-87N5&s=w1RtYvs8dwtfvlTkHqP_P-74ITvUW2IiHLSai7br25U&e=
>> Instituto Superior Técnico - Universidade de Lisboa
>> Campus IST-Taguspark
>> Avenida Professor Cavaco Silva                 Phone: +351  214233231
>> 2744-016 Porto Salvo, Portugal
>>>> On 13 Jan 2023, at 08:13, Schmidhuber Juergen <juergen at idsia.ch> wrote:
>>> Machine learning is the science of credit assignment. My new survey credits the pioneers of deep learning and modern AI (supplementing my award-winning 2015 survey):
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__arxiv.org_abs_2212.11279&d=DwIDaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=wQR1NePCSj6dOGDD0r6B5Kn1fcNaTMg7tARe7TdEDqQ&m=oGn-OID5YOewbgo3j_HjFjI3I2N3hx-w0hoIfLR_JJsn8q5UZDYAl5HOHPY-87N5&s=6E5_tonSfNtoMPw1fvFOm8UFm7tDVH7un_kbogNG_1w&e=
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__people.idsia.ch_-7Ejuergen_deep-2Dlearning-2Dhistory.html&d=DwIDaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=wQR1NePCSj6dOGDD0r6B5Kn1fcNaTMg7tARe7TdEDqQ&m=oGn-OID5YOewbgo3j_HjFjI3I2N3hx-w0hoIfLR_JJsn8q5UZDYAl5HOHPY-87N5&s=XPnftI8leeqoElbWQIApFNQ2L4gDcrGy_eiJv2ZPYYk&e=
>>> This was already reviewed by several deep learning pioneers and other experts. Nevertheless, let me know under juergen at idsia.ch if you can spot any remaining error or have suggestions for improvements.
>>> Happy New Year!
>>> Jürgen