Connectionists: Annotated History of Modern AI and Deep Learning (97 pages, 666+ references)

[Juergen Schmidhuber]

Dec 2025: major update of the little booklet (97 pages, 666+ references):

Annotated History of Modern AI and Deep Learning 

Link:     https://people.idsia.ch/~juergen/deep-learning-history.html 
Tweet:  https://x.com/SchmidhuberAI/status/1606333832956973060 
arXiv:   https://arxiv.org/abs/2212.11279 

Abstract. Machine learning (ML) is the science of credit assignment. It seeks to find patterns in observations that explain and predict the consequences of events and actions. This then helps to improve future performance. Minsky's so-called "fundamental credit assignment problem" (1963) surfaces in all sciences including physics (why is the world the way it is?) and history (which persons/ideas/actions have shaped society and civilisation?). Here I focus on the history of ML itself. Modern artificial intelligence (AI) is dominated by artificial neural networks (NNs) and deep learning, both of which are conceptually closer to the old field of cybernetics than what was traditionally called AI (e.g., expert systems and logic programming). A modern history of AI & ML must emphasize breakthroughs outside the scope of shallow AI text books. In particular, it must cover the mathematical foundations of today's NNs such as the chain rule (1676), the first NNs (circa 1800), the first practical AI (1914), the theory of AI and its limitations (1931-34), and the first working deep learning algorithms (1965-). From the perspective of 2025, I provide a timeline of the most significant events in the history of NNs, ML, deep learning, AI, computer science, and mathematics in general, crediting the individuals who laid the field's foundations. The text contains numerous hyperlinks to relevant overview sites from the AI Blog. It also debunks certain popular yet misleading historical accounts of AI and deep learning and — with a ten-year delay — supplements my 2015 award-winning deep learning survey which provides hundreds of additional references. Finally, I will put things in a broader historical context, spanning from the Big Bang to when the universe will be many times older than it is now. 

Table of Contents

Sec. 1: Introduction
Sec. 2: 1676: The Chain Rule For Backward Credit Assignment
Sec. 3: Circa 1800: First Neural Net (NN) / Linear Regression / Shallow Learning
Sec. 4: 1920-1925: First Recurrent NN (RNN) Architecture. ~1972: First Learning RNNs
Sec. 5: 1958: Multilayer Feedforward NN (without Deep Learning)
Sec. 6: 1965: First Deep Learning
Sec. 7: 1967-68: Deep Learning by Stochastic Gradient Descent
Sec. 8: 1970: Backpropagation. 1982: For NNs. 1960: Precursor.
Sec. 9: 1979: First Deep Convolutional NN (1969: Rectified Linear Units)
Sec. 10: 1980s-90s: Graph NNs / Stochastic Delta Rule (Dropout) / More RNNs / Etc
Sec. 11: Feb 1990: Generative Adversarial Networks / Artificial Curiosity / NN Online Planners
Sec. 12: April 1990: NNs Learn to Generate Subgoals / Work on Command
Sec. 13: March 1991: NNs Learn to Program NNs. Unnormalized Linear Transformers
Sec. 14: April 1991: Deep Learning by Pre-Training (the P in ChatGPT). Distilling NNs
Sec. 15: June 1991: Fundamental Deep Learning Problem: Vanishing/Exploding Gradients
Sec. 16: June 1991: Roots of Long Short-Term Memory / Highway Nets / ResNets
Sec. 17: 1980s-: NNs for Learning to Act Without a Teacher
Sec. 18: It's the Hardware, Stupid!
Sec. 19: But Don't Neglect the Theory of AI (Since 1931) and Computer Science
Sec. 20: The Broader Historic Context from Big Bang to Far Future
Sec. 21: Acknowledgments
Sec. 22: 666+ Partially Annotated References (many more in the award-winning survey)

First paragraph of the introduction:

Over time, certain historic events have become more important in the eyes of certain beholders. For example, the Big Bang of 13.8 billion years ago is now widely considered an essential moment in the history of everything. Until a few decades ago, however, it has remained completely unknown to earthlings, who for a long time have entertained quite erroneous ideas about the origins of the universe (see the final section for more on the world's history). Currently accepted histories of many more limited subjects are results of similarly radical revisions. Here I will focus on the history of artificial intelligence (AI), which also isn't quite what it used to be. ... 

Last paragraph of the final section:

Finally, as already indicated above, future artificial scientists will meticulously study their origins by extracting as much relevant historical data as possible from recent and ancient human-generated papers, letters, emails, recordings, videos, artefacts, and other sources. They will revise current prevalent but misleading narratives created by self-serving AI influencers, just like humanity has recently revised or even completely abandoned millennia-old religious myths about the history of the universe. The present overview of AI history isn't perfect and will surely be revised again, but it's much more accurate than alternative histories out there, and will provide initial guidance to future AI historians!