Connectionists: Annotated History of Modern AI and Deep Learning

[Gary Marcus]

A bit more history, on the possibility that it might be of use to future students of our contentious AI times, and in the spirit of the Elvis quote below:

2015: Gary Marcus writes, somewhat loosely, in a trade book (The Future of the Brain)
> “Hierarchies of features are less suited to challenges such as language, inference, and high-level planning. For example, as Noam Chomsky famously pointed out, language is filled with sentences you haven't seen before. Pure classifier systems don't know what to do with such sentences. The talent of feature detectors -- in  identifying which member of some category something belongs to -- doesn't translate into understanding novel sentences, in which each sentence has its own unique meaning.”

Sometime thereafter: Turing Award winner Geoff Hinton enshrines the quote on his own web page, with ridicule, as “My Favorite Gary Marcus quote”; people in the deep learning community circulate it on Facebook and Twitter, mocking Marcus.

October 2019: Geoff Hinton, based perhaps primarily on the quote, warns a crowd of researchers at Toronto to not waste their time listening to Marcus. (Hinton’s email bounces, because it was sent from the wrong address). Hinton’s view is that language has been solved, by Google Translate; in his eyes, Marcus is a moron. 

[Almost three years pass; ridicule of Marcus continues on major social media]

February 2023: Hinton’s fellow Turing Award winner Yann LeCun unleashes a Tweetstorm, saying that “LLMs such as ChatGPT can eloquently spew complete nonsense. Their grasp of reality is very superficial” and that “ [LLM] make very stupid mistakes of common-sense that a 4 year-old, a chimp, a dog, or a cat would never make. LLMs have a more superficial understanding of the world than a house cat.”

Marcus receives many emails wondering whether LeCun has switched sides. On Twitter, people ask whether Marcus has hacked LeCun’s Twitter account.

The quote from Marcus, at the bottom of Hinton’s home page, remains.





> On Feb 3, 2023, at 02:15, Schmidhuber Juergen <juergen at idsia.ch> wrote:
> 
> PS: the weirdest thing is that later Minsky & Papert published a famous book (1969) [M69] that cited neither Amari’s SGD-based deep learning (1967-68) nor the original layer-by-layer deep learning (1965) by Ivakhnenko & Lapa [DEEP1-2][DL2]. 
> 
> Minsky & Papert's book [M69] showed that shallow NNs without hidden layers are very limited. Duh! That’s exactly why people like Ivakhnenko & Lapa and Amari had earlier overcome this problem through _deep_ learning with many learning layers. 
> 
> Minsky & Papert apparently were unaware of this. Unfortunately, even later they failed to correct their book [T22]. 
> 
> Much later, others took this as an opportunity to promulgate a rather self-serving revisionist history of deep learning [S20][DL3][DL3a][T22] that simply ignored pre-Minsky deep learning.
> 
> However, as Elvis Presley put it, "Truth is like the sun. You can shut it out for a time, but it ain't goin' away.” [T22]
> 
> Juergen
> 
> 
> 
>> On 26. Jan 2023, at 16:29, Schmidhuber Juergen <juergen at idsia.ch> wrote:
>> 
>> And in 1967-68, the same Shun-Ichi Amari trained multilayer perceptrons (MLPs) with many layers by stochastic gradient descent (SGD) in end-to-end fashion. See Sec. 7 of the survey: https://urldefense.proofpoint.com/v2/url?u=https-3A__people.idsia.ch_-7Ejuergen_deep-2Dlearning-2Dhistory.html-232nddl&d=DwIDaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=wQR1NePCSj6dOGDD0r6B5Kn1fcNaTMg7tARe7TdEDqQ&m=4LzhBNueqlX8EkcU7h_DxubfArfr6b5GHokpJlCSdmTq7ZPDMknduDgY5WCt_lhe&s=vtnXTzEYRRA1-iq260_cxSYhH8FdQaIWYVoaGdTkTBw&e= 
>> 
>> Amari's implementation [GD2,GD2a] (with his student Saito) learned internal representations in a five layer MLP with two modifiable layers, which was trained to classify non-linearily separable pattern classes. 
>> 
>> Back then compute was billions of times more expensive than today.   
>> 
>> To my knowledge, this was the first implementation of learning internal representations through SGD-based deep learning. 
>> 
>> If anyone knows of an earlier one then please let me know :)
>> 
>> Jürgen 
>> 
>> 
>>>> On 25. Jan 2023, at 16:44, Schmidhuber Juergen <juergen at idsia.ch> wrote:
>>> 
>>> Some are not aware of this historic tidbit in Sec. 4 of the survey: half a century ago, Shun-Ichi Amari published a learning recurrent neural network (1972) which was later called the Hopfield network.
>>> 
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__people.idsia.ch_-7Ejuergen_deep-2Dlearning-2Dhistory.html-23rnn&d=DwIDaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=wQR1NePCSj6dOGDD0r6B5Kn1fcNaTMg7tARe7TdEDqQ&m=4LzhBNueqlX8EkcU7h_DxubfArfr6b5GHokpJlCSdmTq7ZPDMknduDgY5WCt_lhe&s=E4HvMqgORTTmtoivOznAA1FsqYk0EqbAvZi1jQZPEbM&e= 
>>> 
>>> Jürgen
>>> 
>>> 
>>> 
>>> 
>>>> On 13. Jan 2023, at 11:13, Schmidhuber Juergen <juergen at idsia.ch> wrote:
>>>> 
>>>> Machine learning is the science of credit assignment. My new survey credits the pioneers of deep learning and modern AI (supplementing my award-winning 2015 survey): 
>>>> 
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__arxiv.org_abs_2212.11279&d=DwIDaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=wQR1NePCSj6dOGDD0r6B5Kn1fcNaTMg7tARe7TdEDqQ&m=4LzhBNueqlX8EkcU7h_DxubfArfr6b5GHokpJlCSdmTq7ZPDMknduDgY5WCt_lhe&s=KaU8D1yHizw6UUsIuIba6AKBx5Ok5clZYo32bx-cPAs&e= 
>>>> 
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__people.idsia.ch_-7Ejuergen_deep-2Dlearning-2Dhistory.html&d=DwIDaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=wQR1NePCSj6dOGDD0r6B5Kn1fcNaTMg7tARe7TdEDqQ&m=4LzhBNueqlX8EkcU7h_DxubfArfr6b5GHokpJlCSdmTq7ZPDMknduDgY5WCt_lhe&s=4Qj78cOJPkfxEDnytPDkfrCvsAbE5RvzpOb7t8ooLIw&e= 
>>>> 
>>>> This was already reviewed by several deep learning pioneers and other experts. Nevertheless, let me know under juergen at idsia.ch if you can spot any remaining error or have suggestions for improvements.
>>>> 
>>>> Happy New Year!
>>>> 
>>>> Jürgen
>>>> 
> 
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/connectionists/attachments/20230204/53c1dc6c/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 124355 bytes
Desc: not available
URL: <http://mailman.srv.cs.cmu.edu/pipermail/connectionists/attachments/20230204/53c1dc6c/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: IMG_3771.jpeg
Type: image/jpeg
Size: 820505 bytes
Desc: not available
URL: <http://mailman.srv.cs.cmu.edu/pipermail/connectionists/attachments/20230204/53c1dc6c/attachment-0001.jpeg>