<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 18pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
Dear Juergen,</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 18pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
<br>
</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 18pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">
Thanks for mentioning the Ising model!</div>
<div class="elementToProof">
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 18pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Arial, Helvetica, sans-serif; font-size: 18pt; color: rgb(0, 0, 0);" class="elementToProof">
As you know, it is a <b>binary model</b>, with just two states,<b> </b>and it does not learn.</div>
<div id="Signature">
<div>
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:18pt; color:#000000; font-family:Arial,Helvetica,sans-serif">
<div class="" style="color:rgb(0,0,0); font-style:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px">
<span class="Apple-style-span" style="font-weight:normal; border-collapse:separate; border-spacing:0px">
<div class="" style="word-wrap:break-word"><span class="Apple-style-span" style=""></span></div>
</span></div>
<span id="ms-rterangepaste-end"></span><br>
<p style="margin-top:0px; margin-bottom:0px"></p>
</div>
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:18pt; color:#000000; font-family:Arial,Helvetica,sans-serif">
My Magnum Opus</div>
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:18pt; color:#000000; font-family:Arial,Helvetica,sans-serif" class="ContentPasted0">
<a href="https://www.amazon.com/Conscious-Mind-Resonant-Brain-Makes/dp/0190070552" id="LPlnk136284">https://www.amazon.com/Conscious-Mind-Resonant-Brain-Makes/dp/0190070552</a><br>
</div>
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:18pt; color:#000000; font-family:Arial,Helvetica,sans-serif">
<br>
</div>
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:18pt; color:#000000; font-family:Arial,Helvetica,sans-serif">
reviews some of the early binary neural network models, such as the <b>McCulloch-Pitts</b>,<b> </b><b>Caianiello</b>, and
<b>Rosenblatt </b>models<b>, </b>starting on p. 64, before going on to review early
<b>linear models</b> that included learning, like the <b>Adeline and Madeline</b> models of Bernie
<b>Widrow</b> and the <b>Brain-State-in-a-Box</b> model of Jim <b>Anderson, </b>then
<b>continuous and nonlinear models</b> of various kinds, including models that are still used today.</div>
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:18pt; color:#000000; font-family:Arial,Helvetica,sans-serif">
<br>
</div>
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:18pt; color:#000000; font-family:Arial,Helvetica,sans-serif">
Best,</div>
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:18pt; color:#000000; font-family:Arial,Helvetica,sans-serif">
<br>
</div>
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:18pt; color:#000000; font-family:Arial,Helvetica,sans-serif">
Steve</div>
</div>
</div>
</div>
<div id="appendonsend"></div>
<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Connectionists <connectionists-bounces@mailman.srv.cs.cmu.edu> on behalf of Schmidhuber Juergen <juergen@idsia.ch><br>
<b>Sent:</b> Wednesday, January 25, 2023 11:40 AM<br>
<b>To:</b> connectionists@cs.cmu.edu <connectionists@cs.cmu.edu><br>
<b>Subject:</b> Re: Connectionists: Annotated History of Modern AI and Deep Learning: Early recurrent neural networks for serial verbal learning and associative pattern learning</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt">
<div class="PlainText">Dear Steve, <br>
<br>
thanks - I hope you noticed that the survey mentions your 1969 work! <br>
<br>
And of course it also mentions the origin of this whole recurrent network business: the Ising model or Lenz-Ising model introduced a century ago. See Sec. 4: 1920-1925: First Recurrent NN (RNN) Architecture<br>
<br>
<a href="https://people.idsia.ch/~juergen/deep-learning-history.html#rnn" data-auth="NotApplicable">https://people.idsia.ch/~juergen/deep-learning-history.html#rnn</a><br>
<br>
"The first non-learning RNN architecture (the Ising model or Lenz-Ising model) was introduced and analyzed by physicists Ernst Ising and Wilhelm Lenz in the 1920s [L20][I24,I25][K41][W45][T22]. It settles into an equilibrium state in response to input conditions,
and is the foundation of the first learning RNNs ...”<br>
<br>
Jürgen <br>
<br>
<br>
> On 25. Jan 2023, at 18:42, Grossberg, Stephen <steve@bu.edu> wrote:<br>
> <br>
> Dear Juergen and Connectionists colleagues,<br>
> <br>
> In his attached email below, Juergen mentioned a 1972 article of my friend and colleague, Shun-Ichi Amari, about recurrent neural networks that learn.<br>
> <br>
> Here are a couple of my own early articles from 1969 and 1971 about such networks. I introduced them to explain paradoxical data about serial verbal learning, notably the bowed serial position effect:<br>
> <br>
> Grossberg, S. (1969). On the serial learning of lists. Mathematical Biosciences, 4, 201-253.
<br>
> <a href="https://sites.bu.edu/steveg/files/2016/06/Gro1969MBLists.pdf" data-auth="NotApplicable">
https://sites.bu.edu/steveg/files/2016/06/Gro1969MBLists.pdf</a><br>
> <br>
> Grossberg, S. and Pepe, J. (1971). Spiking threshold and overarousal effects in serial learning. Journal of Statistical Physics, 3, 95-125.
<br>
> <a href="https://sites.bu.edu/steveg/files/2016/06/GroPepe1971JoSP.pdf" data-auth="NotApplicable">
https://sites.bu.edu/steveg/files/2016/06/GroPepe1971JoSP.pdf</a><br>
> <br>
> Juergen also mentioned that Shun-Ichi's work was a precursor of what some people call the Hopfield model, whose most cited articles were published in 1982 and 1984.<br>
> <br>
> I actually started publishing articles on this topic starting in the 1960s. Here are two of them:<br>
> <br>
> Grossberg, S. (1969). On learning and energy-entropy dependence in recurrent and nonrecurrent signed networks. Journal of Statistical Physics, 1, 319-350.
<br>
> <a href="https://sites.bu.edu/steveg/files/2016/06/Gro1969JourStatPhy.pdf" data-auth="NotApplicable">
https://sites.bu.edu/steveg/files/2016/06/Gro1969JourStatPhy.pdf</a><br>
> <br>
> Grossberg, S. (1971). Pavlovian pattern learning by nonlinear neural networks. Proceedings of the National Academy of Sciences, 68, 828-831.
<br>
> <a href="https://sites.bu.edu/steveg/files/2016/06/Gro1971ProNatAcaSci.pdf" data-auth="NotApplicable">
https://sites.bu.edu/steveg/files/2016/06/Gro1971ProNatAcaSci.pdf</a><br>
> <br>
> An early use of Lyapunov functions to prove global limit theorems in associative recurrent neural networks is found in the following 1980 PNAS article:<br>
> <br>
> Grossberg, S. (1980). Biological competition: Decision rules, pattern formation, and oscillations. Proceedings of the National Academy of Sciences, 77, 2338-2342.
<br>
> <a href="https://sites.bu.edu/steveg/files/2016/06/Gro1980PNAS.pdf" data-auth="NotApplicable">
https://sites.bu.edu/steveg/files/2016/06/Gro1980PNAS.pdf</a><br>
> <br>
> Subsequent results culminated in my 1983 article with Michael Cohen, which was in press when the Hopfield (1982) article was published:<br>
> <br>
> Cohen, M.A. and Grossberg, S. (1983). Absolute stability of global pattern formation and parallel memory storage by competitive neural networks. IEEE Transactions on Systems, Man, and Cybernetics, SMC-13, 815-826.<br>
> <a href="https://sites.bu.edu/steveg/files/2016/06/CohGro1983IEEE.pdf" data-auth="NotApplicable">
https://sites.bu.edu/steveg/files/2016/06/CohGro1983IEEE.pdf</a><br>
> <br>
> Our article introduced a general class of neural networks for associative spatial pattern learning, which included the Additive and Shunting neural networks that I had earlier introduced, as well as a Lyapunov function for all of them.
<br>
> <br>
> This article proved global limit theorems about all these systems using that Lyapunov function.<br>
> <br>
> The Hopfield article describes the special case of the Additive model.<br>
> <br>
> His article proved no theorems.<br>
> <br>
> Best to all,<br>
> <br>
> Steve<br>
> <br>
> Stephen Grossberg<br>
> <a href="http://en.wikipedia.org/wiki/Stephen_Grossberg" data-auth="NotApplicable">
http://en.wikipedia.org/wiki/Stephen_Grossberg</a><br>
> <a href="http://scholar.google.com/citations?user=3BIV70wAAAAJ&hl=en" data-auth="NotApplicable">
http://scholar.google.com/citations?user=3BIV70wAAAAJ&hl=en</a><br>
> <a href="https://youtu.be/9n5AnvFur7I" data-auth="NotApplicable">https://youtu.be/9n5AnvFur7I</a><br>
> <a href="https://www.youtube.com/watch?v=_hBye6JQCh4" data-auth="NotApplicable">
https://www.youtube.com/watch?v=_hBye6JQCh4</a><br>
> <a href="https://www.amazon.com/Conscious-Mind-Resonant-Brain-Makes/dp/0190070552" data-auth="NotApplicable">
https://www.amazon.com/Conscious-Mind-Resonant-Brain-Makes/dp/0190070552</a><br>
> <br>
> Wang Professor of Cognitive and Neural Systems<br>
> Director, Center for Adaptive Systems<br>
> Professor Emeritus of Mathematics & Statistics, <br>
> Psychological & Brain Sciences, and Biomedical Engineering<br>
> Boston University<br>
> sites.bu.edu/steveg<br>
> steve@bu.edu<br>
> <br>
> From: Connectionists <connectionists-bounces@mailman.srv.cs.cmu.edu> on behalf of Schmidhuber Juergen <juergen@idsia.ch><br>
> Sent: Wednesday, January 25, 2023 8:44 AM<br>
> To: connectionists@cs.cmu.edu <connectionists@cs.cmu.edu><br>
> Subject: Re: Connectionists: Annotated History of Modern AI and Deep Learning<br>
> <br>
> Some are not aware of this historic tidbit in Sec. 4 of the survey: half a century ago, Shun-Ichi Amari published a learning recurrent neural network (1972) which was later called the Hopfield network.<br>
> <br>
> <a href="https://people.idsia.ch/~juergen/deep-learning-history.html#rnn" data-auth="NotApplicable">
https://people.idsia.ch/~juergen/deep-learning-history.html#rnn</a><br>
> <br>
> Jürgen<br>
> <br>
> <br>
> <br>
> <br>
> > On 13. Jan 2023, at 11:13, Schmidhuber Juergen <juergen@idsia.ch> wrote:<br>
> > <br>
> > Machine learning is the science of credit assignment. My new survey credits the pioneers of deep learning and modern AI (supplementing my award-winning 2015 survey):
<br>
> > <br>
> > <a href="https://arxiv.org/abs/2212.11279" data-auth="NotApplicable">https://arxiv.org/abs/2212.11279</a><br>
> > <br>
> > <a href="https://people.idsia.ch/~juergen/deep-learning-history.html" data-auth="NotApplicable">
https://people.idsia.ch/~juergen/deep-learning-history.html</a><br>
> > <br>
> > This was already reviewed by several deep learning pioneers and other experts. Nevertheless, let me know under juergen@idsia.ch if you can spot any remaining error or have suggestions for improvements.<br>
> > <br>
> > Happy New Year!<br>
> > <br>
> > Jürgen<br>
> > <br>
<br>
<br>
</div>
</span></font></div>
</body>
</html>