<div dir="ltr">Dear Stephen,<div><br></div><div>Thanks for letting us know about your Magnum Opus.</div><div><br></div><div>There is actually a learning algorithm for the Ising model and it works even when you can only observe the states of a subset of the units. It's called the Boltzmann Machine learning algorithm.</div><div><br></div><div>Geoff</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jan 25, 2023 at 1:25 PM Grossberg, Stephen <<a href="mailto:steve@bu.edu">steve@bu.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg7175408667475331088">
<div dir="ltr">
<div style="font-family:Arial,Helvetica,sans-serif;font-size:18pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
Dear Juergen,</div>
<div style="font-family:Arial,Helvetica,sans-serif;font-size:18pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif;font-size:18pt;color:rgb(0,0,0);background-color:rgb(255,255,255)">
Thanks for mentioning the Ising model!</div>
<div>
<div style="font-family:Arial,Helvetica,sans-serif;font-size:18pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Arial,Helvetica,sans-serif;font-size:18pt;color:rgb(0,0,0)">
As you know, it is a <b>binary model</b>, with just two states,<b> </b>and it does not learn.</div>
<div id="m_8906136608709761641Signature">
<div>
<div id="m_8906136608709761641divtagdefaultwrapper" dir="ltr" style="font-size:18pt;color:rgb(0,0,0);font-family:Arial,Helvetica,sans-serif">
<div style="color:rgb(0,0,0);font-style:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<span style="font-weight:normal;border-collapse:separate;border-spacing:0px">
<div style="overflow-wrap: break-word;"><span></span></div>
</span></div>
<span id="m_8906136608709761641ms-rterangepaste-end"></span><br>
<p style="margin-top:0px;margin-bottom:0px"></p>
</div>
<div id="m_8906136608709761641divtagdefaultwrapper" dir="ltr" style="font-size:18pt;color:rgb(0,0,0);font-family:Arial,Helvetica,sans-serif">
My Magnum Opus</div>
<div id="m_8906136608709761641divtagdefaultwrapper" dir="ltr" style="font-size:18pt;color:rgb(0,0,0);font-family:Arial,Helvetica,sans-serif">
<a href="https://www.amazon.com/Conscious-Mind-Resonant-Brain-Makes/dp/0190070552" id="m_8906136608709761641LPlnk136284" target="_blank">https://www.amazon.com/Conscious-Mind-Resonant-Brain-Makes/dp/0190070552</a><br>
</div>
<div id="m_8906136608709761641divtagdefaultwrapper" dir="ltr" style="font-size:18pt;color:rgb(0,0,0);font-family:Arial,Helvetica,sans-serif">
<br>
</div>
<div id="m_8906136608709761641divtagdefaultwrapper" dir="ltr" style="font-size:18pt;color:rgb(0,0,0);font-family:Arial,Helvetica,sans-serif">
reviews some of the early binary neural network models, such as the <b>McCulloch-Pitts</b>,<b> </b><b>Caianiello</b>, and
<b>Rosenblatt </b>models<b>, </b>starting on p. 64, before going on to review early
<b>linear models</b> that included learning, like the <b>Adeline and Madeline</b> models of Bernie
<b>Widrow</b> and the <b>Brain-State-in-a-Box</b> model of Jim <b>Anderson, </b>then
<b>continuous and nonlinear models</b> of various kinds, including models that are still used today.</div>
<div id="m_8906136608709761641divtagdefaultwrapper" dir="ltr" style="font-size:18pt;color:rgb(0,0,0);font-family:Arial,Helvetica,sans-serif">
<br>
</div>
<div id="m_8906136608709761641divtagdefaultwrapper" dir="ltr" style="font-size:18pt;color:rgb(0,0,0);font-family:Arial,Helvetica,sans-serif">
Best,</div>
<div id="m_8906136608709761641divtagdefaultwrapper" dir="ltr" style="font-size:18pt;color:rgb(0,0,0);font-family:Arial,Helvetica,sans-serif">
<br>
</div>
<div id="m_8906136608709761641divtagdefaultwrapper" dir="ltr" style="font-size:18pt;color:rgb(0,0,0);font-family:Arial,Helvetica,sans-serif">
Steve</div>
</div>
</div>
</div>
<div id="m_8906136608709761641appendonsend"></div>
<div style="font-family:Arial,Helvetica,sans-serif;font-size:18pt;color:rgb(0,0,0)">
<br>
</div>
<hr style="display:inline-block;width:98%">
<div id="m_8906136608709761641divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Connectionists <<a href="mailto:connectionists-bounces@mailman.srv.cs.cmu.edu" target="_blank">connectionists-bounces@mailman.srv.cs.cmu.edu</a>> on behalf of Schmidhuber Juergen <<a href="mailto:juergen@idsia.ch" target="_blank">juergen@idsia.ch</a>><br>
<b>Sent:</b> Wednesday, January 25, 2023 11:40 AM<br>
<b>To:</b> <a href="mailto:connectionists@cs.cmu.edu" target="_blank">connectionists@cs.cmu.edu</a> <<a href="mailto:connectionists@cs.cmu.edu" target="_blank">connectionists@cs.cmu.edu</a>><br>
<b>Subject:</b> Re: Connectionists: Annotated History of Modern AI and Deep Learning: Early recurrent neural networks for serial verbal learning and associative pattern learning</font>
<div> </div>
</div>
<div><font size="2"><span style="font-size:11pt">
<div>Dear Steve, <br>
<br>
thanks - I hope you noticed that the survey mentions your 1969 work! <br>
<br>
And of course it also mentions the origin of this whole recurrent network business: the Ising model or Lenz-Ising model introduced a century ago. See Sec. 4: 1920-1925: First Recurrent NN (RNN) Architecture<br>
<br>
<a href="https://people.idsia.ch/~juergen/deep-learning-history.html#rnn" target="_blank">https://people.idsia.ch/~juergen/deep-learning-history.html#rnn</a><br>
<br>
"The first non-learning RNN architecture (the Ising model or Lenz-Ising model) was introduced and analyzed by physicists Ernst Ising and Wilhelm Lenz in the 1920s [L20][I24,I25][K41][W45][T22]. It settles into an equilibrium state in response to input conditions,
and is the foundation of the first learning RNNs ...”<br>
<br>
Jürgen <br>
<br>
<br>
> On 25. Jan 2023, at 18:42, Grossberg, Stephen <<a href="mailto:steve@bu.edu" target="_blank">steve@bu.edu</a>> wrote:<br>
> <br>
> Dear Juergen and Connectionists colleagues,<br>
> <br>
> In his attached email below, Juergen mentioned a 1972 article of my friend and colleague, Shun-Ichi Amari, about recurrent neural networks that learn.<br>
> <br>
> Here are a couple of my own early articles from 1969 and 1971 about such networks. I introduced them to explain paradoxical data about serial verbal learning, notably the bowed serial position effect:<br>
> <br>
> Grossberg, S. (1969). On the serial learning of lists. Mathematical Biosciences, 4, 201-253.
<br>
> <a href="https://sites.bu.edu/steveg/files/2016/06/Gro1969MBLists.pdf" target="_blank">
https://sites.bu.edu/steveg/files/2016/06/Gro1969MBLists.pdf</a><br>
> <br>
> Grossberg, S. and Pepe, J. (1971). Spiking threshold and overarousal effects in serial learning. Journal of Statistical Physics, 3, 95-125.
<br>
> <a href="https://sites.bu.edu/steveg/files/2016/06/GroPepe1971JoSP.pdf" target="_blank">
https://sites.bu.edu/steveg/files/2016/06/GroPepe1971JoSP.pdf</a><br>
> <br>
> Juergen also mentioned that Shun-Ichi's work was a precursor of what some people call the Hopfield model, whose most cited articles were published in 1982 and 1984.<br>
> <br>
> I actually started publishing articles on this topic starting in the 1960s. Here are two of them:<br>
> <br>
> Grossberg, S. (1969). On learning and energy-entropy dependence in recurrent and nonrecurrent signed networks. Journal of Statistical Physics, 1, 319-350.
<br>
> <a href="https://sites.bu.edu/steveg/files/2016/06/Gro1969JourStatPhy.pdf" target="_blank">
https://sites.bu.edu/steveg/files/2016/06/Gro1969JourStatPhy.pdf</a><br>
> <br>
> Grossberg, S. (1971). Pavlovian pattern learning by nonlinear neural networks. Proceedings of the National Academy of Sciences, 68, 828-831.
<br>
> <a href="https://sites.bu.edu/steveg/files/2016/06/Gro1971ProNatAcaSci.pdf" target="_blank">
https://sites.bu.edu/steveg/files/2016/06/Gro1971ProNatAcaSci.pdf</a><br>
> <br>
> An early use of Lyapunov functions to prove global limit theorems in associative recurrent neural networks is found in the following 1980 PNAS article:<br>
> <br>
> Grossberg, S. (1980). Biological competition: Decision rules, pattern formation, and oscillations. Proceedings of the National Academy of Sciences, 77, 2338-2342.
<br>
> <a href="https://sites.bu.edu/steveg/files/2016/06/Gro1980PNAS.pdf" target="_blank">
https://sites.bu.edu/steveg/files/2016/06/Gro1980PNAS.pdf</a><br>
> <br>
> Subsequent results culminated in my 1983 article with Michael Cohen, which was in press when the Hopfield (1982) article was published:<br>
> <br>
> Cohen, M.A. and Grossberg, S. (1983). Absolute stability of global pattern formation and parallel memory storage by competitive neural networks. IEEE Transactions on Systems, Man, and Cybernetics, SMC-13, 815-826.<br>
> <a href="https://sites.bu.edu/steveg/files/2016/06/CohGro1983IEEE.pdf" target="_blank">
https://sites.bu.edu/steveg/files/2016/06/CohGro1983IEEE.pdf</a><br>
> <br>
> Our article introduced a general class of neural networks for associative spatial pattern learning, which included the Additive and Shunting neural networks that I had earlier introduced, as well as a Lyapunov function for all of them.
<br>
> <br>
> This article proved global limit theorems about all these systems using that Lyapunov function.<br>
> <br>
> The Hopfield article describes the special case of the Additive model.<br>
> <br>
> His article proved no theorems.<br>
> <br>
> Best to all,<br>
> <br>
> Steve<br>
> <br>
> Stephen Grossberg<br>
> <a href="http://en.wikipedia.org/wiki/Stephen_Grossberg" target="_blank">
http://en.wikipedia.org/wiki/Stephen_Grossberg</a><br>
> <a href="http://scholar.google.com/citations?user=3BIV70wAAAAJ&hl=en" target="_blank">
http://scholar.google.com/citations?user=3BIV70wAAAAJ&hl=en</a><br>
> <a href="https://youtu.be/9n5AnvFur7I" target="_blank">https://youtu.be/9n5AnvFur7I</a><br>
> <a href="https://www.youtube.com/watch?v=_hBye6JQCh4" target="_blank">
https://www.youtube.com/watch?v=_hBye6JQCh4</a><br>
> <a href="https://www.amazon.com/Conscious-Mind-Resonant-Brain-Makes/dp/0190070552" target="_blank">
https://www.amazon.com/Conscious-Mind-Resonant-Brain-Makes/dp/0190070552</a><br>
> <br>
> Wang Professor of Cognitive and Neural Systems<br>
> Director, Center for Adaptive Systems<br>
> Professor Emeritus of Mathematics & Statistics, <br>
> Psychological & Brain Sciences, and Biomedical Engineering<br>
> Boston University<br>
> <a href="http://sites.bu.edu/steveg" target="_blank">sites.bu.edu/steveg</a><br>
> <a href="mailto:steve@bu.edu" target="_blank">steve@bu.edu</a><br>
> <br>
> From: Connectionists <<a href="mailto:connectionists-bounces@mailman.srv.cs.cmu.edu" target="_blank">connectionists-bounces@mailman.srv.cs.cmu.edu</a>> on behalf of Schmidhuber Juergen <<a href="mailto:juergen@idsia.ch" target="_blank">juergen@idsia.ch</a>><br>
> Sent: Wednesday, January 25, 2023 8:44 AM<br>
> To: <a href="mailto:connectionists@cs.cmu.edu" target="_blank">connectionists@cs.cmu.edu</a> <<a href="mailto:connectionists@cs.cmu.edu" target="_blank">connectionists@cs.cmu.edu</a>><br>
> Subject: Re: Connectionists: Annotated History of Modern AI and Deep Learning<br>
> <br>
> Some are not aware of this historic tidbit in Sec. 4 of the survey: half a century ago, Shun-Ichi Amari published a learning recurrent neural network (1972) which was later called the Hopfield network.<br>
> <br>
> <a href="https://people.idsia.ch/~juergen/deep-learning-history.html#rnn" target="_blank">
https://people.idsia.ch/~juergen/deep-learning-history.html#rnn</a><br>
> <br>
> Jürgen<br>
> <br>
> <br>
> <br>
> <br>
> > On 13. Jan 2023, at 11:13, Schmidhuber Juergen <<a href="mailto:juergen@idsia.ch" target="_blank">juergen@idsia.ch</a>> wrote:<br>
> > <br>
> > Machine learning is the science of credit assignment. My new survey credits the pioneers of deep learning and modern AI (supplementing my award-winning 2015 survey):
<br>
> > <br>
> > <a href="https://arxiv.org/abs/2212.11279" target="_blank">https://arxiv.org/abs/2212.11279</a><br>
> > <br>
> > <a href="https://people.idsia.ch/~juergen/deep-learning-history.html" target="_blank">
https://people.idsia.ch/~juergen/deep-learning-history.html</a><br>
> > <br>
> > This was already reviewed by several deep learning pioneers and other experts. Nevertheless, let me know under <a href="mailto:juergen@idsia.ch" target="_blank">juergen@idsia.ch</a> if you can spot any remaining error or have suggestions for improvements.<br>
> > <br>
> > Happy New Year!<br>
> > <br>
> > Jürgen<br>
> > <br>
<br>
<br>
</div>
</span></font></div>
</div>
</div></blockquote></div>