
<html>

<head>

<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">

<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>

</head>

<body dir="ltr">

<div style="font-family: Arial, Helvetica, sans-serif; font-size: 18pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">

Dear Geoff,</div>

<div style="font-family: Arial, Helvetica, sans-serif; font-size: 18pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">

<br>

</div>

<div style="font-family: Arial, Helvetica, sans-serif; font-size: 18pt; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="elementToProof">

It's good to hear from you!</div>

<div class="elementToProof">

<div style="font-family: Arial, Helvetica, sans-serif; font-size: 18pt; color: rgb(0, 0, 0);" class="elementToProof">

<br>

</div>

<div style="font-family: Arial, Helvetica, sans-serif; font-size: 18pt; color: rgb(0, 0, 0);" class="elementToProof">

I of course know about the <b>Boltzmann Machine</b> learning algorithm that you published with David Ackley and Terry Sejnowski:</div>

<div style="font-family: Arial, Helvetica, sans-serif; font-size: 18pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted11">

<a href="https://onlinelibrary.wiley.com/doi/pdfdirect/10.1207/s15516709cog0901_7" id="LPlnk407215">https://onlinelibrary.wiley.com/doi/pdfdirect/10.1207/s15516709cog0901_7</a><br>

</div>

<div style="font-family: Arial, Helvetica, sans-serif; font-size: 18pt; color: rgb(0, 0, 0);" class="elementToProof">

<br>

</div>

<div style="font-family: Arial, Helvetica, sans-serif; font-size: 18pt; color: rgb(0, 0, 0);" class="elementToProof">

Because your article was published in 1985, I did not include it in a list of early algorithms.</div>

<div style="font-family: Arial, Helvetica, sans-serif; font-size: 18pt; color: rgb(0, 0, 0);" class="elementToProof">

<br>

</div>

<div style="font-family: Arial, Helvetica, sans-serif; font-size: 18pt; color: rgb(0, 0, 0);" class="elementToProof">

I do discuss it, however, in my Magnum Opus on p. 156:</div>

<div style="font-family: Arial, Helvetica, sans-serif; font-size: 18pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0">

<a href="https://www.amazon.com/Conscious-Mind-Resonant-Brain-Makes-ebook/dp/B094W6BBKN/ref=tmm_kin_swatch_0?_encoding=UTF8&qid=&sr=" id="LPlnk280050">https://www.amazon.com/Conscious-Mind-Resonant-Brain-Makes-ebook/dp/B094W6BBKN/ref=tmm_kin_swatch_0?_encoding=UTF8&qid=&sr=</a><br>

</div>

<div id="Signature">

<div>

<div id="divtagdefaultwrapper" dir="ltr" style="font-size:18pt; color:#000000; font-family:Arial,Helvetica,sans-serif">

<div class="" style="color:rgb(0,0,0); font-style:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px">

<span class="Apple-style-span" style="font-weight:normal; border-collapse:separate; border-spacing:0px">

<div class="" style="word-wrap:break-word"><span class="Apple-style-span" style=""></span></div>

</span></div>

<span id="ms-rterangepaste-end"></span><br>

<p style="margin-top:0px; margin-bottom:0px"></p>

</div>

</div>

</div>

</div>

<div id="appendonsend"></div>

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0)" class="elementToProof">

As you know better than I do, there is more to the Boltzmann Machine than an Ising model, as your use of the name Boltzmann, one of the greatest founders of statistical mechanics, suggests.</div>

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0)" class="elementToProof">

<br>

</div>

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1">

In particular, your model requires that an<b> external parameter</b>, such as a formal<b> temperature variable</b>, be slowly adjusted to control the approach to equilibrium.</div>

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1">

<br>

</div>

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1">

The Boltzmann Machine is thus neither an autonomous, nor a non-parametric, algorithm.</div>

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1">

<br>

</div>

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1">

I had also published quite a few models by 1985, notably foundational models on C<b>ompetitive Learning</b> and <b>Adaptive Resonance Theory</b>, or ART, between 1976 and 1980. </div>

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1">

<br>

</div>

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1">

ART can <b><i>autonomously </i>learn to attend, classify, recognize, and predict objects and events in a changing world

</b>that is filled with unexpected events. </div>

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1">

<br>

</div>

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1">

<b>Unsupervised ART</b> models, such as those that I published between 1976 and 1980, do not require any external supervision; e.g.,</div>

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1">

<br>

</div>

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1">

<span style="font-size:18pt;text-align:left;background-color:rgba(0, 0, 0, 0);display:inline !important" class="ContentPasted6">Grossberg, S. (1976). Adaptive pattern classification and universal recoding, II: Feedback, expectation, olfaction, and illusions.</span><span style="color:rgb(85, 85, 85);font-family:Capita;font-size:18px;text-align:left;background-color:rgb(255, 255, 255);display:inline !important"><span style="font-family:Arial, Helvetica, sans-serif;font-size:18pt;color:rgb(0, 0, 0);background-color:rgba(0, 0, 0, 0)" class="ContentPasted6"> </span></span><span style="font-size:18pt;background-color:rgba(0, 0, 0, 0)"><em style="box-sizing:border-box;text-align:left;background-color:rgba(0, 0, 0, 0)" class="ContentPasted6">Biological

 Cybernetics</em></span><span style="font-size:18pt;text-align:left;background-color:rgba(0, 0, 0, 0);display:inline !important" class="ContentPasted6">,</span><span style="color:rgb(85, 85, 85);font-family:Capita;font-size:18px;text-align:left;background-color:rgb(255, 255, 255);display:inline !important"><span style="font-family:Arial, Helvetica, sans-serif;font-size:18pt;color:rgb(0, 0, 0);background-color:rgba(0, 0, 0, 0)" class="ContentPasted6"> </span></span><span style="font-size:18pt;background-color:rgba(0, 0, 0, 0)"><strong style="box-sizing:border-box;font-weight:bold;text-align:left;background-color:rgba(0, 0, 0, 0)" class="ContentPasted6">23</strong></span><span style="font-size:18pt;text-align:left;background-color:rgba(0, 0, 0, 0);display:inline !important" class="ContentPasted6">,

 187-202.</span><span style="color:rgb(85, 85, 85);font-family:Capita;font-size:18px;text-align:left;background-color:rgb(255, 255, 255);display:inline !important"><span style="font-family:Arial, Helvetica, sans-serif;font-size:18pt;color:rgb(0, 0, 0);background-color:rgba(0, 0, 0, 0)" class="ContentPasted6"> </span></span><br>

</div>

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1 ContentPasted7">

<a href="https://sites.bu.edu/steveg/files/2016/06/Gro1976BiolCyb_II.pdf" id="LPNoLPOWALinkPreview_2">https://sites.bu.edu/steveg/files/2016/06/Gro1976BiolCyb_II.pdf</a><br>

</div>

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1 ContentPasted7">

<br>

</div>

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1 ContentPasted7">

<span style="font-size:18pt;text-align:left;background-color:rgba(0, 0, 0, 0);display:inline !important" class="ContentPasted8">Grossberg, S. (1980). How does a brain build a cognitive code?</span><span style="color:rgb(85, 85, 85);font-family:Capita;font-size:18px;text-align:left;background-color:rgb(255, 255, 255);display:inline !important"><span style="font-family:Arial, Helvetica, sans-serif;font-size:18pt;color:rgb(0, 0, 0);background-color:rgba(0, 0, 0, 0)" class="ContentPasted8"> </span></span><span style="font-size:18pt;background-color:rgba(0, 0, 0, 0)"><em style="box-sizing:border-box;text-align:left;background-color:rgba(0, 0, 0, 0)" class="ContentPasted8">Psychological

 Review</em></span><span style="font-size:18pt;text-align:left;background-color:rgba(0, 0, 0, 0);display:inline !important" class="ContentPasted8">,</span><span style="color:rgb(85, 85, 85);font-family:Capita;font-size:18px;text-align:left;background-color:rgb(255, 255, 255);display:inline !important"><span style="font-family:Arial, Helvetica, sans-serif;font-size:18pt;color:rgb(0, 0, 0);background-color:rgba(0, 0, 0, 0)" class="ContentPasted8"> </span></span><span style="font-size:18pt;background-color:rgba(0, 0, 0, 0)"><strong style="box-sizing:border-box;font-weight:bold;text-align:left;background-color:rgba(0, 0, 0, 0)" class="ContentPasted8">87</strong></span><span style="font-size:18pt;text-align:left;background-color:rgba(0, 0, 0, 0);display:inline !important" class="ContentPasted8">,

 1-51.</span><span style="color:rgb(85, 85, 85);font-family:Capita;font-size:18px;text-align:left;background-color:rgb(255, 255, 255);display:inline !important"><span style="font-family:Arial, Helvetica, sans-serif;font-size:18pt;color:rgb(0, 0, 0);background-color:rgba(0, 0, 0, 0)" class="ContentPasted8"> </span></span><br>

</div>

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1 ContentPasted7">

<span style="color:rgb(85, 85, 85);font-family:Capita;font-size:18px;text-align:left;background-color:rgb(255, 255, 255);display:inline !important"><span style="font-family:Arial, Helvetica, sans-serif;font-size:18pt;color:rgb(0, 0, 0);background-color:rgba(0, 0, 0, 0)" class="ContentPasted8 ContentPasted9"><a href="https://sites.bu.edu/steveg/files/2016/06/Gro1980PsychRev.pdf" id="LPlnk437733">https://sites.bu.edu/steveg/files/2016/06/Gro1980PsychRev.pdf</a></span></span></div>

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1 ContentPasted7">

<br>

</div>

<div style="" class="elementToProof ContentPasted1 ContentPasted7"><font color="#555555" face="Capita"><span style="font-size: 18pt; font-family: Arial, Helvetica, sans-serif; color: rgb(0, 0, 0);">See the Appendices, starting on p. 45, for some theorems.</span></font></div>

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1 ContentPasted7">

<br>

</div>

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1">

Starting in 1987, Gail Carpenter and I began to publish ART learning algorithms with a full suite of mathematical theorems and parametric computer simulations, including a proof that these

<b>ART models do not experience catastrophic forgetting:</b></div>

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1">

<br>

</div>

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1">

<span style="font-size:18pt;text-align:left;background-color:rgba(0, 0, 0, 0);display:inline !important" class="ContentPasted2">Carpenter, G.A., and Grossberg, S. (1987). A massively parallel architecture for a self-organizing neural pattern recognition machine.</span><span style="color:rgb(85, 85, 85);font-family:Capita;font-size:18px;text-align:left;background-color:rgb(255, 255, 255);display:inline !important"><span style="font-family:Arial, Helvetica, sans-serif;font-size:18pt;color:rgb(0, 0, 0);background-color:rgba(0, 0, 0, 0)" class="ContentPasted2"> </span></span><span style="font-size:18pt;background-color:rgba(0, 0, 0, 0)"><em style="box-sizing:border-box;text-align:left;background-color:rgba(0, 0, 0, 0)" class="ContentPasted2">Computer

 Vision, Graphics, and Image Processing</em></span><span style="font-size:18pt;text-align:left;background-color:rgba(0, 0, 0, 0);display:inline !important" class="ContentPasted2">,</span><strong style="box-sizing:border-box;font-weight:bold;color:rgb(85, 85, 85);font-family:Capita;font-size:18px;text-align:left;background-color:rgb(255, 255, 255)"><span style="font-family:Arial, Helvetica, sans-serif;font-size:18pt;color:rgb(0, 0, 0);background-color:rgba(0, 0, 0, 0)" class="ContentPasted2"> </span></strong><span style="font-size:18pt;background-color:rgba(0, 0, 0, 0)"><strong style="box-sizing:border-box;font-weight:bold;text-align:left;background-color:rgba(0, 0, 0, 0)" class="ContentPasted2">37</strong></span><span style="font-size:18pt;text-align:left;background-color:rgba(0, 0, 0, 0);display:inline !important" class="ContentPasted2">,

 54-115.</span><span style="color:rgb(85, 85, 85);font-family:Capita;font-size:18px;text-align:left;background-color:rgb(255, 255, 255);display:inline !important"><span style="font-family:Arial, Helvetica, sans-serif;font-size:18pt;color:rgb(0, 0, 0);background-color:rgba(0, 0, 0, 0)" class="ContentPasted2"> </span></span><br>

</div>

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1">

<span style="color:rgb(85, 85, 85);font-family:Capita;font-size:18px;text-align:left;background-color:rgb(255, 255, 255);display:inline !important"><span style="font-family:Arial, Helvetica, sans-serif;font-size:18pt;color:rgb(0, 0, 0);background-color:rgba(0, 0, 0, 0)" class="ContentPasted2 ContentPasted3"><a href="https://sites.bu.edu/steveg/files/2016/06/CarGro1987CVGIP.pdf" id="LPNoLPOWALinkPreview">https://sites.bu.edu/steveg/files/2016/06/CarGro1987CVGIP.pdf</a><br>

</span></span></div>

<div class="_Entity _EType_OWALinkPreview _EId_OWALinkPreview _EReadonly_1"></div>

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1">

<b><br>

</b></div>

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1">

<b>Supervised ARTMAP</b> models, which I began to published with Gail and some of our students starting in 1991, were simulated using challenging benchmark databases and compared with other algorithms. They are "supervised" by environmental feedback, which

 may or may not include a human teacher. <span style="color: rgb(0, 0, 0); font-family: Arial, Helvetica, sans-serif; font-size: 18pt;">See: </span></div>

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1">

<br>

</div>

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1">

<span style="font-size:18pt;text-align:left;background-color:rgba(0, 0, 0, 0);display:inline !important" class="ContentPasted4">Carpenter, G.A., Grossberg, S., and Reynolds, J.H. (1991). ARTMAP: Supervised real-time learning and classification of nonstationary

 data by a self-organizing neural network.</span><span style="color:rgb(85, 85, 85);font-family:Capita;font-size:18px;text-align:left;background-color:rgb(255, 255, 255);display:inline !important"><span style="font-family:Arial, Helvetica, sans-serif;font-size:18pt;color:rgb(0, 0, 0);background-color:rgba(0, 0, 0, 0)" class="ContentPasted4"> </span></span><span style="font-size:18pt;background-color:rgba(0, 0, 0, 0)"><em style="box-sizing:border-box;text-align:left;background-color:rgba(0, 0, 0, 0)" class="ContentPasted4">Neural

 Networks</em></span><span style="font-size:18pt;text-align:left;background-color:rgba(0, 0, 0, 0);display:inline !important" class="ContentPasted4">,</span><span style="color:rgb(85, 85, 85);font-family:Capita;font-size:18px;text-align:left;background-color:rgb(255, 255, 255);display:inline !important"><span style="font-family:Arial, Helvetica, sans-serif;font-size:18pt;color:rgb(0, 0, 0);background-color:rgba(0, 0, 0, 0)" class="ContentPasted4"> </span></span><span style="font-size:18pt;background-color:rgba(0, 0, 0, 0)"><strong style="box-sizing:border-box;font-weight:bold;text-align:left;background-color:rgba(0, 0, 0, 0)" class="ContentPasted4">4</strong></span><span style="font-size:18pt;text-align:left;background-color:rgba(0, 0, 0, 0);display:inline !important" class="ContentPasted4">,

 565-588.</span><span style="color:rgb(85, 85, 85);font-family:Capita;font-size:18px;text-align:left;background-color:rgb(255, 255, 255);display:inline !important"><span style="font-family:Arial, Helvetica, sans-serif;font-size:18pt;color:rgb(0, 0, 0);background-color:rgba(0, 0, 0, 0)" class="ContentPasted4"> </span></span><br>

</div>

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1">

<span style="color:rgb(85, 85, 85);font-family:Capita;font-size:18px;text-align:left;background-color:rgb(255, 255, 255);display:inline !important"><span style="font-family:Arial, Helvetica, sans-serif;font-size:18pt;color:rgb(0, 0, 0);background-color:rgba(0, 0, 0, 0)" class="ContentPasted4 ContentPasted5"><a href="https://sites.bu.edu/steveg/files/2016/06/CarGroRey1991NN.pdf" id="LPNoLPOWALinkPreview_1">https://sites.bu.edu/steveg/files/2016/06/CarGroRey1991NN.pdf</a><br>

</span></span></div>

<div class="_Entity _EType_OWALinkPreview _EId_OWALinkPreview_1 _EReadonly_1"></div>

<br>

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1 ContentPasted10">

as well as many other increasingly powerful ART algorithms that can be downloaded from sites.bu.edu/steveg and <a href="http://techlab.bu.edu/members/gail/publications.html" id="LPlnkOWALinkPreview_2">http://techlab.bu.edu/members/gail/publications.html</a></div>

<div class="_Entity _EType_OWALinkPreview _EId_OWALinkPreview_2 _EReadonly_1"></div>

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1">

<br>

</div>

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1">

Best,</div>

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1">

<br>

</div>

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1">

Steve</div>

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0)" class="elementToProof ContentPasted1">

<br>

</div>

<hr tabindex="-1" style="display:inline-block; width:98%">

<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Geoffrey Hinton <geoffrey.hinton@gmail.com><br>

<b>Sent:</b> Wednesday, January 25, 2023 2:02 PM<br>

<b>To:</b> Grossberg, Stephen <steve@bu.edu><br>

<b>Cc:</b> connectionists@cs.cmu.edu <connectionists@cs.cmu.edu><br>

<b>Subject:</b> Re: Connectionists: Annotated History of Modern AI and Deep Learning: Early binary, linear, and continuous-nonlinear neural networks, some which included learning</font>

<div> </div>

</div>

<div>

<div dir="ltr">Dear Stephen,

<div><br>

</div>

<div>Thanks for letting us know about your Magnum Opus.</div>

<div><br>

</div>

<div>There is actually a learning algorithm for the Ising model and it works even when you can only observe the states of a subset of the units. It's called the Boltzmann Machine learning algorithm.</div>

<div><br>

</div>

<div>Geoff</div>

<div><br>

</div>

</div>

<br>

<div class="x_gmail_quote">

<div dir="ltr" class="x_gmail_attr">On Wed, Jan 25, 2023 at 1:25 PM Grossberg, Stephen <<a href="mailto:steve@bu.edu" data-auth="NotApplicable" data-loopstyle="link">steve@bu.edu</a>> wrote:<br>

</div>

<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">

<div class="x_msg7175408667475331088">

<div dir="ltr">

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">

Dear Juergen,</div>

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">

<br>

</div>

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0); background-color:rgb(255,255,255)">

Thanks for mentioning the Ising model!</div>

<div>

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0)">

<br>

</div>

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0)">

As you know, it is a <b>binary model</b>, with just two states,<b> </b>and it does not learn.</div>

<div id="x_m_8906136608709761641Signature">

<div>

<div id="x_m_8906136608709761641divtagdefaultwrapper" dir="ltr" style="font-size:18pt; color:rgb(0,0,0); font-family:Arial,Helvetica,sans-serif">

<div style="color:rgb(0,0,0); font-style:normal; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px">

<span style="font-weight:normal; border-collapse:separate; border-spacing:0px">

<div style=""><span></span></div>

</span></div>

<span id="x_m_8906136608709761641ms-rterangepaste-end"></span><br>

<p style="margin-top:0px; margin-bottom:0px"></p>

</div>

<div id="x_m_8906136608709761641divtagdefaultwrapper" dir="ltr" style="font-size:18pt; color:rgb(0,0,0); font-family:Arial,Helvetica,sans-serif">

My Magnum Opus</div>

<div id="x_m_8906136608709761641divtagdefaultwrapper" dir="ltr" style="font-size:18pt; color:rgb(0,0,0); font-family:Arial,Helvetica,sans-serif">

<a href="https://www.amazon.com/Conscious-Mind-Resonant-Brain-Makes/dp/0190070552" data-auth="NotApplicable" id="x_m_8906136608709761641LPlnk136284">https://www.amazon.com/Conscious-Mind-Resonant-Brain-Makes/dp/0190070552</a><br>

</div>

<div id="x_m_8906136608709761641divtagdefaultwrapper" dir="ltr" style="font-size:18pt; color:rgb(0,0,0); font-family:Arial,Helvetica,sans-serif">

<br>

</div>

<div id="x_m_8906136608709761641divtagdefaultwrapper" dir="ltr" style="font-size:18pt; color:rgb(0,0,0); font-family:Arial,Helvetica,sans-serif">

reviews some of the early binary neural network models, such as the <b>McCulloch-Pitts</b>,<b> </b><b>Caianiello</b>, and

<b>Rosenblatt </b>models<b>, </b>starting on p. 64, before going on to review early

<b>linear models</b> that included learning, like the <b>Adeline and Madeline</b> models of Bernie

<b>Widrow</b> and the <b>Brain-State-in-a-Box</b> model of Jim <b>Anderson, </b>then

<b>continuous and nonlinear models</b> of various kinds, including models that are still used today.</div>

<div id="x_m_8906136608709761641divtagdefaultwrapper" dir="ltr" style="font-size:18pt; color:rgb(0,0,0); font-family:Arial,Helvetica,sans-serif">

<br>

</div>

<div id="x_m_8906136608709761641divtagdefaultwrapper" dir="ltr" style="font-size:18pt; color:rgb(0,0,0); font-family:Arial,Helvetica,sans-serif">

Best,</div>

<div id="x_m_8906136608709761641divtagdefaultwrapper" dir="ltr" style="font-size:18pt; color:rgb(0,0,0); font-family:Arial,Helvetica,sans-serif">

<br>

</div>

<div id="x_m_8906136608709761641divtagdefaultwrapper" dir="ltr" style="font-size:18pt; color:rgb(0,0,0); font-family:Arial,Helvetica,sans-serif">

Steve</div>

</div>

</div>

</div>

<div id="x_m_8906136608709761641appendonsend"></div>

<div style="font-family:Arial,Helvetica,sans-serif; font-size:18pt; color:rgb(0,0,0)">

<br>

</div>

<hr style="display:inline-block; width:98%">

<div id="x_m_8906136608709761641divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Connectionists <<a href="mailto:connectionists-bounces@mailman.srv.cs.cmu.edu" data-auth="NotApplicable" data-loopstyle="link">connectionists-bounces@mailman.srv.cs.cmu.edu</a>>

 on behalf of Schmidhuber Juergen <<a href="mailto:juergen@idsia.ch" data-auth="NotApplicable" data-loopstyle="link">juergen@idsia.ch</a>><br>

<b>Sent:</b> Wednesday, January 25, 2023 11:40 AM<br>

<b>To:</b> <a href="mailto:connectionists@cs.cmu.edu" data-auth="NotApplicable" data-loopstyle="link">

connectionists@cs.cmu.edu</a> <<a href="mailto:connectionists@cs.cmu.edu" data-auth="NotApplicable" data-loopstyle="link">connectionists@cs.cmu.edu</a>><br>

<b>Subject:</b> Re: Connectionists: Annotated History of Modern AI and Deep Learning: Early recurrent neural networks for serial verbal learning and associative pattern learning</font>

<div> </div>

</div>

<div><font size="2"><span style="font-size:11pt">

<div>Dear Steve, <br>

<br>

thanks - I hope you noticed that the survey mentions your 1969 work! <br>

<br>

And of course it also mentions the origin of this whole recurrent network business: the Ising model or Lenz-Ising model introduced a century ago. See Sec. 4: 1920-1925: First Recurrent NN (RNN) Architecture<br>

<br>

<a href="https://people.idsia.ch/~juergen/deep-learning-history.html#rnn" data-auth="NotApplicable">https://people.idsia.ch/~juergen/deep-learning-history.html#rnn</a><br>

<br>

"The first non-learning RNN architecture (the Ising model or Lenz-Ising model) was introduced and analyzed by physicists Ernst Ising and Wilhelm Lenz in the 1920s [L20][I24,I25][K41][W45][T22]. It settles into an equilibrium state in response to input conditions,

 and is the foundation of the first learning RNNs ...�<br>

<br>

J�rgen <br>

<br>

<br>

> On 25. Jan 2023, at 18:42, Grossberg, Stephen <<a href="mailto:steve@bu.edu" data-auth="NotApplicable" data-loopstyle="link">steve@bu.edu</a>> wrote:<br>

> <br>

> Dear Juergen and Connectionists colleagues,<br>

> <br>

> In his attached email below, Juergen mentioned a 1972 article of my friend and colleague, Shun-Ichi Amari, about recurrent neural networks that learn.<br>

> <br>

> Here are a couple of my own early articles from 1969 and 1971 about such networks. I introduced them to explain paradoxical data about serial verbal learning, notably the bowed serial position effect:<br>

> <br>

> Grossberg, S. (1969). On the serial learning of lists. Mathematical Biosciences, 4, 201-253.

<br>

> <a href="https://sites.bu.edu/steveg/files/2016/06/Gro1969MBLists.pdf" data-auth="NotApplicable">

https://sites.bu.edu/steveg/files/2016/06/Gro1969MBLists.pdf</a><br>

> <br>

> Grossberg, S. and Pepe, J. (1971). Spiking threshold and overarousal effects in serial learning. Journal of Statistical Physics, 3, 95-125.

<br>

> <a href="https://sites.bu.edu/steveg/files/2016/06/GroPepe1971JoSP.pdf" data-auth="NotApplicable">

https://sites.bu.edu/steveg/files/2016/06/GroPepe1971JoSP.pdf</a><br>

> <br>

> Juergen also mentioned that Shun-Ichi's work was a precursor of what some people call the Hopfield model, whose most cited articles were published in 1982 and 1984.<br>

> <br>

> I actually started publishing articles on this topic starting in the 1960s. Here are two of them:<br>

> <br>

> Grossberg, S. (1969). On learning and energy-entropy dependence in recurrent and nonrecurrent signed networks. Journal of Statistical Physics, 1, 319-350.

<br>

> <a href="https://sites.bu.edu/steveg/files/2016/06/Gro1969JourStatPhy.pdf" data-auth="NotApplicable">

https://sites.bu.edu/steveg/files/2016/06/Gro1969JourStatPhy.pdf</a><br>

> <br>

> Grossberg, S. (1971). Pavlovian pattern learning by nonlinear neural networks. Proceedings of the National Academy of Sciences, 68, 828-831.

<br>

> <a href="https://sites.bu.edu/steveg/files/2016/06/Gro1971ProNatAcaSci.pdf" data-auth="NotApplicable">

https://sites.bu.edu/steveg/files/2016/06/Gro1971ProNatAcaSci.pdf</a><br>

> <br>

> An early use of Lyapunov functions to prove global limit theorems in associative recurrent neural networks is found in the following 1980 PNAS article:<br>

> <br>

> Grossberg, S. (1980). Biological competition: Decision rules, pattern formation, and oscillations. Proceedings of the National Academy of Sciences, 77, 2338-2342.

<br>

> <a href="https://sites.bu.edu/steveg/files/2016/06/Gro1980PNAS.pdf" data-auth="NotApplicable">

https://sites.bu.edu/steveg/files/2016/06/Gro1980PNAS.pdf</a><br>

> <br>

> Subsequent results culminated in my 1983 article with Michael Cohen, which was in press when the Hopfield (1982) article was published:<br>

> <br>

> Cohen, M.A. and Grossberg, S. (1983). Absolute stability of global pattern formation and parallel memory storage by competitive neural networks. IEEE Transactions on Systems, Man, and Cybernetics, SMC-13, 815-826.<br>

>  <a href="https://sites.bu.edu/steveg/files/2016/06/CohGro1983IEEE.pdf" data-auth="NotApplicable">

https://sites.bu.edu/steveg/files/2016/06/CohGro1983IEEE.pdf</a><br>

> <br>

> Our article introduced a general class of neural networks for associative spatial pattern learning, which included the Additive and Shunting neural networks that I had earlier introduced, as well as a Lyapunov function for all of them.

<br>

> <br>

> This article proved global limit theorems about all these systems using that Lyapunov function.<br>

> <br>

> The Hopfield article describes the special case of the Additive model.<br>

> <br>

> His article proved no theorems.<br>

> <br>

> Best to all,<br>

> <br>

> Steve<br>

> <br>

> Stephen Grossberg<br>

> <a href="http://en.wikipedia.org/wiki/Stephen_Grossberg" data-auth="NotApplicable">

http://en.wikipedia.org/wiki/Stephen_Grossberg</a><br>

> <a href="http://scholar.google.com/citations?user=3BIV70wAAAAJ&hl=en" data-auth="NotApplicable">

http://scholar.google.com/citations?user=3BIV70wAAAAJ&hl=en</a><br>

> <a href="https://youtu.be/9n5AnvFur7I" data-auth="NotApplicable">https://youtu.be/9n5AnvFur7I</a><br>

> <a href="https://www.youtube.com/watch?v=_hBye6JQCh4" data-auth="NotApplicable">

https://www.youtube.com/watch?v=_hBye6JQCh4</a><br>

> <a href="https://www.amazon.com/Conscious-Mind-Resonant-Brain-Makes/dp/0190070552" data-auth="NotApplicable">

https://www.amazon.com/Conscious-Mind-Resonant-Brain-Makes/dp/0190070552</a><br>

> <br>

> Wang Professor of Cognitive and Neural Systems<br>

> Director, Center for Adaptive Systems<br>

> Professor Emeritus of Mathematics & Statistics, <br>

>        Psychological & Brain Sciences, and Biomedical Engineering<br>

> Boston University<br>

> <a href="http://sites.bu.edu/steveg" data-auth="NotApplicable" data-loopstyle="link">

sites.bu.edu/steveg</a><br>

> <a href="mailto:steve@bu.edu" data-auth="NotApplicable" data-loopstyle="link">steve@bu.edu</a><br>

> <br>

> From: Connectionists <<a href="mailto:connectionists-bounces@mailman.srv.cs.cmu.edu" data-auth="NotApplicable" data-loopstyle="link">connectionists-bounces@mailman.srv.cs.cmu.edu</a>> on behalf of Schmidhuber Juergen <<a href="mailto:juergen@idsia.ch" data-auth="NotApplicable" data-loopstyle="link">juergen@idsia.ch</a>><br>

> Sent: Wednesday, January 25, 2023 8:44 AM<br>

> To: <a href="mailto:connectionists@cs.cmu.edu" data-auth="NotApplicable" data-loopstyle="link">

connectionists@cs.cmu.edu</a> <<a href="mailto:connectionists@cs.cmu.edu" data-auth="NotApplicable" data-loopstyle="link">connectionists@cs.cmu.edu</a>><br>

> Subject: Re: Connectionists: Annotated History of Modern AI and Deep Learning<br>

>  <br>

> Some are not aware of this historic tidbit in Sec. 4 of the survey: half a century ago, Shun-Ichi Amari published a learning recurrent neural network (1972) which was later called the Hopfield network.<br>

> <br>

> <a href="https://people.idsia.ch/~juergen/deep-learning-history.html#rnn" data-auth="NotApplicable">

https://people.idsia.ch/~juergen/deep-learning-history.html#rnn</a><br>

> <br>

> J�rgen<br>

> <br>

> <br>

> <br>

> <br>

> > On 13. Jan 2023, at 11:13, Schmidhuber Juergen <<a href="mailto:juergen@idsia.ch" data-auth="NotApplicable" data-loopstyle="link">juergen@idsia.ch</a>> wrote:<br>

> > <br>

> > Machine learning is the science of credit assignment. My new survey credits the pioneers of deep learning and modern AI (supplementing my award-winning 2015 survey):

<br>

> > <br>

> > <a href="https://arxiv.org/abs/2212.11279" data-auth="NotApplicable">https://arxiv.org/abs/2212.11279</a><br>

> > <br>

> > <a href="https://people.idsia.ch/~juergen/deep-learning-history.html" data-auth="NotApplicable">

https://people.idsia.ch/~juergen/deep-learning-history.html</a><br>

> > <br>

> > This was already reviewed by several deep learning pioneers and other experts. Nevertheless, let me know under

<a href="mailto:juergen@idsia.ch" data-auth="NotApplicable" data-loopstyle="link">

juergen@idsia.ch</a> if you can spot any remaining error or have suggestions for improvements.<br>

> > <br>

> > Happy New Year!<br>

> > <br>

> > J�rgen<br>

> > <br>

<br>

<br>

</div>

</span></font></div>

</div>

</div>

</blockquote>

</div>

</div>

</body>

</html>