<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><font class="Apple-style-span" face="Arial" size="5">Dear John,</font><div><font class="Apple-style-span" face="Arial" size="5"><br></font></div><div><font class="Apple-style-span" face="Arial" size="5">Thanks for your questions. I reply below.</font></div><div><font class="Apple-style-span" face="Arial" size="5"><br></font><div><div><font class="Apple-style-span" face="Arial" size="5">On Apr 5, 2014, at 10:51 AM, Juyang Weng wrote:</font></div><font class="Apple-style-span" face="Arial" size="5"><br class="Apple-interchange-newline"></font><blockquote type="cite">
<meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type">
<div bgcolor="#FFFFFF" text="#000000"><font class="Apple-style-span" face="Arial" size="5">
Dear Steve,<br>
<br>
This is one of my long-time questions that I did not have a chance
to ask you when I met you many times before. <br>
But they may be useful for some people on this list. <br>
Please accept my apology of my question implies any false impression
that I did not intend.<br>
<br>
(1) Your statement below seems to have confirmed my understanding:
<br>
Your top-down process in ART in the late 1990's is basically for
finding an acceptable match <br>
between the input feature vector and the stored feature vectors
represented by neurons (not meant for the nearest match). <br></font></div></blockquote><div><font class="Apple-style-span" face="Arial" size="5"><br></font></div><font class="Apple-style-span" face="Arial" size="5">ART has developed a lot since the 1990s. A non-technical but fairly comprehensive review article was published in 2012 in <i>Neural Networks</i> and can be found at <a href="http://cns.bu.edu/~steve/ART.pdf">http://cns.bu.edu/~steve/ART.pdf</a>.</font></div><div><font class="Apple-style-span" face="Arial" size="5"><br></font></div><div><font class="Apple-style-span" face="Arial" size="5">I do not think about the top-down process in ART in quite the way that you state above. My reason for this is summarized by the acronym CLEARS for the processes of Consciousness, Learning, Expectation, Attention, Resonance, and Synchrony. </font><span class="Apple-style-span" style="font-family: Arial; font-size: x-large; ">All the CLEARS processes come into this story, and </span><span class="Apple-style-span" style="font-family: Arial; font-size: x-large; ">ART top-down mechanisms contribute to all of them. For me, the most fundamental issues concern how ART dynamically self-stabilizes the memories that are learned within the model's bottom-up adaptive filters and top-down expectations. </span></div><div><font class="Apple-style-span" face="Arial" size="5"><br></font></div><div><font class="Apple-style-span" face="Arial" size="5">In particular, during learning, a big enough mismatch can lead to hypothesis testing and search for a new, or previously learned, category that leads to an acceptable match. The criterion for what is "big enough mismatch" or "acceptable match" is regulated by a vigilance parameter that can itself vary in a state-dependent way.</font></div><div><font class="Apple-style-span" face="Arial" size="5"><br></font></div><div><font class="Apple-style-span" face="Arial" size="5">After learning occurs, a bottom-up input pattern typically directly selects the best-matching category, without any hypothesis testing or search. And even if there is a reset due to a large initial mismatch with a previously active category, a single reset event may lead directly to a matching category that can directly resonate with the data. </font></div><div><font class="Apple-style-span" face="Arial" size="5"><br></font></div><div><font class="Apple-style-span" face="Arial" size="5">I should note that all of the foundational predictions of ART now have substantial bodies of psychological and neurobiological data to support them. See the review article if you would like to read about them.</font></div><div><font class="Apple-style-span" face="Arial" size="5"><br></font><blockquote type="cite"><div bgcolor="#FFFFFF" text="#000000"><font class="Apple-style-span" face="Arial" size="5">
The currently active neuron is the one being examined by the top
down process<br></font></div></blockquote><div><font class="Apple-style-span" face="Arial" size="5"><br></font></div><font class="Apple-style-span" face="Arial" size="5">I'm not sure what you mean by "being examined", but perhaps my comment above may deal with it.</font></div><div><font class="Apple-style-span" face="Arial" size="5"><br></font></div><div><font class="Apple-style-span" face="Arial" size="5">I should comment, though, about your use of the word "currently active neuron". I assume that you mean at the category level. </font></div><div><font class="Apple-style-span" face="Arial" size="5"><br></font></div><div><font class="Apple-style-span" face="Arial" size="5">In this regard, there are two ART's. The first aspect of ART is as a cognitive and neural theory whose scope, which includes perceptual, cognitive, and adaptively timed cognitive-emotional dynamics, among other processes, is illustrated by the above referenced 2012 review article in <i>Neural Networks</i>. In the biological theory, there is no general commitment to just one "currently active neuron". One always considers the neuronal population, or populations, that represent a learned category. Sometimes, but not always, a winner-take-all category is chosen. </font></div><div><font class="Apple-style-span" face="Arial" size="5"><br></font></div><div><font class="Apple-style-span" face="Arial" size="5">The 2012 review article illustrates some of the large data bases of psychological and neurobiological data that have been explained in a principled way, quantitatively simulated, and successfully predicted by ART over a period of decades. ART-like processing is, however, certainly not the only kind of computation that may be needed to understand how the brain works. The paradigm called Complementary Computing that I introduced awhile ago makes precise the sense in which ART may be just one kind of dynamics supported by advanced brains. This is also summarized in the review article.<br></font><div><font class="Apple-style-span" face="Arial" size="5"><br></font></div><div><font class="Apple-style-span" face="Arial" size="5">The second aspect of ART is as a series of algorithms that mathematically characterize key ART design principles and mechanisms in a focused setting, and provide algorithms for large-scale applications in engineering and technology. ARTMAP, fuzzy ARTMAP, and distributed ARTMAP are among these, all of them developed with Gail Carpenter. Some of these algorithms use winner-take-all categories to enable the proof of mathematical theorems that characterize how underlying design principles work. In contrast, the distributed ARTMAP family of algorithms, developed by Gail Carpenter and her colleagues, allows for distributed category representations without losing the benefits of fast, incremental, self-stabilizing learning and prediction in response to a large non-stationary databases that can include many unexpected events. </font></div><div><font class="Apple-style-span" face="Arial" size="5"><br></font></div><div><font class="Apple-style-span" face="Arial" size="5">See, e.g., <a href="http://techlab.bu.edu/members/gail/articles/115_dART_NN_1997_.pdf">http://techlab.bu.edu/members/gail/articles/115_dART_NN_1997_.pdf</a> and <a href="http://techlab.bu.edu/members/gail/articles/155_Fusion2008_CarpenterRavindran.pdf">http://techlab.bu.edu/members/gail/articles/155_Fusion2008_CarpenterRavindran.pdf</a>.</font></div><div><font class="Apple-style-span" face="Arial" size="5"><br></font></div><div><font class="Apple-style-span" face="Arial" size="5">I should note that FAST learning is a technical concept: it means that each adaptive weight can converge to its new equilibrium value on EACH learning trial. That is why ART algorithms can often successfully carry out one-trial incremental learning of a data base. This is not true of many other algorithms, such as back propagation, simulated annealing, and the like, which all experience catastrophic forgetting if they try to do fast learning. Almost all other learning algorithms need to be run using slow learning, that allows only a small increment in the values of adaptive weights on each learning trial, to avoid massive memory instabilities, and work best in response to stationary data. Such algorithms often fail to detect important rare cases, among other limitations. ART can provably learn in either the fast or slow mode in response to non-stationary data.</font></div><div><font class="Apple-style-span" face="Arial" size="5"><br></font></div><blockquote type="cite"><div bgcolor="#FFFFFF" text="#000000"><font class="Apple-style-span" face="Arial" size="5">
in a sequential fashion: one neuron after another, until an
acceptable neuron is found.<br>
<br>
(2) The input to the ART in the late 1990's is for a single feature
vector as a monolithic input. <br>
By monolithic, I mean that all neurons take the entire input feature
vector as input. <br>
I raise this point here because neuron in ART in the late 1990's
does not have an explicit local sensory receptive field (SRF), <br>
i.e., are fully connected from all components of the input vector.
A local SRF means that each neuron is only connected to a small
region <br>
in an input image. <br></font></div></blockquote><div><font class="Apple-style-span" face="Arial" size="5"><br></font></div><font class="Apple-style-span" face="Arial" size="5">Various ART algorithms for technology do use fully connected networks. They represent a single-channel case, which is often sufficient in applications and which simplifies mathematical proofs. However, the single-channel case is, as its name suggests, not a necessary constraint on ART design. </font></div><div><font class="Apple-style-span" face="Arial" size="5"><br></font></div><div><font class="Apple-style-span" face="Arial" size="5">In addition, many ART biological models do not restrict themselves to the single-channel case, and do have receptive fields. These include the LAMINART family of models that predict functional roles for many identified cell types in the laminar circuits of cerebral cortex. These models illustrate how variations of a shared laminar circuit design can carry out very different intelligent functions, such as 3D vision (e.g., 3D LAMINART), speech and language (e.g., cARTWORD), and cognitive information processing (e.g., LIST PARSE). They are all summarized in the 2012 review article, with the archival articles themselves on my web page <a href="http://cns.bu.edu/~steve">http://cns.bu.edu/~steve</a>. </font></div><div><font class="Apple-style-span" face="Arial" size="5"><br></font></div><div><font class="Apple-style-span" face="Arial" size="5">The existence of these laminar variations-on-a-theme provides an existence proof for the exciting goal of designing a family of chips whose specializations can realize all aspects of higher intelligence, and which can be consistently connected because they all share a similar underlying design. Work on achieving this goal can productively occupy lots of creative modelers and technologists for many years to come.</font></div><div><font class="Apple-style-span" face="Arial" size="5"><br></font></div><div><font class="Apple-style-span" face="Arial" size="5">I hope that the above replies provide some relevant information, as well as pointers for finding more.</font></div><div><font class="Apple-style-span" face="Arial" size="5"><br></font></div><div><font class="Apple-style-span" face="Arial" size="5">Best,</font></div><div><font class="Apple-style-span" face="Arial" size="5"><br></font></div><div><font class="Apple-style-span" face="Arial" size="5">Steve</font></div><div><font class="Apple-style-span" face="Arial" size="5"><br></font></div><div><font class="Apple-style-span" face="Arial" size="5"><br></font></div><div><font class="Apple-style-span" face="Arial" size="5"><br></font></div><div><blockquote type="cite"><div bgcolor="#FFFFFF" text="#000000">
<font class="Apple-style-span" face="Arial" size="5"><br>
My apology again if my understanding above has errors although I
have examined the above two points carefully <br>
through multiple your papers.<br>
<br>
Best regards,<br>
<br>
-John<br><br></font><pre class="moz-signature" cols="72"><font class="Apple-style-span" face="Arial"><span class="Apple-style-span" style="font-size: 18px;">Juyang (John) Weng, Professor
Department of Computer Science and Engineering
MSU Cognitive Science Program and MSU Neuroscience Program
428 S Shaw Ln Rm 3115
Michigan State University
East Lansing, MI 48824 USA
Tel: 517-353-4388
Fax: 517-432-1061
Email: <a class="moz-txt-link-abbreviated" href="mailto:weng@cse.msu.edu">weng@cse.msu.edu</a>
URL: <a class="moz-txt-link-freetext" href="http://www.cse.msu.edu/~weng/">http://www.cse.msu.edu/~weng/</a>
----------------------------------------------
</span></font></pre>
</div>
</blockquote></div><font class="Apple-style-span" face="Arial" size="5"><br></font><div>
<font class="Apple-style-span" face="Arial" size="5"><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><div><div><div>Stephen Grossberg</div><div>Wang Professor of Cognitive and Neural Systems</div><div>Professor of Mathematics, Psychology, and Biomedical Engineering</div><div><div>Director, Center for Adaptive Systems <a href="http://www.cns.bu.edu/about/cas.html">http://www.cns.bu.edu/about/cas.html</a></div></div><div><a href="http://cns.bu.edu/~steve">http://cns.bu.edu/~steve</a></div><div><a href="mailto:steve@bu.edu">steve@bu.edu</a></div></div></div></div><div><br></div></div></span></div></span><br class="Apple-interchange-newline"></span><br class="Apple-interchange-newline">
</font></div>
<br></div></body></html>