Connectionists: Scientific Integrity, the 2021 Turing Lecture, etc.

[Tsvi Achler]

Hi Asim,

I love your globalist-localist dichotomy as it refers to learning.  In
localist learning only the neuron that is associated with the learning
label needs to be modified to learn. Feed forward methods are globalist
because if you want to modify a label you have to rehearse with all the old
and the new and many weights may change anywhere within the network.

Unfortunately when I try to publish papers using this distinction I find
globalist and localist can mean many things to related literatures.  Do you
have good references that nail down the definitions and history from the
beginning of connectionism? I would love to put that up and clarify it
within a general search, maybe even place it on Wikipedia.

I find the localist-globalist distinction very important.. Regulatory
feedback learns in a localist way. This is why it does not need the same
rehearsal as feed forward networks.

Regulatory feedback may also help address the criticism by those who
subscribe to globalist methods and criticise the need of non-distributed
single grandmother cells. Regulatory feedback can support a localist method
that is also distributed.
Due to regulatory feedback, if multiple neurons encode the same thing then
they create a linearly dependent type of situation. You can create many
neurons that encode the same thing (let's say grandmother) and get the same
result as if you had one neuron. In this paradigm all of the neurons that
encode the same thing can simply be summed up and they act as a single
neuron.   Thus any combination of neurons can be used in total to represent
the same thing and if one neuron dies this does not affect the sum at all.

This linearly dependent situation also yields interesting results
especially when you put a regulatory feedback network into the paradigm of
LTP and LTD experiments.  Within LTP & LTD activation and induction, you
can get changes where a different linearly dependent neuron takes priority,
looking like synaptic change but occurring without any synaptic change.
This can greatly change our understanding of brain recognition and design
of experiments.

Although I may not expect the community to immediately adopt this paradigm
and results, I do expect it to be accepted as a possibility to be
thought about and analyzed, and placed in literature where it will be
read.  Instead it is snuffed out.
You won't be able to read it in a publication.

For example, I had a conference paper on this linear dependence accepted a
while ago, but since I did not have funding, I could not provide the $500
the conference demanded. As a pay-to-play paywall journal they did not care
about the situation. Moreover as a long term non-funded researcher I am
no longer willing to put my time and efforts into publishing work where it
will not be read or where a cost to the author is demanded even if there is
no funding.  If the connectionist community and academia really wants to
support new ideas, it has to equitably provide funding, make publications
free, or make things less political.

If there are enough people interested and subscribed to view videos I can
make a video to show the dynamics that create this situation just like I
did for regulatory feedback.

Sincerely
-Tsvi

On Sat, Nov 6, 2021, 1:57 AM Asim Roy <ASIM.ROY at asu.edu> wrote:

> Over a period of more than 25 years, I have had the opportunity to argue
> about the brain in both public forums and private discussions. And they
> included very well-known scholars such as Walter Freeman (UC-Berkeley),
> Horace Barlow (Cambridge; great grandson of Charles Darwin), Jay McClelland
> (Stanford), Bernard Baars (Neuroscience Institute), Christof Koch (Allen
> Institute), Teuvo Kohonen (Finland) and many others, some of whom are on
> this list. And many became good friends through these debates.
>
>
>
> We argued about many issues over the years, but the one that baffled me
> the most was the one about localist vs. distributed representation. Here’s
> the issue. As far as I know, although all the Nobel prizes in the field of
> neurophysiology – from Hubel and Wiesel (simple and complex cells) and
> Moser and O’Keefe (grid and place cells) to the current one on discovery of
> temperature and touch sensitive receptors and neurons - are about finding
> “meaning” in single or a group of dedicated cells, the distributed
> representation theory has yet to explain these findings of “meaning.”
> Contrary to the assertion that the field is open-minded, I think most in
> this field are afraid the to cross the red line.
>
>
>
> Horace Barlow was the exception. He was perhaps the only neuroscientist
> who was willing to cross the red line and declare that “grandmother cells
> will be found.” After a debate on this issue in 2012, which included Walter
> Freeman and others, Horace visited me in Phoenix at the age of 91 for
> further discussion.
>
>
>
> If the field is open minded, would love to hear how distributed
> representation is compatible with finding “meaning” in the activations of
> single or a dedicated group of cells.
>
>
>
> Asim Roy
>
> Professor, Arizona State University
>
> Lifeboat Foundation Bios: Professor Asim Roy
> <https://lifeboat.com/ex/bios.asim.roy>
>
>
>
>
>
> *From:* Connectionists <connectionists-bounces at mailman.srv.cs.cmu.edu> *On
> Behalf Of *Adam Krawitz
> *Sent:* Friday, November 5, 2021 10:01 AM
> *To:* connectionists at cs.cmu.edu
> *Subject:* Re: Connectionists: Scientific Integrity, the 2021 Turing
> Lecture, etc.
>
>
>
> Tsvi,
>
>
>
> I’m just a lurker on this list, with no skin in the game, but perhaps that
> gives me a more neutral perspective. In the spirit of progress:
>
>
>
>    1. If you have a neural network approach that you feel provides a new
>    and important perspective on cognitive processes, then write up a paper
>    making that argument clearly, and I think you will find that the community
>    is incredibly open to that. Yes, if they see holes in the approach they
>    will be pointed out, but that is all part of the scientific exchange.
>    Examples of this approach include: Elman (1990) Finding Structure in Time,
>    Kohonen (1990) The Self-Organizing Map, Tenenbaum et al. (2011) How to Grow
>    a Mind: Statistics, Structure, and Abstraction (not neural nets, but a
>    “new” approach to modelling cognition). I’m sure others can provide more
>    examples.
>    2. I’m much less familiar with how things work on the applied side,
>    but I have trouble believing that Google or anyone else will be dismissive
>    of a computational approach that actually works. Why would they? They just
>    want to solve problems efficiently. Demonstrate that your approach can
>    solve a problem more effectively (or at least as effectively) as the
>    existing approaches, and they will come running. Examples of this include:
>    Tesauro’s TD-Gammon, which was influential in demonstrating the power of
>    RL, and LeCun et al.’s convolutional NN for the MNIST digits.
>
>
>
> Clearly communicate the novel contribution of your approach and I think
> you will find a receptive audience.
>
>
>
> Thanks,
>
> Adam
>
>
>
>
>
> *From:* Connectionists <connectionists-bounces at mailman.srv.cs.cmu.edu> *On
> Behalf Of *Tsvi Achler
> *Sent:* November 4, 2021 9:46 AM
> *To:* gary at ucsd.edu
> *Cc:* connectionists at cs.cmu.edu
> *Subject:* Re: Connectionists: Scientific Integrity, the 2021 Turing
> Lecture, etc.
>
>
>
> Lastly Feedforward methods are predominant in a large part because they
> have financial backing from large companies with advertising and clout like
> Google and the self-driving craze that never fully materialized.
>
>
>
> Feedforward methods are not fully connectionist unless rehearsal for
> learning is implemented with neurons.  That means storing all patterns,
> mixing them randomly and then presenting to a network to learn.  As far as
> I know, no one is doing this in the community, so feedforward methods are
> only partially connectionist.  By allowing popularity to predominate and
> choking off funds and presentation of alternatives we are cheating
> ourselves from pursuing other more rigorous brain-like methods.
>
>
>
> Sincerely,
>
> -Tsvi
>
>
>
>
>
> On Tue, Nov 2, 2021 at 7:08 PM Tsvi Achler <achler at gmail.com> wrote:
>
> Gary- Thanks for the accessible online link to the book.
>
>
>
> I looked especially at the inhibitory feedback section of the book which
> describes an Air Conditioner AC type feedback.
>
> It then describes a general field-like inhibition based on all activations
> in the layer.  It also describes the role of inhibition in sparsity and
> feedforward inhibition,
>
>
>
> The feedback described in Regulatory Feedback is similar to the AC
> feedback but occurs for each neuron individually, vis-a-vis its inputs.
>
> Thus for context, regulatory feedback is not a field-like inhibition, it
> is very directed based on the neurons that are activated and their inputs.
> This sort of regulation is also the foundation of Homeostatic Plasticity
> findings (albeit with changes in Homeostatic regulation in experiments
> occurring in a slower time scale).  The regulatory feedback model describes
> the effect and role in recognition of those regulated connections in real
> time during recognition.
>
>
>
> I would be happy to discuss further and collaborate on writing about the
> differences between the approaches for the next book or review.
>
>
>
> And I want to point out to folks, that the system is based on politics and
> that is why certain work is not cited like it should, but even worse these
> politics are here in the group today and they continue to very
> strongly influence decisions in the connectionist community and holds us
> back.
>
>
>
> Sincerely,
>
> -Tsvi
>
>
>
> On Mon, Nov 1, 2021 at 10:59 AM gary at ucsd.edu <gary at eng.ucsd.edu> wrote:
>
> Tsvi - While I think Randy and Yuko's book
> <https://urldefense.com/v3/__https:/www.amazon.com/dp/0262650541/__;!!IKRxdwAv5BmarQ!P43fgF97h1EkMmUyqwIyGb3BiM6QvDDIayyZy_zt_11O7NVqPb6YiU7U4snDyk0$>is
> actually somewhat better than the online version (and buying choices on
> amazon start at $9.99), there *is* an online version.
> <https://urldefense.com/v3/__https:/compcogneuro.org/__;!!IKRxdwAv5BmarQ!P43fgF97h1EkMmUyqwIyGb3BiM6QvDDIayyZy_zt_11O7NVqPb6YiU7UH2qn4go$>
>
>
> Randy & Yuko's models take into account feedback and inhibition.
>
>
>
> On Mon, Nov 1, 2021 at 10:05 AM Tsvi Achler <achler at gmail.com> wrote:
>
> Daniel,
>
>
>
> Does your book include a discussion of Regulatory or Inhibitory Feedback
> published in several low impact journals between 2008 and 2014 (and in
> videos subsequently)?
>
> These are networks where the primary computation is inhibition back to the
> inputs that activated them and may be very counterintuitive given today's
> trends.  You can almost think of them as the opposite of Hopfield networks.
>
>
>
> I would love to check inside the book but I dont have an academic budget
> that allows me access to it and that is a huge part of the problem with how
> information is shared and funding is allocated. I could not get access to
> any of the text or citations especially Chapter 4: "Competition, Lateral
> Inhibition, and Short-Term Memory", to weigh in.
>
>
>
> I wish the best circulation for your book, but even if the Regulatory
> Feedback Model is in the book, that does not change the fundamental problem
> if the book is not readily available.
>
>
>
> The same goes with Steve Grossberg's book, I cannot easily look inside.
> With regards to Adaptive Resonance I dont subscribe to lateral inhibition
> as a predominant mechanism, but I do believe a function such as vigilance
> is very important during recognition and Adaptive Resonance is one of
> a very few models that have it.  The Regulatory Feedback model I have
> developed (and Michael Spratling studies a similar model as well) is built
> primarily using the vigilance type of connections and allows multiple
> neurons to be evaluated at the same time and continuously during
> recognition in order to determine which (single or multiple neurons
> together) match the inputs the best without lateral inhibition.
>
>
>
> Unfortunately within conferences and talks predominated by the Adaptive
> Resonance crowd I have experienced the familiar dismissiveness and did not
> have an opportunity to give a proper talk. This goes back to the larger
> issue of academic politics based on small self-selected committees, the
> same issues that exist with the feedforward crowd, and pretty much all of
> academia.
>
>
>
> Today's information age algorithms such as Google's can determine
> relevance of information and ways to display them, but hegemony of the
> journal systems and the small committee system of academia developed in the
> middle ages (and their mutual synergies) block the use of more modern
> methods in research.  Thus we are stuck with this problem, which especially
> affects those that are trying to introduce something new and
> counterintuitive, and hence the results described in the two National
> Bureau of Economic Research articles I cited in my previous message.
>
>
>
> Thomas, I am happy to have more discussions and/or start a different
> thread.
>
>
>
> Sincerely,
>
> Tsvi Achler MD/PhD
>
>
>
>
>
>
>
> On Sun, Oct 31, 2021 at 12:49 PM Levine, Daniel S <levine at uta.edu> wrote:
>
> Tsvi,
>
>
>
> While deep learning and feedforward networks have an outsize popularity,
> there are plenty of published sources that cover a much wider variety of
> networks, many of them more biologically based than deep learning.  A
> treatment of a range of neural network approaches, going from simpler to
> more complex cognitive functions, is found in my textbook *Introduction
> to Neural and Cognitive Modeling* (3rd edition, Routledge, 2019).  Also
> Steve Grossberg's book *Conscious Mind, Resonant Brain* (Oxford, 2021)
> emphasizes a variety of architectures with a strong biological basis.
>
>
>
>
>
> Best,
>
>
>
>
>
> Dan Levine
> ------------------------------
>
> *From:* Connectionists <connectionists-bounces at mailman.srv.cs.cmu.edu> on
> behalf of Tsvi Achler <achler at gmail.com>
> *Sent:* Saturday, October 30, 2021 3:13 AM
> *To:* Schmidhuber Juergen <juergen at idsia.ch>
> *Cc:* connectionists at cs.cmu.edu <connectionists at cs.cmu.edu>
> *Subject:* Re: Connectionists: Scientific Integrity, the 2021 Turing
> Lecture, etc.
>
>
>
> Since the title of the thread is Scientific Integrity, I want to point out
> some issues about trends in academia and then especially focusing on the
> connectionist community.
>
>
>
> In general analyzing impact factors etc the most important progress gets
> silenced until the mainstream picks it up Impact Factiors in novel
> research www.nber.org/.../working_papers/w22180/w22180.pdf
> <https://urldefense.com/v3/__https:/nam12.safelinks.protection.outlook.com/?url=https*3A*2F*2Fwww.nber.org*2Fsystem*2Ffiles*2Fworking_papers*2Fw22180*2Fw22180.pdf*3Ffbclid*3DIwAR1zHhU4wmkrHASTaE-6zwIs6gI9-FxZcCED3BETxUJlMsbN_2hNbmJAmOA&data=04*7C01*7Clevine*40uta.edu*7Cb1a267e3b6a64ada666208d99ca37f6d*7C5cdc5b43d7be4caa8173729e3b0a62d9*7C1*7C0*7C637713048300122043*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000&sdata=9o*2FzcYY8gZVZiAwyEL5SVI9TEzBWfKf7nfhdWWg8LHU*3D&reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUl!!IKRxdwAv5BmarQ!P43fgF97h1EkMmUyqwIyGb3BiM6QvDDIayyZy_zt_11O7NVqPb6YiU7UD9hRGNg$>  and
> often this may take a generation
> https://www.nber.org/.../does-science-advance-one-funeral...
> <https://urldefense.com/v3/__https:/nam12.safelinks.protection.outlook.com/?url=https*3A*2F*2Fwww.nber.org*2Fdigest*2Fmar16*2Fdoes-science-advance-one-funeral-time*3Ffbclid*3DIwAR1Lodsf1bzje-yQU9DvoZE2__S6R7UPEgY1_LxZCSLdoAYnj-uco0JuyVk&data=04*7C01*7Clevine*40uta.edu*7Cb1a267e3b6a64ada666208d99ca37f6d*7C5cdc5b43d7be4caa8173729e3b0a62d9*7C1*7C0*7C637713048300132034*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000&sdata=DgxnJTT7MsN5KCzZlA7VAHKrHXVsRsYhopJv0FCwbtw*3D&reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUlJSUl!!IKRxdwAv5BmarQ!P43fgF97h1EkMmUyqwIyGb3BiM6QvDDIayyZy_zt_11O7NVqPb6YiU7UapVS1t0$>
>   .
>
>
>
> The connectionist field is stuck on feedforward networks and variants such
> as with inhibition of competitors (e.g. lateral inhibition), or other
> variants that are sometimes labeled as recurrent networks for learning time
> where the feedforward networks can be rewound in time.
>
>
>
> This stasis is specifically occuring with the popularity of deep
> learning.  This is often portrayed as neurally plausible connectionism but
> requires an implausible amount of rehearsal and is not connectionist if
> this rehearsal is not implemented with neurons (see video link for further
> clarification).
>
>
>
> Models which have true feedback (e.g. back to their own inputs) cannot
> learn by backpropagation but there is plenty of evidence these types of
> connections exist in the brain and are used during recognition. Thus they
> get ignored: no talks in universities, no featuring in "premier" journals
> and no funding.
>
>
>
> But they are important and may negate the need for rehearsal as needed in
> feedforward methods.  Thus may be essential for moving connectionism
> forward.
>
>
>
> If the community is truly dedicated to brain motivated algorithms, I
> recommend giving more time to networks other than feedforward networks.
>
>
>
> Video:
> https://www.youtube.com/watch?v=m2qee6j5eew&list=PL4nMP8F3B7bg3cNWWwLG8BX-wER2PeB-3&index=2
> <https://urldefense.com/v3/__https:/nam12.safelinks.protection.outlook.com/?url=https*3A*2F*2Fwww.youtube.com*2Fwatch*3Fv*3Dm2qee6j5eew*26list*3DPL4nMP8F3B7bg3cNWWwLG8BX-wER2PeB-3*26index*3D2&data=04*7C01*7Clevine*40uta.edu*7Cb1a267e3b6a64ada666208d99ca37f6d*7C5cdc5b43d7be4caa8173729e3b0a62d9*7C1*7C0*7C637713048300132034*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000&sdata=EaEp5zLZ7HkDhsBHmP3x3ObPl8j14B8*2BFcOkkNEWZ9w*3D&reserved=0__;JSUlJSUlJSUlJSUlJSUlJSUlJSUlJSUl!!IKRxdwAv5BmarQ!P43fgF97h1EkMmUyqwIyGb3BiM6QvDDIayyZy_zt_11O7NVqPb6YiU7UzMnNL04$>
>
>
>
> Sincerely,
>
> Tsvi Achler
>
>
>
>
>
>
>
> On Wed, Oct 27, 2021 at 2:24 AM Schmidhuber Juergen <juergen at idsia.ch>
> wrote:
>
> Hi, fellow artificial neural network enthusiasts!
>
> The connectionists mailing list is perhaps the oldest mailing list on
> ANNs, and many neural net pioneers are still subscribed to it. I am hoping
> that some of them - as well as their contemporaries - might be able to
> provide additional valuable insights into the history of the field.
>
> Following the great success of massive open online peer review (MOOR) for
> my 2015 survey of deep learning (now the most cited article ever published
> in the journal Neural Networks), I've decided to put forward another piece
> for MOOR. I want to thank the many experts who have already provided me
> with comments on it. Please send additional relevant references and
> suggestions for improvements for the following draft directly to me at
> juergen at idsia.ch:
>
>
> https://people.idsia.ch/~juergen/scientific-integrity-turing-award-deep-learning.html
> <https://urldefense.com/v3/__https:/nam12.safelinks.protection.outlook.com/?url=https*3A*2F*2Fpeople.idsia.ch*2F*juergen*2Fscientific-integrity-turing-award-deep-learning.html&data=04*7C01*7Clevine*40uta.edu*7Cb1a267e3b6a64ada666208d99ca37f6d*7C5cdc5b43d7be4caa8173729e3b0a62d9*7C1*7C0*7C637713048300142030*7CUnknown*7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0*3D*7C1000&sdata=mW3lH7SqKg4EuJfDwKcC2VhwEloC3ndh6kI5gfQ2Ofw*3D&reserved=0__;JSUlJX4lJSUlJSUlJSUlJSUlJQ!!IKRxdwAv5BmarQ!P43fgF97h1EkMmUyqwIyGb3BiM6QvDDIayyZy_zt_11O7NVqPb6YiU7UNznV_Qo$>
>
> The above is a point-for-point critique of factual errors in ACM's
> justification of the ACM A. M. Turing Award for deep learning and a
> critique of the Turing Lecture published by ACM in July 2021. This work can
> also be seen as a short history of deep learning, at least as far as ACM's
> errors and the Turing Lecture are concerned.
>
> I know that some view this as a controversial topic. However, it is the
> very nature of science to resolve controversies through facts. Credit
> assignment is as core to scientific history as it is to machine learning.
> My aim is to ensure that the true history of our field is preserved for
> posterity.
>
> Thank you all in advance for your help!
>
> Jürgen Schmidhuber
>
>
>
>
>
>
>
>
>
> --
>
> Gary Cottrell 858-534-6640 FAX: 858-534-7029
>
> Computer Science and Engineering 0404
> IF USING FEDEX INCLUDE THE FOLLOWING LINE:
> CSE Building, Room 4130
> University of California San Diego                                      -
> 9500 Gilman Drive # 0404
> La Jolla, Ca. 92093-0404
>
> Email: gary at ucsd.edu
> Home page: http://www-cse.ucsd.edu/~gary/
> <https://urldefense.com/v3/__http:/www-cse.ucsd.edu/*gary/__;fg!!IKRxdwAv5BmarQ!P43fgF97h1EkMmUyqwIyGb3BiM6QvDDIayyZy_zt_11O7NVqPb6YiU7U-G68mLE$>
>
> Schedule: http://tinyurl.com/b7gxpwo
> <https://urldefense.com/v3/__http:/tinyurl.com/b7gxpwo__;!!IKRxdwAv5BmarQ!P43fgF97h1EkMmUyqwIyGb3BiM6QvDDIayyZy_zt_11O7NVqPb6YiU7UcMz40H8$>
>
>
>
> *Listen carefully,*
> *Neither the Vedas*
> *Nor the Qur'an*
> *Will teach you this:*
> *Put the bit in its mouth,*
> *The saddle on its back,*
> *Your foot in the stirrup,*
> *And ride your wild runaway mind*
> *All the way to heaven.*
>
> *-- Kabir*
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/connectionists/attachments/20211106/19f3ffb4/attachment.html>