Connectionists: Stephen Hanson in conversation with Geoff Hinton

[Asim Roy]

There was another recent attempt to take down the grandmother cell idea:

Frontiers | The Value of Failure in Science: The Story of Grandmother Cells in Neuroscience | Neuroscience (frontiersin.org)<https://www.frontiersin.org/articles/10.3389/fnins.2019.01121/full>

And here’s my commentary defending grandmother cells:

Frontiers | Commentary: The Value of Failure in Science: The Story of Grandmother Cells in Neuroscience | Neuroscience (frontiersin.org)<https://www.frontiersin.org/articles/10.3389/fnins.2020.00059/full>

By the way, we have had vigorous private arguments over the years about grandmother cells and many were involved – from Jay McClelland and Christof Koch to Walter Freeman and Bernard Baars. As far as I can tell, the brain uses abstractions at the single cell level that can be argued to be symbols. Short arguments are in the commentary, based on observations by neurophysiologists themselves.

Asim Roy
Professor, Information Systems
Arizona State University
Lifeboat Foundation Bios: Professor Asim Roy<https://urldefense.proofpoint.com/v2/url?u=https-3A__lifeboat.com_ex_bios.asim.roy&d=DwMFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=wQR1NePCSj6dOGDD0r6B5Kn1fcNaTMg7tARe7TdEDqQ&m=waSKY67JF57IZXg30ysFB_R7OG9zoQwFwxyps6FbTa1Zh5mttxRot_t4N7mn68Pj&s=oDRJmXX22O8NcfqyLjyu4Ajmt8pcHWquTxYjeWahfuw&e=>
Asim Roy | iSearch (asu.edu)<https://urldefense.proofpoint.com/v2/url?u=https-3A__isearch.asu.edu_profile_9973&d=DwMFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=wQR1NePCSj6dOGDD0r6B5Kn1fcNaTMg7tARe7TdEDqQ&m=waSKY67JF57IZXg30ysFB_R7OG9zoQwFwxyps6FbTa1Zh5mttxRot_t4N7mn68Pj&s=jCesWT7oGgX76_y7PFh4cCIQ-Ife-esGblJyrBiDlro&e=>


From: Connectionists <connectionists-bounces at mailman.srv.cs.cmu.edu> On Behalf Of Gary Marcus
Sent: Friday, February 4, 2022 12:53 PM
To: Stephen José Hanson <jose at rubic.rutgers.edu>
Cc: connectionists at mailman.srv.cs.cmu.edu; Stevan Harnad <harnad at ecs.soton.ac.uk>; Francesca Rossi2 <Francesca.Rossi2 at ibm.com>; Artur Garcez <arturdavilagarcez at gmail.com>; Anima Anandkumar <anima at caltech.edu>; Luis Lamb <lamb at inf.ufrgs.br>; Gadi Singer <gadi.singer at intel.com>; Josh Tenenbaum <josh.tenenbaum at gmail.com>; AIhub <aihuborg at gmail.com>
Subject: Re: Connectionists: Stephen Hanson in conversation with Geoff Hinton

Steve,

The phrase I always liked was “poverty of the imagination arguments”; I share your disdain for them. But that’s why I think you should be careful of any retreat into biological plausibility. As even Jay McClelland has acknowledged, we do know that some humans some of the time manipulate symbols. So wetware-based symbols are not literally biologically impossible; the real question for cognitive neuroscience is about the scope and development of symbols.

For engineering, the real question is, are they useful. Certainly for software engineering in general, they are indispensable.

Beyond this, none of the available AI approaches map particularly neatly onto what we know about the brain, and none of what we know about the brain is understood well enough to solve AI.  All the examples you point to, for instance, are actually controversial, not decisive. As you probably know, for example, Nancy Kanwisher has a different take on domain-specificity than you do (https://web.mit.edu/bcs/nklab/<https://urldefense.com/v3/__https:/web.mit.edu/bcs/nklab/__;!!IKRxdwAv5BmarQ!JCW7V01kHqhTTrA2Up45kCtD9LH_yvmF66ZxASm8db91n2BEu2yuciVpdlGuWSU$>), with evidence of specialization early in life, and Jeff Bowers has argued that the grandmother cell hypothesis has been dismissed prematurely  (https://jeffbowers.blogs.bristol.ac.uk/blog/grandmother-cells/<https://urldefense.com/v3/__https:/jeffbowers.blogs.bristol.ac.uk/blog/grandmother-cells/__;!!IKRxdwAv5BmarQ!JCW7V01kHqhTTrA2Up45kCtD9LH_yvmF66ZxASm8db91n2BEu2yuciVpI7KDNxg$>); there’s also a long literature on the possible neural realization of rules, both in humans and other animals.

I don’t know what the right answers are there, but nor do I think that neurosymbolic systems are beholden to them anymore than CNNs are bound to whether or not the brain performs back-propagation.

Finally, as a reminder,  “Distributed” per se in not the right question; in some technical sense ASCII encodings are distributed, and about as symbolic as you can get. The proper question is really what you do with your encodings; the neurosymbolic approach is trying to broaden the available range of options.

Gary

On Feb 4, 2022, at 07:04, Stephen José Hanson <jose at rubic.rutgers.edu<mailto:jose at rubic.rutgers.edu>> wrote:


Well I don't like counterfactual arguments or ones that start with "It can't be done with neural networks.."--as this amounts to the old Rumelhart saw, of "proof by lack of imagination".

I think my position and others (I can't speak for Geoff and won't) is more of a "purist" view that brains have computationally complete representational power to do what ever is required of human level mental processing.  AI symbol systems are remote descriptions of this level of processing.     Looking at 1000s of brain scans, one begins to see a pattern of interacting large and smaller scale networks, probably related to Resting state and the Default Mode networks in some important competitive way.   But what one doesn't find is modular structure (e.g. face area.. nope)  or evidence of "symbols"  being processed.    Research on Numbers is interesting in this regard, as number representation should provide some evidence of  discrete symbol processing as would  letters.   But again the processing states from brain imaging  more generally appear to be distributed representations of some sort.

One other direction has to do with prior rules that could be neurally coded and therefore provide an immediate bias in learning and thus dramatically reduce the number of examples required for  asymptotic learning.     Some of this has been done with pre-training-- on let's say 1000s of videos that are relatively generic, prior to learning on a small set of videos related to a specific topic-- say two individuals playing a monopoly game.  In that case, no game-like videos were sampled in the pre-training, and the LSTM was trained to detect change point on 2 minutes of video, achieving a 97% match with human parsers.    In these senses I have no problem with this type of hybrid training.

Steve
On 2/4/22 9:07 AM, Gary Marcus wrote:
The whole point of the neurosymbolic approach is to develop systems that can accommodate both vectors and symbols, since neither on their own seems adequate.

If there are arguments against trying to do that, we would be interested.


On Feb 4, 2022, at 4:17 AM, Stephen José Hanson <jose at rubic.rutgers.edu><mailto:jose at rubic.rutgers.edu> wrote:


Geoff's position is pretty clear.   He said in the conversation we had and in this thread,  "vectors of soft features",

Some of my claim is in several of the conversations with Mike Jordan and Rich Sutton, but briefly,  there are a number of
very large costly efforts from the 1970s and 1980s, to create, deploy and curate symbol AI systems that were massive failures.  Not counterfactuals,  but factuals that failed.   The MCC comes to mind with Adm Bobby Inmann's  national US mandate to counter the Japanese so called"Fifth-generation AI systems"  as a massive failure of symbolic AI.

--------------------

In 1982, Japan launched its Fifth Generation Computer Systems project (FGCS), designed to develop intelligent software that would run on novel computer hardware. As the first national, large-scale artificial intelligence (AI) research and development (R&D) project to be free from military influence and corporate profit motives, the FGCS was open, international, and oriented around public goods.
On 2/3/22 6:34 PM, Francesca Rossi2 wrote:

Hi all.



Thanks Gary for adding me to this thread.



I also would be interested in knowing why Steve thinks that NS AI did not work in the past, and why this is an indication that it cannot work now or in the future.



Thanks,

Francesca.

------------------



Francesca Rossi

IBM Fellow and AI Ethics Global Leader

T.J. Watson Research Center, Yorktown Heights, USA

+1-617-3869639



________________________________________

From: Artur Garcez <arturdavilagarcez at gmail.com><mailto:arturdavilagarcez at gmail.com>

Sent: Thursday, February 3, 2022 6:00 PM

To: Gary Marcus

Cc: Stephen José Hanson; Geoffrey Hinton; AIhub; connectionists at mailman.srv.cs.cmu.edu<mailto:connectionists at mailman.srv.cs.cmu.edu>; Luis Lamb; Josh Tenenbaum; Anima Anandkumar; Francesca Rossi2; Swarat Chaudhuri; Gadi Singer

Subject: [EXTERNAL] Re: Connectionists: Stephen Hanson in conversation with Geoff Hinton



It would be great to hear Geoff's account with historical reference to his 1990 edited special volume of the AI journal on connectionist symbol processing. Judging from recent reviewing for NeurIPS, ICLR, ICML but also KR, AAAI, IJCAI (traditionally ZjQcmQRYFpfptBannerStart

This Message Is From an External Sender

This message came from outside your organization.

ZjQcmQRYFpfptBannerEnd



It would be great to hear Geoff's account with historical reference to his 1990 edited special volume of the AI journal on connectionist symbol processing.



Judging from recent reviewing for NeurIPS, ICLR, ICML but also KR, AAAI, IJCAI (traditionally symbolic), there is a clear resurgence of neuro-symbolic approaches.



Best wishes,

Artur





On Thu, Feb 3, 2022 at 5:00 PM Gary Marcus <gary.marcus at nyu.edu<mailto:gary.marcus at nyu.edu><mailto:gary.marcus at nyu.edu><mailto:gary.marcus at nyu.edu>> wrote:

Steve,



I’d love to hear you elaborate on this part,



 Many more shoes will drop in the next few years.  I for one don't believe one of those shoes will be Hybrid approaches to AI,  I've seen that movie before and it didn't end well.





I’d love your take on why you think the impetus towards hybrid models ended badly before, and why you think that the mistakes of the past can’t be corrected. Also it’ would be really instructive to compare with deep learning, which lost steam for quite some time, but reemerged much stronger than ever before. Might not the same happen with hybrid models?



I am cc’ing some folks (possibly not on this list) who have recently been sympathetic to hybrid models, in hopes of a rich discussion.  (And, Geoff, still cc’d, I’d genuinely welcome your thoughts if you want to add them, despite our recent friction.)



Cheers,

Gary





On Feb 3, 2022, at 5:10 AM, Stephen José Hanson <jose at rubic.rutgers.edu<mailto:jose at rubic.rutgers.edu><mailto:jose at rubic.rutgers.edu><mailto:jose at rubic.rutgers.edu>> wrote:





I would encourage you to read the whole transcript, as you will see the discussion does intersect with a number of issues you raised in an earlier post on what is learned/represented in DLs.



Its important for those paying attention to this thread, to realize these are still very early times.    Many more shoes will drop in the next few years.  I for one don't believe one of those shoes will be Hybrid approaches to AI,  I've seen that movie before and it didn't end well.



Best and hope you are doing well.



Steve


--
<signature.png>
--
[cid:image001.png at 01D81AD9.13EC7A80]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/connectionists/attachments/20220206/6fee2cf5/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 19957 bytes
Desc: image001.png
URL: <http://mailman.srv.cs.cmu.edu/pipermail/connectionists/attachments/20220206/6fee2cf5/attachment.png>