Connectionists: Early history of symbolic and neural network approaches to AI

[Gary Marcus]

The notion of constraints is very interesting in connection with the discussion of understanding. Sora for example appears not to be able to oblige objects to respect gravity, solidity, or object permanence. (Basic things human infants understand). There is no way to restrict the things that GPT says to those that have been verified, etc. Dave Touretzky’s example show a failure to obey an even simpler constraint.




> On Feb 19, 2024, at 6:04 AM, Johan Suykens <johan.suykens at esat.kuleuven.be> wrote:
> 
> 
> Thanks Gary for your clarifications.
> 
> Related to your mentioning of
> 
>  "AI/models of human cognition require algebraic operations over variables in order to be robust, in addition to other mechanisms"
> 
> I would like to inform that in our recent NeurIPS 2023 paper https://urldefense.proofpoint.com/v2/url?u=https-3A__arxiv.org_abs_2305.19798&d=DwIDaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=wQR1NePCSj6dOGDD0r6B5Kn1fcNaTMg7tARe7TdEDqQ&m=ycCLlME4xVoD3WTyo8LuvWjVNcRbCdUopSwCRNCTp8GoUfWcnbbNa5IyXgNnHSKe&s=CwBhpfeHo4p3bhV5w6tCk6vGcRBC91P4K2J_70AzOLI&e=  (Primal-Attention: Self-attention through Asymmetric Kernel SVD in Primal Representation) we have described self-attention of transformers through a modified form of kernel singular value decomposition, related to asymmetric kernels (where the kernel function is the dot product between a query and key feature map) with primal and dual representations. In this way low rank representations are obtained together with efficient training of transformers in primal form. Singular value decomposition is a well-known method and major tool in linear algebra. This is a kernel version of it.
> 
> I hope this may possibly bridge the gap between polarized viewpoints in the debate.
> 
> Within the same framework it is also possible to consider e.g. pairwise constraints. An example is kernel spectral clustering where pairwise constraints yielding models that better align with human understanding (as e.g. in the paper "A regularized formulation for spectral clustering with pairwise constraints" https://urldefense.proofpoint.com/v2/url?u=https-3A__ieeexplore.ieee.org_document_5178772&d=DwIDaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=wQR1NePCSj6dOGDD0r6B5Kn1fcNaTMg7tARe7TdEDqQ&m=ycCLlME4xVoD3WTyo8LuvWjVNcRbCdUopSwCRNCTp8GoUfWcnbbNa5IyXgNnHSKe&s=W3_Nmc1GjxWl6JIZsgxc2xvzlHi4VpG-flVwvI3MAwY&e=  ). In Fig.7 an illustration is given of an image segmentation with "horses + the sky + the ground". However, the sun is shining and therefore there is also the "shadow of the horses" on the ground. This leads to an ambiguity: should the shadow of the horse be part of the horse or be part of the ground? By specifying a few additional must-link and cannot-link constraints between pixels, one can quickly improve the model and make it compatible with our "human understanding" of this image.
> 
> It seems that one of the main issues in the debate is then whether we need such constraints yes or no. If there are enough additional images showing e.g. also the sun on the image or other example images that could clarify the context, possibly one doesn't need to have the additional constraints. However, the additional constraints can definitely help in quickly interfacing the model with the real world and making it compatible with "human understanding".
> 
> Best regards,
> Johan Suykens
> 
> 
> 
> 
>> On 2024-02-18 18:30, Gary Marcus wrote:
>> - certainly in broad agreement; never argued that symbols alone were sufficient.
>> - have always since my first paper in 1992 advocated for hybrid models, with both associative and symbolic-components
>> - as argued in my 2001 book, i think the real question is not whether we need symbols per se (which localist output units represent) but rather AI/models of human cognition require algebraic operations over variables in order to be robust, in addition to other mechanisms.
>>>> On Feb 18, 2024, at 09:25, poole <poole at cs.ubc.ca> wrote:
>>> Thanks Gary.
>>> These are both worthwhile reading!
>>> I don’t think symbolic = logic.  McCulloch and Pitts were interested in representing logical operations.
>>> “Symbolic" follows the tradition of Hobbes (1588–1679) who claimed that thinking was symbolic reasoning, like talking out loud or working out an answer with pen and paper [see Haugeland, J. Artificial Intelligence: The Very Idea. MIT Press  1985].  Newell and Simon [1976] defined a symbol is a meaningful pattern that can be manipulated. A symbol system creates, copies, modifies, and destroys symbols.
>>> Graphical models and believe networks typically have symbolic random variables.
>>> It is very common for modern neural networks to have symbolic inputs or outputs, e.g., words, knowledge graphs, molecular structure, game moves,…
>>> I don’t think Gary would disagree that there needs to be some non-symbols (e.g, hidden units in neural networks).
>>> Arguments for symbols — the most compelling one for me is that organizations (which are much more intelligent than individuals) reason in terms of symbols (words, diagrams, spreadsheets) — are not diminished by the need for non-symbols.
>>> David
>>> (references from artint.info where many of these issues are discussed).
>>>> On Feb 17, 2024, at 5:42 PM, Gary Marcus <gary.marcus at nyu.edu> wrote:
>>>> [CAUTION: Non-UBC Email]adding some pointers to David’s remarks: McCulloch and Pitts in 1943 were very much trying to bridge the symbolic and neural world. It’s clear even in the abstract (article below). Tensions between symbolic and neural approaches were in full force by Minsky and Papert 1969, and resurfaced in the 1980s.
>>>> I don’t have a clear sense of where things were in Turing’s time per se, but both approaches were countenanced in the 1955 proposal for the Dartmouth conference link below; Rosenblatt had gathered steam by 1958 as noted.
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__web.archive.org_web_20070826230310_http-3A__www-2Dformal.stanford.edu_jmc_history_dartmouth_dartmouth.html&d=DwIFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=wQR1NePCSj6dOGDD0r6B5Kn1fcNaTMg7tARe7TdEDqQ&m=i1FPhi6c93rU7h_AikZJEx36pDsvgNzWMEKlOMHY0uqMdsAuRf5w_wtT_l9EhRaQ&s=qB3xwLa8UE9Dg_o7t21jk3miiMP1icWlWBL_XGQ49-Q&e=
>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__home.csulb.edu_-7Ecwallis_382_readings_482_mccolloch.logical.calculus.ideas.1943.pdf&d=DwIFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=wQR1NePCSj6dOGDD0r6B5Kn1fcNaTMg7tARe7TdEDqQ&m=i1FPhi6c93rU7h_AikZJEx36pDsvgNzWMEKlOMHY0uqMdsAuRf5w_wtT_l9EhRaQ&s=6jCRjchELvEA_Dzv8hAwzcYSMdGNQiQNU_z3DO6ddp4&e=
>>>>>> On Feb 17, 2024, at 4:40 PM, poole <poole at cs.ubc.ca> wrote:
>>>>> 
>>>>>> On Feb 17, 2024, at 1:08 PM, David H Kirshner <dkirsh at lsu.edu> wrote:
>>>>>> [CAUTION: Non-UBC Email]
>>>>>> You’re right, David.
>>>>>> I should have said “Back in Alan Turing’s time when the possibility of AI meant the possibility of symbolic AI, ….”
>>>>> In Turing's time, from what I can see (I wan’t alive then ;^) neural networks were more trendy than symbolic approaches. Turing’s paper was 1950. McCulloch and Pitts seminal work was 1943. Minsky’s thesis on neural networks was written in 1952. (Schmidhuber has great resources on the history of NNs and AI on his website).
>>>>> There was lots of neural network hype in the 1950’s:
>>>>> "The Navy revealed the embryo of an electronic computer today that it expects will be able to walk, talk, see, write, reproduce itself and be conscious of its existence. …The service said it would …build the first of its Perceptron thinking machines that will be able to read and write. It is expected to be finished in about a year at a cost of $100,000."
>>>>> – New York Times [1958]
>>>>> It was later in the 1950’s that they came to realize that AI needed representations, lead by Minsky and McCarthy, whick lead to the rise of symbolic approaches.. (It is interesting that a major NN conference ICLR is about representations).
>>>>> I am sure there are people who know the history better than me, who might like to provide more persoective.
>>>>> David
>>>>>> ——
>>>>>> David Poole,
>>>>>> Department of Computer Science,
>>>>>> University of British Columbia,
>>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__cs.ubc.ca_-7Epoole&d=DwIFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=wQR1NePCSj6dOGDD0r6B5Kn1fcNaTMg7tARe7TdEDqQ&m=GaLFSysAmNesHoiuLMLNQzFZKcQfTn2lkEhxM8Xrc205FjMs-5qx1lSBZ4u9kagl&s=XFNzK3B_BUrmI0gbNaacNriRQ53tTCzjlNaL2JBAmRg&e=
>>>>>> poole at cs.ubc.ca