Connectionists: Early history of symbolic and neural network approaches to AI

[Johan Suykens]

Thanks Gary for your clarifications.

Related to your mentioning of

   "AI/models of human cognition require algebraic operations over 
variables in order to be robust, in addition to other mechanisms"

I would like to inform that in our recent NeurIPS 2023 paper 
https://arxiv.org/abs/2305.19798 (Primal-Attention: Self-attention 
through Asymmetric Kernel SVD in Primal Representation) we have 
described self-attention of transformers through a modified form of 
kernel singular value decomposition, related to asymmetric kernels 
(where the kernel function is the dot product between a query and key 
feature map) with primal and dual representations. In this way low rank 
representations are obtained together with efficient training of 
transformers in primal form. Singular value decomposition is a 
well-known method and major tool in linear algebra. This is a kernel 
version of it.

I hope this may possibly bridge the gap between polarized viewpoints in 
the debate.

Within the same framework it is also possible to consider e.g. pairwise 
constraints. An example is kernel spectral clustering where pairwise 
constraints yielding models that better align with human understanding 
(as e.g. in the paper "A regularized formulation for spectral clustering 
with pairwise constraints" https://ieeexplore.ieee.org/document/5178772 
). In Fig.7 an illustration is given of an image segmentation with 
"horses + the sky + the ground". However, the sun is shining and 
therefore there is also the "shadow of the horses" on the ground. This 
leads to an ambiguity: should the shadow of the horse be part of the 
horse or be part of the ground? By specifying a few additional must-link 
and cannot-link constraints between pixels, one can quickly improve the 
model and make it compatible with our "human understanding" of this 
image.

It seems that one of the main issues in the debate is then whether we 
need such constraints yes or no. If there are enough additional images 
showing e.g. also the sun on the image or other example images that 
could clarify the context, possibly one doesn't need to have the 
additional constraints. However, the additional constraints can 
definitely help in quickly interfacing the model with the real world and 
making it compatible with "human understanding".

Best regards,
Johan Suykens




On 2024-02-18 18:30, Gary Marcus wrote:
> - certainly in broad agreement; never argued that symbols alone were 
> sufficient.
> - have always since my first paper in 1992 advocated for hybrid models, 
> with both associative and symbolic-components
> - as argued in my 2001 book, i think the real question is not whether 
> we need symbols per se (which localist output units represent) but 
> rather AI/models of human cognition require algebraic operations over 
> variables in order to be robust, in addition to other mechanisms.
> 
>> On Feb 18, 2024, at 09:25, poole <poole at cs.ubc.ca> wrote:
>> 
>> Thanks Gary.
>> These are both worthwhile reading!
>> 
>> I don’t think symbolic = logic.  McCulloch and Pitts were interested 
>> in representing logical operations.
>> 
>> “Symbolic" follows the tradition of Hobbes (1588–1679) who claimed 
>> that thinking was symbolic reasoning, like talking out loud or working 
>> out an answer with pen and paper [see Haugeland, J. Artificial 
>> Intelligence: The Very Idea. MIT Press  1985].  Newell and Simon 
>> [1976] defined a symbol is a meaningful pattern that can be 
>> manipulated. A symbol system creates, copies, modifies, and destroys 
>> symbols.
>> 
>> Graphical models and believe networks typically have symbolic random 
>> variables.
>> 
>> It is very common for modern neural networks to have symbolic inputs 
>> or outputs, e.g., words, knowledge graphs, molecular structure, game 
>> moves,…
>> 
>> I don’t think Gary would disagree that there needs to be some 
>> non-symbols (e.g, hidden units in neural networks).
>> 
>> Arguments for symbols — the most compelling one for me is that 
>> organizations (which are much more intelligent than individuals) 
>> reason in terms of symbols (words, diagrams, spreadsheets) — are not 
>> diminished by the need for non-symbols.
>> 
>> David
>> (references from artint.info where many of these issues are 
>> discussed).
>> 
>>> On Feb 17, 2024, at 5:42 PM, Gary Marcus <gary.marcus at nyu.edu> wrote:
>>> 
>>> [CAUTION: Non-UBC Email]adding some pointers to David’s remarks: 
>>> McCulloch and Pitts in 1943 were very much trying to bridge the 
>>> symbolic and neural world. It’s clear even in the abstract (article 
>>> below). Tensions between symbolic and neural approaches were in full 
>>> force by Minsky and Papert 1969, and resurfaced in the 1980s.
>>> 
>>> I don’t have a clear sense of where things were in Turing’s time per 
>>> se, but both approaches were countenanced in the 1955 proposal for 
>>> the Dartmouth conference link below; Rosenblatt had gathered steam by 
>>> 1958 as noted.
>>> 
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__web.archive.org_web_20070826230310_http-3A__www-2Dformal.stanford.edu_jmc_history_dartmouth_dartmouth.html&d=DwIFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=wQR1NePCSj6dOGDD0r6B5Kn1fcNaTMg7tARe7TdEDqQ&m=i1FPhi6c93rU7h_AikZJEx36pDsvgNzWMEKlOMHY0uqMdsAuRf5w_wtT_l9EhRaQ&s=qB3xwLa8UE9Dg_o7t21jk3miiMP1icWlWBL_XGQ49-Q&e=
>>> 
>>> 
>>> 
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__home.csulb.edu_-7Ecwallis_382_readings_482_mccolloch.logical.calculus.ideas.1943.pdf&d=DwIFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=wQR1NePCSj6dOGDD0r6B5Kn1fcNaTMg7tARe7TdEDqQ&m=i1FPhi6c93rU7h_AikZJEx36pDsvgNzWMEKlOMHY0uqMdsAuRf5w_wtT_l9EhRaQ&s=6jCRjchELvEA_Dzv8hAwzcYSMdGNQiQNU_z3DO6ddp4&e=
>>> 
>>> 
>>> 
>>>>> On Feb 17, 2024, at 4:40 PM, poole <poole at cs.ubc.ca> wrote:
>>>> 
>>>> 
>>>>> On Feb 17, 2024, at 1:08 PM, David H Kirshner <dkirsh at lsu.edu> 
>>>>> wrote:
>>>>> 
>>>>> [CAUTION: Non-UBC Email]
>>>>> You’re right, David.
>>>>> I should have said “Back in Alan Turing’s time when the possibility 
>>>>> of AI meant the possibility of symbolic AI, ….”
>>>> 
>>>> In Turing's time, from what I can see (I wan’t alive then ;^) neural 
>>>> networks were more trendy than symbolic approaches. Turing’s paper 
>>>> was 1950. McCulloch and Pitts seminal work was 1943. Minsky’s thesis 
>>>> on neural networks was written in 1952. (Schmidhuber has great 
>>>> resources on the history of NNs and AI on his website).
>>>> 
>>>> There was lots of neural network hype in the 1950’s:
>>>> 
>>>> "The Navy revealed the embryo of an electronic computer today that 
>>>> it expects will be able to walk, talk, see, write, reproduce itself 
>>>> and be conscious of its existence. …The service said it would …build 
>>>> the first of its Perceptron thinking machines that will be able to 
>>>> read and write. It is expected to be finished in about a year at a 
>>>> cost of $100,000."
>>>> – New York Times [1958]
>>>> 
>>>> It was later in the 1950’s that they came to realize that AI needed 
>>>> representations, lead by Minsky and McCarthy, whick lead to the rise 
>>>> of symbolic approaches.. (It is interesting that a major NN 
>>>> conference ICLR is about representations).
>>>> 
>>>> I am sure there are people who know the history better than me, who 
>>>> might like to provide more persoective.
>>>> 
>>>> David
>>>> 
>>>> 
>>>>> ——
>>>>> David Poole,
>>>>> Department of Computer Science,
>>>>> University of British Columbia,
>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__cs.ubc.ca_-7Epoole&d=DwIFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=wQR1NePCSj6dOGDD0r6B5Kn1fcNaTMg7tARe7TdEDqQ&m=GaLFSysAmNesHoiuLMLNQzFZKcQfTn2lkEhxM8Xrc205FjMs-5qx1lSBZ4u9kagl&s=XFNzK3B_BUrmI0gbNaacNriRQ53tTCzjlNaL2JBAmRg&e=
>>>>> poole at cs.ubc.ca
>>>> 
>>>> 
>>