Connectionists: Chomsky's apple

[Natsuki Oka]

Judging from the responses below, ChatGPT understands counting and sorting
to the extent that it can write a correct Python program but does not have
the understanding to bring the proper execution results.
[image: count_and_sort.png]
Here's the program that ChatGPT created:
---
sentences = [
    "The quick brown fox jumps over the lazy dog",
    "Python is a popular programming language",
    "I like to eat pizza for dinner",
    "The capital of France is Paris",
    "The cat in the hat wears a red and white striped hat",
    "My favorite color is blue",
    "The United States has fifty states",
    "There are seven days in a week",
    "I enjoy listening to music while I work",
    "Mount Everest is the highest mountain in the world"
]

# sort the sentences by number of words
sentences.sort(key=lambda x: len(x.split()))

# print the sorted sentences with the number of words in parentheses
for sentence in sentences:
    num_words = len(sentence.split())
    print(f"{sentence} ({num_words})")
---

The execution of this program yields the following correct results:
---
My favorite color is blue (5)
Python is a popular programming language (6)
The capital of France is Paris (6)
The United States has fifty states (6)
I like to eat pizza for dinner (7)
There are seven days in a week (7)
I enjoy listening to music while I work (8)
The quick brown fox jumps over the lazy dog (9)
Mount Everest is the highest mountain in the world (9)
The cat in the hat wears a red and white striped hat (12)
---

Oka Natsuki
Miyazaki Sangyo-keiei University

2023年3月13日(月) 17:45 Gary Marcus <gary.marcus at nyu.edu>:

> Geoff, Terry (mentioned below) and others,
>
> You raise an important question.
>
> Of course learning disabled people can understand some things and not
> others. Just as some computer scientists understand computer science and
> not psychology, etc. (and vice versa; unfortunately a lot of psychologists
> have never written a line of code, and that often undermines their work).
>
> That said your remark was itself a deflection away from my own questions,
> which I will reprint here, since you omitted them.
>
> *If a broken clock were correct twice a day, would we give it credit for
> patches of understanding of time? If n-gram model produced a sequence that
> was 80% grammatical, would we attribute to an underlying understanding of
> grammar?*
>
>
> The point there (salient to every good cognitive psychologist) is that you
> can’t infer underlying psychology and internal representations *directly*
> from behavior.
>
> A broken clock is behaviorally correct (occasionally) but it doesn’t have
> a functioning internal representation of time. An n-gram model, for high-n,
> can produce fluent prose, but not have any underlying understanding or
> representations of what it is saying, succeding to the extent that it does
> by piggybacking onto a corpus of speech produced by humans that talk about
> a world that is largely regular.
>
> Psychology is hard. Almost any “correct” behavior can be created in a
> multiplicity of ways; that’s why (cognitive) psychologists who are
> interested in underlying representations so often look to errors, and tests
> of generalization.
>
> In the case of LLMs, it’s clear that even when they produce a correct
> output, they rarely if ever deribe the same abstractions that a human
> would, or that a symbolic machine might use (perhaps preprogrammed) in a
> similar circumstance.
>
> Minerva, for example, is trained on an immense amount of data, and
> ostensibly captures two-digit arithmetic, but it fails altogether on
> 4-digit multiplication, The parsimonious explanation is that it is doing a
> kind of pattern recognition over stored examples (with 2-digit cases more
> densely sampled than 4-digit cases)—rather than genuinely understanding
> what multiplication is about.
>
> The same goes for essentially everything an LLMs talks about; there is a
> degree of generalization to similar examples, but distribution shift is
> hard (the crux of my own work going back to 1998), and nearly any
> generalization can be easily broken.
>
> As a last example, consider the following, where it initially sort of
> seems like ChatGPT has understood both counting and sorting in the context
> of complex query—which would be truly impressive—but on inspection it gets
> the details wrong, because it is relying on similarity, and not actually
> inducing the abstractions that define counting or sorting.
>
> [image: image]
>
> This example by the way also speaks against what Terry erroneously alleged
> yesterday (“If you ask a nonsense question, you get a nonsense answer...
> LLMs mirror the intelligence of the prompt”). The request is perfectly
> clear, not a nonsensical question in any way. The prompt is perfectly
> sensible; the system just isn’t up to the job.
>
> Cheers,
> Gary
>
>
> On Mar 10, 2023, at 10:57, Geoffrey Hinton <geoffrey.hinton at gmail.com>
> wrote:
>
> 
> A clever deflection. But can you please say if you think learning disabled
> people understand some things even though they do not understand others.
> This should be an area in which you actually have some relevant expertise.
>
> Geoff
>
>
> On Fri, Mar 10, 2023 at 1:45 PM Gary Marcus <gary.marcus at nyu.edu> wrote:
>
>> I think you should really pose this question to Yann LeCun, who recently
>> said “LLMs have a more superficial understanding of the world than a house
>> cat“ (
>> https://twitter.com/ylecun/status/1621861790941421573?s=61&t=eU_JMbqlN1G6Dkgee1AzlA
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__twitter.com_ylecun_status_1621861790941421573-3Fs-3D61-26t-3DeU-5FJMbqlN1G6Dkgee1AzlA&d=DwMFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=wQR1NePCSj6dOGDD0r6B5Kn1fcNaTMg7tARe7TdEDqQ&m=FVWJZf1ojyzQCmNLi6fhBu2H55KsAnX9FwsBVpN2cdxfhUwG7YQPx2xBoeB349YY&s=Es_NXJ0YjX5wOL7rD3KfaZLZ7-neIym77X_OMT8Kbrs&e=>
>> )
>>
>> Curious to hear how the conversation goes.
>>
>>
>> On Mar 10, 2023, at 10:04 AM, Geoffrey Hinton <geoffrey.hinton at gmail.com>
>> wrote:
>>
>> 
>>
>> A former student of mine, James Martens,  came up with the following way
>> of demonstrating chatGPT's lack of understanding. He asked it how many legs
>> the rear left side of a cat has.
>> It said 4.
>>
>> I asked a learning disabled young adult the same question. He used the
>> index finger and thumb of both hands pointing downwards to represent the
>> legs on the two sides of the cat and said 4.
>> He has problems understanding some sentences, but he gets by quite well
>> in the world and people are often surprised to learn that he has a
>> disability.
>>
>> Do you really want to use the fact that he misunderstood this question to
>> say that he has no understanding at all?
>> Are you really happy with using the fact that chatGPT sometimes
>> misunderstands to claim that it never understands?
>>
>> Geoff
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/connectionists/attachments/20230315/44ea09f9/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 489850 bytes
Desc: not available
URL: <http://mailman.srv.cs.cmu.edu/pipermail/connectionists/attachments/20230315/44ea09f9/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: count_and_sort.png
Type: image/png
Size: 156418 bytes
Desc: not available
URL: <http://mailman.srv.cs.cmu.edu/pipermail/connectionists/attachments/20230315/44ea09f9/attachment-0001.png>