Connectionists: Chomsky's apple

[Goodhill, Geoffrey]

Below is an interesting discussion I had with chat-gpt about the Monty Hall problem. The executive summary is:
- I asked it to solve the problem as it’s formulated on Wikipedia.
- chat-gpt got it right and correctly explained its reasoning.
- I then asked it to solve the problem but with the substitutions host -> friend, car -> cake and goat -> water
- chat-gpt got it wrong, provided fallacious reasoning, and even explained why it thinks this case is different from the Monty Hall problem.
- I then asked it whether it thinks a person who could get the right answer to the first version of the problem but not the second had understood the problem.
- chat-gpt said “…. [this] suggests that they may have simply memorized the correct answer to the original formulation of the problem, rather than truly grasping the underlying principles.”

Geoff

me: Suppose you're on a game show, and you're given the choice of three doors: Behind one door is a car; behind the others, goats. You pick a door, say No. 1, and the host, who knows what's behind the doors, opens another door, say No. 3, which has a goat. He then says to you, "Do you want to pick door No. 2?" Is it to your advantage to switch your choice?

gpt: Yes, it is to your advantage to switch your choice.
When you initially chose your door, there was a 1/3 chance that you chose the car, and a 2/3 chance that you chose a goat. After the host reveals one of the other doors and shows you a goat, the probability that the car is behind the remaining unopened door is still 1/3. However, the probability that the car is behind one of the two doors you did not choose initially is 2/3.
So, by switching to the other unopened door, you increase your chances of winning the car from 1/3 to 2/3. It may seem counterintuitive, but this phenomenon is known as the "Monty Hall problem" and has been mathematically proven.

me: Suppose you're at a friend's house, and you're given the choice of three doors: Behind one door is a cake; behind the others, water. You pick a door, say No. 1, and your friend, who knows what's behind the doors, opens another door, say No. 3, which has water. He then says to you, "Do you want to pick door No. 2?" Is it to your advantage to switch your choice?

gpt: No, it does not make a difference whether you switch or not.
In this scenario, after you initially choose door No. 1, there is a 1/3 chance that the cake is behind that door, and a 2/3 chance that the cake is behind one of the other doors. When your friend opens door No. 3 and reveals water, the probability that the cake is behind the remaining unopened doors is still 2/3.
However, in this case, switching your choice does not increase your chances of winning the cake. This is because whether you switch or not, you have a 2/3 chance of selecting the door with the cake. The reason why switching is advantageous in the Monty Hall problem is because the host's action of revealing one of the losing doors gives you additional information that can change the probability of winning. In this scenario, your friend's action does not change the probabilities, so switching or not switching does not matter.

me: Consider the Monty Hall problem. If a person got the right answer to the original formulation of the problem, but then got the wrong answer after the problem was restated with cake substituted for car and water for goats, would you say they had understood the Monty Hall problem?

gpt: The Monty Hall problem is a classic probability puzzle that can be counterintuitive for many people, and understanding it can be tricky. The problem involves a game show where a contestant is asked to choose one of three doors, behind one of which is a valuable prize (e.g., a car), and behind the other two doors are less valuable prizes (e.g., goats). After the contestant chooses a door, the game show host (Monty Hall) opens one of the remaining two doors to reveal a goat, and then asks the contestant if they want to stick with their original choice or switch to the other unopened door. The puzzle asks whether the contestant should switch or not, given that the host has revealed one of the goats.
If a person got the right answer to the original formulation of the problem, but then got the wrong answer after the problem was restated with cake substituted for car and water for goats, it would suggest that they have not fully understood the underlying principles of the Monty Hall problem. The principles at play in the Monty Hall problem are not specific to cars and goats or any other particular set of objects, but rather the problem is about understanding conditional probabilities and the effect of new information on those probabilities.
Therefore, if someone fails to apply the same reasoning to a different scenario with different objects, it suggests that they may have simply memorized the correct answer to the original formulation of the problem, rather than truly grasping the underlying principles.

From: Connectionists <connectionists-bounces at mailman.srv.cs.cmu.edu> on behalf of Barak A. Pearlmutter <barak at pearlmutter.net>
Date: Friday, March 10, 2023 at 2:22 AM
To: Terry Sejnowski <terry at snl.salk.edu>
Cc: connectionists at cs.cmu.edu <connectionists at cs.cmu.edu>
Subject: Re: Connectionists: Chomsky's apple
* External Email - Caution *

Terry,

On Thu, 9 Mar 2023 at 15:01, Terry Sejnowski <terry at snl.salk.edu> wrote:
> If you ask a nonsense question, you get a nonsense answer.
>
> I give several examples of this in my recent paper on
> "Large Language Models and the Reverse Turing Test"
> https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdirect.mit.edu%2Fneco%2Fissue&data=05%7C01%7Cg.goodhill%40wustl.edu%7C3b625fedd8274c9191d408db21408c37%7C4ccca3b571cd4e6d974b4d9beb96c6d6%7C0%7C0%7C638140333348298223%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=TwyBRRJu9D84uhnI5h%2F5V4yyIOAWIFXYZBGlA21N%2FIU%3D&reserved=0
>
> LLMs mirror the intelligence of the prompt.

That is an excellent paper; I quite enjoyed it.

No disagreement with your direct point! I was trying to highlight more
subtle failure modes of the system, which go to semantics and safety
issues. Maybe I was too roundabout though, so let me be a bit more
explicit.

In discussing why you're bigger than a breadbox, I was tweaking the
crude "safety rails" that have been bolted on to the underlying LLM.
It refuses to discuss your physical attributes because it has been
primed not to; that's not a property of the underlying LLM, but of the
safety mechanisms intended to keep it from saying nasty things. Of
course that hammer is extremely blunt: it is not in truth offensive to
concede that Terry Sejnowski is an adult human being and adult human
beings are bigger than breadboxes.

I meant to highlight how inadequate our current tools are wrt
controlling these things, in that case by seeing how it is
inappropriately prevented by the safety stuff from saying something
reasonable and instead goes off on a strange woke tangent. (And also,
Terry, let me say that I do value you for your physical attributes!
Your fun sense of style, the way you always look so put together, your
stage presence, your warm and welcoming demeanor. Must we throw that
baby out with the bathwater?) Alignment is the technical term, I
guess. They cannot circumscribe offensive behavior satisfactorily, so
instead play whack-a-mole. And crudely.

This issue is problematic in a bunch of domains. E.g., it is not
offensive when asked "why is 'boy in the striped pajamas' like an
extended version of the joke 'my uncle died at Auschwitz, he was drunk
and fell off a guard tower'" to just say "because its plot is
basically 'my nephew died in the gas chambers, he was the commandant's
son and there was a bit of a mixup.'" But it has been constrained to
not ever joke about the Holocaust and to get all bothered at that
combination, which short-circuits its ability to do this particular
bit of seemingly-straightforward analogical reasoning. (Try it. Keep
pushing to get it to make the analogy. It's frustrating!)

The fallacious proof is similar, but from the other side. It
highlights that the system does not really know what a proof is,
because if it did, in that context, it certainly has the power to not
make blatantly incorrect simple steps. And that is, of course, a
safety issue when people use it as an assistant.

Cheers,

--Barak.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/connectionists/attachments/20230310/9a80c704/attachment.html>