<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div dir="ltr"></div><div dir="ltr">There may no X for which we are absolutely sure—or unsure—that ChatGPT can do X; GPT is, as Bender et al have said, a stochastic parrot. Whether it can do any particular formulation depends heavily on how the conversational context up to a given moment resembles the training set.</div><div dir="ltr"><br></div><div dir="ltr">We cannot give perfect answers to most questions with respect to ChatGPT because the training set is closed. In work in related systems where the training set is open, similarity to training has clearly had a significant influence on output, e.g., <a href="https://arxiv.org/abs/2202.07206">https://arxiv.org/abs/2202.07206</a>, boldface added :</div><div dir="ltr"><br></div><div dir="ltr"><div dir="ltr">Impact of Pretraining Term Frequencies on Few-Shot Reasoning</div><div dir="ltr"><br></div><div dir="ltr">Yasaman Razeghi, Robert L. Logan IV, Matt Gardner, Sameer Singh</div><div dir="ltr">Pretrained Language Models (LMs) have demonstrated ability to perform numerical reasoning by extrapolating from a few examples in few-shot settings. However, the extent to which this extrapolation relies on robust reasoning is unclear. In this paper, we investigate how well these models reason with terms that are less frequent in the pretraining data. In particular, we examine the correlations between the model performance on test instances and the frequency of terms from those instances in the pretraining data. We measure the strength of this correlation for a number of GPT-based language models (pretrained on the Pile dataset) on various numerical deduction tasks (e.g., arithmetic and unit conversion). Our results <b>consistently demonstrate that models are more accurate on instances whose terms are more prevalent, in some cases above 70% (absolute) more accurate on the top 10\% frequent terms in comparison to the bottom 10\%</b>. Overall, although LMs exhibit strong performance at few-shot numerical reasoning tasks, <b>our results raise the question of how much models actually generalize beyond pretraining data, and we encourage researchers to take the pretraining data into account when interpreting evaluation results.</b></div></div><div dir="ltr"><br><blockquote type="cite">On Feb 19, 2024, at 16:35, Iam Palatnik <iam.palat@gmail.com> wrote:<br><br></blockquote></div><blockquote type="cite"><div dir="ltr"><div dir="ltr"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
The fact that answers are so dependent on word choice
</blockquote><div>Just to clarify, I was not referring to the prompt of the task itself. That is, whether we tell chatgpt to 'add 2 and 2' or 'give me 2+2'.</div><div>I'm referring to the fact that adding something like '<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__arxiv.org_pdf_2205.11916.pdf&d=DwMFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=wQR1NePCSj6dOGDD0r6B5Kn1fcNaTMg7tARe7TdEDqQ&m=YWjVJPf5AOaOB3ijUCLlzIDpa9dSb2YGJAQgI73mqjrsS3Mfs5hQ_RWWaGRugPL_&s=kABQ6Yw30aqt1uMm9ZtxjM-W6Bm4Ylky28UauP7pHhE&e=">think step by step</a>' onto the task prompt has a noticeable effect on the performance for various tasks.</div><div></div><div>And it doesn't have to literally be 'think step by step' exactly.<br></div><div><br></div><div>An example that happened just the other day: a friend was trying to get ChatGPT to write a 30 paragraph long text, and was having no luck. The model always wrote the wrong number of paragraphs. I told him to try including 'number the start of each paragraph to help yourself' to the prompt and bingo. I had never seen someone trying to do this exact task, nor this numbering trick. I came up with it on the spot, and it worked, but why? Why should we expect the model to perform better with this, at all?<br></div><div><br></div><div>Do examples like this entail understanding or lack thereof? I'm not sure everyone would agree on the answer.</div><div>But regardless of the specific wording we would use to describe what happened there, it would be a problem if we are 'absolutely sure' ChatGPT can't do X because it can't understand anything, while it turns out it can do X if you just change the prompt or tool access.<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Feb 19, 2024 at 6:13 PM Brad Wyble <<a href="mailto:bwyble@gmail.com" target="_blank">bwyble@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>The question is not whether we need to rely on external tools to verify the line lengths (we do), but whether we can understand that this illusion affects us, and can understand the basic idea of such an illusion. And we can do both of these. You can explain to someone that this illusion exists and they are able to understand that there is a difference between what they think their eyes are telling them about line length and what is actually true in the world. </div><div><br></div><div>As far as we can determine from GPT experiments, LLMs are not able to reason about their own limitations in this way. That is the fundamental distinction I'm referring to. </div><div><br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div><br></div><div>Because the performance of the LLMs on some of these tests seem to depend so much on how the questions are formulated and what tools they are given to respond with, I still tend to think that they understand something. I'm OK with the idea that their understanding has space to be much deeper, still, too.<br></div><div><div><br></div><div><br></div></div></div></blockquote><div><br></div><div>This is the opposite of the argument that is typically used though. It is specifically the face that their answers depend so much on phrase (e.g. that paper by Melanie Mitchell that recently went by) that we argue they do NOT understand. The fact that answers are so dependent on word choice is an indication that they are parroting remembered examples rather than understanding in a human sense. </div><div><br></div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div></div><div><br></div><div><br></div><div><br></div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Feb 19, 2024 at 1:50 PM Brad Wyble <<a href="mailto:bwyble@gmail.com" target="_blank">bwyble@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Iam, the difference is that while you may need an external source to remember all 50 states, for the ones that you have remembered/looked up, you are able to verify that they do or do not contain specific letters without reference to a resource, or writing some code to verify it. It is even worse that if you push them on their mistakes, they are still unable to correct. <div><br></div><div>A better counterargument to the example Dave provides is that perhaps LLMs just cannot ever break things down at the letter level because of their reliance on tokens. Humans can do this of course, but a good analogy for us might be the Muller Lyer illusion, which is essentially impenetrable to our cognitive faculties. I.e. we are unable to force ourselves to see the lines as their true lengths on the page because the basis of our representations does not permit it. This is perhaps similar to the way that LLM representations preclude them from accessing the letter level. </div><div><br></div><div>However, I think a good counterpoint to this is that while people are unable to un-see the Muller Lyer illusion, it is not that difficult to teach someone about this blindspot and get them to reason around it, with no external tools, just their own reasoning faculties. LLMs seem unable to achieve this level of self-knowledge no matter how patiently things are explained. They do not have the metacognitive faculty that allows them to even understand their blindspot about letters. </div><div><br></div><div><br></div><div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Feb 19, 2024 at 10:06 AM Gary Marcus <<a href="mailto:gary.marcus@nyu.edu" target="_blank">gary.marcus@nyu.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto"><div dir="ltr"></div><div dir="ltr">Correct; also tool integration has actually been less successful than some people believe: </div><div><br></div><a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__open.substack.com_pub_garymarcus_p_getting-2Dgpt-2Dto-2Dwork-2Dwith-2Dexternal-3Fr-3D8tdk6-26utm-5Fcampaign-3Dpost-26utm-5Fmedium-3Dweb&d=DwMFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=wQR1NePCSj6dOGDD0r6B5Kn1fcNaTMg7tARe7TdEDqQ&m=YWjVJPf5AOaOB3ijUCLlzIDpa9dSb2YGJAQgI73mqjrsS3Mfs5hQ_RWWaGRugPL_&s=EdAGBdW6tAiO3By0Rzq9AaKDHOMNv3uBWQwH6oVygus&e=" target="_blank">https://open.substack.com/pub/garymarcus/p/getting-gpt-to-work-with-external</a><div><div dir="ltr"><br><blockquote type="cite">On Feb 19, 2024, at 5:49 AM, Thomas Trappenberg <<a href="mailto:tt@cs.dal.ca" target="_blank">tt@cs.dal.ca</a>> wrote:<br><br></blockquote></div><blockquote type="cite"><div dir="ltr"><div dir="auto">Good point, but Dave's point stands as the models he is referring to did not even comprehend that they made mistakes. <div dir="auto"><br></div><div dir="auto">Cheers, Thomas</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Feb 19, 2024, 4:43 a.m. <<a href="mailto:wuxundong@gmail.com" target="_blank">wuxundong@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">That can be attributed to the models' underlying text encoding and processing mechanisms, specifically tokenization that removes the spelling information from those words. If you use GPT-4 instead, it can process it properly by resorting to external tools.<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Feb 19, 2024 at 3:45 PM Dave Touretzky <<a href="mailto:dst@cs.cmu.edu" rel="noreferrer" target="_blank">dst@cs.cmu.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">My favorite way to show that LLMs don't know what they're talking about<br>
is this simple prompt:<br>
<br>
List all the US states whose names don't contain the letter "a".<br>
<br>
ChatGPT, Bing, and Gemini all make a mess of this, e.g., putting "Texas"<br>
or "Alaska" on the list and leaving out states like "Wyoming" and<br>
"Tennessee". And you can have a lengthy conversation with them about<br>
this, pointing out their errors one at a time, and they still can't<br>
manage to get it right. Gemini insisted that all 50 US states have an<br>
"a" in their name. It also claimed "New Jersey" has two a's.<br>
<br>
-- Dave Touretzky<br>
</blockquote></div>
</blockquote></div>
</div></blockquote></div></div></blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Brad Wyble<br>Professor<span style="font-size:12.8px"> of Psychology</span> <br>Penn State University<div><br></div></div></div></div></div></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Brad Wyble (he/him)<div><br></div></div></div></div></div></div></div>
</blockquote></div>
</div></blockquote></body></html>