Connectionists: Statistics versus “Understanding” in Generative AI.

[Gary Marcus]

💯


> On Feb 19, 2024, at 08:41, Dave Touretzky <dst at cs.cmu.edu> wrote:
> 
> Even GPT 3.5 can answer many questions about the spelling of words, so I
> don't think the problem is that the tokenization has removed this
> information.  I think the problem is that a stack of attention heads
> isn't good at applying a novel, arbitrary rule consistently to a long
> list of items.
> 
> Also, while the average human would be forgiven if they missed a few
> entries in the list of US states that do not contain "a" in their name,
> no normal human would claim that "Alaska" or "Texas" belongs in this
> list, nor would they persist in such an error after it was pointed out
> to them, the way ChatGPT 4 and Gemini do.
> 
> This isn't an argument against the possibility of AGI.  It's an argument
> that "attention is all you need" might be a bit of an overstatement.
> 
> -- Dave
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/connectionists/attachments/20240219/1c3f55c2/attachment.html>