Connectionists: LLM memory and reasoning/intelligence. Symposium on LLM mellontology (futurology)?

[pitas at csd.auth.gr]

Dear AI Colleagues,

 

in the International AI Doctoral Academy (AIDA <https://www.i-aida.org/> ), we recently organized a series of events (lectures etc), towards clarifying LLM issues for the

Benefit of our PhD students and postdocs.  I have also done some research myself (though not an LLM expert), as I wanted to add a section on LLMs in my recent book (Oct 2022)

I.Pitas, “Artificial Intelligence Science and Society Part A: Introduction to AI Science and Information Technology“, Amazon/Createspece,  <https://www.amazon.com/dp/9609156460?ref_=pe_3052080_397514860> https://www.amazon.com/dp/9609156460?ref_=pe_3052080_397514860

 

I attach some thoughts (parts of the above mentioned book section) on the issue of LLM memory and intelligence that may be of interest to you. Of course, critical comments are welcomed.

 

As such discussions in Connectionists are very interesting, but fragmented by the nature of this medium, I propose that they continue in the form of a rather informal ‘LLM mellontology’ symposium 

that can summarize the related findings. If there is an interest in co-organizing such a (e-?)symposium, let me know. 

 

AIDA has recently organized several very successful such symposia, the last one (2023) on Computational Politics <https://icarus.csd.auth.gr/ai-mellontology-symposium-2023/> .

Though I am really oversubscribed, if there is sufficient interest, I could consider co-organizing such an event in Greece, which is very beautiful in summer (2023). 

Any other location proposal (or e-symposium organization) is welcomed.

Best regards

Ioannis Pitas, AIDA Chair

 

a.	LLMs and Memory (I still have some reservations on whether LLMs are really closed systems or have access to external knowledge sources).

 

LLMs have been around for a while. However, they did caption massive people’s attention and enjoyed huge popularity with the huge success of ChatGPT of OpenAI (2022). It showcased the production of smooth text, in response to user needs and prompts. It could also enter into ‘intelligent’ dialogs with even experienced users. Such text and replies could have the form of small essays containing rather accurate factual information (though it may hallucinate facts at times).  Of course, a closed LLM system having billions of parameters should have a finite storage capacity. It has been designed to implicitly  'store' grammar and syntax knowledge, in the form of word correlations. However, it is quite puzzling how it can store rather well factual knowledge, as it can infinite (in theory). This storage capacity of LLMs should be studied more carefully. It can be proven that such a study could give us hints on how biological (including human) memory mechanisms function.  Furthermore, the use of ‘closed LLM systems’ in conjunction with classical knowledge storage, query and  retrieval mechanisms, e.g., in the form of knowledge graphs, can both expand their application domain and offer new superior ways to search content.  Such questions become even more interesting, if we take into account that the 'knowledge' storage capacity of much simpler DNNs has not really been studied well in the literature.

 

b.	LLMs and reasoning/intelligence.

 

It is understood that LLMs were not designed to address logical inference/reasoning. However, they create the impression to the average user that they have such a capacity, possibly at a limited level. For example, ChatGPT can also solve mathematical problems and produce programming code. A big question is whether such inference mechanisms are just an illusion or a revolutionary approach in solving a core AI research issues: reconciling Symbolic AI with Machine Learning, while advancing the former. As language (text) include millions of inference examples, it could be revealed that LLMs have the capacity to learn to learn symbolic AI from examples. After all, human inference follows certain thought patterns that can be easily mimicked (predicted) by LLMs through massive training. For example, LLMs can be proven to be proficient in forms of Logic Programming, as they already do in other programming tasks. This prospect may create jitter to Mathematical Logic scientists, but it can be proven to be a viable way forward. Any LLM ‘inference by example’ capacity may also hint ways that humans use to learn to think. Even in the absence of formal education, humans learn from their mothers, relatives and peers how to think, based on countless discussions and arguments on every day topics. At the very end, the debate on whether LLMs are intelligent or not may just evolve around a non-issue. For great many people, it is enough that machines appear to be intelligent and possibly better at that than themselves. However, stronger research efforts are needed answer such fundamental questions on machine intelligence.

 



-- 
This email has been checked for viruses by Avast antivirus software.
www.avast.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/connectionists/attachments/20230326/adc6de60/attachment.html>