Apr 16 at 4pm, NSH 3305 -- Mingjie Sun, CMU, -- Massive Activations in Large Language Models

[Victor Akinwande]

Dear all,

We look forward to seeing you next Tuesday (04/16) from 04:00-5:00 PM (ET) for
the next talk of this semester's CMU AI Seminar, sponsored by SambaNova
Systems (https://sambanova.ai). The seminar will be held in *NSH 3305 *with
pizza provided and will be streamed on Zoom.

To learn more about the seminar series or to see the future schedule,
please visit the seminar website (http://www.cs.cmu.edu/~aiseminar/).

Next Tuesday (04/16), Mingjie Sun (CMU) will be giving a talk titled
"Massive Activations in Large Language Models".

*Talk Abstract: *
In the 2020s, Transformers have dominated the deep learning landscape,
powering almost all advanced AI systems. Despite their promising
capabilities, their inner workings are often overlooked and poorly
understood. In this talk, we delve into an intriguing phenomenon we observe
in Large Language Models (LLMs): very few activations within the hidden
states exhibit exceptionally high magnitudes, e.g., 100,000 times greater
than others. We call them massive activations. We present our investigation
of massive activations in LLMs and show how they are closely connected to
the self-attention mechanism — the core building block of Transformers.
Last, we go beyond the language domain and discuss the presence of massive
activations in Vision Transformers.

*Speaker Bio: *
Mingjie Sun is a Ph.D. student in the Computer Science Department at CMU.
His research focuses on improving the efficiency and empirical
understanding of foundation models.


*In person: NSH 3305Zoom Link:
 https://cmu.zoom.us/j/99510233317?pwd=ZGx4aExNZ1FNaGY4SHI3Qlh0YjNWUT09
<https://cmu.zoom.us/j/99510233317?pwd=ZGx4aExNZ1FNaGY4SHI3Qlh0YjNWUT09>*


- Victor & Asher
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/ai-seminar-announce/attachments/20240412/88116a0e/attachment.html>