Apr 16 at 4pm, NSH 3305 -- Mingjie Sun, CMU, -- Massive Activations in Large Language Models

[Victor Akinwande]

Quick reminder that this talk is happening later today at NSH 3305.

See you there!

On Fri, Apr 12, 2024 at 10:22 AM Victor Akinwande <vakinwan at andrew.cmu.edu>
wrote:

> Dear all,
>
> We look forward to seeing you next Tuesday (04/16) from 04:00-5:00 PM
> (ET) for the next talk of this semester's CMU AI Seminar, sponsored by
> SambaNova Systems (https://sambanova.ai). The seminar will be held in *NSH
> 3305 *with pizza provided and will be streamed on Zoom.
>
> To learn more about the seminar series or to see the future schedule,
> please visit the seminar website (http://www.cs.cmu.edu/~aiseminar/).
>
> Next Tuesday (04/16), Mingjie Sun (CMU) will be giving a talk titled
> "Massive Activations in Large Language Models".
>
> *Talk Abstract: *
> In the 2020s, Transformers have dominated the deep learning landscape,
> powering almost all advanced AI systems. Despite their promising
> capabilities, their inner workings are often overlooked and poorly
> understood. In this talk, we delve into an intriguing phenomenon we observe
> in Large Language Models (LLMs): very few activations within the hidden
> states exhibit exceptionally high magnitudes, e.g., 100,000 times greater
> than others. We call them massive activations. We present our investigation
> of massive activations in LLMs and show how they are closely connected to
> the self-attention mechanism — the core building block of Transformers.
> Last, we go beyond the language domain and discuss the presence of massive
> activations in Vision Transformers.
>
> *Speaker Bio: *
> Mingjie Sun is a Ph.D. student in the Computer Science Department at CMU.
> His research focuses on improving the efficiency and empirical
> understanding of foundation models.
>
>
> *In person: NSH 3305Zoom Link:
>  https://cmu.zoom.us/j/99510233317?pwd=ZGx4aExNZ1FNaGY4SHI3Qlh0YjNWUT09
> <https://cmu.zoom.us/j/99510233317?pwd=ZGx4aExNZ1FNaGY4SHI3Qlh0YjNWUT09>*
>
>
> - Victor & Asher
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/ai-seminar-announce/attachments/20240416/9def2443/attachment.html>