[CMU AI Seminar] Apr 20 at 12pm (Zoom) -- Misha Khodak (CMU) -- Factorized Layers Revisited: Compressing Deep Neural Networks Without Playing the Lottery -- AI Seminar sponsored by Fortive

[Shaojie Bai]

Dear all,

We look forward to seeing you *next Tuesday (4/20)* from *1**2:00-1:00 PM
(U.S. Eastern time)* for the next talk of our *CMU AI seminar*, sponsored
by Fortive <https://careers.fortive.com/>.

To learn more about the seminar series or see the future schedule, please
visit the seminar website <http://www.cs.cmu.edu/~aiseminar/>.
<http://www.cs.cmu.edu/~aiseminar/>

On 4/20, *Misha Khodak* (CMU CSD) will be giving a talk on "*Factorized
Layers Revisited: Compressing Deep Neural Networks Without Playing the
Lottery*".

*Title*: Factorized Layers Revisited: Compressing Deep Neural Networks
Without Playing the Lottery

*Talk Abstract*: Machine learning models are rapidly growing in size,
leading to increased training and deployment costs. While the most popular
approach for training compressed models is trying to guess good "lottery
tickets" or sparse subnetworks, we revisit the low-rank factorization
approach, in which weights matrices are replaced by products of smaller
matrices. We extend recent analyses of optimization of deep networks to
motivate simple initialization and regularization schemes for improving the
training of these factorized layers. Empirically these methods yield higher
accuracies than popular pruning and lottery ticket approaches at the same
compression level. We further demonstrate their usefulness in two settings
beyond model compression: simplifying knowledge distillation and training
Transformer-based architectures such as BERT. This is joint work with Neil
Tenenholtz, Lester Mackey, and Nicolo Fusi.

*Speaker Bio*: Misha Khodaka is a PhD student in Carnegie Mellon
University's Computer Science Department advised by Nina Balcan and Ameet
Talwalkar. His research focuses on foundations and applications of machine
learning, most recently neural architecture search, meta-learning, and
unsupervised representation learning. He recently spent time as an intern
with Nicolo Fusi at Microsoft Research - New England and previously
received an AB in Mathematics and an MSE in Computer Science from Princeton
University, where he worked with Sanjeev Arora.

*Zoom Link*:
https://cmu.zoom.us/j/93099996457?pwd=b3BSSHp2RWZWQjZ0SUE4ZkdKSDk4UT09

Thanks,
Shaojie Bai (MLD)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/ai-seminar-announce/attachments/20210415/2ee69d68/attachment.html>