[AI Seminar] AI Seminar -- Vaishnavh Nagarajan, Uniform Convergence May Be Unable to Explain Generalization in Deep Learning

[Han Zhao]

Dear faculty and students:

We look forward to seeing you next Tuesday, Nov. 5th, at noon in *NSH 3305 *for
our AI Seminar sponsored by Apple. To learn more about the seminar series,
please visit the website <http://www.cs.cmu.edu/~aiseminar/>.
On Tuesday, Vaishnavh Nagarajan will give the following talk:
*Title: *Uniform Convergence May Be Unable to Explain Generalization in
Deep Learning

*Abstract:* In this talk, I will present our work that casts doubt on the
ongoing pursuit of using uniform convergence to explain generalization in
deep learning.

Over the last couple of years, research in deep learning theory has focused
on developing newer and more refined generalization bounds (using
Rademacher complexity, covering numbers, PAC-Bayes etc.,) to help us
understand why overparameterized deep networks generalize well. Although
these bounds are quite different on the surface, essentially, they are
'implementations' of a single learning-theoretic technique called uniform
convergence.

While it is well-known that many of these existing bounds are numerically
large, through a variety of experiments, we first bring to light another
crucial and more concerning aspect of these bounds: in practice, these
bounds can increase with the dataset size. Guided by these observations, we
then present specific scenarios where uniform convergence provably fails to
explain generalization in deep learning. That is, in these scenarios, even
though a deep network trained by stochastic gradient descent (SGD)
generalizes well, any uniform convergence bound would be vacuous, however
carefully it is applied.

Through our work, we call for going beyond uniform convergence to explain
generalization in deep learning.

This is joint work with Zico Kolter.
-- 

*Han ZhaoMachine Learning Department*


*School of Computer ScienceCarnegie Mellon UniversityMobile: +1-*
*412-652-4404*
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/ai-seminar-announce/attachments/20191103/a268ea31/attachment.html>