NIPS*96 Postconference Workshop

[Jenny Orr]

                         CALL FOR SPEAKERS

                   NIPS*96 Postconference Workshop 

        TRICKS OF THE TRADE: How to Make Algorithms REALLY Work

                   Snowmass (Aspen), Colorado USA
                        Saturday Dec 7th, 1996


ORGANIZERS:

   Jenny Orr        Willamette University     gorr at willamette.edu
   Klaus Muller     GMD First,  Germany	      klaus at first.gmd.de
   Rich Caruana     Carnegie Mellon           caruana at cs.cmu.edu


OBJECTIVES:

Using neural networks to solve difficult problems often requires as
much art as science.  Researchers and practitioners acquire, through
experience and word-of-mouth, techniques and heuristics that help them
succeed.  Often these ``tricks'' are theoretically well motivated.
Sometimes they're the result of trial and error.  In this workshop we
ask you to share the ``tricks'' you have found helpful. Our focus will
be mainly regression and classification.


WHAT IS A TRICK?

A technique, rule-of-thumb, or heuristic that:

   - is easy to describe and understand
   - can make a real difference in practice
   - is not (yet) part of well documented  technique
   - has broad application
   - may or may not (yet) have a theoretical explanation

Examples of well known tricks include: early stopping, using symmetric
sigmoids, on-line calculation of the largest eigenvalue of the Hessian
without computing the hessian to determine optimal learning speed, ...


POTENTIAL TOPICS:

 - architecture design: picking layers, nodes, connectivity, 
                        modularity, activation functions, ...

 - model parameters &   learning rates, momentum, annealing schedules,
   speeding learning:   on-line, batch, conjugate gradient,
                        approximating the Hessian, ...

 - training/test sets:  sizes, dealing with too much/little data, 
                        noisy and/or missing data, active sampling,
                        skewed samples, ...

 - generalization:      which smoothers/regularizers to use and when 
                        to use them, network capacity, learning rate, 
                        net initialization, output representation,
                        SSE vs. cross-entropy, ...

 - training problems:   symmetry breaking, bootstrapping large nets, 
                        no negative instances, ...



WORKSHOP FORMAT:

Our goal is to create an enjoyable, quick moving one-day workshop with
lot's of ideas and discussion.  Each three-hour session will have 5-10
short presentations (10 mins max) and 1-2 longer presentations (30
mins max).  The long presentations will allow speakers to present
collections of ``tricks'' focussed on particular topics such as how to
speed up backprop, when and what regularization to use, ...  The short
presentations will give the rest of us an opportunity to present
isolated ``tricks'' with a minimum of presentation overhead.  To help
keep things "light", we ask that short presentations use 5 or fewer
slides.


SUBMISSIONS:

We already have a number of speakers lined up (see below), but we are
looking for more contributions.  If you'd like to give a presentation,
please email a short (1 page or less) description of the trick(s) to
gorr at willamette.edu.  If you wish to discuss a single trick, the total
presentation time will be 10 minutes, or less.  If you wish to discuss
a group of related tricks, please say so and briefly describe all the
tricks, and the total presentation should be 20-30 minutes, or less.

We will review the submissions and include as many as there is time
for in the schedule.  If possible, please discuss what the trick is
used for, when it is and is not applicable, sample problems you have
used it on, how well it seems to work in practice, and any explanation
you have, theoretical or otherwise, for why it seems to work.  Keep in
mind that this is a workshop, and "tricks" do not have to be fully
fleshed out methods with rigorous theoretical or empirical evidence.
If you've found a technique that's sometimes useful, the odds are
others will find it interesting and useful, too.  In order to list
your presentation in the workshop brochure, we need to have the title
and abstract by September 27.

IMPORTANT DATES:

Deadline for listing title in workshop brochure:   27 September, 1996
Final Deadline for submission of abstracts:         7 October, 1996
Notification of acceptance:         	           15 October, 1996 


FOLLOW-UP TO WORKSHOP:

To help insure that presenters get credit for divulging their tricks,
we'll ask presenters to prepare concise, one page write-ups of their
tricks.  These will be compiled into a report and/or published on the
web and made available to anyone interested.  Tricks that as yet have
no known theoretical explanation will be grouped together to form a
set of open problems.  We are also considering the possibility of
publishing a collection of tricks as a short book or special journal
issue.


CONFIRMED SPEAKERS:

Yann LeCun, 
Larry Yaeger
Hans Georg Zimmermann
Patrice Simard
Eric Wan
Rich Caruana
Nici Schraudolph
Shumeet Baluja
David Cohn


General info about the Postconference workshops can be found on the
NIPS homepage: http://www.cs.cmu.edu/afs/cs.cmu.edu/Web/Groups/NIPS/

If you have any questions about this workshop, don't hesitate to
contact one of the organizers.  We look forward to seeing you in
Snowmass!

-Jenny, Klaus, and Rich.


TOP-TEN REASONS WHY YOU SHOULD PRESENT A TRICK:

10: someone should get credit for the tricks we all end up using, 
    it might as well be you  (who's responsible for early stopping?)
 9: so you can bond with others who discovered the same trick
 8: because you couldn't get it on Letterman's Stupid Human Tricks
 7: no one will believe you're an expert if you don't use tricks
 6: your trick sucks, but you'll feel better using the other tricks
    you learn if you present something
 5: so you show everyone how very clever you are
 4: because you'll feel more comfortable using an unsupported trick
    if you can get others to use it, too
 3: so those of us who see flaws in your tricks can flame you alive
 2: you'll feel less guilty skiing if you present something at a 
    workshop
 1: because you really don't want to write a whole paper about it