Connectionists: NIPS 07 workshop on approximate inference for continuous/hybrid models: Call for contributions

[Matthias Seeger]

*** Apologies for Multiple Postings ***

================== CALL FOR CONTRIBUTIONS ==================

Neural Information Processing Systems (NIPS) 2007 Workshop:

   Approximate Bayesian Inference in Continuous/Hybrid Models

Dates: 7-8 December, 2007. Whistler, CA

Organizers:
Matthias Seeger, Max-Planck Biological Cybernetics, Tuebingen
David Barber,    University College London
Neil Lawrence,   University of Manchester
Onno Zoeter,     Microsoft Research Cambridge

WWW:    http://intranet.cs.man.ac.uk/ai/nips07/
E-mail: abichm at gmail.com

Abstract Submission Deadline: October 21, 2007
Notification of Acceptance:   November 1, 2007

Sponsored by the Pascal Network of Excellence and by
Microsoft Research Cambridge

================== CALL FOR CONTRIBUTIONS ==================

Abstract:

The workshop will provide a forum to discuss unsolved issues, both
practical and theoretical, pertaining to the application of approximate
Bayesian inference in continuous variable and hybrid models. The emphasis
of the workshop will be in understanding the particular difficulties in
this class of models and the differential strengths and weaknesses of
available deterministic (variational) approximation techniques, as opposed
to the arguably better understood field of approximate inference in discrete
variable systems.

The target audience are practitioners, providing insight into and analysis
of problems with certain methods or comparative studies of several methods,
as well as theoreticians interested in characterizing the hardness of
continuous distributions or proving relevant properties of an established
method. We welcome contributions in the areas of Statistics (e.g. Markov
Chain
Monte Carlo methods), Information Geometry, Optimal Filtering, or other
related fields if an effort is made of bridging the gap towards variational
techniques.

Format:

The workshop will be single-day, comprising of a tutorial introduction,
invited talks (20 to 30 mins), and presentations of contributed work, with
time for discussions. Depending on quality and compatibility with workshop
aims, slots for brief talks and posters will be allocated.

We intend to have an interactive workshop, and will give priority to
contributions of novel ideas not yet established in Machine Learning,
and to critical and careful empirical comparative studies over polished
applications of established methods to standard problems. We encourage the
applicant to try to address some of the aims listed below or on the
workshop website.

We encourage contributions from related fields such as
* Statistics (e.g. Markov Chain Monte Carlo methods)
* Information Geometry
* Filtering, Dynamical Systems
if they can motivate the potential applicability to analyzing variational
inference techniques. Contributions of this sort could be tutorial in
nature.

Contributions should be communicated to the program committee (the
organizers) in form of an extended abstract (up to 8 pages in the NIPS
conference paper style), sent to the mail address stated at the beginning
of this mail. Submissions sent after the deadline (see beginning of mail) or
violating the format constraint will not be reviewed.

Motivation:

Many of the most important problems in Machine Learning and related
application areas are most naturally and succinctly treated using
continuous variable models. Several important continuous latent variable
models come to mind, each underlying a host of applications:

- Gaussian Process Models with non Gaussian likelihoods
- Sparse Linear Models (Bayesian ICA, Relevance Vector Machine, Sparse
  Image Coding, Non-negative Image Coding, Compressed Sensing)
- Dynamical Systems (Filters, Smoothers, Tracking, Switching Models,
  Latent State Space Models)

Several variational inference approximations have been applied
successfully to continuous models. However, most of these techniques
originated in Statistical Physics or Information Theory, where systems of
interest typically consist of discrete variables throughout, and
theoretical analyzes, convergence, or performance guarantees are
predominantly available for the discrete case and focus primarily on
discrete attributes such as graph topology.

Properties of variational approximations are much less well understood
when applied to continuous models. In some applications (such as gene
or metabolic network identification), continuous variables are
artificially discretized in order to allow the use of better understood
discrete techniques. In others, Monte Carlo techniques predominate. In
both cases, potential users of variational methods are probably deterred
by the general lack of theoretical understanding available.

In many applications (for example the models listed above), variational
methods are the state of the art today. However, as opposed to the
situation with discrete models, we still lack adequate understanding of
which characteristics of a given realistic model make accurate
variational inference simple or hard, whether such properties are
transferable or specific to certain methods, and which empirical
signatures allow such difficulties to be detected in a given method.

Continuous models give rise to difficulties not present in discrete
ones. On the one hand, the Gaussian family allows efficient Bayesian
computations with many variables, even if no independence structure is
present. However, non-Gaussian continuous families are usually not
closed under marginalization, an essential operation for any message
passing scheme, so that projections become necessary. Also,
intractable integrals often require additional convex bounding or
numerical quadrature. Both steps introduce errors typically not present
in discrete variable schemes.

Target Audience:

We welcome participants to share their experiences on practical
problems. However, in contrast to the usual `success stories' for an
established method, we invite descriptions of practical difficulties
in applying approximate inference methods, and how these were
analyzed and dealt with in the application in a principled manner.

While the goal is to understand deterministic (variational)
approximations better, we welcome contributions of researchers working
on Monte Carlo approximate inference if an effort is made towards
bridging the gap between the fields. In this context, we encourage
contributions of tutorial nature or of preliminary ideas.

The workshop is therefore intended to appeal both to practitioners with
insight into the difficulties in approximate inference in continuous
systems, and to theorists with an interest in characterizing the
complexity of posterior distributions or in analyzing properties of
approximate inference methods.

Aims:

The aim of the workshop is to study deterministic approximation
methods and characterizations of inference complexity in continuous
and hybrid systems.

Specifically, several important practical open issues are:

* Variational mean field Bayes methods are very frequently used, due to
  their generic derivation, ease of implementation, and numerical
  stability. However, growing evidence suggests that the methods may in
  practice be often severely biased, giving rise to adverse effects such
  as over-pruning of parameters. Why is this the case, and can it be
  improved upon?

* Learning hyperparameters requires inference as a subroutine. Which
  estimation biases do different methods imply? Can they be corrected?

* When and why might Expectation Propagation (EP) be superior to
  Variational mean field Bayes?

* Continuous families are not closed under marginalization, and
  projections (moment matching, variational KL minimization) are often
  needed. What properties do different projections have, and how do
  they affect the final solution?

* EP sometimes has severe numerical stability issues. For which models
  can its convergence be guaranteed? Do numerical problems reveal the
  hardness of a problem, or do they arise from specific shortcomings of
  the method?

* Which methods are numerically (in)stable, and on which problems?
  Are the difficulties inherent, or can the methods be stabilized?
  How do stable methods behave on problems where others are inherently
  instable?

* Is posterior multi-modality the only property that makes a problem
  hard? Which practically relevant properties of a non-Gaussian
  distribution render its approximation by a Gaussian difficult?

* Markov chain Monte Carlo (MCMC) is provably efficient for
  log-concave posterior distributions. What is the role of
  log-concavity in current variational methods (many Gaussian Process
  and Sparse Linear Models are log-concave)?

* Most methods make use of ideas coming from other communities
  (numerical quadrature, convex duality, scale mixtures). How have
  approximation errors been quantified there, and can we transfer these
  ideas?


================== CALL FOR CONTRIBUTIONS ==================

WWW:    http://intranet.cs.man.ac.uk/ai/nips07/
E-mail: abichm at gmail.com

Abstract Submission Deadline: October 21, 2007
Notification of Acceptance:   November 1, 2007