thesis: latent var. models, dim. reduction & missing data reconstr.

[Miguel . Carreira-Perpin]

Dear connectionists,

I am pleased to make my PhD thesis available online (abstract below):

  Continuous latent variable models for dimensionality reduction and
		    sequential data reconstruction
		     Miguel A. Carreira-Perpinan
	  333 pages, 130 figures, 24 tables, 445 references

This thesis may be of interest to researchers working on probabilistic
models for data analysis, in particular dimensionality reduction,
inverse problems and missing data problems. Applications are given mainly
for speech processing (electropalatography, acoustic-to-articulatory
mapping). It also contains extensive surveys of all these areas.

The thesis can be retrieved in PostScript and PDF formats from:

  http://www.dcs.shef.ac.uk/~miguel/papers/phd-thesis.html

or

  http://www.giccs.georgetown.edu/~miguel/papers/phd-thesis.html

Also available there are:
- Matlab software for several of the models and algorithms discussed
- A BibTeX file with the references

Best regards,
Miguel

-- 
Miguel A Carreira-Perpinan
Department of Neuroscience             Tel. (202) 6878679
Georgetown University Medical Center   Fax  (202) 6870617
3900 Reservoir Road NW                 mailto:miguel at giccs.georgetown.edu
Washington, DC 20007, USA              http://www.giccs.georgetown.edu/~miguel


----------------------------------8<----------------------------------
  CONTINUOUS LATENT VARIABLE MODELS FOR DIMENSIONALITY REDUCTION AND
		    SEQUENTIAL DATA RECONSTRUCTION

		     Miguel A. Carreira-Perpinan

	Dept. of Computer Science, University of Sheffield, UK
			    February 2001


			       Abstract
                               ========

Continuous latent variable models (cLVMs) are probabilistic models
that represent a distribution in a high-dimensional Euclidean space
using a small number of continuous, latent variables. This thesis
explores, theoretically and practically, the ability of cLVMs for
dimensionality reduction and sequential data reconstruction.

The first part of the thesis reviews and extends the theory of cLVMs:
definition in terms of a prior distribution in latent space, a mapping
to data space and a noise model; maximum likelihood parameter
estimation with an expectation-maximisation (EM) algorithm; specific
cLVMs (factor analysis, principal component analysis (PCA),
independent component analysis, independent factor analysis and the
generative topographic mapping (GTM)); mixtures of cLVMs;
identifiability, interpretability and visualisation; and derivation of
mappings for dimensionality reduction and reconstruction and their
properties, such as continuity, for each cLVM. We extend GTM to
diagonal noise and give a corresponding EM algorithm.

We also describe a discrete LVM for binary data, Bernoulli mixtures,
widely used in practice. We show that their log-likelihood surface has
no singularities, unlike other mixture models, which makes EM
estimation practical; and that their theoretical non-identifiability
is rarely realised in actual estimates, which makes them
interpretable.

The second part deals with dimensionality reduction. We define the
problem and give an extensive, critical review of nonprobabilistic
methods for it: linear methods (PCA, projection pursuit), nonlinear
autoassociators, kernel methods, local dimensionality reduction,
principal curves, vector quantisation methods (elastic net,
self-organising map) and multidimensional scaling methods. We then
empirically evaluate, in terms of reconstruction error, computation
time and visualisation, several latent-variable methods for
dimensionality reduction of binary electropalatographic (EPG) data:
PCA, factor analysis, mixtures of factor analysers, GTM and Bernoulli
mixtures. We compare these methods with earlier, nonadaptive EPG data
reduction methods and derive 2D maps of EPG sequences for use in
speech research and therapy.

The last part of this thesis proposes a new method for missing data
reconstruction of sequential data that includes as particular case the
inversion of many-to-one mappings. We define the problem, distinguish
it from inverse problems, and show when both coincide. The method is
based on multiple pointwise reconstruction and constraint
optimisation. Multiple pointwise reconstruction uses a Gaussian
mixture joint density model for the data, conveniently implemented
with a nonlinear cLVM (GTM). The modes of the conditional distribution
of missing values given present values at each point in the sequence
represent local candidate reconstructions. A global sequence
reconstruction is obtained by efficiently optimising a constraint,
such as continuity or smoothness, with dynamic programming. We give a
probabilistic interpretation of the method. We derive two algorithms
for exhaustive mode finding in Gaussian mixtures, based on
gradient-quadratic search and fixed-point search, respectively; as
well as estimates of error bars for each mode and a measure of
distribution sparseness. We discuss the advantages of the method over
previous work based on the conditional mean or on universal mapping
approximators (including ensembles and recurrent networks),
conditional distribution estimation, vector quantisation and
statistical analysis of missing data. We study the performance of the
method with synthetic data (a toy example and an inverse kinematics
problem) and real data (mapping between EPG and acoustic data). We
describe the possible application of the method to several well-known
reconstruction or inversion problems: decoding of neural population
activity for hippocampal place cells; wind field retrieval from
scatterometer data; inverse kinematics and dynamics of a redundant
manipulator; acoustic-to-articulatory mapping; audiovisual mappings
for speech recognition; and recognition of occluded speech.


			 Contents (abridged)
                         ===================

1.  Introduction
2.  The continuous latent variable modelling formalism
3.  Some properties of finite mixtures of multivariate Bernoulli
    distributions
4.  Dimensionality reduction
5.  Dimensionality reduction of electropalatographic (EPG) data
6.  Inverse problems and mapping inversion
7.  Sequential data reconstruction
8.  Exhaustive mode finding in Gaussian mixtures
9.  Experiments with synthetic data
10. Experiments with real-world data: the acoustic-to-articulatory
    mapping problem
11. Conclusions
Appendices
----------------------------------8<----------------------------------