Two TechReps on ICA and PP

[Aapo Hyvarinen]

The following technical reports on independent component analysis
and projection pursuit are available at:

http://www.cis.hut.fi/~aapo/pub.html


Aapo Hyvarinen:
INDEPENDENT COMPONENT ANALYSIS BY MINIMIZATION OF MUTUAL INFORMATION

Independent component analysis (ICA) is a statistical method for
transforming an observed multidimensional random vector into
components that are statistically as independent from each other as
possible.  In this paper, the linear version of the ICA problem is
approached from an information-theoretic viewpoint, using Comon's
framework of minimizing mutual information of the components.  Using
maximum entropy approximations of differential entropy, we introduce a
family of new contrast (objective) functions for ICA, which can also
be considered 1-D projection pursuit indexes. The statistical
properties of the estimators based on such contrast functions are
analyzed under the assumption of the linear mixture model.  It is
shown how to choose optimal contrast functions according to different
criteria. Novel algorithms for maximizing the contrast functions are
then introduced. Hebbian-like learning rules are shown to result from
gradient descent methods.  Finally, in order to speed up the
convergence, a family of fixed-point algorithms for maximization of
the contrast functions is introduced.


Aapo Hyvarinen:
NEW APPROXIMATIONS OF DIFFERENTIAL ENTROPY FOR
INDEPENDENT COMPONENT ANALYSIS AND PROJECTION PURSUIT 
(To appear in NIPS*97)

We derive a first-order approximation of the density of maximum
entropy for a continuous 1-D random variable, given a number of simple
constraints. This results in a density expansion which is somewhat
similar to the classical polynomial density expansions by
Gram-Charlier and Edgeworth. Using this approximation of density, an
approximation of 1-D differential entropy is derived. The
approximation of entropy is both more exact and more robust against
outliers than the classical approximation based on the polynomial
density expansions, without being computationally more expensive. The
approximation has applications, for example, in independent component
analysis and projection pursuit.