New Paper on Mutual Information

[Marcus Hutter]

Dear Colleagues,

I would like to announce a new paper on Mutual Information
beyond the point/frequency estimate, which today appeared
in the online NIPS-Proceedings
http://www-2.cs.cmu.edu/~nips/2001papers/
since it is possibly of general interest.

Best wishes

Marcus Hutter

===========================================================================
"Distribution of Mutual Information"

The mutual information of two random variables i and j with joint
probabilities t_ij is commonly used in learning Bayesian nets as
well as in many other fields. The chances t_ij are usually
estimated by the empirical sampling frequency n_ij/n leading to a
point estimate I(n_ij/n) for the mutual information. To answer
questions like ``is I(n_ij/n) consistent with zero?'' or ``what is
the probability that the true mutual information is much larger
than the point estimate?'' one has to go beyond the point estimate.
In the Bayesian framework one can answer these questions by
utilizing a (second order) prior distribution p(t) comprising
prior information about t. From the prior p(t) one can compute the
posterior p(t|n), from which the distribution p(I|n) of the mutual
information can be calculated. We derive reliable and quickly
computable approximations for p(I|n). We concentrate on the mean,
variance, skewness, and kurtosis, and non-informative priors. For
the mean we also give an exact expression. Numerical issues and
the range of validity are discussed.

Downloadable also at: http://www.idsia.ch/~marcus/ai/xentropy.htm in various
formats.

===========================================================================
Dr. Marcus Hutter, IDSIA
Istituto Dalle Molle di Studi sull'Intelligenza Artificiale
Galleria 2 CH-6928 Manno(Lugano) - Switzerland
Phone:   +41-91-6108668 Fax:  +41-91-6108661
E-mail marcus at idsia.ch   http://www.idsia.ch/~marcus