information function vs. squared error

[Mark Plumbley]

Thanasis,

The "G" function you mentioned, based on an Entropy method, is probably
the one developed by Pearmutter and Hinton as a procedure for unsupervised
learning of binary units [1].  More recently, Linsker [2,3] and Plumbley
and Fallside [4] considered the principle of maximum information
transmission (or minimum information loss) for continuous units,  relating
this to Principal Component methods for linear units.

Unfortunately, these are mainly about unsupervised learning, rather than
Backprop specifically, although in [4] we do look at the way the
mean-squared error criterion places an *upper-bound* on the information loss
through a supervised network.  This bound will be tightest when the errors
on all the output units are independent and have the same variance (or the
same entropy for non-additive-Gaussian errors).  *If* you can choose the
target representation used by Backprop so that the errors are likely to
have these properties, it should perform closer to the (information-
theoretic) optimal.

Hope this is some help,

Mark.

References:

[1] B. A. Pearlmutter and G. E. Hinton: "G-Maximization: An Unsupervised
Learning Procedure for Discovering Regularities". In Proceedings of the
Conference on `Neural Networks for Computing'. American Institute of
Physics, 1986.

[2] R. Linsker: "Towards an Organisational Principle for a Layered
Perceptual Network". In "Neural Information Processing Systems
(Denver, CO. 1987)" (Ed. D. Z. Anderson), pp. 485-494.
American Institute of Physics, 1988.

[3] R. Linsker: "Self-Organization in a Perceptual Network". IEEE Computer,
vol. 21 (3), March 1988, pp. 105-117.

[4] M. D. Plumbley and F. Fallside: "An Information-Theoretic Approach to
Unsupervised Connectionist Models". Tech. Report CUED/F-INFENG/TR.7.
Cambridge University Engineering Department, 1988. Also in "Proceedings
of the 1988 Connectionist Models Summer School", pp. 239-245.
Morgan-Kaufmann, San Mateo, CA.

+--------------------------------------------+---------------------------+
| Mark Plumbley                              | Cambridge University      |
|  JANET: mdp at uk.ac.cam.eng.dsl              |   Engineering Department, |
|  ARPANET:                                  | Trumpington Street,       |
|     mdp%dsl.eng.cam.ac.uk at nss.cs.ucl.ac.uk | Cambridge  CB2 1PZ        |
|  Tel: +44 223 332754  Fax: +44 223 332662  | UK                        |
+--------------------------------------------+---------------------------+