principal components

[Gary Cottrell]

In reply to:

>Date: Fri, 8 Nov 91 10:02:34 -0800
>From: Ray White <white at teetot.acusd.edu>
>To: Connectionists at CS.CMU.EDU
>
>(Where "that" refers to my 'Competitive Hebbian Learning', to be published
>in Neural Networks, 1992, in response to Yoshio Yamamoto.)
>
>The Sanger paper that I think of in this connection is the 'Neural Networks '
>paper, T. Sanger (1989) Optimal unsupervised learning..., Neural Networks, 2,
>459-473.
>As I understand it, Sanger's 'Generalized Hebbian learning' trains units
>to find successively, the principle components of the input, starting with
>the most important and working on down, depending on the number of units
>you use.
>
>Competitive Hebbian Learning, on the other hand, is a
>simpler algorithm which trains units to learn simultaneously (approximately)
>orthogonal linear combinations of the components of the input.  With this
>algorithm, one does not get the princple components nicely separated out,
>but one does get trained units of roughly equal importance.

>   Ray White	(white at teetot.acusd.edu)
>   Depts. of Physics & Computer Science
>   University of San Diego
>   

Back prop when used with linear nets, does just this also. Since the
optimal technique is PCA in the linear case with a quadratic cost
function, bp is just a way of directly performing this and is not an
improvement over Karhunen-Loeve (except, perhaps in being space
efficient).  More recently, Mathilde Mougeot has used the fact that bp
is doing PCA to discover a fast algorithm for the quadratic case, and
she also has shown that bp can be effectively used for other norms.

References:
Baldi & Hornik, (1988) Neural Networks and Principal Components
Analysis: Learning from examples without local minima. Neural Networks,
Vol 2, No 1.

Cottrell, G.W. and Munro, P. (1988) Principal components analysis
of images via back propagation. Invited paper in \fIProceedings of
the Society of Photo-Optical Instrumentation Engineers\fP,
Cambridge, MA.

Mougeot, M., Azencott, R. & Angeniol, B. (1991) Image compression with back
propagation: Improvement of the visual restoration using different cost
functions. Neural Networks vol 4. number 4 pp 467-476.

onward and upward,
Gary Cottrell 619-534-6640 Sec'y: 619-534-5288 FAX: 619-534-7029
Computer Science and Engineering 0114
University of California San Diego 
La Jolla, Ca. 92093
gary at cs.ucsd.edu (INTERNET)
gcottrell at ucsd.edu (BITNET, almost anything)