New Book

[Colin Fyfe]

I have taken extracts from 8 of the PhDs completed in this university
over the last few years and published them together as a book. I hope
this may interest some in the connectionists community, and would be
happy to receive any feedback.

Colin Fyfe

-----------------------------------------------
Hebbian Learning and Negative Feedback Networks
Colin Fyfe

Springer ISBN 1-85233-883-0


The central idea of this book is that artificial neural networks which
use negative feedback of activation can use simple Hebbian learning to
self-organise in such a way that they uncover interesting structure in
data sets. The network in its simplest form performs a Principal
Component Analysis. Extensions to the network are shown to perform
Exploratory Projection Pursuit: they find low-dimensional filters of
the data which reveal interesting structure in the data. For example,
we might search for outliers from the main body of the data or
clusters within the data set and this search can be performed in a
hierarchical manner – we find one cluster in the midst of many and
then re-project the data from this one cluster to attempt to find
sub-clusters. There are two main ways of performing these searches and
these are contrasted and compared and a composite method created which
exhibits useful properties from the two underlying methods.

The network can also be used to find independent components of a data
set in a number of different ways. For example, one extension to the
basic network is shown to perform a type of Factor Analysis – it
identifies a set of factors which when OR-ed together will construct
the data set. Other methods are used to perform Independent Component
Analysis which is extensively used in blind source separation –
extracting one signal from a linear mixture of signals. The network
can also be used for clustering in a topology preserving manner: there
are several ways of clustering using this network in such a way that
similar data points are clustered close to one another and only
similar data points are treated this way.

In part 2 of the book, twinned networks are introduced: these networks
have two input data streams on which they self-organise using simple
Hebbian learning with negative feedback again. In their basic form,
the networks are shown to perform Canonical Correlation Analysis, the
statistical technique which finds those filters onto which projections
of the two data streams have greatest correlation. Various extensions
of the basic methods are devised in order to create methods which
react to more than two data streams at a time or which deal with
problems such as multicollinearity. A further extension is the
twinning of the Exploratory Projection Pursuit methods from the first
part of the book so that the new network identifies shared structure
across two data streams. This new network is also shown to perform
Independent Component Analysis. A final chapter deviates somewhat from
the rest of the book since its emphasis is on an extension of the
Principal Curve algorithm so that we now have two curves learning on
two data streams simultaneously.

Since the scope of the book is the development of new algorithms, all
algorithms which are derived analytically, are illustrated on
artificial data before being used on real data sets. Where it is of
interest, the results are compared with those from standard statistics
or from alternative artificial neural networks.  Other than the final
chapter, all networks are biologically plausible in that they use
locally available data to self-organise to extract information from
data sets.

Table of Contents
{1}Introduction
{1.1}Artificial Neural Networks
{1.2}The Organisation of this Book

Part I Single Stream Networks

 {2}Background
{2.1}Hebbian Learning
{2.2}Quantification of Information
{2.3}Principal Component Analysis
{2.4}Weight Decay in Hebbian Learning
{2.5}ANNs and PCA
{2.6}Anti-Hebbian Learning
{2.7}Independent Component Analysis
{2.8}Conclusion

 {3}The Negative Feedback Network
{3.1}Introduction
{3.2}The $VW$ Model
{3.3}Using Distance Differences
{3.4}Minor Components Analysis
{3.5}Conclusion

 {4}Peer-Inhibitory Neurons
{4.1}Analysis of Differential Learning Rates
{4.2}Differential Activation Functions
{4.3}Emergent Properties of the Peer-Inhibition Network
{4.4}Conclusion

 {5}Multiple Cause Data
{5.1}Non-negative Weights
{5.2}Factor Analysis
{5.3}Conclusion

 {6}Exploratory Data Analysis
{6.1}Exploratory Projection Pursuit
{6.2}The Data and Sphering
{6.3}The Projection Pursuit Network
{6.4}Other Indices
{6.5}Using Exploratory Projection Pursuit
{6.6}Independent Component Analysis
{6.7}Conclusion

 {7}Topology Preserving Maps
{7.1}Background
{7.2}The Classification Network
{7.3}The Scale Invariant Map
{7.4}The Subspace Map
{7.5}The Negative Feedback Coding Network
{7.6}Conclusion

 {8}Maximum Likelihood Hebbian Learning
{8.1}The Negative Feedback Network and Cost Functions
{8.2}$\epsilon $-Insensitive Hebbian Learning
{8.3}The Maximum Likelihood EPP Algorithm
{8.4}A Combined Algorithm
{8.5}Conclusion

Part II Dual Stream Networks

 {9}Two Neural Networks for Canonical Correlation Analysis
{9.1}Statistical Canonical Correlation Analysis
{9.2}The First Canonical Correlation Network
{9.3}Experimental Results
{9.4}A Second Neural Implementation of CCA
{9.5}Simulations
{9.6}Linear Discriminant Analysis
{9.7}Discussion

 {10}Alternative Derivations of CCA Networks
{10.1}A Probabilistic Perspective
{10.2}Robust CCA
{10.3}A Model Derived from Becker's Model 1
{10.4}Discussion

 {11}Kernel and Nonlinear Correlations
{11.1}Nonlinear Correlations
{11.2}The Search for Independence
{11.3}Kernel Canonical Correlation Analysis
{11.4}Relevance Vector Regression
{11.5}Appearance-Based Object Recognition
{11.6}Mixtures of Linear Correlations

 {12}Exploratory Correlation Analysis
{12.1}Exploratory Correlation Analysis
{12.2}Experiments
{12.3}Connection to CCA
{12.4}FastECA
{12.5}Local Filter Formation From Natural Stereo Images
{12.6}Twinned Maximum Likelihood Learning
{12.7}Unmixing of Sound Signals
{12.8}Conclusion

 {13}Multicollinearity and Partial Least Squares
{13.1}The Ridge Model
{13.2}Application to CCA
{13.3}Extracting Multiple Canonical Correlations
{13.4}Experiments on Multicollinear Data
{13.5}A Neural Implementation of Partial Least Squares
{13.6}Conclusion

 {14}Twinned Principal Curves
{14.1}Twinned Principal Curves
{14.2}Properties of Twinned Principal Curves
{14.3}Twinned Self-Organising Maps
{14.4}Discussion

 {15}The Future
{15.1}Review
{15.2}Omissions
{15.3}Current and Future Work

Appendices
A Negative Feedback Artificial Neural Networks
{A.1}The Interneuron Model
{A.2}Other Models
{A.3}Related Biological Models

 B Previous Factor Analysis Models
{B.1} F\"{o}ldi\'{a}k's Sixth Model
{B.2}Competitive Hebbian Learning
{B.3}Multiple Cause Models
{B.4}Predictability Minimisation
{B.5}Mixtures of Experts
{B.6}Probabilistic Models

C Related Models for ICA
{C.1}Jutten and Herault
{C.2}Nonlinear PCA
{C.3}Information Maximisation
{C.4}Penalised Minimum Reconstruction Error
{C.5}FastICA

D Previous Dual Stream Approaches
{D.1}The I-Max Model
{D.2}Stone's Model
{D.3}Kay's Neural Models
{D.4}Borga's Algorithm

 E Data Sets
{E.1}Artificial Data Sets
{E.2}Real Data Sets



Legal disclaimer
--------------------------

The information transmitted is the property of the University of
Paisley and is intended only for the person or entity  to which it
is addressed and may contain confidential and/or privileged material.
Statements and opinions expressed in this  e-mail may not represent
those of the company.  Any review, retransmission, dissemination and
other use of, or taking of any action in reliance upon, this
information by persons or entities other than the intended recipient
is prohibited.  If you received this in error, please contact the
sender immediately and delete the material from any computer. 

--------------------------