Connectionists: Preprint: Incremental Online Learning in High Dimensions

[Sethu Vijayakumar]

The following paper is available for download from:
http://homepages.inf.ed.ac.uk/svijayak/publications/vijayakumar-NeuCom2005.pdf 

or as featured (free) article from the MIT Press website:
http://mitpress.mit.edu/catalog/item/default.asp?sid=22065875-6E38-4AAA-BB11-E00879BDE665&ttype=4&tid=31 


Incremental Online Learning in High Dimensions,
Neural Computation, vol. 17, no. 12, pp. 2602-2634 (2005)

Locally weighted projection regression (LWPR) is a new algorithm for 
incremental nonlinear
function approximation in high-dimensional spaces with redundant and 
irrelevant input dimensions.
At its core, it employs nonparametric regression with locally linear 
models. In order to stay
computationally efficient and numerically robust, each local model 
performs the regression
analysis with a small number of univariate regressions in selected 
directions in input space in
the spirit of partial least squares regression. We discuss when and how 
local learning
techniques can successfully work in high-dimensional spaces and review 
the various techniques
for local dimensionality reduction before finally deriving the LWPR 
algorithm. The properties
of LWPR are that it (1) learns rapidly with second-order learning 
methods based on incremental
training, (2) uses statistically sound stochastic leave-one-out cross 
validation for learning without
the need to memorize training data, (3) adjusts its weighting kernels 
based on only local
information in order to minimize the danger of negative interference of 
incremental learning,
(4) has a computational complexity that is linear in the number of 
inputs, and (5) can deal with a
large number of—possibly redundant—inputs, as shown in various empirical 
evaluations with up
to 90 dimensional data sets. For a probabilistic interpretation, 
predictive variance and confidence
intervals are derived. To our knowledge, LWPR is the first truly 
incremental spatially localized
learning method that can successfully and efficiently operate in very 
high-dimensional spaces.

Software (MATLAB/C++) implementation of the LWPR algorithm can be found at:
http://homepages.inf.ed.ac.uk/svijayak/software/LWPR/

-- 
------------------------------------------------------------------
Sethu Vijayakumar, Ph.D.          Assistant Professor(UK Lecturer)
Director, IPAB, School of Informatics, The University of Edinburgh
2107F JCMB, The Kings Buildings, Edinburgh EH9 3JZ, United Kingdom
URL: http://homepages.inf.ed.ac.uk/svijayak Ph: +44(0)131 651 3444
SLMC Research Group URL: http://www.ipab.informatics.ed.ac.uk/slmc
------------------------------------------------------------------
Adjunct Assistant Professor, 
Department of Computer Science,  University of Southern California
------------------------------------------------------------------