Tech Report on regularisation in sequential learning

[J.F. Gomes De Freitas]

Hi

A technical report on regularisation in sequential learning is now
available at

ftp://svr-ftp.eng.cam.ac.uk/pub/reports/freitas_tr307.ps.gz
(SVR - Cambridge)

The paper covers topics such as Bayesian inference with hierarchical models, 
extended Kalman filtering, regularisation, adaptive learning rates and automatic
relevance determination. It is a longer version of a recent NIPS
publication and feedback will be gratefully appreciated. I hope you find
it interesting too.

ABSTRACT:

In this paper, we show that a hierarchical Bayesian modelling approach
to sequential learning leads to many interesting attributes such as
regularisation and automatic relevance determination. We identify
three inference levels within this hierarchy, namely model selection,
parameter estimation and noise estimation. In environments where data 
arrives sequentially ,techniques such as cross-validation to achieve 
regularisation or model selection are not possible. The Bayesian 
approach, with extended Kalman filtering at the parameter estimation 
level, allows for regularisation within a minimum variance framework.A 
multi-layer perceptron is used to generate the extended Kalman filter 
nonlinear measurements mapping. We describe several algorithms at the 
noise estimation level, which allow us to implement adaptive  
regularisation and automatic relevance determination of model inputs and 
basis functions. An important contribution of this paper is to show the 
theoretical links between adaptive noise estimation in extended Kalman 
filtering, multiple adaptive learning rates and multiple smoothing 
regularisation coefficients. 

Thanks
Nando de Freitas
_______________________________________________________________________________

JFG de Freitas (Nando)
Speech, Vision and Robotics Group
Information Engineering
Cambridge University
CB2 1PZ England

Tel (01223) 302323 (H)          
    (01223) 332754 (W)
_______________________________________________________________________________