Paper: efficient online training of curved models using ancillary statistics

[zhuh]

The following paper is accepted for 1996 International Conference
on Neural Information Processing, Hong Kong, Sept. 1996.

ftp://cs.aston.ac.uk/neural/zhuh/ac1.ps.Z


	Using Ancillary Statistics in On-Line Learning Algorithms

		    Huaiyu Zhu and Richard Rohwer
		   Neural Computing Research Group
 		    Dept of Comp. Sci. Appl. Math.
	  	  Aston Univ., Birmingham B4 7ET, UK

				Abstract

  Neural networks are usually curved statistical models.  They do
  not have finite dimensional sufficient statistics, so on-line
  learning on the model itself inevitably loses information.  In this
  paper we propose a new scheme for training curved models, inspired
  by the ideas of ancillary statistics and adaptive critics.  At each
  point estimate an auxiliary flat model (exponential family) is
  built to locally accommodate both the usual statistic (tangent to
  the model) and an ancillary statistic (normal to the model).  The
  auxiliary model plays a role in determining credit assignment
  analogous to that played by an adaptive critic in solving temporal
  problems.  The method is illustrated with the Cauchy model and the
  algorithm is proved to be asymptotically efficient.


--
Huaiyu Zhu, PhD                   email: H.Zhu at aston.ac.uk
Neural Computing Research Group   http://neural-server.aston.ac.uk/People/zhuh
Dept of Computer Science          ftp://cs.aston.ac.uk/neural/zhuh
    and Applied Mathematics       tel: +44 121 359 3611 x 5427
Aston University,                 fax: +44 121 333 6215
Birmingham B4 7ET, UK