meta-learning

[A. Steven Younger]

Dear Connectionists,

In a connectionist message on Aug 30, 2001, Juergen Schmidhuber 
graciously drew attention to some work on meta-learning (learning how to 
learn) done by Sepp Hochreiter and myself. This work was presented by 
Sepp at the ICANN conference in August of this year [5].

For those of you that are interested, we have a companion paper that I 
presented at the IJCNN conference in July of this year [9]. This paper 
describes our work from a different perspective - that of  "Fixed-Weight 
Learning." 

Some other related papers are:

 L. A. Feldkamp, et al. in 1996 [3] used a similar Fixed-Weight Learning 
method  to meta-learn time series prediction problems.

 The  possibility of this type of meta-learning was conjectured by N. E. 
Cotter and P. R. Conwell in their 1990 and 1991 [1,2] original papers on 
Fixed-Weight Learning. Some examples of neural networks of this type 
were presented by myself, Conwell and Cotter in a 1999 paper [8].

 Another type of neural network that has embedded learning was presented 
by Schmidhuber in 1993 [6].

 An explanation of the Long Short-Term Memory (LSTM) was presented by 
Hochreiter and Schmidhuber in 1997  [4].

 A good presentation on the principles of meta-learning can be found in 
Schmidhueber, et al. in 1996 [7]

 This list is by no means exhaustive, but it should give a starting 
point for anyone interested in this topic.

 References:

 [1] N. E. Cotter and P. R. Conwell. "Fixed-Weight Networks Can Learn." 
In International Joint Conference on Neural Networks held in San Diego 
1990, IEEE, New York, 1990, pp. II.553- 559

 [2] N. E. Cotter and P. R. Conwell. "Learning Algorithms and Fixed 
Dynamics." In International Joint Conference on Neural Networks held in 
Seattle 1991. by IEEE. New York: IEEE 1991, pp. I.799- 804.

 [3] L. A. Feldkamp, G. V. Puskorius and P. C. Moore. "Adaptation from 
Fixed-Weight Dynamic Networks. "Proceedings of International Conference 
of Neural Networks 96, IEEE-1996

 [4] Sepp Hochreiter and Juergen Schmidhuber, "Long Short-Term Memory." 
Neural Computation.  9(8)  pp. 1735-1780, 1997

 [5] Sepp Hochreiter, A. Steven Younger and Peter R. Conwell, "Learning 
to learn using gradient descent", Lecture Notes on Comp. Sci. 2130, 
Proc. Intl. Conf. on Artificial Neural Networks (ICANN-2001), editors  
G. Dorffner and H. Bischof and K. Hornik", Springer: Berlin, Heidelberg, 
pp. 87-94, 2001

 [6] Juergen Schmidhuber.  "A neural network that embeds its own 
meta-levels." In Proc. Of the International Conference on Neural 
Networks '93, San Fransisco, IEEE 1993

 [7] Juergen Schmidhuber, Jieyu Zhao, and Marco Wiering. "Simple 
Principles of Meta-Learning", TR-IDSIA-69-96 http://www.idsia.ch

 [8] A. Steven Younger, P. R. Conwell, and N. E. Cotter. "Fixed-Weight 
On-Line Learning."  IEEE Transactions on Neural Networks. Vol.10 No. 2, 
March 1999 pp. 272-283

[9] A. Steven Younger, Sepp Hochreiter, and Peter R. Conwell. 
"Meta-Learning with Backpropagation."  IJCNN'01 International Joint 
Conference on Neural Networks 2001. IEEE-2001, pp.2001-2006

  
-------------------------------------------------------------------------


A. Steven Younger
syounger at boulder.net
Department of Computer Science                   Data Fusion Corporation
University of Colorado at Boulder                Northglenn CO
Boulder CO