Papers on Unsupervised Temporal Sequence Processing

[Guilherme de Alencar Barreto]

Dear Connectionists,

The following three papers, on unsupervised temporal sequence processing,
are available from  
http://www.sel.eesc.sc.usp.br/lasi/www/gbarreto/publicacoes.htm


1) Arajo, A.F.R. and Barreto, G.A. (2002). Context in temporal sequence 
processing: A self-organizing approach and its application to robotics. 
IEEE Transactions on Neural Networks, Vol. 13, No. 1, pp. 45-57, January 
Issue. 

Abstract:

A self-organizing neural network for learning and recall of complex 
temporal sequences is developed and applied to robot trajectory planning. 
We consider trajectories with both repeated and shared states. Both cases 
give rise to ambiguities during reproduction of stored trajectories which 
are resolved via temporal context  information. Feedforward weights 
encode spatial features of the input trajectories, while the temporal 
order is learned by lateral weights through a time-delayed Hebbian 
learning rule. After training is completed, the network model operates in 
an anticipative fashion by always recalling the successor of the current 
input state. Redundancy in sequence representation improves the 
robustness of the network to noise and faults. The network uses memory 
resources efficiently by reusing neurons that have previously stored 
repeated/shared states. Simulations have been carried out to evaluate the 
performance 
of the network in terms of trajectory reproduction, convergence time and 
memory usage, tolerance to fault and noise, and sensitivity to trajectory 
sampling rate. The results show that the network model is fast, accurate 
and robust. Its performance is discussed in comparison with other neural 
networks models. 

Keywords: Context, temporal sequences, self-organization, Hebbian 
learning, robotics, trajectory planning. 


2) Barreto, G.A. and Arajo, A.F.R. (2001). Time in self-organizing maps: 
An overview of models. International Journal of Computer Research, 
Special Issue on Neural Networks: Past, Present and Future, 10(2):139-179.

Abstract:

We review a number of neural models of self-organizing feature maps 
designed to process sequential patterns in engineering and cognitive 
applications. This type of pattern inherently holds information of both a 
spatial and a temporal nature. The latter includes the temporal order, 
relative duration of the time interval, and temporal correlations of the 
items in the sequence. We present the main concepts related to the 
processing of spatiotemporal sequences and then discuss how the time 
dimension can be incorporated into the network dynamics through the use 
of various short-term memory models. The vast majority of the models are 
based on Kohonen's self-organizing map, being organized according to the 
network architecture and learning rules, and presented in nearly 
chronological order. We conclude the paper by suggesting possible 
directions for further research on temporal sequence processing through 
self-organizing maps. 

Keywords: Self-organizing maps, unsupervised learning, time dimension, 
temporal sequence, short-term memory, temporal context. 


3) Barreto, G.A. and Arajo, A.F.R. (2001). Unsupervised learning and 
temporal context to recall complex robot trajectories. International 
Journal of Neural Systems, 11(1):11-22. 

Abstract:

An unsupervised neural network is proposed to learn and recall complex 
robot trajectories. Two cases are considered: (i) A single trajectory in 
which a particular arm configuration (state) may occur more than once, 
and (ii) trajectories sharing states with each other. Ambiguities occur 
in both cases during recall of such trajectories. The proposed model 
consists of two groups of synaptic weights trained by competitive and 
Hebbian learning laws. They are responsible for encoding spatial and 
temporal features of the input sequences, respectively. Three mechanisms 
allow the network to deal with repeated or shared states: local and 
global context units, neurons disabled from learning, and redundancy. The 
network reproduces the current and the next state of the learned 
sequences and is able to resolve ambiguities. The model was simulated 
over various sets of robot trajectories in order to evaluate learning and 
recall, trajectory sampling effects and robustness. 


Guilherme de A. Barreto
Dept. of Electrical Engineering 
University of So Paulo (USP)
So Carlos, SP, BRAZIL
FAX: 55- 16 - 273 9372
PHONE: 55- 16 - 273 9357