Connectionists: Initialization and self-organized optimization of recurrent neural network connectivity

[Oliver.Obst@csiro.au]

Dear Colleagues

as a subscriber to this list you may be interested in our paper

Initialization and self-organized optimization of recurrent neural network connectivity
Joschka Boedecker, Oliver Obst, N. Michael Mayer, and Minoru Asada
HFSP J. 2009 October; 3(5): 340–349. Published online 2009 October 26. doi: 10.2976/1.3240502.

which is now available free of charge from
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2801534/pdf/HJFOA5-000003-000340_1.pdf

Comments and questions are welcome.

Abstract:
Reservoir computing (RC) is a recent paradigm in the field of recurrent neural networks. Networks in RC have a sparsely and randomly connected fixed hidden layer, and only output connections are trained. RC networks have recently received increased attention as a mathematical model for generic neural microcircuits to investigate and explain computations in neocortical columns. Applied to specific tasks, their fixed random connectivity, however, leads to significant variation in performance. Few problem-specific optimization procedures are known, which would be important for engineering applications, but also in order to understand how networks in biology are shaped to be optimally adapted to requirements of their environment. We study a general network initialization method using permutation matrices and derive a new unsupervised learning rule based on intrinsic plasticity (IP). The IP-based learning uses only local learning, and its aim is to improve network performance in a self-organized way. Using three different benchmarks, we show that networks with permutation matrices for the reservoir connectivity have much more persistent memory than the other methods but are also able to perform highly nonlinear mappings. We also show that IP-based on sigmoid transfer functions is limited concerning the output distributions that can be achieved.

--  
Oliver Obst                     form follows function (Louis Sullivan).
Adaptive Systems Team   CSIRO ICT Centre  http://research.ict.csiro.au/
+61 2 9372 4710                                  http://oliver.obst.eu/