Papers on associative memory in neuronal networks
Tue Jun 6 06:52:25 EDT 2006
PAPERS ON ASSOCIATIVE MEMORY IN NEURONAL NETWORKS
We would like to bring to your attention a series of recent
theoretical papers about associative memory in neuronal networks,
now available over internet.
By analysis and simulation experiments these studies explore the
computational function of biophysical mechanisms, such as spike
synchronization, gamma oscillations, NMDA transmission
characteristics, and activity feedback in local cortical networks
or over reciprocal projections.
The simulation models vary widely in their degree of biophysical
realism ranging from binary sparse associative memories to networks
of compartmental neurons.
List of manuscripts and abstracts, see below. Postscript versions are
available on our web pages:
http://www.informatik.uni-ulm.de/ni/mitarbeiter/FSommer/FSommernew.html
http://personal-homepages.mis.mpg.de/wenneker/index.html
Fritz and Thomas
----------------------------------------------------------------------------
Dr. Friedrich T. Sommer
Department of Neural Information Processing
University of Ulm
D-89069 Ulm
Germany
Tel. 49(731)502-4154
FAX 49(731)502-4156
FRITZ at NEURO.INFORMATIK.UNI-ULM.DE
----------------------------------------------------------------------------
____________________________________________________________________________
Dr.Thomas Wennekers
Max-Planck-Institute for Mathematics in the Sciences
Inselstrasse 22-26
04103 Leipzig
Germany
Phone: +49-341-9959-533
Fax: +49-341-9959-555
Email: Thomas.Wennekers at mis.mpg.de
____________________________________________________________________________
-----------------------------------------------------------------------------
LIST OF MANUSCRIPTS:
(1) Sommer, F.T. and Wennekers, T.
Modeling studies on the computational function of fast temporal
structure in neuronal network activity
submitted to J.Neurophysiol. (Paris)
(2) Sommer, F.T. and Wennekers, T.:
Associative memory in a pair of cortical cell groups with reciprocal
connections
Acctepted at the Computational Neuroscience Meeting CNS 2000 , Brugge,
Belgium, July 2000.
(3) Vollmer, U., Wennekers, T. and Sommer, F.T.:
Coexistence of short and long term memory in a model network of
realistic neurons
Accepted at the Computational Neuroscience Meeting CNS 2000,
Bruegge, Belgium
(4) Sommer, F.T.:
On cell assemblies in a cortical column
Neurocomputing 2000, to appear
(5) Wennekers, T. and Sommer, F.T.:
Gamma-oscillations support optimal retrieval in associative memories of
two-compartment neurons.
Neurocomputing 26-27, 573-578, 1999.
(6) Sommer, F.T. and Palm, G.:
Improved Bidirectional Retrieval of Sparse Patterns Stored by
Hebbian Learning
Neural Networks 12 (2) (1999) 281 - 297
(7) Sommer, F.T.; Wennekers, Th.; Palm, G.:
Bidirectional completion of cell assemblies in the cortex.
In: J.M.Bower (ed) Computational Neuroscience: Trends in
Research. Plenum Press, New York, 1998.
(8) Sommer, F.T. and Palm, G.:
Bidirectional Retrieval from Associative Memory
in Advances in Neural Information Processing Systems 10, MIT
Press, Cambridge, MA (1998) 675 - 681
=========================================================================================
ABSTRACTS:
-----------------------------------------------------------------------------------------
(1)
Modeling studies on the computational function of fast temporal
structure in cortical circuit activity
Friedrich T. Sommer and Thomas Wennekers
The interplay between experiments and theoretical approaches can
support the exploration of the function of neuronal circuits in the cortex.
In this review we exemplify such a proceeding with a study on the
functional role of spike timing and gamma-oscillations, and their
relation to associative activity feedback through cortex-intrinsical synaptic
connections. We first discern the theoretical approaches in general that have
been most important in brain research, in particular, those approaches focusing
on the biophysical, the functional, and the computational aspect. It is
demonstrated how results from computational model studies on
different levels of abstraction can constrain the functionality of associative
memory expected in real cortical neuronal circuits.
These constraints will be used to implement a computational model of associative
memory on the base of biophysically elaborated compartmental neurons developed
by Pinsky and Rinzel \cite{AN:PinskyRinzel94}. We run simulation experiments for
two network architectures: a single interconnected pool of cells (say a cortical
column), and two such reciprocally connected pools.
In our biophysical model individual cell populations correspond to entities
formed by
Hebbian coincidence learning. When recalled by stimulating some cells in the
population the stored patterns are extremely quickly completed and coded by
events of synchronized single spikes. These fast associations are executed
with an efficiency comparable to optimally tuned technical associative
networks. The maximum repetition frequency for these association processes lies
in the gamma-range.
If a stimulus changes fast enough to switch between different memory patterns
within one gamma period, a single association takes place without periodic
firing of individual cells. Gamma-band firing and phase locking are therefore
not primary coding features. They appear, however, with tonic stimulation or if
feedback loops in the network provide a reverberation. The latter can
improve (clean up) the recall iteratively. In the reciprocal wiring
architecture bidirectional reverberations do not express in a rigid phase
locking between the pools. Bursting turns out as a supportive mechanism
for bidirectional associative memory.
Sommer, F.T. and Wennekers, T.
Modeling studies on the computational function of fast temporal
structure in neuronal network activity
submitted to J.Neurophysiol. (Paris)
-----------------------------------------------------------------------------------------
(2)
Associative memory in a pair of cortical cell groups with reciprocal projections
Friedrich T. Sommer and Thomas Wennekers
We examine the functional hypothesis of bidirectional associative memory in
a pair of reciprocally projecting cortical cell groups.
Our simulation model features two-compartment neurons and synaptic weights
formed by Hebbian learning of pattern pairs.
After stimulation of a learned memory in one group we recorded the network
activation. At high synaptic memory load (0.14 bit/synapse)
we varied the number of cells receiving stimulation input (input activity).
The network ``recalled'' patterns by synchronized regular gamma spiking.
Stimulated cells also expressed bursts that fascilitated the recall
with low input activity.
Performance was evaluated for one-step retrieval based on monosynaptic
transmission expressed after ca. 35ms, and for {\it bidirectional retrieval}
involving iterative activity propagation.
One-step retrieval performed comparably to the technical Willshaw model with
small input activity, but worse in other cases. In 80\% of the trials with low
one-step performance iterative retrieval improved the result.
It achieved higher overall performance after recall times of 60--260ms.
Keyword: population coding; associative memory; Hebbian synapses; reciprocal
cortical wiring
Sommer, F.T. and Wennekers, T.:
Associative memory in a pair of cortical cell groups with reciprocal
connections
Acctepted at the Computational Neuroscience Meeting CNS 2000 , Brugge,
Belgium, July 2000.
-----------------------------------------------------------------------------------------
(3)
Coexistence of short and long term memory in a model network of
realistic neurons
Urs Vollmer, Thomas Wennekers, Friedrich T. Sommer
NMDA-mediated synaptic currents are believed to influence LTP.
A recent model \cite{Lisman98} demonstrates that they can instead
support short term memory based on rhythmic spike activity.
We examine this effect in a more realistic model that uses two-compartment
neurons experiencing fatigue and also includes long-term memory by synaptic LTP.
We find that the network does support both modes of operation without
any parameter changes, but depending on the input patterns.
Short term memory
functionality might facilitate Hebbian learning through LTP
by holding a new pattern while synaptic potentiation occurs.
We also find that susceptibility of the short term memory against new input
is time-dependent and reaches a maximum around the time constant of
neuronal fatigue (200--400~ms).
This corresponds well to
the time scale of the syllabic rhythm and various psychophysical phenomena.
Keywords: Short-term memory; associative memory;
population coding; NMDA-activated channels.
Vollmer, U., Wennekers, T. and Sommer, F.T.:
Coexistence of short and long term memory in a model network of
realistic neurons
Accepted at the Computational Neuroscience Meeting CNS 2000,
Bruegge, Belgium
-----------------------------------------------------------------------------------------
(4)
On cell assemblies in a cortical column
Friedrich T. Sommer
Recent experimental evidence for temporal coding of cortical cell populations
\cite{AN:Riehleetal97,AN:Donoghueetal98} recurs to Hebb's classical cell assembly notion.
Here the properties of columnar cell assemblies are estimated, using the assumptions
about biological parameters of Wickens \& Miller \cite{FS:WickensMiller97}, but
extending and correcting their predictions: Not the combinatorical constraint as
they assume, but synaptic saturation and the requirement of low activation outside
the assembly limit assembly size and number.
As will be shown, i) columnar assembly processing can be still information
theoretically efficient, and ii) at efficient parameter settings several assemblies
can be ignited in a column at the same time. The feature ii) allows faster and more
flexible access to the information contained in the set of stored cell assemblies.
Keyword}s: population coding; associative memory; Hebbian synapses, columnar connectivity
Sommer, F.T.:
On cell assemblies in a cortical column
Neurocomputing 2000, to appear
-----------------------------------------------------------------------------------------
(5)
Gamma-oscillations support optimal retrieval in associative memories of
two-compartment neurons
Thomas Wennekers and Friedrich T. Sommer
Theoretical studies concerning iterative retrieval in conventional
associative memories suggest that cortical gamma-oscillations may
constitute sequences of fast associative processes each restricted
to a single period. By providing a rhythmic threshold modulation suppressing
cells that are uncorrelated with a stimulus, interneurons significantly
contribute to this process. This hypothesis is tested in the
present paper utilizing a network of two-compartment model neurons
developed by Pinsky and Rinzel. It is shown that gamma-oscillations
can simultaneously support an optimal speed for single pattern
retrieval, an optimal repetition frequency for consecutive retrieval
processes, and a very high memory capacity.
Keywords: gamma-oscillations; threshold control; associative memory
Wennekers, T. and Sommer, F.T.:
Gamma-oscillations support optimal retrieval in associative memories of
two-compartment neurons.
Neurocomputing 26-27, 573-578, 1999.
-----------------------------------------------------------------------------------------
(6)
Improved Bidirectional Retrieval of Sparse Patterns Stored
by Hebbian Learning
Friedrich T. Sommer and Guenther Palm
The Willshaw model is asymptotically the most efficient neural associative memory (NAM),
but its finite version is hampered by high retrieval errors. Iterative retrieval
has been proposed in a large number of different models to improve performance in
auto-association tasks.
In this paper bidirectional retrieval for the hetero-associative memory task is
considered:
We define information efficiency as a general performance measure for bidirectional
associative memory (BAM) and determine its asymptotic bound for the bidirectional
Willshaw model. For the finite Willshaw model an efficient new bidirectional retrieval
strategy is proposed, the appropriate combinatorial model analysis is derived, and
implications of the proposed sparse BAM for applications and brain theory are discussed.
The distribution of the dendritic sum in the finite Willshaw model given by
\citet{FS:Buckingham92} allows no fast numerical evaluation.
We derive a combinatorial formula with a highly reduced evaluation time that is
used in the improved error analysis of the basic model and for estimation of
the retrieval error in the naive model extension where bidirectional retrieval is
employed in the hetero-associative Willshaw model.
The analysis rules out the naive BAM extension as a promising improvement.
A new bidirectional retrieval algorithm -- called {\em crosswise bidirectional} (CB)
retrieval -- is presented. The cross talk error is significantly reduced without
employing more complex learning procedures or dummy augmentation in the pattern
coding as proposed in other refined BAM models \citep{FS:Wangetal90,FS:Leungetal95}.
The improved performance of CB retrieval is shown by a combinatorial analysis of the
first step and by simulation experiments: It allows very efficient hetero-associative
mapping as well as auto-associative completion for sparse patterns -- the experimentally
achieved information efficiency is close to the asymptotic bound.
The different retrieval methods in hetero-associative Willshaw matrix are discussed
as Boolean linear optimization problems.
The improved BAM model opens interesting new perspectives, for instance, in
Information Retrieval it allows efficient data access providing segmentation of
ambiguous user input, relevance feedback and relevance ranking.
Finally, we discuss BAM models as functional model for reciprocal
cortico-cortical pathways, and the implication of this for a more flexible version
of Hebbian cell-assemblies.
Keywords: Bidirectional associative memory, Hebbian learning, iterative
retrieval, combinatorial analysis, cell-assemblies, neural information retrieval
(6) Sommer, F.T. and Palm, GT.:
Improved Bidirectional Retrieval of Sparse Patterns Stored by
Hebbian Learning
Neural Networks 12 (2) (1999) 281 - 297
-----------------------------------------------------------------------------------------
(7)
Bidirectional Completion of Cell Assemblies in the Cortex
Friedrich T. Sommer T. Wennekers and G. Palm
Reciprocal pathways are presumedly the dominant wiring organization
for cortico-cortical long range projections\refnote{\cite{AN:FellemanVanEssen91}}.
This paper examines the hypothesis that synaptic modification and
activation flow in a reciprocal cortico-cortical pathway correspond to
learning and retrieval in a bidirectional associative memory (BAM):
Unidirectional activation flow may provide the fast estimation of
stored information, whereas bidirectional activation flow might establish
an improved recall mode. The idea is tested in a network of binary neurons
where pairs of sparse memory patterns have been stored in bidirectional
synapses by fast Hebbian learning (Willshaw model). We assume that cortical
long-range connections shall be efficiently used, i.e., in many different
hetero-associative projections corresponding in technical terms to a high
memory load. While the straight-forward BAM extension of the Willshaw model
does not improve the performance at high memory load, a new bidirectional
recall method (CB-retrieval) is proposed accessing patterns with highly
improved fault tolerance and also allowing segmentation of ambiguous input.
The improved performance is demonstrated in simulations. The consequences and
predictions of such a cortico-cortical pathway model are discussed. A brief outline
of the relations between a theory of modular BAM operation and common ideas about
cell assemblies is given.
Sommer, F.T.; Wennekers, Th.; Palm, G.:
Bidirectional completion of cell assemblies in the cortex.
In: J.M.Bower (ed) Computational Neuroscience: Trends in
Research. Plenum Press, New York, 1998.
-----------------------------------------------------------------------------------------
(8)
Bidirectional Retrieval from Associative Memory
Friedrich T. Sommer and G. Palm
Similarity based fault tolerant retrieval in neural associative memories (NAM) has
not lead to wiedespread applications. A drawback of the efficient Willshaw model
for sparse patterns \cite{FS:Steinbuch61,FS:Willshaw69}, is that the high asymptotic
information capacity is of little practical use because of high cross talk noise
arising in the retrieval for finite sizes. Here a new bidirectional iterative
retrieval method for the Willshaw model is presented, called crosswise bidirectional
(CB) retrieval, providing enhanced performance. We discuss its asymptotic capacity limit,
analyze the first step, and compare it in experiments with the Willshaw model.
Applying the very efficient CB memory model either in information retrieval systems
or as a functional model for reciprocal cortico-cortical pathways requires more
than robustness against random noise in the input: Our experiments show also the
segmentation ability of CB-retrieval with addresses containing the superposition
of pattens, provided even at high memory load.
Sommer, F.T. and Palm, G.:
Bidirectional Retrieval from Associative Memory
in Advances in Neural Information Processing Systems 10, MIT
Press, Cambridge, MA (1998) 675 - 681
-----------------------------------------------------------------------------------------
=========================================================================================
More information about the Connectionists
mailing list