TR: Virtual Memories and Massive Generalization

[Paul Smolensky]

           Virtual Memories and Massive Generalization
             in Connectionist Combinatorial Learning

                Olivier Brousse & Paul Smolensky
                Department of Computer Science &
                 Institute of Cognitive Science
                University of Colorado at Boulder

We report a series of experiments on connectionist learning  that
addresses  a particularly pressing set of objections on the plau-
sibility of connectionist learning as a model of human  learning.
Connectionist  models  have typically suffered from rather severe
problems of inadequate generalization (where generalizations  are
significantly  fewer  than  training  inputs) and interference of
newly learned items with previously learned items. Taking  a  cue
from  the  domains in which human learning dramatically overcomes
such problems, we see that indeed connectionist learning can  es-
cape  these  problems in *combinatorially structured domains.* In
the simple combinatorial domain of letter sequences, we find that
a  basic  connectionist learning model trained on 50 6-letter se-
quences can correctly generalize to over 10,000 novel  sequences.
We  also discover that the model exhibits over 1,000,000 *virtual
memories*: new items which, although  not  correctly  generalized,
can  be  learned in a few presentations while leaving performance
on the previously learned items intact.  Virtual memories can  be
thought  of  states  which are not harmony maxima (energy minima)
but which can become so with a  few  presentations,  without  in-
terfering with existing  harmony  maxima.   Like generalizations,
virtual memories in combinatorial memories are largely novel com-
binations of familiar subpatterns extracted from the contexts in
which they appear in the training set.  We conclude that, in com-
binatorial domains like language,  connectionist learning  is not 
as  harmful to the  empiricist position as  typical connectionist
learning experiments might suggest.

Submitted to the annual meeting of the Cognitive Science Society.

Please send requests to conn_tech_report at boulder.Colorado.EDU and
request report CU-CS-431-89.  These will be available for mailing
shortly.