Connectionist symbol processing: any progress?

[Dave_Touretzky@cs.cmu.edu]

I'd like to start a debate on the current state of connectionist
symbol processing?  Is it dead?  Or does progress continue?

A few years ago I contributed a short article on "Connectionist and
symbolic representations" to Michael Arbib's Handbook of Brain Theory
and Neural Networks (MIT Press, 1995).  In that article I explained
concepts such as coarse coding, distributed representations, and
RAAMs, and how people had managed to do elementary kinds of symbol
processing tasks in this framework.  The problem, though, was that we
did not have good techniques for dealing with structured information
in distributed form, or for doing tasks that require variable binding.
While it is possible to do these things with a connectionist network,
the result is a complex kludge that, at best, sort of works for small
problems, but offers no distinct advantages over a purely symbolic
implementation.  The cases where people had shown interesting
generalization behavior in connectionist nets involved simple
vector-based representations, without nested structures or variable
binding.

People had gotten some interesting effects with localist networks, by
doing spreading activation and a simple form of constraint
satisfaction.  A good example is the spreading activation models of
word disambiguation developed in the 1980s by Jordan Pollack and Dave
Walts, and by Gary Cottrell.  But the heuristic reasoning enabled by
spreading activation models is extremely limited.  This approach does
not create new structure on the fly, or deal with structured
representations or variable binding.  Those localist networks that did
attempt to implement variable binding did so in a discrete, symbolic
way that did not advance the parallel constraint
satisfaction/heuristic reasoning agenda of earlier spreading
activation research.

So I concluded that connectionist symbol processing had reached a
plateau, and further progress would have to await some revolutionary
new insight about representations.  The last really significant work
in the area was, in my opinion, Tony Plate's holographic reduced
representations, which offered a glimpse of how structured information
might be plausibly manipulated in distributed form.  (Tony received an
IJCAI-91 best paper award for this work.  For some reason, the journal
version did not appear until 1995.)  But further incremental progress
did not seem possible.

People still do cognitive modeling using connectionist networks.  And
there is some good work out there.  One of my favorite examples is
David Plaut's use of attractor neural networks to model deep and
surface dyslexia -- an area pioneered by Geoff Hinton and Tim
Shallice.  But like most connectionist cognitive models, it relies on
a simple feature vector representation.  The problems of structured
representations and variable binding have remained unsolved.  No one
is trying to build distributed connectionist reasoning systems any
more, like the connectionist production system I built with Geoff
Hinton, or Mark Derthick's microKLONE.

Today, Michael Arbib is working on the second edition of his handbook,
and I've been asked to update my article on connectionist symbol
processing.  Is it time to write an obituary for a research path that
expired because the problems were too hard for the tools available?
Or are there important new developments to report?

I'd love to hear some good news.

-- Dave Touretzky


References:

Arbib, M. A. (ed) (1995) Handbook of Brain Theory and Neural Networks.
Cambridge, MA: MIT Press.

Plaut, D. C., McClelland, J. L., Seidenberg, M. S., and Patterson,
K. (1996) Understandig normal and impaired word reading: computational
principles in quasi-regular domains.  Psychological Review,
103(1):56-115.

Plate, T. A. (1995) Holographic reduced representations.  IEEE
Transactions on Neural Networks, 6(3):623.

Touretzky, D. S. and Hinton, G. E.  (1988) A distributed connectionist
production system.  Cognitive Science, vol. 12, number 3, pp. 423-466.

Touretzky, D. S. (1995) Connectionist and symbolic representations.
In M.  A. Arbib (ed.), Handbook of Brain Theory and Neural Networks,
pp. 243-247.  MIT Press.