Connectionist symbol processing: any progress?

[Jamie Henderson]

Dave Touretzky writes:
>I'd like to start a debate on the current state of connectionist
>symbol processing?  Is it dead?  Or does progress continue?
..
>The problem, though, was that we
>did not have good techniques for dealing with structured information
>in distributed form, or for doing tasks that require variable binding.
>While it is possible to do these things with a connectionist network,
>the result is a complex kludge that, at best, sort of works for small
>problems, but offers no distinct advantages over a purely symbolic
>implementation.  The cases where people had shown interesting
>generalization behavior in connectionist nets involved simple
>vector-based representations, without nested structures or variable
>binding.

I just gave a paper at the COLING-ACL'98 conference, which is the main
international conference for Computational Linguistics.  The paper is on
learning to do syntactic parsing using a connectionist architecture that
extends SRNs with Temporal Synchrony Variable Binding (ala SHRUTI).  This
architecture does generalize in a structural way, with variable binding.
Crucially, the paper evaluates this learning method on a real corpus of
naturally occurring text, and gets results that approach the state of the
art in the field (which is all statistical methods these days).  I received
a surprisingly positive response to this paper.  I got comments like "I've
never taken connectionist NLP seriously, but you're playing the same game as
us".  "The game" is training and testing on large corpora of real text, not
toy domains.  The winner is the method with the lowest error rate.

I see three morals in this:
  - Connectionist approaches to processing structural information have made
	significant progress, to the point that they can now be justified on
	purely empirical/engineering grounds. 
  - Connectionist methods do solve problems that current non-connectionist
	methods have (ad-hoc independence assumptions, sparse data, etc.),
	and people working in learning know it. 
  - Connectionist NLP researchers should be using modern empirical methods,
	and they will be taken seriously if they do.

The paper is available from my web page (http://www.dcs.ex.ac.uk/~jamie/).
Below is the reference and abstract.

					- Jamie Henderson


Henderson, J. and Lane, P. (1998) A Connectionist Architecture for Learning
to Parse.  In Proceedings of the 36th Annual Meeting of the Association for
Computational Linguistics and 17th International Conference on Computational
Linguistics, University of Montreal, Canada.

Abstract:
We present a connectionist architecture and demonstrate that it can learn
syntactic parsing from a corpus of parsed text.  The architecture can
represent syntactic constituents, and can learn generalizations over
syntactic constituents, thereby addressing the sparse data problems of
previous connectionist architectures.  We apply these Simple Synchrony
Networks to mapping sequences of word tags to parse trees.  After training
on parsed samples of the Brown Corpus, the networks achieve precision and
recall on constituents that approaches that of statistical methods for this
task.
(7 pages)


-------------------------------
Dr James Henderson
Department of Computer Science
University of Exeter
Exeter EX4 4PT, U.K.
http://www.dcs.ex.ac.uk/~jamie/
jamie at dcs.ex.ac.uk
-------------------------------