Connectionist symbol processing: any progress?

[Rob.Callan@solent.ac.uk]

Dear Bryan

I was interested to read your reponse to Tony Plate's message:

"Tony Plate's response is interesting and I, for one, will have to give
it some thought.  I am not certain that

> concepts, respectively.  For example, one can build a distributed
> representation for a shape configuration#33 of "circle above
> triangle" as: config33 = vertical + circle + triangle > +
> ontop*circle +
>   below*triangle > > By using an appropriate multiplication
>   operation (I used > circular, or wrapped, convolution), the
>   reduced > representation of the compositional concept (e.g.,
>   config33) > has the same dimension as its components, and can
>   readily be > used as a component in other higher-level relations.
>   Quite

is inherently different from a spatial approach and, hence, a localist
approach itself.  You need to have enough dimensionality to represent
the key features as well as enough to multiply them out by the key
relational features -- quite a few dimensions, even if some of that
dimensionality is pushed off into numerical precision..."

I think this point "You need to have enough dimensionality to represent the
key features"  has often been overlooked. I am speaking in particular about
RAAM's of which I have most experience. One of the great attractions of
reduced representations is their potential to be used in holistic
processes. However, it appears that the greater the 'reduction' the harder
it is for holistic processing. Boden & Niklasson (1995) showed that for a
set of tree structures encoded with a RAAM, the structure was maintained
but the influence of constituents was not necessarily available for
holistic processing. About 3 years ago we developed (S)RAAM (simplified
RAAM - see Callan & Palmer-Brown 1997) which uses PCA and a recursive
procedure to produce matrices that simulate the first and second
weight-layers of  a RAAM. Unlike RAAMs, (S)RAAMs cannot reduce the
'representational width' beyond the redundancy present in the training set.
One of my student's (John Flackett) has repeated Boden and Niklasson's
experminent with (S)RAAM and results (unsurprisingly) show a significant
improvement over their RAAM. The action of the recursive process also
appears to impose a weighting of the constituents but this is to be further
explored. The weighting may prove useful for some tasks (e.g., planning)
and so is not necessarily a bad thing for all forms of holistic processing.

It is also clear to me that (S)RAAMs have no capability to exhibit 'strong
systematicity' and I believe the same is true of RAAMs. I am not ruling out
the possibility of strong systematic behavior when RAAMs etc., are used in
a modular system (some impressive results were demonstrated by  Niklasson &
Sharkey 1997). For the general reading list, two recent papers that offer
some interesting discussion are:

Steven Phillips - examines systematicity in feedforwad and recurrent
networks - ref below

James Hammerton - general discussion and definition of holistic computation
- ref below

Callan R, Plamer-Brown D (1997). (S)RAAM: An Analytical Technique for Fast
and Reliable Derivation pf Connectionist Symbol structure Representations.
Connection Science, Vol 9, No 2.

BodenM, Niklasson L (1995). Feature of Distributed Representations for
Tree-structures: A Study of RAAM. Presented at the 2nd Swedish Conference
on Connectionism. Published in Current trends in Connectionism (Niklasson &
Boden eds.) Lawrence Erlbaum Associates.

Niklasson L, sharkey N E (1997) Systematicity and generalization in
compositional connectionist representations, in G Dorffner (ed), Neural
Networks and a New artificial Intelligence. International Thomson Computer
Press.

Phillips S (1998). are feedforward and Recurrent Networks Systematic?
Analysis and Implications for a Connectinist Cognitive Architecture.
Connection Science, Vol 10, No 2.

Hammerton J (1998) Holistic Computation: Reconstructing a Muddled Concept.
Connection Science, Vol 10, No 1.