Connectionist symbol processing: any progress?

[Lev Goldfarb]

On Sun, 16 Aug 1998, Tony Plate wrote:

> One of things that has recently renewed my interest in the
> idea of using distributed representations for processing
> complex information was finding out about Latent Semantic
> Analysis/Indexing (LSA/LSI) at NIPS*97.  LSA is a method
> for taking a large corpus of text and constructing vector
> representations for words in such a way that similar words
> are represented by similar vectors.  LSA works by
> representing a word by its context (harkenning back to a
> comment I recently saw attributed to Firth 1957: "You shall
> know a word by the company it keeps" :-), and then reducing
> the dimensionality of the context using singular value
> decomposition (SVD) (v. closely related to principal
> component analysis (PCA)).  The vectors constructed by LSA
> can be of any size, but it seems that moderately high
> dimensions work best: 100 to 300 elements.

In connection with the above, in my Ph.D. (published as "A new approach to
pattern recognition", in Progress in Pattern recognition 2, ed. Kanal and
Rosenfeld, North-Holland, 1985, pp. 241-402) I have proposed to replace
the PATTERN RECOGNITION PROBLEM formulated in an input space with a
distance measure defined on it by the corresponding problem in a UNIQUELY
constructed pseudo-Euclidean vector space (through a uniquely constructed
isometric, i.e. distance preserving, embedding of the training set, using,
of course SVD of the distance matrix). The classical recognition
techniques can then be generalized to the pseudo-Euclidean space and the
PATTERN RECOGNITION PROBLEM can then be solved more efficiently than in a
general distance space setting. The model is OK, IF YOU HAVE THE RIGHT
DISTANCE MEASURE, i.e. if you have the distance measure that capture the
CLASS representation and therefore provides a good separation of the class
from its complement. 

However, in general, WHO WILL GIVE YOU THE "RIGHT" DISTANCE MEASURE? I now
believe that the construction of the "right" distance measure is a more
basic, INDUCTIVE LEARNING, PROBLEM. In a classical vector space setting,
this problem is obscured because of the rigidity of the representation
space (and, as I have mentioned earlier, because of the resulting
uniqueness of the metric), which apparently has not raised any
substantiated suspicions in non-cognitive sciences. I strongly believe
that this is due to the fact that the classical measurement processes are
based on the concept of number and therefore as long as we rely on such
measurement processes we are back where we started from--vector space
representation.
 

On Sun, 16 Aug 1998, Mitsu Hadeishi wrote:
 
> It is quite often possible to describe one representation in terms of another; symbolic in
> terms of numbers, and vice-versa.  What does this prove?  You can say numbers are an
> alphabet with only one letter; I can describe alphabets with numbers, too.
> 
> The real question is, which representation is natural for any given problem.  Obviously
> symbolic representations have value and are parsimonious for certain problem domains, or
> they wouldn't have evolved in nature.  But to say your discovery, great as it might be, is
> the only "natural" representation seems rather strange.  Clearly, mechanics can be
> described rather elegantly using numbers, and there are lots of beautiful symmetries and
> so forth using that description.  I am willing to believe other descriptions may be better
> for other situations, but I do not believe that it is reasonable to say that one can be
> certain that any given representation is *clearly* more natural than another.  It depends
> on the situation.  Symbolic representations have evolved, but so have numeric
> representations.  They have different applications, and you can transform between them.
> Is one fundamentally "better" than another?  Maybe better for this or that problem, but I
> do not believe it is reasonable to say they are better in some absolute sense.
> 
> I am a "representation agnostic."

(Mitsu, my apologies for the paragraph in italics in my last message: I
didn't intend to "shout".) 

Concluding my brief discussion of the "connectionist symbol processing", I
would like to say that I'm not at all a "representation agnostic".
Moreover, I believe that the above "agnostic" position is a defensive
mechanism that the mind has developed in the face of the mess that has
been created out of the representation issues during the last 40 years. 
During this time, with the full emergence of computers, on the one hand,
the role of non-numeric representations has begun to increase (see, for
example, "Forms of Representation", ed. Donald Peterson, Intellect Books,
1996) and, at the same time, partly due to the disproportionate and
inappropriate influence of the computability theory (again, related to the
former), the concept of representation became relativized, as Mitsu so
succinctly and quite representatively articulated above and throughout the
entire discussion.

Computability theory (and, ironically, the entire logic) has not dealt
with the representational issues, because, basically, it has ignored the
nature of intelligent computational processes, and thus, for example, the
central, I believe, issue of how to construct the inductive class
representation has not been addressed within it. 

My purpose for participating in this brief discussion (spread over the
several messages) has been to strongly urge both theoretical and applied
cognitive scientists to take the representation issue much more seriously
and treat it with all the respect one can muster, i.e. to assume that the
input, or representation, space is all we have and all we will ever have,
and, as the mathematical (not logical) tradition of the past several
thousand years strongly suggests, the operations of the representation
space "make"  this space. All other operations not related to the original
space operations become then essentially invisible. 

For us, this path leads (unfortunately, very slowly) to a considerably
more "non-numeric"  mathematics that has been historically the case so
far, and, at the same time, it inevitably leads to the "symbolic", or
inductive, measurement processes, in which the outcome of the measurement
process is not a number but a structured entity which we call "struct".
Such measurement processes appear to be far-reaching generalizations of
the classical, or numeric, measurement processes.  

Best regards and cheers,
                         Lev