Connectionist symbol processing: any progress?

[Mitsu Hadeishi]

Lev Goldfarb wrote:

> Mitsu, I'm afraid, I failed to see what is wrong with my (quoted)
> question. First, I suggested in it that to do inductive learning properly
> ONE MUST HAVE AN EXPLICIT AND MEANINGFUL DISTANCE FUNCTION ON THE INPUT
> SPACE.

The point I am making is simply that after one has transformed the input space, two points
which begin "close together" (not infinitesimally close, but just close) may end up far
apart and vice versa.  The mapping can be degenerate, singular, etc.  Why is the metric on
the initial space, then, so important, after all these transformations?  Distance measured
in the input space may have very little correlation with distance in the output space.

Also, again, you continue to fail to address the fact that the input may be presented in
time sequence (i.e., a series of n-tuples).  What about that?  In fact the structure of
the whole thing may end up looking very much like your symbolic model.

> >                                                In a sufficiently complex
> > network, you can pretty much get any arbitrary map you like from the input space to
> > the output, and the error measure is biased by the specific nature of the training
> > set (for example), and is measured on the output of the network AFTER it has gone
> > through what amounts to an arbitrary differentiable transformation.  By this time,
> > the "metric" on the original input space can be all but destroyed.  Add recurrency
> > and you even get rid of the fixed dimensionality of the input space.  In the quote
> > above, it appears you are implying that there is some direct relationship between
> > the metric on the initial input space and the operation of the learning algorithm.
> > I do not see how this is the case.
>
> YES, INDEED, I AM STRONGLY SUGGESTING THAT THERE MUST BE A DIRECT
> CONNECTION "BETWEEN THE METRIC ON THE INITIAL INPUT SPACE AND THE
> OPERATIONS OF THE LEARNING ALGORITHM". IN OTHER WORDS, THE SET OF CURRENT
> OPERATIONS ON THE REPRESENTATION SPACE (WHICH, OF COURSE, CAN NOW BE
> DYNAMICALLY MODIFIED DURING LEARNING) SHOULD ALWAYS BE USED FOR DISTANCE
> COMPUTATION.
>
> What is the point of, first, changing the symbolic representation to the
> numeric representation, and, then, applying to this numeric representation
> "very strange", symbolic, operations? I absolutely fail to see the need
> for such an artificial contortion.

If your problem is purely symbolic you may be right, but what if it isn't?  (Also: no need
to shout.)

> > Well, what one mathematician calls natural and the other calls artificial may be
> > somewhat subject to taste as well as rational argument.  At this point one can get
> > into the realm of mathematical aesthetics or philosophy rather than hard science.
> > >From my point of view, symbolic representations can be seen as merely emergent
> > phenomena or patterns of behavior of physical feedback systems (i.e., looking at
> > cognition as essentially a bounded feedback system---bounded under normal
> > conditions, unless the system goes into seizure (explodes mathematically---well, it
> > is still bounded but it tries to explode!), of course.)  From this point of view
> > both symbols and fuzziness and every other conceptual representation are neither
> > "true" nor "real" but simply patterns which tend to be, from an
> > information-theoretic point of view, compact and useful or efficient
> > representations.  But they are built on a physical substrate of a feedback system,
> > not vice-versa.
> >
> > However, it isn't the symbol, fuzzy or not, which is ultimately general, it is the
> > feedback system, which is ultimately a physical system of course.  So, while we may
> > be convinced that your formalism is very good, this does not mean it is more
> > fundamentally powerful than a simulation approach.  It may be that your formalism is
> > in fact better for handling symbolic problems, or even problems which require a
> > mixture of fuzzy and discrete logic, etc., but what about problems which are not
> > symbolic at all?  What about problems which are both symbolic and non-symbolic (not
> > just fuzzy, but simply not symbolic in any straightforward way?)
> >
> > The fact is, intuitively it seems to me that some connectionist approach is bound to
> > be more general than a more special-purpose approach.  This does not necessarily
> > mean it will be as good or fast or easy to use as a specialized approach, such as
> > yours.  But it is not at all convincing to me that just because the input space to a
> > connectionist network looks like R(n) in some superficial way, this would imply that
> > somehow a connectionist model would be incapable of doing symbolic processing, or
> > even using your model per se.
>
> The last paragraphs betray your classical physical bias based on our
> present (incidentally vector-space based) mathematics. As you can see from
> my home page, I do not believe in it any more: we believe that the
> (inductive) symbolic representation is a more basic and much more adequate
> (evolved during the evolution) form of representation, while the numeric
> form is a very special case of the latter when the alphabet consists of a
> single letter.

It is quite often possible to describe one representation in terms of another; symbolic in
terms of numbers, and vice-versa.  What does this prove?  You can say numbers are an
alphabet with only one letter; I can describe alphabets with numbers, too.

The real question is, which representation is natural for any given problem.  Obviously
symbolic representations have value and are parsimonious for certain problem domains, or
they wouldn't have evolved in nature.  But to say your discovery, great as it might be, is
the only "natural" representation seems rather strange.  Clearly, mechanics can be
described rather elegantly using numbers, and there are lots of beautiful symmetries and
so forth using that description.  I am willing to believe other descriptions may be better
for other situations, but I do not believe that it is reasonable to say that one can be
certain that any given representation is *clearly* more natural than another.  It depends
on the situation.  Symbolic representations have evolved, but so have numeric
representations.  They have different applications, and you can transform between them.
Is one fundamentally "better" than another?  Maybe better for this or that problem, but I
do not believe it is reasonable to say they are better in some absolute sense.

I am a "representation agnostic."  I certainly am not going to say that numeric
representations are the "only" valid basis, or even that they are foundational (to me that
would be incoherent).  All representations I believe are kind of stable information points
reached as a result of dynamic feedback; in other words, they survive because they have
evolutionary value.  Whether you call this or that representation "real" or "better" to me
is a matter of application and parsimony.  The ultimate test is seeing how simple a
description of a model is in any given representation.  If the description is complex and
long, the representation is not efficient; if it is short, it isn't.  However, for
generality one might choose a less parsimonious representation so you can gain expressive
power over a greater range of models.  Whether your model is better than connectionist
models I do not know, but I do not think it is necessary to think of it as some kind of
absolute choice.  May the best representation win, as it were (it is a matter of survival
of the fittest representation.)

Mitsu


> By the way, I'm not the only one to doubt the adequacy of the classical
> form of representation. For example, here are two quotes from Erwin
> Schrodinger's book "Science and Humanism" (Cambridge Univ. Press), a
> substantial part of which is devoted to a popular explication of the
> following ideas:
>
> "The observed facts (about particles and light and all sorts of radiation
> and their mutual interaction) appear to be REPUGNANT to the classical
> ideal of continuous description in space and time."
>
> "If you envisage the development of physics in THE LAST HALF-CENTURY, you
> get the impression that the discontinuous aspect of nature has been forced
> upon us VERY MUCH AGAINST OUR WILL. We seemed to feel quite happy with the
> continuum. Max Plank was seriously frightened by the idea of a
> discontinuous exchange of energy . . ."
>
> (italics are in the original)
>
> Cheers,
>          Lev