Connectionist symbol processing: any progress?

[Jonathan Baxter]

Forwarded message:
> From owner-neuroz at munnari.oz.au  Tue Aug 18 19:25:34 1998
> Date: Mon, 17 Aug 1998 18:09:01 -0300 (ADT)
> From: Lev Goldfarb <goldfarb at unb.ca>
> X-Sender: goldfarb at sol.sun.csd.unb.ca
> Reply-To: Lev Goldfarb <goldfarb at unb.ca>
> To: connectionists at cs.cmu.edu, inductive at unb.ca
> Subject: Re: Connectionist symbol processing: any progress? 
> In-Reply-To: <199808160830.UAA08508 at rialto.mcs.vuw.ac.nz>
> Message-Id: <Pine.SOL.3.96.980817100125.682J-100000 at sol.sun.csd.unb.ca>
> Mime-Version: 1.0
> Content-Type: TEXT/PLAIN; charset=US-ASCII
> 
> 

On Monday August 17, Lev Goldfarb wrote:


> 
> However, in general, WHO WILL GIVE YOU THE "RIGHT" DISTANCE MEASURE? I now
> believe that the construction of the "right" distance measure is a more
> basic, INDUCTIVE LEARNING, PROBLEM. In a classical vector space setting,
> this problem is obscured because of the rigidity of the representation
> space (and, as I have mentioned earlier, because of the resulting
> uniqueness of the metric), which apparently has not raised any
> substantiated suspicions in non-cognitive sciences. I strongly believe
> that this is due to the fact that the classical measurement processes are
> based on the concept of number and therefore as long as we rely on such
> measurement processes we are back where we started from--vector space
> representation.

Just to add a note on this point of "what is the right distance measure" and
"where do you get it": it is  reasonably clear that if you are faced with just 
a single learning problem finding the right distance measure is equivalent 
to the learning problem itself. After all, in a classification setting
the perfect distance measure would set the distance between examples
belonging to the same class to zero, and the distance between examples
belonging to different clesses to some positive
number. One-nearest-neighbour classification with such a distance
metric and a training set containing at least one example of each
class will have zero error. 

In contrast, in a "learning to learn" setting where a learner is faced
with a (potentially infinite) *sequence* of learning tasks one can ask
that the learner learns a distance metric that is in some sense
appropriate for all the tasks. I think it is this sort of metric that
people are thinking of when they talk about "the right distance
measure". For example, in my life I have to learn to recognize
thousands of faces, not just a single face. If I learn a distance
measure that works for just a single face (say, just distinguish my
father from everybody else) then that distance measure is unlikely to
be the "right" measure for faces; it would most likely focus on some
idiosyncratic feature of his face in order to make the
distinction and would thus be unusable for distinuishing faces that
don't possess such a feature. However, if I learn a distance measure
that works for a large variety of faces, then that distance measure is
more likely to focus on the "true" invariants of people's faces and
hence has more chance of being the "right" measure. 

Anyway, to cut a long story short, you can formalize this idea of
learning the right distance measure for a number of related tasks---I
had papers on this in NIPS and ICML last year. You can also get them
from my web page: 

http://wwwsyseng.anu.edu.au/~jon/papers/nips97.ps.gz
http://wwwsyseng.anu.edu.au/~jon/papers/icml97.ps.gz.

This idea has turned up in a number of different guises in various places
(here is a few):

Shimon Edelman. Representation, Similarity and the Chorus of
Protoypes. Minds and Machines, 5, 1995.

Oehler and Gray. Combining image compression and classification using
vector quantization. IEEE Transactions on PAMI. 17(5): 461--473. 1995.

Thrun and Mitchell. Learning one more thing. TR CS-94-184, CMU, 1994.

Cheers,

Jon

-------------
Jonathan Baxter	
Department of Systems Engineering
Research School of Information Science and Engineering
Australian National University
http://keating.anu.edu.au/~jon
Tel: +61 2 6279 8678
Fax: +61 2 6279 8688