Distributed Representations

[Bo Xu]

----------------------------Original message----------------------------
Two days ago I mentioned (strict) local representation, binary distributed
representation, and analog distributed representation.
 
As an attempt to answer Ali Minai's question, I try to give my understandings
on representations as follows:
 
(1). In my opinion, the key points underlying the definitions of representa-
tions are the correspondences between the objects (or items, entities, etc.)
to be represented by the units (or nodes, neurons, etc.) of the network and
the units.  The objects can be classified according to the properties they
have.  More than one object probably can possess the same property.  In this
case, these objects should be classified into the same group with this
property.  The units can represent different properties of the objects, or
different objects within the same property group.
     As mentioned in my mail two days ago, there are four kinds of
correspondences existed for the relationships between the objects and units:
one-to-one, multiple-to-one, one-to-multiple, and multiple-to-multiple.
     If we define the (strict) local representation as the one that represents
the one-to-one correspondence only, then all the other three correspondences
can be called distributed representations.
     However, since there are three different correspondences in distributed
representation, the word "distributed representation" will probably be a too
broad or too general concept if we try to use one definition "distributed
representation" to refer to all the three correspondences.  It is perhaps
due to this too general word or concept that brought about the confusion on the
advantages and disadvantages of local representation vs. distributed
representation.
 
(2). In an attempt to clarify these confusions, I think it is necessary
to give more specific definitions to all these four different correspondences.
     Followings are my attempt to define these representations:
 
Local Representation ---- The one-to-one correspondence in which each object
      is represented by one unit, and each unit represents only one object.
      Units in local representation always take binary values.
 
Binary Distributed Representation ---- The one-to-multiple correspondence
      in which each object is represented by multiple units and each unit
      is employed to represent only one object.  The unit takes only binary
      values here because it represents only one object, there is no need
      for it to take analog values.
 
Analog Distributed Representation ---- The multiple-to-one correspondence
      in which multiple objects with the same property are represented by
      one unit and each unit represents multiple objects with the same
      property only.  Here the unit takes different analog values for
      different objects within this property group. Different analog
      values are used to differentiate these different objects within the
      same property group.
 
Mixed Distributed Representation ---- The multiple-to-multiple correspondence
       in which multiple objects of multiple properties are represented by
       one unit and each unit represents multiple objects with multiple
       properties. Here, the units take either binary or analog values
       depending on the properties and the object they represent.
 
 
I am not sure whether the above definitions clarify these concepts and
reduce the confusions on these problems or not.
 
Welcome your comments on above statements.
 
 
Bo Xu
Dept. of Physiology and Biophysics
School of Medicine
Indiana University
ITGT500 at INDYCMS.BITNET