Distributed vs Localist Representations

[Ross Gayler]

Randall Stark (rstark at aipna.edinburgh.ac.uk) writes:

>One aspect of this issue which seems implicit in much of this discussion
>is the notion that distributed representation can be considered
>a *relative* property.  Thus the "room schema" network is "distributed"
>relative to rooms, but "localist" relative to ovens.

A related point was raised by Paul Smolensky in his work on variable binding
using tensor representations.  By his definition a representation is
distributed if enitities of external interest (objects, attributes, values
or whatever) are represented as patterns across multiple units.  The point
Paul makes is that in much connectionist work the variables are localised
while the values are distributed.  That is, the set of units is typically
divided into disjoint groups that function as registers or variables.
Each variable is able to hold a pattern of activations that is a distributed
value.

He proposed a mechanism in which the variables are not disjoint sets of units
but instead are patterns that are bound to the patterns representing values.
Using this scheme a binding of a variable with a value is itself represented
as a pattern distributed over units and multiple bindings can be simultaneously
represented on the same units.  The nice point about this is that it puts
variables and values on an equal footing, they are both patterns.  In fact the
system does not need to distinguish between them from a processing perspective.
Whether something is a variable or a value is a question of how it is used,
not how it is represented or implemented.

Ross Gayler
ross at psych.psy.uq.oz.au