Distributed v Local

[Chris Lacher]

Remark related to the distributed/local representation conversation.

We've been studying expert networks lately:  discrete-time computational
networks with symbolic-level node functionality.  The digraph structure
comes from a rule base in an expert system, and the synaptic and node
functionality mimics the inferential dynamics of the ES.  The nodes
represent things like AND, NOT, NOR and EVIDENCE ACCUMULATION.  (We
announced a preprint to connectionists on this topic in January 91.)  We
have some of these things that actually reason like expert systems and can
learn from data.  Clearly these nets use "local" representation in the sense
that single components of the architecture have identifiable meaning outside
the net.  (Although, the identifiable meaning is that of a process that
makes sense at a cognitive level rather than an object or attribute as in
the more common interpretation of local representation.)

These symbolic-level processors can be replaced with relatively small
networks of sub-symbolic processors that do the common "output:=
squash(weighted sum of inputs)" kind of processing.  One can imagine either
feed-forward or recurrent subnets that accomplish this.

The expert networks tend to be sparsely connected, O(n), while the
subnets will generally be highly connected, O(n^2).

Wildly speculate the existence of a system containing N symbolic processors
with O(N) interconnectivity, each processor actually consisting of K
sub-symbolic processors with O(K^2) intraconnectivity, and assume there are
overall 10^10 sub-symbolic processors and 10^13 connections (about the
numbers for the cerebral cortex).  Then we have

                        NK   = 10^10   and
                        NK^2 = 10^13

which yields N = 10^7 and K = 10^3. That is, an organization of about 10^7
sparsely connected subnetworks each being a highly intraconnected network of
around 1000 sub-symbolic processors. 

Aren't the columnar structures in the CC highly connected subnets of about a
thousand neurons?

                                 --- Chris Lacher


(My coworkers include colleague Susan Hruska, recent PhD Dave Kuncicky, and
a number of current graduate students.  They should not be held responsible
for this note, however.  Susan and Dave are in Seattle at IJCNN.  I'm here
in Tallahassee trying to survive dog days.)