Information Capacity and Local vs Distributed

[Arun Jagota]

Dear Connectionists,
I think Information Capacity* (IC) (Abu-Mostafa, Jacques 85) is a useful
quantitative criterion for L vs D, illustrated by the following trivial 
example. 

You are given k pebbles, to be placed in k-of-n locations. 
location has pebble => `1', otherwise `0'.
IC == # distinct vectors that can be stored = C(n,k)    (n choose k)

For this e.g, its nice that the Binomial distribution quantifies IC for L vs D. 
The IC of k ~ n/2 (distributed) is by far superior. 

k = 1     ==> Local,  IC = n
k is n/2  ==> distributed, IC = C(n,n/2) is maximum
k = n-1   ==> over-distributed, IC = n

With (threshold-element) connectionist nets, the analogy holds, but the
(hidden or output layer) units [locations] are not independent. I would 
think there is scope for theory and empirical work along these lines. I 
have seen IC work on symmetric nets but even here I am unaware of work on IC 
as a function of k. I am unaware (haven't looked) of any work on FF nets.

* - IC is actually defined as log of how I have shown 
Sincerely,
Arun Jagota
jagota at cs.buffalo.edu