No subject

[Ken Johnson]

In response to the debate on Distributed vs. Local Representations.....

Everyone in this field has a view point colored by their academic background.  So here is mine.

The fundamental issues associated with information representations was in many ways dealt with by Shannon. If we consider neural activity value spread across a vector of neurons a resource then one can conjur up images of 'neural representation bandwidth'..  The usage of this bandwidth is determined by noise power, integration time, and a bunch of other signal/system properties.  In general, given a finite amount of noise, and a given number of neurons, a distributed representation is more 'efficient' than a local representation.  In this case efficiency would be the ability to pack more codes into a given coding scheme or 'code space'.

An equally important issue is that of neural code processing. Representation of the information is more or less useless without a processing system capable of transforming neural codes from one form to another form in a hierarchical system such as the brain.  In this case we have Ashby's Law of Requisite Variety.  I can't find my copy of the reference, but its by John Porter circa 1983-1987.  In this work he goes into a discussion and analysis wherein he shows that a neural system's capacity for control and information processing cannot exceed its capabilities as a communication channel.  Hence, he throws that ultimate capabilities of a neural processor back to Shannon's description.

In addition to these philosophical and theoretical reasons for my preference of distributed codes I've got reams of data from Neocognitron simulations which clearly show that proper operation of the machine REQUIRES distributed codes that use the representation space wisely.  References to this work can be found in the Proceedings of the IJCNN 1988 in SanDiego, 1988 in Boston, and 1990 Washington.  What we found was an important dichotomy.  Neural codes for similar features had to be close together in codes space to be grouped into new features by higher level processes.  Without this characteristic pattern classification would not group very similar patterns together.  On the other hand, differences between patterns had to be far apart in representation space in order to be discriminated accurately.  Hence, we see proper code organization required similar codes be close while different codes needed to be far apart.  One should expect this property if the goal of the system !
is representationaly richness rat

The above arguments lead me to believe that neural coding is one of the fundamental issues that needs to be invesgtigated more throughly.  Correct utilization of neural representation bandwidth is something we don't use very well.  In fact, I'll state that we don't use it at all.  The notion of bandwidth immediately suggests time as a representational dimension we don't use.  Feedforward systems don't use time to increase the bandwidth of a representation - they are static. Almost all feedback and recurrent systems we see are allowed to 'reach equilibrium' before the 'neural code' is interpreted.  Thus, the code is again static.  Why not use state trajectories, and temporal modulation as means of enhancing neural representation bandwidth and  increasing the processing capabilities of neural systems?

Ken Johnson
kbj at risc.rockwell.com