distributed vs. local encoding schemes

[crr@shum.huji.ac.il]

Terry Sejnowski mentioned the kinds of hidden units that we found in NETtalk.
As for the input/output representations, we ran a number of 
experiments using both local (one unit per letter/phoneme, but
more than one unit on per window) and distributed 
representations (more than one unit on per letter/phoneme).
Learning times are generally faster with distributed representations simply 
because the net inputs and resulting error gradients are larger.
(However it might be possible to boost the learning rate for the
local representation to match the distributed one.  I don't know if
this would affect generalization or not since I didn't try it.)
Using a representation that "makes sense" for the particular domain
(such as using an articulatory feature code for the phonemes -- or is this 
local because the units represent features?)
also leds to faster learning, and is more resistant to damage than
a "random" encoding of the phonemes.

Charlie Rosenberg