Distributed Representations

[Ken Laws]

I'm not sure this is the same concept, but there were several
papers at the last IJCAI showing that neural networks worked
better than decision trees.  The reason seemed to be that
neural decisions depend on all the data all the time, whereas
local decisions use only part of the data at one time.

I've never put much stock in the military reliability claims.
A bullet through the chip or its power supply will be a real
challenge.  Noise tolerance is important, though, and I suspect
that neural systems really are more tolerant.

Terry Sejnowski's original NETtalk work has always bothered me.
He used a neural network to set up a mapping from an input
bit string to 27 output bits, if I recall.  I have never seen
a "control" experiment showing similar results for 27 separate
discriminant analyses, or for a single multivariate discriminant.
I suspect that the results would be far better.  The wonder of
the net was not that it worked so well, but that it worked
at all.

I have come to believe strongly in "coarse-coded" representations,
which are somewhat distributed.  (I have no insight as to whether
fully distributed representations might be even better.  I suspect
that their power is similar to adding quadratic and higher-order
terms to a standard statistical model.)  The real win in
coarse coding occurs if the structure of the code models
structure in the data source (or perhaps in the problem
to be solved).

                                        -- Ken Laws
-------