ACAM

[alexis%yummy.mitre.org@gateway.mitre.org]

> Doesn't Hinton's weight decay scheme help to keep the activations
> from going to their extreme values of 0 and 1?  This would seem
> to make a feed forward net more analog-like, since the sigmoids
> try to remain more linear rather than step-like.

Actually not.  Weight decay causes weights that are no longer being used
to disappear; but unless it's turned up unduely high it does not stop
you from getting out of the linear region of the nodes (which is good
since generally a network of linear nodes is rather boring from a 
computational point of view).  Weight decay does make it easier though,
to develop analog mappings with a feedforward network.