Subtractive network design

[John K. Kruschke]

The dichotomy between additive and subtractive schemes for modifying
network architectures is based on the notion that nodes which are not
"in" the network consume no computation or memory; i.e., what gets
added or subtracted is the *presence* of the node.  An alternative
construal is that what gets added or subtracted is not the node itself
but its *participation* in the functionality of the network.  As a
trivial example, a node can be present but not participate if all the
weights leading out of it are zero. 

Under the first construal (presence), subtractive schemes can be more
expensive to implement in hardware or software than "additive"
schemes, because the additive schemes spend nothing on nodes which
aren't there yet.  Under the second construal (functional
participation), the two schemes consume equal amounts of resources,
because all the nodes are processed all the time.  In this latter case,
arguments for or against one type of scheme must come from other
constraints; e.g., ability to generalize, learning speed, neural
plausibility, or even (gasp!) human performance. 

Architecture modification schemes can be both additive and
subtractive. For example, Kruschke and Movellan (1991) described an
algorithm in which individual nodes from a large pool of candidates
can have their functional participation gradually suppressed
(subtracted) or resurrected (added).  Other methods for manipulating
the functional participation of hidden nodes are described in the
other papers listed below. 

Kruschke, J. K., & Movellan, J. R. (1991).  Benefits of gain: Speeded 
learning and minimal hidden layers in back propagation networks.
IEEE Transactions on Systems, Man and Cybernetics, v.21, pp.273-280.

Kruschke, J. K. (1989b).  Distributed bottlenecks for improved 
generalization in back-propagation networks.  International Journal of 
Neural Networks Research and Applications, v.1, pp.187-193.

Kruschke, J. K. (1989a). Improving generalization in back-propagation 
networks with distributed bottlenecks. In:  Proceedings of the IEEE 
International Joint Conference on Neural Networks, Washington D.C. 
June 1989, v.1, pp.443-447.

Kruschke, J. K. (1988).  Creating local and distributed bottlenecks in 
hidden layers of back-propagation networks.  In: D. Touretzky, 
G. Hinton, & T. Sejnowski (eds.),  Proceedings of the 1988 
Connectionist Models Summer School, pp.120-126.  San Mateo, CA: 
Morgann Kaufmann.

-------------------------------------------------------------------
 John K. Kruschke                Asst. Prof. of Psych. & Cog. Sci.
 Dept. of Psychology            internet: kruschke at ucs.indiana.edu
 Indiana University               bitnet: kruschke at iubacs
 Bloomington, IN 47405-4201       office: (812) 855-3192
 USA                                 lab: (812) 855-9613 
===================================================================