Subtractive network design

[Hans Henrik Thodberg]

I would like a discussion on the virtue of
subtractive versus additive methods in design of neural networks.

It is widely accepted that if several networks process 
the training data correctly, the smallest of them will 
generalise best. The design problem is therefore to find 
these mimimal nets.

Many workers have chosen to construct the networks by adding nodes or
weights until the training data is processed correctly (cascade 
correlation, adding hidden units during training, meiosis). This
philosophy is natural in our culture. We are used to custruct
things by pieces.

I would like to advocate an alternative method. One trains a (too)
large network, and then SUBTRACTS nodes or weights (while retraining)
until the network starts to fail to process the training data correctly.

Neural networks are powerful because they can form global or
distributed representations of a domain. The global structures  
are more economic, i.e they use fewer weights, and therefore 
generalise better. 

My point is that subtractive shemes are more likely to find 
these global descriptions. These structures so to speek condense out of
the more complicated structures under the force of subtraction.

I would like to hear your opinion on this claim! 

I give here some references on subtractive schemes:

Y.Le Cun, J.S.Denker and S.A.Solla, "Optimal Brain Damage", 
NIPS 2, p.598-605
 
H.H.Thodberg, "Improving Generalization of Neural Networks through 
Pruning", Int. Journal of Neural Systems, 1, 317-326, (1991). 

A.S.Weigend, D.E.Rumelhart and B.A.Huberman, "Generalization by 
Weight Elimination with Application to Forecasting", NIPS 3, p. 
877-882. 

------------------------------------------------------------------
Hans Henrik Thodberg                Email: thodberg at nn.meatre.dk
Danish Meat Research Institute      Phone: (+45) 42 36 12 00
Maglegaardsvej 2, Postboks 57       Fax:   (+45) 42 36 48 36
DK-4000 Roskilde, Denmark
------------------------------------------------------------------