Subtractive network design

[Hans Henrik Thodberg]

 Scott_Fahlman writes (Mon Nov 18)
 
   "A subtractive scheme can also lead to a network of about the right
   complexity, and you cite a couple of excellent studies that demonstrate
   this.  But I don't see why these should be better than additive methods"

Well, I agree that what is needed is comparative studies of additive and
subtractive methods, so if anybody out there has this, please post it!

Meanwhile, I believe that one can get some understanding by
appealing to pictures and analogies:

My general picture of the mechanisms of a subtractive network design is the
following: A network which has learned the training data and is too large 
is still rather uncontrained. The network is flexible towards rearranging 
its internal representations in response to some external pressure. This "poly-
morphic soup" is now subjected to the pruning. My favourite pruning technique 
is brute and efficient (but also time-consuming). It removes one connection 
at a time tentatively. If the error after some retrinign is no worse than 
before (apart from a small allowable error increase), the connection is 
considered pruned. Otherwise the network state prior to the removal is 
reestablished.

This gradually forces the network to collaps into simpler networks. It is
like an annealing process. By approaching the minimal solution from "above",
i.e. from the more complicated networks, one is more likely to find the 
optimal network, since one is guided by the hopefully wide basin of attraction. 
Since the basin is not covering everything, one must train and prune with new 
initial weights/topology (see Int. Journ. Neur. Syst. for more details).

A additive method does not have this nice pool of resources cooperating
in a plastic manner. Suppose that you were to develop the first car in the
world by additive methods. Adding one wheel at a time would not lead you to
the Honda Civic, because a one- or two-wheeled Civic would be as bad as a 
zero-wheeled. However a twenty-wheeled polymorphic monster-car could be 
pruned to a Civic. 

Another analogy to subtractive methods is the brainstorming. 
Out of a wild discussion, where many complicated ideas are flying through the
room, can suddenly emerge a simple and beatiful solution to the problem.
The additive approach would correspond to a strict analytical incremental 
thought process.

I view the reluctance towards subtractive methods as part of the 
old discussion between AI and connectionism. We (certainly in Denmark) were 
brought up with LEGO-bricks, learning that everything can be contructed 
from its parts. We are not used to projecting solutions out of chaos or 
complexity. We like to be in control, and it seems like a waste to through 
away part of your model. 

------------------------------------------------------------------
Hans Henrik Thodberg                Email: thodberg at nn.meatre.dk
Danish Meat Research Institute      Phone: (+45) 42 36 12 00
Maglegaardsvej 2, Postboks 57       Fax:   (+45) 42 36 48 36
DK-4000 Roskilde, Denmark
------------------------------------------------------------------