Subtractive network design

[P.Refenes@cs.ucl.ac.uk]

reduce the generality of a network and thus improve its generalisation.
Depending on size and training times, fixed geometry networks often 
develop (near-) duplicate and/or (near-) reduntant functionality. 
Prunning techniques aim to remove this functionality from the network
and they do quite well here. There are however two problems: firstly,
these are not the only cases of increased functionality, and secondly, 
the removal of near zero connections often ignores the knock-on effects 
on generalisation due the accumulated influence that these connections 
might have. 

It is often conjectured that hidden unit size is the culprit for bad
generalisation. This is not strictly so. The true culprit is the high
degree of freedom in exploring the search space which also depends on
other parameters such as training times. The solution proposed by Scott 
Fahlman i.e. to use the cross-validation performance as an indicator of 
when to stop is not complete, because as soon as you do this the cross-
validation dataset becomes part of the training dataset (the fact that 
we are not using it for the backward pass is irrelevant). So any improvement 
in generalisation is probably due to the fact that we are using a larger
training dataset (again the fact that we are doing manually, should not
divert us). My view is that this  method should be treated as a "good 
code of professional practise" when reporting results, rather than as a
panacea.

Paul Refenes