Methods for improving generalization (was Re: some questions on ...)

[lange@ira.uka.de]

Dear Mr. Hicks,

in your mail to Mr. Grossman you mentioned the "Soft Weight-Sharing" algorithm
and stated, that this algorithm would do some adaption to the data.
I don't think, that this is right: Soft Weight-Sharing is just a bit more
complicated than Weight-Decay or other things (so some improvements have
been made). But Soft Weight-Sharing does not really adapt to the data,
because you have to tune the same parameters as in normal Weight-Decay:
the parameters, that are used to handle the strength of the penalty-term.
The article of Nowlan and Hinton "Simplifying Neural Networks by Soft Weight-
Sharing" does not mention a method to do this automatically - so no "real"
adaption to the data is made.

Maybe the methods of MacKay ("Bayesian Interpolation", Neural Comp. 4 (1992),
page 415-447) could be used to get a fully-automatic adaption. A combination
of this method with Weight-Decay or Soft Weight-Sharing would perhaps be
data-adaptive; but Soft Weight-Sharing alone has still a parameter, that is
not adapted by the data.

Yours,
Frank Lange