No subject

[dhw@santafe.edu]

Tony Chan writes:



>>>
I would like to raise the following question: Is there such a thing as
the best method to learn a random idea?  More precisely, given n
random (machine) learning problems, is there such a thing as the best
method for dealing with these problems that will give the best overall
performance.  There may be some best methods for dealing with some
specific types of learning problems but is there one that would deal
with any learning problem and give the best overall performance?
>>>

The answer depends on the precise way the problem is phrased. But in
general, the answer is (provably) no, at least as far as
off-training set error is concerned.

For example, if the prior distribution over target functions is
uniform, then all algorithms have the exact same average off-training
set performance. Moreover, in a broad number of contexts, it is always
true that if "things" (be they priors, training sets, or whatever) are
such that algorithm 1 will outperform algorithm 2, then one can always
set up those "things" differently, so that algorithm 2 outperforms
algorithm 1, at least as far as off-training set behavior is concerned.

Many of the results in the literature which appear to dispute this
are simply due to use of an error function which is not restricted to
being off-training set. In other words, there's always a "win" 
if you perform rationally on the training set (e.g., reproduce it
exactly, when there's no noise), if your error function gives you
points for performing rationally on the training set. In a certain
sense, this is trivial, and what's really interesting is off-training
set behavior. In any case, this automatic on-training set win is all
those aforementioned results refer to; in particular, they imply essentially
nothing concerning performance off of the training set.