New paper

[David Wolpert]

	******* DO NOT FORWARD TO OTHER LISTS *****************



The following paper has been placed in neuroprose:



A RIGOROUS INVESTIGATION OF "EVIDENCE" AND "OCCAM FACTORS" IN 
BAYESIAN REASONING


by David H. Wolpert


Abstract: This paper first reviews the reasoning behind the Bayesian 
"evidence" procedure for setting parameters in the probability 
distributions involved in inductive inference. This paper then 
proves that the evidence procedure is incorrect. More precisely, 
this paper proves that the assumptions going into the evidence 
procedure do not, as claimed, "let the data determine the distributions". 
Instead, those assumptions simply amount to an implicit replacement of 
the original distributions, containing free parameters, with new 
distributions, none of whose parameters are free. For example, as used 
by MacKay [1991] in the context of neural nets, the evidence procedure 
is a means for using the training set to determine the free parameter 
alpha in the the prior distribution P({wi}) proportional to 
exp(alpha x S), where the N wi are the N weights in the network, and S 
is the sum of the squares of those weights. As this paper proves, in 
actuality the assumptions going into MacKay's use of the evidence 
procedure do not result in a distribution P({wi}) proportional to 
exp(alpha x S), for some alpha, but rather result in a parameter-less 
distribution, P({wi}) proportional to (S) ** [-(N/2 + 1)]. This paper 
goes on to prove that if one makes the assumption of an "entropic prior" 
with unknown parameter value, in addition to the assumptions used in 
the evidence procedure, then the prior is completely fixed, but in a 
form which can not be entropic. (This calls into question the 
self-consistency of the numerous arguments purporting to derive an 
entropic prior "from first principles".) Finally, this paper goes 
on to investigate the Bayesian first-principles "proof" of Occam's 
razor involving Occam factors. This paper proves that that "proof" is 
flawed.



To retrieve this file, do the following:

unix> ftp archive.cis.ohio-state.edu
Name (archive.cis.ohio-state.edu:dhw): anonymous
Password: neuron
ftp> cd pub/neuroprose
ftp> binary
ftp> get wolpert.evidence.ps.Z
ftp> quit
unix> uncompress wolpert.evidence.ps.Z
unix> lpr wolpert.evidence.ps