Query on "infinite" priors

[Radford Neal]

Craig Hicks. hicks at cs.titech.ac.jp writes:

> Can we say that a uniform prior over an infinite domain exists?  For
> example, the uniform prior over all natural numbers...
> 
> I ask because a true Bayesian approach to some problems may require
> the prior to be defined.  If there is no prior, then we can't say we
> are taking a Bayesian approach.  If an infinite uniform prior does not
> exist, then we cannot take the approach that "no prior knowledge" =
> "infinite uniform prior".  I.e., it would imply that any Bayesian
> approach involving the prior MUST begin with some assumptions about
> the prior (i.e., it must be formed from a summable/integrable function).

This is a long-standing issue in Bayesian inference.  These "infinite"
priors are usually called "improper" priors, while those that can be
normalized are called "proper" priors.  

Some Bayesians like to use improper priors, as long as the posterior
turns out to be proper (which is often, but not always, the case).
Other Bayesians eschew improper priors, because strange things can
sometimes occur when you use them.  

One strangeness is that a Bayesian procedure based on an improper
prior can be "inadmissible" - ie, be uniformly worse than some other
procedure with respect to expected performance, for any state of the
world.  A famous example (Stein's paradox) is estimation of the mean
of a vector of three or more independent components having Gaussian
distributions, with the aim of minimizing the expected squared error.
The Bayesian estimate with an improper uniform prior is just the
sample mean, which turns out to be inadmissible.  In contrast Bayesian
procedures based on proper priors that are nowhere zero are always
admissible.

There should be lots of stuff on this in Bayesian textbooks, such as
Smith and Bernardo's recent book on "Bayesian Theory" (though I don't
have a copy handy to verify just what they say).

----------------------------------------------------------------------------
Radford M. Neal                                       radford at cs.toronto.edu
Dept. of Statistics and Dept. of Computer Science radford at utstat.toronto.edu
University of Toronto                     http://www.cs.toronto.edu/~radford
----------------------------------------------------------------------------