"How Good Were Those Probability Predictions?" -- Paper Available

[David Rosen]

Announcing the following paper available in the neuroprose archive:

		How Good Were Those Probability Predictions?
	     The Expected Recommendation Loss (ERL) Scoring Rule

			       David B. Rosen

      To appear in: Maximum Entropy and Bayesian Methods.  (Proceedings
	 of the Thirteenth International Workshop, August 1993.)  G.
     Heidbreder, ed.  Kluwer, Dordrecht, The Netherlands, 1996.  8 pages.

We present a new way to choose an appropriate scoring rule for evaluating the
performance of a "soft classifier", i.e.  of a supplier of predicted
(inferred/estimated/learned/guessed) probabilities.  A scoring rule
(probability loss function) is a function of a single such prediction and the
corresponding outcome event (true class); its expectation over the data space
is the generalization performance of ultimate interest, while its sum or
average over some benchmark test data set is an empirical performance measure.

A user of probability predictions can apply his own decision threshold,
preferring to err on one side, for example, to the extent that the
consequences of an erroneous decision are more severe on the other side; this
process is the subject of decision theory/analysis.  We are not able to
specify in advance, with certainty, these relative consequences, i.e.  the
user's cost matrix (indexed by decision and outcome event) defining his
decision-making problem.  So we represent this uncertainty itself by a
distribution, from which we think of the cost matrix as being drawn.
Specifying this distribution determines a uniquely appropriate scoring rule.
We can interpret and characterize common scoring rules, such as the
logarithmic (cross-entropy), quadratic (squared error or Brier), and the
"0-1" misclassification score, as representing different assumptions about
the probability that the predictions will be used in various decision-making
problems.  We discuss the connection to the theory of proper (truth- or
honesty-rewarding) scoring rules.

     PostScript and plain-text versions are available via this Web page:
	       http://www.scs.unr.edu/~cbmr/people/rosen/erl/

    The paper is in Jordan Pollack's NEUROPROSE anonymous ftp archive as:
   ftp://archive.cis.ohio-state.edu/pub/neuroprose/rosen.exp-rec-loss.ps.Z
      (This supersedes an unannounced early version rosen.scoring.ps.Z)

		       Hardcopies cannot be provided.
--
David B Rosen  <rosen at unr.edu> OR <d.rosen at ieee.org>
New York Medical College