Combining human and machine experts

[Padhraic J. Smyth]

A tangent to the discussion on combining estimators and experts
is a pointer to the statistical literature on the topic of combining
the subjective ratings of multiple human experts. This problem arises
in medical diagnosis and in remote sensing applications where it
is not uncommon to have multiple opinions and one does not know
which expert to believe. Quite a bit of work has been published in
this area, here's some pointers to a few of many papers on the topic:

J. S. Uebersax,  ``Statistical modeling of expert
ratings on medical treatment appropriateness," {\it J. Amer. Statist. Assoc.},
vol.88, no.422, pp.421--427, 1993.

A.  Agresti, ``Modelling patterns of agreement
and disagreement," {\it Statistical Methods in Medical Research},
 vol.1, pp.201--218, 1992.

S. French, "Group consensus probability distributions: a critical
 survey," in Bayesian Statistics 2, pp.183-202, Bernardo, DeGroot,
 Lindley, Smith (eds), Elsevier Science (North-Holland), 1985.
 
A. P. Dawid and A. M. Skene,
 ``Maximum likelihood estimation of observer error-rates using the EM
 algorithm," {\it Applied Statistics}, vol.28, no.1, pp.20--28, 1979.

and a paper we had at last year's NIPS:
P. Smyth, M.C. Burl, U. M. Fayyad, P. Perona, P. Baldi, `Inferring
ground truth from subjectively-labeled images of Venus,'  
to appear in NIPS 7.
(can be gotten from my home page: http://www-aig.jpl.nasa.gov/mls/home/pjs)


Its interesting to note the differences between combining algorithmic
"experts" and human experts: as a function of the input data,
the algorithms are usually deterministic while the humans are usually
non-deterministic, i.e., given the same data more than once they can produce
different estimates. Humans are certainly more difficult to model than
algorithms for combination purposes since they can "drift" over
time, be affected by non-data factors, and so forth - not implying of
course that combining algorithmic experts is necessarily easy !

I have not seen any work on combining both human
and algorithmic predictions, I would be interested if anyone knows
of such work.

Padhraic Smyth