Paper in Neuroprose: estimating generalization with cross-validation

[Mark Plutowski]

FTP-host: archive.cis.ohio-state.edu
FTP-filename: /pub/neuroprose/pluto.imse.ps.Z

|This is related to the generalization debate on Machine Learning List.|


By popular request, the following paper has been placed in Neuroprose.

Title: 	   "Cross-validation estimates integrated mean squared error." 

Authors:   Plutowski , M. (1,3), S. Sakata (2), H. White (2,3). 
	   (1) Computer Science and Engineering
	   (2) Economics
	   (3) Institute for Neural Computation
	   (All at UCSD).


A discussion on the Machine Learning List prompted the question 

"Have theoretical conditions been established under which 
cross-validation is justified?"  

The answer is "Yes."  The statistical literature abounds with
application-specific and model-specific demonstrations that
cross-validation is statistically accurate and precise for use as 
a real-world estimate of an ideal measure of generalization known
as Integrated Mean Squared Error (IMSE).  IMSE is the
average mean squared error, averaged over all training sets of 
a particular size.   IMSE is closely related to Prediction Risk,
(aka statistical risk) therefore such results are applicable to 
statistical risk as well (as averaged over training sets of a
particular size).  See Plutowski's thesis in Neuroprose/Thesis for
explicit relationship between IMSE and statistical risk.

This paper extends such results to apply to nonlinear regression 
in general.  Strong convergence (w.p.1) and unbiasedness are proved.

The key assumption is that the training and test data be
independent and identically distributed (i.i.d.) - therefore, 
data must be drawn from the same space (stationarity), 
independent of previously sampled datum.

Note that if training data are explicitly excluded from the
test sample, then the i.i.d. assumption does not hold,
since in this case the test sample is drawn from a space that 
depends upon (is conditioned by) particular choice of training sample.  

Therefore, the measure of generalization referred to in the raging 
debate on the Machine Learning List would not meet the conditions 
employed by the results in this paper.


Filename:  pluto.imse.ps.Z.  
Title: 	   "Cross-validation estimates integrated mean squared error." 
File size: 97K compressed, 242K uncompressed.  
	   17 single-spaced pages (8 pages of text, the remainder is a
	   mathematical appendix).

Email contact: pluto at cs.ucsd.edu.  

SUBJECT: theorems proving cross-validation is a statistically 
accurate and precise estimator of an ideal measure of generalization.
Abridged version of this appeared in NIPS 6.


= Mark Plutowski


PS:  Sorry, no hard copies available.