Paper available by ftp

[Ron Meir]

FTP-host: archive.cis.ohio-state.edu
FTP-filename: /pub/neuroprose/meir.bias_variance.ps.Z 

The following technical report is available by anonymous ftp.

18 printed pages. 

------------------------------------------------------------------------
	   Bias, Variance and the Combination of Estimators;
		     The case of Linear Least Squares

				Ronny Meir
		   Department of Electrical Engineering
		 	         Technion
			        Haifa 32000
			          Israel
		  	 rmeir at ee.technioin.ac.il 

We consider the effect of combining several least squares estimators 
on the solution to a regression problem. 
Computing the exact bias and variance curves as
a function of the sample size we are able to quantitatively compare the effect
of the combination on the bias and variance separately, and thus on the
expected error which is the sum of the two. 
First, we show that by splitting the data set into several independent
parts and training each estimator on a different subset, the performance
can in some cases be significantly improved. We find three basic regions 
of interest. For a small number of noisy samples the estimation 
quality is dramatically improved by combining several independent 
estimators. For intermediate sample sizes, however, the
effect of combining estimators can in fact be deletarious, tending to increase
the bias too much. For large sample sizes both the single and 
the combined estimator approach the same limit. 
Our results are derived analytically
for the case of linear least-squares regression, and are valid for systems
of large input dimensions. A definite conclusion of our work is that 
substantial improvement in the quality of least-squares estimation is possible
by decreasing the variance at the cost of an increase in bias. This gain is
especially pronounced for small and noisy data sets. 
We stress, however, that the approach of estimator combination is not a panacea 
for constructing improved estimators and must be applied with care. 

-------------------------------------------------------------------- 
ftp instructions: 

% ftp archive.cis.ohio-state.edu
Name: anonymous
password: your full email address
ftp> cd pub/neuroprose
ftp> binary
ftp> get meir.bias_variance.ps.Z
ftp> bye
% uncompress meir.bias_variance.ps.Z
% lpr meir.bias_variance.ps