Boosting methods for regression and classification.

[Jonathan Baxter]

> 
> 
> 
>                   *** Technical Report Available ***
> 
> 
>                     Greedy Function Approximation:
>                      A Gradient Boosting Machine
> 
>                          Jerome H. Friedman
>                          Stanford University
> 
>                               ABSTRACT
> 
> Function approximation is viewed from the perspective of numerical
> optimization in function space, rather than parameter space. A
> connection is made between stagewise additive expansions and
> steepest-descent minimization. A general gradient-descent "boosting"
> paradigm is developed for additive expansions based on any fitting
> criterion. 
> Specific algorithms are presented for least-squares,
> least-absolute-deviation, and Huber-M loss functions for regression,
> and multi-class logistic likelihood for classification. Special
> enhancements are derived for the particular case where the individual
> additive components are decision trees, and tools for interpreting
> such "TreeBoost" models are presented. Gradient boosting of decision
> trees produces competitive, highly robust, interpretable procedures
> for regression and classification, especially appropriate for mining
> less than clean data. Connections between this approach and the
> boosting methods of Freund and Shapire 1996, and Friedman, Hastie, and
> Tibshirani 1998 are discussed.
> 
> 
> Available from: "http://www-stat.stanford.edu/~jhf/ftp/trebst.ps"


There was also some discussion of the connection between boosting and
gradient descent in function space at the NIPS workshop on large
margins in December. I have put the slides of my talk on the
subject---"AnyBoost: Boosting with (almost) arbitrary cost functions
and steps"---on my web page for those who are interested
(http://syseng.anu.edu.au/~jon/anyboost.ps).


Cheers,

Jon
-------------
Jonathan Baxter	
Research Fellow
Department of Systems Engineering
Research School of Information Science and Engineering
Australian National University
http://syseng.anu.edu.au/~jon
Tel: +61 2 6279 8678
Fax: +61 2 6279 8688


> 
> 
>