Boosting methods for regression and classification.

[Jerry.Friedman@cmis.CSIRO.AU]

                  *** Technical Report Available ***


                    Greedy Function Approximation:
                     A Gradient Boosting Machine

                         Jerome H. Friedman
                         Stanford University

                              ABSTRACT

Function approximation is viewed from the perspective of numerical
optimization in function space, rather than parameter space. A
connection is made between stagewise additive expansions and
steepest-descent minimization. A general gradient-descent "boosting"
paradigm is developed for additive expansions based on any fitting
criterion. Specific algorithms are presented for least-squares,
least-absolute-deviation, and Huber-M loss functions for regression,
and multi-class logistic likelihood for classification. Special
enhancements are derived for the particular case where the individual
additive components are decision trees, and tools for interpreting
such "TreeBoost" models are presented. Gradient boosting of decision
trees produces competitive, highly robust, interpretable procedures
for regression and classification, especially appropriate for mining
less than clean data. Connections between this approach and the
boosting methods of Freund and Shapire 1996, and Friedman, Hastie, and
Tibshirani 1998 are discussed.


Available from: "http://www-stat.stanford.edu/~jhf/ftp/trebst.ps"