Connectionists: e-print available: supervised learning under concept drift

[Michael Biehl]

Dear Connectionists.
We have studied models of supervised learning under concept drift
using  methods from the statistical  physics of learning. We consider
different
drift scenarios in prototype-based classification and investigate the
influence
of drift and weight decay in layered neural networks for regression.

A corresponding e-print  is available at
https://arxiv.org/abs/2005.10531:


*Supervised Learning in the Presence of Concept DriftA modelling framework*
M. Straat, F. Abadi, Z. Kan, C. Göpfert, B. Hammer, M. Biehl

Abstract
We present a modelling framework for the investigation of supervised
learning in non-stationary environments. Specifically, we model two example
types of learning systems: prototype-based Learning Vector Quantization
(LVQ) for classification and shallow, layered neural networks for
regression tasks. We investigate so-called student teacher scenarios in
which the systems are trained from a stream of high-dimensional, labeled
data. Properties of the target task are considered to be non-stationary due
to drift processes while the training is performed. Different types of
concept drift are studied, which affect the density of example inputs only,
the target rule itself, or both. By applying methods from statistical
physics, we develop a modelling framework for the mathematical analysis of
the training dynamics in non-stationary environments.
Our results show that standard LVQ algorithms are already suitable for the
training in non-stationary environments to a certain extent. However, the
application of weight decay as an explicit mechanismof forgetting does not
improve the performance under the considered drift processes. Furthermore,
we investigate gradient-based training of layered neural networks with
sigmoidal activation functions and compare with the use of rectified linear
units (ReLU). Our findings show that the sensitivity to concept drift and
the effectiveness of weight decay differs significantly between the two
types of activation function.


------------------------------------------------------------

Michael Biehl
Bernoulli Institute for
Mathematics, Computer Science
and Artificial Intelligence
P.O. Box 407, 9700 AK Groningen
The Netherlands

Tel. +31 50 363 3997

www.cs.rug.nl/~biehl
m.biehl at rug.nl
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/connectionists/attachments/20200523/b6e55f64/attachment.html>