two technical reports

[Nici Schraudolph]

Dear colleagues,

the following two papers are available by anonymous ftp:


Technical Report IDSIA-32-98
(to be presented at ICANN'98)

Slope Centering: Making Shortcut Weights Effective
--------------------------------------------------
Nicol N. Schraudolph

Shortcut connections are a popular architectural feature of multi-layer
perceptrons.  It is generally assumed that by implementing a linear
sub-mapping, shortcuts assist the learning process in the remainder of
the network.  Here we find that this is not always the case: shortcut
weights may also act as distractors that slow down convergence and can
lead to inferior solutions.

This problem can be addressed with slope centering, a particular form of
gradient factor centering.  By removing the linear component of the error
signal at a hidden node, slope centering effectively decouples that node
from the shortcuts that bypass it.  This eliminates the possibility of
destructive interference from shortcut weights, and thus ensures that
the benefits of shortcut connections are fully realized.

ftp://ftp.idsia.ch/pub/nic/slope.ps.gz



Technical Report IDSIA-33-98
(submitted to NIPS*98)

Accelerated Gradient Descent by Factor-Centering Decomposition
--------------------------------------------------------------
Nicol N. Schraudolph

Gradient factor centering is a new methodology for decomposing neural
networks into biased and centered subnets which are then trained in
parallel.  The decomposition can be applied to any pattern-dependent factor
in the network's gradient, and is designed such that the subnets are more
amenable to optimization by gradient descent than the original network:
biased subnets because of their simplified architecture, centered subnets
due to a modified gradient that improves conditioning.

The architectural and algorithmic modifications mandated by this
approach include both familiar and novel elements, often in prescribed
combinations.  The framework suggests for instance that shortcut
connections -- a well-known architectural feature -- should work
best in conjunction with slope centering, a new technique described
herein.  Our benchmark experiments bear out this prediction, and show
that factor-centering decomposition can speed up learning significantly
without adversely affecting the trained network's generalization ability.

ftp://ftp.idsia.ch/pub/nic/facede.ps.gz


Best wishes,

--
    Dr. Nicol N. Schraudolph     Tel: +41-91-970-3877
    IDSIA                        Fax: +41-91-911-9839
    Corso Elvezia 36
    CH-6900 Lugano              http://www.idsia.ch/~nic/
    Switzerland