Smoothing Regularizers for PBF NN

[Thorsteinn Rognvaldsson]

New tech report available:

SMOOTHING REGULARIZERS FOR PROJECTIVE BASIS FUNCTION NETWORKS

By:

JOHN E. MOODY & THORSTEINN S. ROGNVALDSSON

Dept. of Computer Science and Engineering
Oregon Graduate Institute of Science and Technology
P.O. Box 91000 Portland, Oregon 97291-1000, U.S.A.

Emails:
moody at cse.ogi.edu
denni at cse.ogi.edu

(Direct correspondence to Prof. Moody)


---------
Abstract:

Smoothing regularizers for radial basis functions have been studied
extensively, but no general smoothing regularizers for PROJECTIVE
BASIS FUNCTIONS (PBFs), such as the widely-used sigmoidal PBFs, have 
heretofore been proposed. We derive new classes of algebraically-simple 
m:th-order smoothing regularizers for networks of projective basis  
functions.
Our simple algebraic forms enable the direct enforcement of smoothness
without the need for e.g. costly Monte Carlo integrations of the  
smoothness
functional.

We show that our regularizers are highly correlated with the
values of standard smoothness functionals, and thus suitable
for enforcing smoothness constraints onto PBF networks.

The regularizers are tested on illustrative sample problems and
compared to quadratic weight decay. The new regularizers are shown to
yield better generalization errors than weight decay when the implicit 
assumptions in the latter are wrong. Unlike weight decay, the new
regularizers distinguish between the roles of the input and output
weights and capture the interactions between them.


--------------------------------------------------
Instructions for retrieving your own personal copy:

WWW:
http://www.cse.ogi.edu/~denni/publications.html


FTP:
% ftp neural.cse.ogi.edu
(username=anonymous, password=your email)
> cd pub/neural/papers/
> get moodyRogn96.smooth_long.ps.Z
> quit
% uncompress moodyRogn96.smooth_long.ps.Z
% lpr moodyRogn96.smooth_long.ps

(assumes you have a UNIX system)