Paper : Learning theory and algebraic analysis and geometry

[Sumio Watanabe]

Dear connectionists,

I would like to announce the following paper
accepted for publication in Neural Computation. 
This paper firstly clarifies the algebraic geometrical
structure of the hierarchical learning machines. 
Questions and comments are welcome. 

+++++

Title:  
Algebraic analysis for non-identifiable learning machines.

Cite: 
http://watanabe-www.pi.titech.ac.jp/~swatanab/appendix.html

ABSTRACT:
This paper clarifies learning efficiency of a non-identifiable
learning machine such as a multi-layer neural network 
whose true parameter set is an analytic set with 
singular points. By using a concept in algebraic analysis, 
we rigorously prove that the free energy or the Bayesian 
stochastic complexity is asymptotically equal to 
$\lambda_{1}\log n -(m_{1}-1)\log\log n + $constant,  
where $\lambda_{1}$ is a rational number, $m_{1}$ is a 
natural number, and $n$ is the number of training samples. 
Also we show an algorithm to calculate $\lambda_{1}$ and 
$m_{1}$ based on the resolution of singularities in algebraic
geometry. In regular models, $2\lambda_{1}$ is equal to 
the number of parameters and $m_{1}=1$, whereas in 
non-regular models such as mutilayer networks
$2\lambda_{1}$ is not larger than the number of 
parameters and $m_{1}\geq 1$. Since the increase of 
the free energy is equal to the generalization error, the 
non-identifiable learning machines are the better models
than the regular ones if Bayesian or ensemble training is 
applied.

+++++

Sincerely, 

Dr. Sumio Watanabe,  Associate Professor
Advanced Information Processing Division
Precision and Intelligence Laboratory
Tokyo Institute of Technology
E-mail: swatanab at pi.titech.ac.jp
(Fax)+81-45-924-5018