Overfitting in learning discrete patterns

[Fabien Moutarde]

  Hi everybody.

 Here are some questions and comments about the overfitting experiment reported
by Charles X. Ling.

I would like to know how were the weights initialized ?
Were they taken from uniform distribution in some fixed
interval whatever the network architecture ? Which interval ?

I ask this because when we tried to provoke some overfitting
on a very simple problem, we realized that the initial amplitude of
weights is a crucial factor : if you begin learning
with some neurons already in their non linear regime somewhere
in learning space, then the initial function realized by the
network is not smooth, and the irregularities are likely
to remain between learning points and to produce overfitting.
This implies that the bigger the network, the lower the initial 
weights should be. 
Ling's problem is a discrete value one, and the above remarks were
related to a continuous function approximation, however
I think it is possible that his results are more a consequence
of his initialization procedure than of the training itself.

  Any further comments, and details on the experiment ?

    Fabien.



 
+===========================================+
|                                           |
| Fabien Moutarde                           |      +---------------+
| Alcatel Alsthom Recherche                 |      | A L C A T E L | 
| Route de Nozay                            |      +---------------+
| 91460 Marcoussis                          |      | A L S T H O M |
| FRANCE                                    |      +===============+
|                                           |          RECHERCHE  
| tel: 33-1-64.49.16.98                     |       
| fax: 33-1-64.49.06.95                     |       
| e-mail: moutarde at aar.alcatel-alsthom.fr   |      
|                                           |   
+===========================================+