Identity Mappings

[Bruce Krulwich]

Geoff Hinton wrote recently:

    The potential advantage of using "encoder" networks is that the code in 
    the middle can be developed without any supervision.
  ...
    If the codes from several encoder networks are then used as the input
    vector for a "higher level" network, one can get a multilayer, modular,
    unsupervised learning procedure that should scale up better to really
    large problems.
    
This brings out a point I've wanted to discuss for a while: Are "encoder
nets" any better than, say, competitive learning (or recirculation, or maybe
GMax?) for a task such as this??  It seems to me that feed-forward I/O nets
are the wrong model for learning correlations, especially if the encodings
themselves are what is going to be used for further computation.

More generally and to the point, could it be that the success in backprop (in
applications and analysis) has resulted in stagnation by tying people to the
idea of feed-forward nets??


Bruce Krulwich
krulwich at cs.yale.edu

 
-------