Connectionists: Deep Learning in Target Space

[Fairbank, Michael H]

Dear Connectionists,

We'd like to highlight to you our new paper "Deep Learning in Target Space" at JMLR: https://jmlr.org/papers/v23/20-040.html

It offers a potentially new paradigm on how to train neural networks - to search through the space of activations of the hidden nodes (i.e. "target space") as opposed to the conventional backpropagation of the gradient in "weight space".  We argue that this stabilises the gradient descent process, a process we call "cascade untangling", meaning deeper neural networks can be trained with less training data and potentially less CPU time.  The work is an updated form of the Moving Targets algorithm by Rohwer (1990) and some follow up work, where we have fixed a lot of technical issues that existing in those earlier works.  For some quick highlights of the work, see our paper's Figs 2 and 4. Results are included on deep RNN problems, CNN standard benchmarks, and some MLP experiments.  The RNN experimental results are especially interesting in that we achieve performance of LSTM networks without using any memory gates - this hopefully motivates the paradigm shift from weight space to target space.  Admittedly though, the target space training steps come with an extra computational cost.

We would welcome discussion.


Dr Michael Fairbank, and co-authors

University of Essex
UK
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/connectionists/attachments/20220222/bb94fcc1/attachment.html>