<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Dear Connectionists, <br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
We'd like to highlight to you our new paper "Deep Learning in Target Space" at JMLR: https://jmlr.org/papers/v23/20-040.html
<div><br>
</div>
<div>It offers a potentially new paradigm on how to train neural networks - to search through the space of activations of the hidden nodes (i.e. "target space") as opposed to the conventional backpropagation of the gradient in "weight space". We argue that
this stabilises the gradient descent process, a process we call "cascade untangling", meaning deeper neural networks can be trained with less training data and potentially less CPU time. The work is an updated form of the Moving Targets algorithm by Rohwer
(1990) and some follow up work, where we have fixed a lot of technical issues that existing in those earlier works. For some quick highlights of the work, see our paper's Figs 2 and 4. Results are included on deep RNN problems, CNN standard benchmarks, and
some MLP experiments. The RNN experimental results are especially interesting in that we achieve performance of LSTM networks without using any memory gates - this hopefully motivates the paradigm shift from weight space to target space. Admittedly though,
the target space training steps come with an extra computational cost.<br>
</div>
<div><br>
</div>
</div>
<div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
We would welcome discussion. <br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="Signature">
<div>
<div></div>
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Helvetica,sans-serif">
<p class="x_MsoNormal" style="margin-top:0px; margin-bottom:0px"><span style="">Dr Michael Fairbank, and co-authors</span></p>
<span style="">University of Essex</span> <br>
</div>
<div dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Helvetica,sans-serif">
UK<br>
<p style="margin-top:0px; margin-bottom:0px"></p>
</div>
</div>
</div>
</div>
</body>
</html>