<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">Dear all:<div><br></div><div>Though it doesn’t quite go back to the fishermen of Northern Spain, it’s worth noting that multi-layer (and hence deep) nets are discussed in some detail by Rosenblatt in his “Principles of Neurodynamics” (1962 Spartan Books), specifically section 15, page 313 et seq, </div><div><br></div><div>Clearly, he did not use the term “deep learning”: he talks about “adaptive pre terminal networks” when referring to alterations of weights in earlier layers. </div><div><br></div><div>—Leslie Smith</div><div><br><div><div>On 13 Mar 2015, at 16:53, Schmidhuber Juergen <<a href="mailto:juergen@idsia.ch">juergen@idsia.ch</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><span style="font-family: Monaco; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">Sorry, but the “semantics of what researchers nowadays call deep learning" are much older. In RNNs, the deepest of all NNs, your "strictly unsupervised followed by supervised finetuning” goes back to Schmidhuber's hierarchical deep RNN stacks of 1991 (the neural history compressors). They were largely replaced (still in the 1990s) by deep supervised LSTM RNNs. History repeated itself between 2006 and 2010, when deep unsupervised FNN stacks (kudos to Hinton et al) were replaced by deep standard supervised FNNs, as you pointed out. (It's hardly clear, however, that the re-popularization of supervised NNs wouldn't have occurred without the work on unsupervised NNs.)</span><br style="font-family: Monaco; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br style="font-family: Monaco; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span style="font-family: Monaco; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">Antoine Bordes' Google-generated graph seems to indicate that the usage of the term went up right after Aizenberg et al.’s book came out (2000). As Yoshua Bengio pointed out, however, it includes all kinds of ancient usages of “Deep Learning,” and is not limited to NN-specific usage in the sense of this thread.<span class="Apple-converted-space"> </span></span><br style="font-family: Monaco; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br style="font-family: Monaco; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span style="font-family: Monaco; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">Again, I am just trying locate the introduction of the term. It's an interesting question in its own right, outside of when the principles of deep learning came into being.<span class="Apple-converted-space"> </span></span><br style="font-family: Monaco; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br style="font-family: Monaco; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span style="font-family: Monaco; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">Juergen<span class="Apple-converted-space"> </span></span><br style="font-family: Monaco; font-size: 11px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"></blockquote></div><br><div apple-content-edited="true">
<div style="color: rgb(0, 0, 0); letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><div>Leslie Smith</div><div><a href="mailto:l.s.smith@cs.stir.ac.uk">l.s.smith@cs.stir.ac.uk</a></div><div>Professor of Computing,</div><div>Computing Science and Mathematics,</div><div>University of Stirling</div><div>Stirling FK9 4LA,</div><div>Scotland, UK</div><div>+44 (0) 1786 467435</div><div><br></div></div><br class="Apple-interchange-newline"><br class="Apple-interchange-newline">
</div>
<br></div><DIV align=left><HR>
<DIV align=left><FONT face=Arial size=2>The University of Stirling has been ranked in the top 12 of UK universities for graduate employment*.</FONT></DIV
><DIV align=left><FONT face=Arial size=2>94% of our 2012 graduates were in work and/or further study within six months of graduation.</FONT></DIV>
<DIV align=left><FONT face=Arial size=2>*The Telegraph</FONT></DIV>
<FONT face=Arial color=gray size=2>The University of Stirling is a charity registered in Scotland, number SC 011159.<BR></FONT>
</DIV>
</body></html>