<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Thanks. Hm, sure, “deep neural nets” are old, and Ivakhnenko’s deep nets worked well even in the 1960s. But what I’d like to know is: who was the first to use the term “deep learning” in an NN publication? </div><div class=""><br class=""></div><div class="">Aizenberg et al (2000) wrote about “deep learning of the features of threshold Boolean functions, one of the most important objects considered in the theory of perceptrons …” </div><div class=""><br class=""></div><div class="">Brian Mingus, however, pointed me to a paper by Rina Dechter (1986). Brian wrote: "Deep learning as compared to shallow learning is terminology used in the study of constraint satisfaction. Constraint satisfaction networks then became RBMs. I would argue this is a good basis for the origin of the modern usage. I like this paper for provenance: <a href="http://www.aaai.org/Papers/AAAI/1986/AAAI86-029.pdf" class="">http://www.aaai.org/Papers/AAAI/1986/AAAI86-029.pdf</a> "</div><div class=""><br class=""></div><div class="">But perhaps the term occurred even earlier in the NN literature?</div><div class=""><br class=""></div><div class="">Juergen</div><div class=""><br class=""></div><div class=""><br class=""></div><br class=""><div><blockquote type="cite" class=""><div class="">On 12 Mar 2015, at 21:16, Geoffrey Hinton <<a href="mailto:geoffrey.hinton@gmail.com" class="">geoffrey.hinton@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class="">I think the current popularity of the term started with the paper by<br class="">Hinton Osindero and Teh in 2006 called "A fast learning algorithm for<br class="">deep belief nets". After this paper there was a lot of talk about<br class="">deep belief nets. In about 2007 the term "deep belief net" started<br class="">changing its meaning and was used (rather sloppily) to refer to deep<br class="">neural nets that were pre-trained as deep belief nets. The term gained<br class="">a lot of popularity because these nets were used to make good acoustic<br class="">models and that triggered the re-introduction of neural nets into<br class="">mainline speech recognizers. People eventually made a clear<br class="">terminological distinction between deep belief nets (DBNs) and deep<br class="">neural nets that were initialized as deep belief nets (DNNs or<br class="">DBN-DNNs). Then they discovered that with large datasets and sensible<br class="">initial scales for the weights the pre-training was not needed and<br class="">they generalized DNNs to any old deep neural net.<br class=""><br class="">Its clearly true that people had previously used the term deep neural<br class="">net but that was not the origin of the resurgence of the term in about<br class="">2007.<br class=""><br class="">Its pretty obvious by now that deep neural networks of the type that<br class="">people were using in the 1980's work very well when they have enough<br class="">data and enough computation, and its pretty obvious that the deep<br class="">convnets that Yann has been using since about 1987 are deep neural<br class="">nets, so what does it matter where the name came from? Deep neural<br class="">nets are finally living up to their promise so lets all enjoy it.<br class=""><br class="">Geoff<br class=""><br class=""><br class=""><br class=""><br class="">On Thu, Mar 12, 2015 at 1:58 PM, Schmidhuber Juergen <<a href="mailto:juergen@idsia.ch" class="">juergen@idsia.ch</a>> wrote:<br class=""><blockquote type="cite" class="">Dear connectionists,<br class=""><br class="">to my knowledge, the ancient term "Deep Learning" was introduced to the NN field by Aizenberg & Aizenberg & Vandewalle's book (2000): "Multi-Valued and Universal Binary Neurons: Theory, Learning and Applications."<br class=""><br class="">Is anyone aware of older NN papers using it?<br class=""><br class="">(Of course, the field itself is much older - Ivakhnenko started his work on deep learning networks in the mid 1960s.)<br class=""><br class="">Thanks!<br class=""><br class="">Juergen<br class=""><br class=""><a href="http://people.idsia.ch/~juergen/whatsnew.html" class="">http://people.idsia.ch/~juergen/whatsnew.html</a><br class=""></blockquote></div></blockquote></div><br class=""></body></html>