<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">See a recent book by Stellan Ohlsson from either Oxford or Cambridge Press with the title in question.<div><br></div><div>Mark</div><div><br><div apple-content-edited="true">
-------------------------------------------------------<br>Mark Orr<br>Research Associate Professor<br>Social and Decision Analytics Laboratory<br>Virginia Tech-National Capital Region<br>900 North Glebe Rd.<br>Arlington, VA 22203<br>p: 571-858-3116<br>f: 571-858-3015<br><a href="mailto:morr9@vbi.vt.edu">morr9@vbi.vt.edu</a><br><br><br>
</div>
<br><div><div>On Mar 13, 2015, at 12:17 AM, Ali Minai <<a href="mailto:minaiaa@gmail.com">minaiaa@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div dir="ltr"><div><div><div>Juergen,<br><br></div>I would say that the instances you point out are not really examples of "deep learning" in the sense the term is being used today. The way we use it now, it refers really to "learning in deep networks", whereas "deep learning" (as opposed to "shallow learning") would mean learning something in a deep sense, e.g., at a conceptual, relational or causal level, rather than in a shallow sense, e.g., at a purely correlational level. This latter sense of "deep learning" may also be implicit in some "deep learning" models, but I don't think the "deep" today refers to this aspect of depth.<br><br></div>Any discussion of early "deep networks" must surely also refer to Fukushima's Neocognitron.<br><br></div>Ali<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Mar 12, 2015 at 5:35 PM, Juergen Schmidhuber <span dir="ltr"><<a href="mailto:juergen@idsia.ch" target="_blank">juergen@idsia.ch</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div>Thanks. Hm, sure, “deep neural nets” are old, and Ivakhnenko’s deep nets worked well even in the 1960s. But what I’d like to know is: who was the first to use the term “deep learning” in an NN publication? </div><div><br></div><div>Aizenberg et al (2000) wrote about “deep learning of the features of threshold Boolean functions, one of the most important objects considered in the theory of perceptrons …” </div><div><br></div><div>Brian Mingus, however, pointed me to a paper by Rina Dechter (1986). Brian wrote: "Deep learning as compared to shallow learning is terminology used in the study of constraint satisfaction. Constraint satisfaction networks then became RBMs. I would argue this is a good basis for the origin of the modern usage. I like this paper for provenance: <a href="http://www.aaai.org/Papers/AAAI/1986/AAAI86-029.pdf" target="_blank">http://www.aaai.org/Papers/AAAI/1986/AAAI86-029.pdf</a> "</div><div><br></div><div>But perhaps the term occurred even earlier in the NN literature?</div><span class="HOEnZb"><font color="#888888"><div><br></div><div>Juergen</div></font></span><div><div class="h5"><div><br></div><div><br></div><br><div><blockquote type="cite"><div>On 12 Mar 2015, at 21:16, Geoffrey Hinton <<a href="mailto:geoffrey.hinton@gmail.com" target="_blank">geoffrey.hinton@gmail.com</a>> wrote:</div><br><div>I think the current popularity of the term started with the paper by<br>Hinton Osindero and Teh in 2006 called "A fast learning algorithm for<br>deep belief nets". After this paper there was a lot of talk about<br>deep belief nets. In about 2007 the term "deep belief net" started<br>changing its meaning and was used (rather sloppily) to refer to deep<br>neural nets that were pre-trained as deep belief nets. The term gained<br>a lot of popularity because these nets were used to make good acoustic<br>models and that triggered the re-introduction of neural nets into<br>mainline speech recognizers. People eventually made a clear<br>terminological distinction between deep belief nets (DBNs) and deep<br>neural nets that were initialized as deep belief nets (DNNs or<br>DBN-DNNs). Then they discovered that with large datasets and sensible<br>initial scales for the weights the pre-training was not needed and<br>they generalized DNNs to any old deep neural net.<br><br>Its clearly true that people had previously used the term deep neural<br>net but that was not the origin of the resurgence of the term in about<br>2007.<br><br>Its pretty obvious by now that deep neural networks of the type that<br>people were using in the 1980's work very well when they have enough<br>data and enough computation, and its pretty obvious that the deep<br>convnets that Yann has been using since about 1987 are deep neural<br>nets, so what does it matter where the name came from? Deep neural<br>nets are finally living up to their promise so lets all enjoy it.<br><br>Geoff<br><br><br><br><br>On Thu, Mar 12, 2015 at 1:58 PM, Schmidhuber Juergen <<a href="mailto:juergen@idsia.ch" target="_blank">juergen@idsia.ch</a>> wrote:<br><blockquote type="cite">Dear connectionists,<br><br>to my knowledge, the ancient term "Deep Learning" was introduced to the NN field by Aizenberg & Aizenberg & Vandewalle's book (2000): "Multi-Valued and Universal Binary Neurons: Theory, Learning and Applications."<br><br>Is anyone aware of older NN papers using it?<br><br>(Of course, the field itself is much older - Ivakhnenko started his work on deep learning networks in the mid 1960s.)<br><br>Thanks!<br><br>Juergen<br><br><a href="http://people.idsia.ch/~juergen/whatsnew.html" target="_blank">http://people.idsia.ch/~juergen/whatsnew.html</a><br></blockquote></div></blockquote></div><br></div></div></div></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature">Ali A. Minai, Ph.D.<br>Professor<br>Complex Adaptive Systems Lab<br>Department of Electrical Engineering & Computing Systems<br>University of Cincinnati<br>Cincinnati, OH 45221-0030<br><br>Phone: (513) 556-4783<br>Fax: (513) 556-7326<br>Email: <a href="mailto:Ali.Minai@uc.edu" target="_blank">Ali.Minai@uc.edu</a><br> <a href="mailto:minaiaa@gmail.com" target="_blank">minaiaa@gmail.com</a><br><br>WWW: <a href="http://www.ece.uc.edu/%7Eaminai/" target="_blank">http://www.ece.uc.edu/~aminai/</a></div>
</div>
</blockquote></div><br></div></body></html>