<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 TRANSITIONAL//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; CHARSET=UTF-8">
<META NAME="GENERATOR" CONTENT="GtkHTML/4.6.6">
</HEAD>
<BODY>
I think there is a confusion here about "Deep Learning". I think Deep Learning is not so much about the words<BR>
"deep" and "learning", but the architecture of the learning system--or neural network.<BR>
<BR>
Those of us who were doing Auto-Encoders back in the 80s, had tried many time to increase the number of hidden layers,<BR>
especially doing some sort of language task or grammer task. I was particularly inspired to try this after seeing Yann's<BR>
early work on conv-nets, in which he employed many layers. Also Geoff had a "kinship network" that consisted of 6 layers or so.. (cog sci,'86), ....so we had all tried to train auto-encoders with multiple layers.. and miserably failed using back-propagations with sigmoidal activation functions... <BR>
<BR>
So the actual concept of Deep Learning is more about this representational compression and abstraction (not so much<BR>
theory about this part, I think). So again, really not about "deep" or "learning".. etc.. I think searching for these words in documents from the 60s will miss the critical aspects of the multiple layers of representational structure that is extracted and used to generalize. This is really an extension of stuff that was happening in the mid to late 1980s.<BR>
<BR>
Cheers<BR>
<BR>
Steve<BR>
<TABLE CELLSPACING="0" CELLPADDING="0" WIDTH="100%">
<TR>
<TD>
-- <BR>
Stephen José Hanson<BR>
Director RUBIC (Rutgers Brain Imaging Center)<BR>
Professor of Psychology<BR>
Member of Cognitive Science Center (NB)<BR>
Member EE Graduate Program (NB)<BR>
Member CS Graduate Program (NB)<BR>
Rutgers University
<PRE>
email: jose@rubic.rutgers.edu
web: psychology.rutgers.edu/~jose
lab: www.rumba.rutgers.edu
fax: 866-434-7959
voice: 973-353-3313 (RUBIC)
</PRE>
<BR>
<BR>
</TD>
</TR>
</TABLE>
</BODY>
</HTML>