Connectionists: First Deep Learning Networks in 1965

[Schmidhuber Juergen]

Who created the first Deep Learning networks?  

To my knowledge, this was done by Olexiy Hryhorovych (Alexey Grigoryevich) Ivakhnenko and colleagues in 1965. Here a brief summary:

Networks trained by the Group Method of Data Handling (GMDH) (Ivakhnenko and Lapa, 1965; Ivakhnenko et al., 1967; Ivakhnenko, 1968, 1971) were perhaps the first Deep Learning systems of the Feedforward Multilayer Perceptron type. The units of GMDH nets may have polynomial activation functions implementing Kolmogorov-Gabor polynomials (more general than other widely used neural network activation functions). Given a training set, layers are incrementally grown and trained by regression analysis (e.g., Legendre, 1805; Gauss, 1809, 1821), then pruned with the help of a separate validation set (using today’s terminology), where Decision Regularisation is used to weed out superfluous units. The numbers of layers and units per layer can be learned in problem-dependent fashion. To my knowledge, this was the first example of hierarchical representation learning in NNs. A paper of 1971 already described a deep GMDH network with 8 layers (Ivakhnenko, 1971). There have been numerous applications of GMDH-style nets, e.g. (Ikeda et al., 1976; Farlow, 1984; Madala and Ivakhnenko, 1994; Ivakhnenko, 1995; Kondo, 1998; Kordik et al., 2003; Witczak et al., 2006; Kondo and Ueno, 2008) … 

Precise references and more history in:

Deep Learning in Neural Networks: An Overview 
PDF & LATEX source & complete public BIBTEX file under
http://www.idsia.ch/~juergen/deep-learning-overview.html

Juergen Schmidhuber