Connectionists: Deep Belief Nets (2006) / Neural History Compressor (1991) or Hierarchical Temporal Memory

[Juergen Schmidhuber]

Gary (Marcus), you wrote: "it is unrealistic to expect that all the relevant information can be extracted by any general purpose learning device." You might be a bit too pessimistic about general purpose systems. Unbeknownst to many NN researchers, there are _universal_ problem solvers that are time-optimal in various theoretical senses [10-12] (not to be confused with universal incomputable AI [13]). For example, there is a meta-method [10] that solves any well-defined problem as quickly as the unknown fastest way of solving it, save for an additive constant overhead that becomes negligible as problem size grows. Note that most problems are large; only few are small. (AI researchers are still in business because many are interested in problems so small that it is worth trying to reduce the overhead.)

Several posts addressed the subject of evolution (Gary Marcus, Ken Stanley, Brian Mingus, Ali Minai, Thomas Trappenberg). Evolution is another a form of learning, of searching the parameter space. Not provably optimal in the sense of the methods above, but often quite practical. It is used all the time for reinforcement learning without a teacher. For example, an RNN with over a million weights recently learned through evolution to drive a simulated car based on a high-dimensional video-like visual input stream [14,15]. The RNN learned both control and visual processing from scratch, without being aided by unsupervised techniques (which may speed up evolution by reducing the search space through compact sensory codes). 

Jim, you wrote: "this could actually be an interesting opportunity for some cross disciplinary thinking about how one would use an active sensory data acquisition controller to select the sensory data that is ideal given an internal model of the world." Well, that's what intrinsic reward-driven curiosity and attention direction is all about - reward the controller for selecting data that maximises learning/compression progress of the world model - lots of work on this since 1990 [16,17]. (See also posts on developmental robotics by Brian Mingus and Gary Cottrell.)

[10]  Marcus Hutter. The Fastest and Shortest Algorithm for All Well-Defined Problems. International Journal of Foundations of Computer Science, 13(3):431-443, 2002. (On J. Schmidhuber's SNF grant 20-61847.)
[11] http://www.idsia.ch/~juergen/optimalsearch.html
[12] http://www.idsia.ch/~juergen/goedelmachine.html
[13] http://www.idsia.ch/~juergen/unilearn.html
[14] J. Koutnik, G. Cuccu, J. Schmidhuber, F. Gomez. Evolving Large-Scale Neural Networks for Vision-Based Reinforcement Learning. Proc. GECCO'13, Amsterdam, July 2013.
[15] http://www.idsia.ch/~juergen/compressednetworksearch.html
[16] http://www.idsia.ch/~juergen/interest.html
[17] http://www.idsia.ch/~juergen/creativity.html

Juergen