Connectionists: Deep Belief Nets (2006) / Neural History Compressor (1991) or Hierarchical Temporal Memory

[Andrew Ross]

I'm probably preaching to at least some of the choir here, I don't think we
should be too beholden to the brain when we're developing software models
to simulate some of the things it does. It honestly seems way too
complicated. And I'd like to offer some software development-inspired
intuitions about why I feel that way. In software engineering, you always
want to be working at a high level of generality, and avoiding special
cases whenever possible. They complicate the code unnecessarily, make it
harder to maintain or reproduce in the future, and make it less clear for
others to understand. Simplicity is crucial for the sustainability of
software, and I would posit this is highly relevant for life, as well.

The reason why I think simulating the brain is hard is because our brain
gives us a general mechanism for solving problems, but at every point along
the way evolution has sneaked in a bunch of special cases. If the input is
a snake, respond in the following way. If the input is an attractive-ish
person, respond in this other manner, etc etc. These extra logic branches
complicate any efforts to model cognition in a useful manner.

Human nature is filled with such an inestimable amount of baggage that
figuring out something very general like the process by which we learn to
abstract from examples seems impossible just from examining the brain, no
matter how well we learn to subdivide its regions. We have all this
specific machinery for recognizing and responding to specific things, and
it seems extraordinarily difficult to fully isolate any of those systems
from the rest. Any individual brain too might have its own peculiarities;
you'd have to examine a whole bunch and figure out principle components to
have any real basis for understanding what you see.

But I think the brain has to be simple. We could not possibly survive and
reproduce so reliably were it not. The biological underpinnings of
cognition have to be super robust against the efforts of entropy to
introduce randomness into our progeny, and that robustness can only come
with a simplicity that I think must be (computationally) tractable.

This is why I am sympathetic to efforts that use the brain for inspiration
(figuratively), but don't try to actually emulate it. No, we will not be
producing anything resembling a human intelligence this way, and there may
be certain things we find it easy to recognize because of hardwiring that a
simplified machine would have a more difficult time with -- but we know
that even people with intense brain damage can find pretty astounding
workarounds. The ability to abstract is the ultimate workaround, and it's
what allows us to survive even if some of those crucial special cases we
have baked in fail to get transmitted to the next generation.

The goal is not to simulate a human being; I don't want a program that will
love, worry as much as I do, or be at all influenced by the belief that it
possesses an elbow. Instead I want a machine that is capable of recognizing
cats, and possibly forming a more general concepts of "animals" which can
be distinguished from pictures of trees or jars of mustard. And beyond
pictures, I want a computer program that can begin to understand relations
like "this thing is inside this other thing" and be able to identify even
very general similarities between different inputs, divorced as much as
possible from the specificities of the modality of input it receives.

I recognize that's far off, but I feel like we should be focusing on
generalization, and how we can simulate some of the most fundamental
mechanisms of cognition, without getting hung up on, you know, our actual
field of study.

Andrew


On Mon, Feb 10, 2014 at 6:56 PM, Gary Marcus <gary.marcus at nyu.edu> wrote:

>
> On Feb 10, 2014, at 4:51 PM, Brian J Mingus <brian.mingus at colorado.edu>
> wrote:
>
> That said, evolution is a blind designer. A human being can out-design
> billions of years of evolution in a few years with nice supercomputer and
> plenty of lab subjects. So, if your goal is to understand exactly what a
> human being is, you might study human development. But if your goal is to
> create something more sophisticated than a human without the annoyance of
> studying exactly how a human develops intelligence
>
> [a reasonable goal, depending on your research program]
>
> , you might use deep networks with pretraining that automatically extract
> features that evolution baked in.
>
>
> the key question is whether extracting features is, in itself, enough to
> replicate (or even better) the blind handiwork of evolution. my own guess
> is "absolutely not".  Evolution has a done a fine job with evo-crafting
> features -- which deep networks might plausibly hope to match--  but that
> there's a lot of highly-selected circuitry *downstream* that probably
> cannot readily be captured through the mere acquisition of hierarchies of
> features.
>
> On the left is a figure from Solari and Stoner<http://www.frontiersin.org/Journal/10.3389/fnana.2011.00065/full>'s
> magnificent cognitive consilience diagram<http://www.frontiersin.org/files/cognitiveconsilience/index.html>,
> which I encourage all students of cortical neuroscience to contemplate
> (click to zoom in). On the right is a figure representing Google's cat
> detector, a state of the art unsupervised learner, yet still no match for
> humans when it comes to invariance or in the use of top-down visual
> information. Is the one on the right genuinely a useful approximation of
> the one of the left?
>
> In my own view there is an impedance mismatch between most current models
> and the intricacy of biological reality.
>
> Cheers,
> Gary
>
>  <http://www.frontiersin.org/files/cognitiveconsilience/index.html>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/connectionists/attachments/20140210/63672c07/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PastedGraphic-1.png
Type: image/png
Size: 265368 bytes
Desc: not available
URL: <http://mailman.srv.cs.cmu.edu/pipermail/connectionists/attachments/20140210/63672c07/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PastedGraphic-5.tiff
Type: image/tiff
Size: 37416 bytes
Desc: not available
URL: <http://mailman.srv.cs.cmu.edu/pipermail/connectionists/attachments/20140210/63672c07/attachment-0001.tiff>