Connectionists: Brain-like computing fanfare and big data fanfare

[Thomas G. Dietterich]

Dear Brian,

 

Please keep in mind that MDL, Ockham's razor, PCA, and similar
regularization approaches focus on the problem of *prediction* (or,
equivalently, compression).  Given a fixed amount of data and a flexible
class of models, these principles tell us how to modulate the expressiveness
of the model to maximize predictive accuracy.  I would characterize it as
follows: "Which deliberately incorrect model should we adopt in order to
optimize predictive accuracy?"

 

One stance toward creating an AI system is to pursue this purely functional
approach and model a person as an input-output mapping (with latent state
variables, as appropriate).  Such an approach might be very useful both for
engineering and for science.  From a scientific perspective, it would tell
us that if we build a system with certain properties, it can exhibit this
input-output behavior.

 

But it would not be a satisfactory theory of neuroscience for two reasons.
First, it only provides sufficient conditions but does not show they are
necessary. There might be other ways of producing the behavior, and the
brain might implement one of those instead. Second, even if it could be made
into a necessary and sufficient condition (e.g., by proving that all systems
lacking certain properties would NOT exhibit the desired behavior), it would
still not explain how the chemistry and biology of the brain produces the
required properties.  To fall back on the old bird vs. airplane analogy, the
accomplishments of the Wright brothers (and the field of aerodynamics)
provided a theory of how flight could be achieved. But we are still learning
at the biological level how birds actually do it.

 

 

 

-- 

Thomas G. Dietterich, Distinguished Professor Voice: 541-737-5559

School of Electrical Engineering              FAX: 541-737-1300

  and Computer Science                        URL: eecs.oregonstate.edu/~tgd

US Mail: 1148 Kelley Engineering Center 

Office: 2067 Kelley Engineering Center

Oregon State Univ., Corvallis, OR 97331-5501

 

 

From: Connectionists [mailto:connectionists-bounces at mailman.srv.cs.cmu.edu]
On Behalf Of Brian J Mingus
Sent: Saturday, January 25, 2014 8:23 PM
To: Brad Wyble
Cc: connectionists at mailman.srv.cs.cmu.edu
Subject: Re: Connectionists: Brain-like computing fanfare and big data
fanfare

 

Hi Brad et al.,  - thanks very much for this fun and entertaining
philosophical discussion:)

 

With regards to turtles all the way down, and also with regards to choosing
the appropriate level of analysis for modeling, I'd like to reiterate a
position I made earlier but  in which I didn't really provide enough detail.

 

There exists a formalization of Ockham's razor in a field called Algorithmic
Information Theory, and this formalization is the Minimum Description Length
(MDL). 

 

This perspective essentially says that we are searching for the optimal
compression of all of the data relating to the brain. This means that we
don't want to overcompress relevant distinctions, but we don't want to
undercompress redundancies. This optimal compression, when represented as a
computer program that outputs all of the brain data (aka a model), has a
description length known as the Kolmogorov complexity. 

 

Now there is something weird about what I have just described, which is that
the resulting model will produce not just the data for a single brain, but
the data for every brain - a kind of meta-brain. And this is not quite what
we are looking for. And due to the turtles problem it is probably ill-posed,
in that the length of the description may be infinite as we zoom in to finer
levels of detail.

 

So we need to provide some relevant constraints on the problem to make it
tractable. Based on what I just described, the MDL for your brain is your
brain. This is essentially because we haven't defined a utility function,
and we haven't done that because we aren't quite sure what exactly it is we
are doing, or what we are looking for, when modeling the brain.

 

To begin fixing this problem, we can rotate this perspective into a tool
that we are all probably familiar with - factor analysis, i.e., PCA. What we
are essentially looking for, first and foremost, is a model that explains
the first principle component of just one person's comprehensive brain
dataset (which includes behavioral data). Then we want to study this
component (which is tantamount to a model of the brain) and see what it can
do.

 

What will this first principle component look like? Now we need to define
what exactly it is that we are after. I would argue that our model should be
composed of neuron-like elements connected in networks, and that when we
look at the statistical properties of these networks, they should be quite
similar to what we see in humans. 

 

Most importantly, however, I would argue that this model, when raised as a
human, should exhibit some distinctly human traits. It should not pass a
trivial turing test, but rather a deep turing test. After having been raised
as and with human beings, but not exposed to any substantial philosophy,
this model should independently invent consciousness philosophy.

 

As you might imagine, our abstract high level model brain which captures the
first principle component of the brain data might not be able to do this.
Thus, we will start adding in more components that explain more of the
variance, iteratively increasing our description length. This is a
distinctly top-down approach, in which we only add relevant detail as it
becomes obvious that the current model just isn't quite human. 

 

This approach follows a scientific gradient advocated for by Ockham's razor,
in that we start with the simplest description (brain model) that explains
the most amount of variance, and gradually increase the size of the
description until it finally reinvents consciousness philosophy and can live
among humans.

 

In my admittedly biased experience, the first appropriate level of analysis
is approximately point-neuron deep neural network architectures. However,
this might actually be too low level - we might want to start with even more
abstract, modern day NIPS-level models, and confirm that, although they can
behave like humans, they can't reinvent consciousness philosophy and are
thus more akin to zombie-like automata. 

 

Of course, with sufficient computing power our modeling approach can be
somewhat more sloppy - we can begin experimenting with the synthesis of
different levels of analysis right away.

 

However, before we do any of this "for real" we probably want to
comprehensively discuss the ethics of raising beings that are ultimately
similar to humans, but are not quite human, and further, the ethics of
raising digital humans. 

 

Lastly, to touch back to the original topic - Big Data - I think it's clear
that the more data we have, the merrier. However, it also makes sense to
follow the Ockham gradient. Ultimately, we are really just not as close to
creating a human being as it may seem, and so it is probably safe, for the
time being, to collect data from all levels of analysis willy nilly.
However, when it comes time to actually build the human, we should be more
careful, for the sake of the being we create. Indeed, perhaps we should be
sure that it will reinvent consciousness philosophy before we ever turn it
on in the first place.

 

If anyone has an idea of how to do that, I would be extremely interested to
hear about it.

 

Brian Mingus

 

Graduate student

Department of Psychology and Neuroscience

University of Colorado at Boulder

http://grey.colorado.edu/mingus

 

On Sat, Jan 25, 2014 at 7:52 PM, Brad Wyble <bwyble at gmail.com> wrote:

Jim, 

 

Great debate!  There are several good points here..

 

First, I agree with you that models with tidy, analytical solutions are
probably not the ultimate answer, as biology is unlikely to exhibit behavior
that coincides with mathematical formalisms that are easy to represent in
equations. In fact, I think that seeking such solutions can get in the way
of progress in some cases.  

 

I also agree with you that community models are a good idea, and I am not
advocating that everyone should build their own model.  But I think that we
need a hierarchy of such community models at multiple levels of abstraction,
with clear ways of translating ideas and constraints from each level to the
next.  The goal of computational neuroscience is not to build the ultimate
model, but to build a shared understanding in the minds of the entire body
of neuroscientists with a minimum of communication failures.   

 

Next,  I think that you're espousing a purely bottom-up approach to
modelling the brain. ( i.e. that if we just build it, understanding will
follow from the emergent dynamics). I very much admire your strong position,
but I really can't agree with it.  I return to the question of how we will
even know what the bottom floor is in such an approach  You seem to imply in
previous emails that it's a channel/cable model, but someone else might
argue that we'd have to represent interactions at the atomic level to truly
capture the dynamics of the circuit.  So if that's the only place to start,
how will we ever make serious progress?  The computational requirements to
simulate even a single neuron at the atomic level on a super cluster is
probably a decade away. And once we'd accomplished that, someone might point
out a case in which subatomic interactions play a functional role in the
neuron and then we've got to wait another 10 years to be able to model a
single neuron again?    

 

To me, it really looks like turtles all the way down which means that we
have to choose our levels of abstraction with an understanding that there
are important dynamics at lower levels that will be missed.  However if we
build in constraints from the behavior of the system, such abstract models
can nevertheless provide a foothold for climbing a bit higher in our
understanding.    

 

Is there some reason that you think channels are a sufficient level of
detail?  (or maybe I've mischaracterized your position)

 

-Brad

 

 

 

 

On Sat, Jan 25, 2014 at 7:09 PM, james bower <bower at uthscsa.edu> wrote:

About to sign off here - as have probably already taken too much bandwidth.
(although it has been a long time)

 

But just for final clarity on the point about physics - I am not claiming
that the actual tools etc, developed by physics mostly to study
non-biological and mostly 'simpler' systems (for example, systems were the
elements (unlike neurons) aren't 'individualized'  - and therefore can be
subjected to a certain amount of averaging (ie. thermodynamics), will apply.

 

But I am suggesting (all be it in an oversimplified way)  that the
transition from a largely folkloric, philosophically (religiously) driven
style of physics, to the physics of today was accomplished in the 15th
century by the rejection of the curve fitting, 'simplified' and self
reflective Ptolemic model of the solar system. (not actually, it turns out
for that reason, but because the Ptolemaic model has become too complex and
impure - the famous equint point).   Instead, Newton, Kepler, etc, further
developed a model that actually valued the physical structure of that
system, independent of the philosophical, self reflecting previous set of
assumptions.  I know, I know that this is an oversimplified description of
what happened, but, it is very likely that Newtons early (age 19) discovery
of what approximated the least squares law in the 'realistic model' he had
constructed of the earth moon system (where it was no problem and pretty
clearly evident that the moon orbited the earth in a regular way), lead in
later years to his development of mechanics - which, clearly provided an
important "community model" of the sort we completely  lack in neuroscience
and seem to me continue to try to avoid.  

 

I have offered for years to buy the beer at the CNS meeting if all the
laboratories describing yet another model of the hippocampus or the visual
cortex would get together to agree on a single model they would all work on.
No takers yet.  The paper I linked to in my first post describes how that
has happened for the Cerebellar Purkinje cell, because of GENESIS and
because we didn't block others from using the model, even to criticize us.
However, when I sent that paper recently to a computational neuroscience I
heard was getting into Purkinje cell modeling, he wrote back to say he was
developing his own model thank you very much.

 

The proposal that we all be free to build our own models - and everyone is
welcome, is EXACTLY the wrong direction. 

 

We need more than calculous - and although I understand their attractiveness
believe me, models that can be solved in close formed solutions are not
likely to be particularly useful in biology, where the averaging won't work
in the same way. The relationship between scales is different, lots of
things are different - which means the a lot of the tools will have to be
different too. And I even agree that some of the tools developed by
engineering, where one is actually trying to make things that work, might
end up being useful, or even perhaps more useful.  However, the transition
to paradigmatic science I believe will critically depend on the acceptance
of community models (they are the 'paradigm'), and the models most likely
with the most persuasive force as well as the ones mostly likelihood of
revealing unexpected functional relationships, are ones that FIRST account
for the structure of the brain, and SECOND are used to explore function
(rather than what is usually the other way around).

 

As described in the paper I posted, that is exactly what has happened
through long hard work (since 1989) using the Purkinje cell model.

 

In the end, unless you are a duelist (which I suspect many actually are, in
effect), brain computation involves nothing beyond the nervous system and
its physical and physiological structure.  Therefore, that structure will be
the ultimate reference for how things really work, no matter what level of
scale you seek to describe.

 

>From 30 years of effort, I believe even more firmly now than I did back
then, that, like Newton and his friends, this is where we should start -
figuring out the principles and behavior from the physics of the elements
themselves.

 

You can claim it is impossible - you can claim that models at other levels
of abstraction can help, however, in the end 'the truth' lies in the
circuitry in all its complexity.  But you can't just jump into the
complexity, without a synergistic link to models that actually provide
insights at the detailed level of the data you seek to collect.  

 

IMHO.

 

Jim

 

(no ps)

 

 

 

 

 

 

 

On Jan 25, 2014, at 4:44 PM, Dan Goodman <dg.connectionists at thesamovar.net>
wrote:





The comparison with physics is an interesting one, but we have to remember
that neuroscience isn't physics. For a start, neuroscience is clearly much
harder than physics in many ways. Linear and separable phenomena are much
harder to find in neuroscience, and so both analysing and modelling data is
much more difficult. Experimentally, it is much more difficult to control
for independent variables in addition to the difficulty of working with
living animals.

So although we might be able to learn things from the history of physics -
and I tend to agree with Axel Hutt that one of those lessons is to use the
simplest possible model rather than trying to include all the biophysical
details we know to exist - while neuroscience is in its pre-paradigmatic
phase (agreed with Jim Bower on this) I would say we need to try a diverse
set of methodological approaches and see what wins. In terms of funding
agencies, I think the best thing they could do would be to not insist on any
one methodological approach to the exclusion of others.

I also share doubts about the idea that if we collect enough data then
interesting results will just pop out. On the other hand, there are some
valid hypotheses about brain function that require the collection of large
amounts of data. Personally, I think that we need to understand the
coordinated behaviour of many neurons to understand how information is
encoded and processed in the brain. At present, it's hard to look at enough
neurons simultaneously to be very sure of finding this sort of coordinated
activity, and this is one of the things that the HBP and BRAIN initiative
are aiming at.

Dan

 

 

 

Dr. James M. Bower Ph.D.

Professor of Computational Neurobiology

Barshop Institute for Longevity and Aging Studies.

15355 Lambda Drive

University of Texas Health Science Center 

San Antonio, Texas  78245

 

Phone:  210 382 0553 <tel:210%20382%200553> 

Email: bower at uthscsa.edu

Web: http://www.bower-lab.org

twitter: superid101

linkedin: Jim Bower

 

CONFIDENTIAL NOTICE:

The contents of this email and any attachments to it may be privileged or
contain privileged and confidential information. This information is only
for the viewing or use of the intended recipient. If you have received this
e-mail in error or are not the intended recipient, you are hereby notified
that any disclosure, copying, distribution or use of, or the taking of any
action in reliance upon, any of the information contained in this e-mail, or

any of the attachments to this e-mail, is strictly prohibited and that this
e-mail and all of the attachments to this e-mail, if any, must be

immediately returned to the sender or destroyed and, in either case, this
e-mail and all attachments to this e-mail must be immediately deleted from
your computer without making any copies hereof and any and all hard copies
made must be destroyed. If you have received this e-mail in error, please
notify the sender by e-mail immediately.

 

 





 

-- 

Brad Wyble
Assistant Professor
Psychology Department
Penn State University

 

http://wyblelab.com

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/connectionists/attachments/20140126/89c46970/attachment.html>