Connectionists: Brain-like computing fanfare and big data fanfare

Sat Jan 25 15:39:40 EST 2014

I’ve enjoyed this provocative exchange. It reminds me very much of the
discussion in the late 80s about whether to invest $3B in sequencing the
human genome. The lab scientists who studied individual genes and metabolic
pathways complained that this lacked any detailed guiding hypotheses and
would just be a waste of money.  The sequencing advocates promised that it
would be revolutionary. It turned out that they were right. In fact, the
technology was much more effective and much cheaper than they had dared to
hope.

The current BRAIN initiative is investing a similar amount of money in
developing new sensing and measurement technologies.  If we look at the
history of science, we see many cases in which new instruments led to giant
leaps forward: telescopes, microscopes, x-ray crystallography, MRI,
interferometry, rapid throughput DNA, RNA, and protein sequencing. Of course
every new technology will have biases and artifacts, and part of the process
of developing a new technology is understanding these and how to compensate
for them.  Often, this advances the science as well.

Sometimes it turns out that the instruments are only very indirectly
capturing the underlying processes. This is where the big data argument gets
interesting. If we look at the kind of data collected on the internet, it is
almost always of this type.  For many ecommerce applications, it suffices to
build a predictive model of customer behavior. And this is the main
contribution of applied machine learning. The bigger the data sets, the more
accurate these predictive models can become. 

Unfortunately, such models are not very useful in science, because the goal
of neuroscience, for example, is not to just to predict human behavior but
to understand how that behavior is generated.  If we only have “indirect”
measurements, we must fit models where the “real” variables are latent.
Making sound scientific inferences about latent variables is extremely
difficult and relies on having very good models of how the latent variables
produce the observed signal. Issues of identifiability and bias must be
addressed, and there are also very challenging computational problems,
because latent variable models typically exhibit many local optima.
Regularization, the favorite tool of machine learning, usually worsens the
biases and limits our ability to draw statistical inferences.  If your model
is fundamentally not identifiable, then it doesn’t matter how big your data
are.

Every scientific experiment (or expenditure) involves risk, and every
scientist must “place bets” on what will work. I’ll place mine on improving
our measurement technology, because I think it can get us closer to
measuring the critical causal variables.  But I agree with Jim that not
enough effort goes into the unglamorous process of figuring out what it is
that our instruments are actually measuring. And if we don’t understand
that, then our scientific inferences are likely to be wrong. 

-- 

Thomas G. Dietterich, Distinguished Professor Voice: 541-737-5559

School of Electrical Engineering              FAX: 541-737-1300

  and Computer Science                        URL: eecs.oregonstate.edu/~tgd

US Mail: 1148 Kelley Engineering Center 

Office: 2067 Kelley Engineering Center

Oregon State Univ., Corvallis, OR 97331-5501

From: Connectionists [mailto:connectionists-bounces at mailman.srv.cs.cmu.edu]
On Behalf Of james bower
Sent: Saturday, January 25, 2014 9:05 AM
To: jose at psychology.rutgers.edu
Cc: Connectionists
Subject: Re: Connectionists: Brain-like computing fanfare and big data
fanfare

Hi Jose,

Ah, neuroimaging - don’t get me started.  Not all, but a great deal of
neuroimaging has become a modern form of phrenology IMHO, distorting not
only neuroscience, but it turns out, increasingly business too.  To wit:

At present I am actually much more concerned (and involved) in the use of
brain imaging in what has come to be called "Neuro-marketing”.  Many on this
list are perhaps not aware, but while we worry about the effect of over
interpretation of neuroimaging data within neuroscience, the effect of this
kind of data in the business world is growing and not good. Although my
guess is that  those of you in the United States might have noted the rather
large and absurd marketing campaign by Lumosity and the sellers of other
‘brain training” games.  A number of neuroscientists are actually now
getting in this business.

As some of you know, almost as long as I have been involved in computational
neuroscience, I have also been involved in exploring the use of games for
children’s learning.  In the game/learning world, the misuse of neuroscience
and especially brain imaging has become excessive.  It wouldn’t be
appropriate to belabor this point on this list - although the use of
neuroscience by the NN community does, in my view, often cross over into a
kind of neuro-marketing.  

For those that are interested in the more general abuses of neuro-marketing,
here is a link to the first ever session I organized in the game development
world based on my work as a neurobiologist:

http://www.youtube.com/watch?v=Joqmf4baaT8
<http://www.youtube.com/watch?v=Joqmf4baaT8&list=PL1G85ERLMItAA0Bgvh0PoZ5iGc
6cHEv6f&index=16> &list=PL1G85ERLMItAA0Bgvh0PoZ5iGc6cHEv6f&index=16

As set up for that video, you should know that in his keynote address the
night before, Jessi Schelll (of Schell Games and CMU) introduced his talk by
saying that he was going to tell the audience what neuroscientists have
figured out about human brains, going on to claim that they (we) have
discovered that human brains come in two forms, goat brains and sheep
brains. Of course the talk implied that the goats were in the room and the
sheep were out there to be sold to.  (although as I noted on the twitter
feed at the time, there was an awful lot of ‘baaing’ going on in the
audience  :-)  ).

Anyway, the second iteration of my campaign to try to bring some balance and
sanity to neuro-marketing, will take place at SxSW in Austin in march, in
another session I have organized on the subject.

http://schedule.sxsw.com/2014/events/event_IAP22511

If you happen to be in Austin for SxSW feel free to stop by.  :-)

The larger point, I suppose,  is that while we debate these things within
our own community, our debate and our claims have unintended consequences in
larger society, with companies like Lumosity, in effect marketing to the
baby boomers the idea (false) that  using “the science of neuroplasticity’
and doing something as simple as playing games “designed by neuroscientists”
can revert their brains to teen age form.  fRMI and Neuropsychology used
extensively as evidence.

Perhaps society has become so accustomed to outlandish claims and over
selling that they won’t hold us accountable.

Or perhaps they will.

Jim

 p.s.  (always a ps)  I have also recently proposed that we declare a
moratorium on neuroimaging studies until we at least know how the signal is
related to actual neural-activity.  Seems rather foolish to base so much
speculation and interpretation on a signal we don’t understand.  Easy enough
to poo poo cell spikers - but to my knowledge, there is no evidence that
neural computing is performed through the generation of areas of red,
yellow, green and blue.  :-)

On Jan 25, 2014, at 9:43 AM, Stephen José Hanson
<jose at psychology.rutgers.edu> wrote:

Indeed.  Its like we never stopped arguing about this for the last 30 years!
Maybe this is a brain principle
integrated fossilized views of the brain principles.

I actually agree with John.. and disagree with you JIm... surprise
surprise...seems like old times..

The most disconcerting thing about the emergence the new new neural network
field(s)
is that the NIH Connectome RFPs contain language about large scale network
functions...and
yet when Program managers are directly asked whether fMRI or any
neuroimaging methods
would be compliant with the RFP.. the answer is "NO".

So once the neuroscience cell spikers get done analyzing 1000 or 10000 or
even a 1M neurons
at a circuit level.. we still won't know why someone makes decisions about
the shoes they wear; much
less any other mental function!   Hopefully neuroimaging will be relevant
again.

Just saying.

Cheers.

Steve
PS.  Hi Gary!  Dijon!

Stephen José Hanson
Director RUBIC (Rutgers Brain Imaging Center)
Professor of Psychology
Member of Cognitive Science Center (NB)
Member EE Graduate Program (NB)
Member CS Graduate Program (NB)
Rutgers University 

email: jose at psychology.rutgers.edu
web: psychology.rutgers.edu/~jose
lab: www.rumba.rutgers.edu
fax: 866-434-7959
voice: 973-353-3313 (RUBIC)

On Fri, 2014-01-24 at 17:31 -0600, james bower wrote:

Well, well - remarkable!!!  an actual debate on connectionists - just like
the old days - in fact REMARKABLY like the old days. 

Same issues - how ‘brain-like’ is ‘brain-like’ and how much hype is
‘brain-like’ generating by itself. How much do engineers really know about
neuroscience, and how much do neurobiologists really know about the brain
(both groups tend to claim they know a lot  - now and then). 

I went to the NIPS meeting this year for the first time in more than 25
years.  Some of the older timers on connectionists may remember that I was
one of the founding members of NIPS - and some will also remember that a few
years of trying to get some kind of real interaction between neuroscience
and then ‘neural networks’ lead me to give up and start, with John Miller,
the CNS meetings - focused specifically on computational neuroscience.
Another story -  

At NIPS this year, there was a very large focus on “big data” of course,
with "machine learning" largely replaced "Neural Networks" in most talk
titles.  I was actually a panelist (most had no idea of my early involvement
with NIPS) on big data in on-line learning (generated by Ed-X, Kahn, etc)
workshop.  I was interested, because for 15 years I have also been running
Numedeon Inc, whose virtual world for kids, Whyville.net
<http://whyville.net/>  was the first game based immersive worlds, and is
still one of the biggest and most innovative.  (no MOOCs there). 

>From the panel I made the assertion, as I had, in effect,  many years ago,
that if you have a big data problem - it is likely you are not taking
anything resembling a ‘brain-like’ approach to solving it.  The version
almost 30 years ago, when everyone was convinced that the relatively simple
Hopfield Network could solve all kinds of hard problems, was my assertion
that, in fact, simple ‘Neural Networks, or simple Neural Network learning
rules were unlikely to work very well, because, almost certainly, you have
to build a great deal of knowledge about the nature of the problem into all
levels (including the input layer) of your network to get it to work. 

Now, many years later, everyone seems convinced that you can figure things
out by amassing an enormous amount of data and working on it. 

It has been a slow revolution (may actually not even be at the revolutionary
stage yet), BUT it is very likely that the nervous system (like all model
based systems) doesn’t collect tons of data to figure out with feedforward
processing and filtering, but instead, collects the data it thinks it needs
to confirm what it already believes to be true.  In other words, it
specifically avoids the big data problem at all cost.  It is willing to
suffer the consequence that occasionally (more and more recently for me),
you end up talking to someone for 15 minutes before you realize that they
are not the person you thought they were. 

An enormous amount of engineering and neuroscience continues to think that
the feedforward pathway is from the sensors to the inside - rather than
seeing this as the actual feedback loop.  Might to some sound like a
semantic quibble,  but I assure you it is not. 

If you believe as I do, that the brain solves very hard problems, in very
sophisticated ways, that involve, in some sense the construction of complex
models about the world and how it operates in the world, and that those
models are manifest in the complex architecture of the brain - then
simplified solutions are missing the point. 

What that means inevitably, in my view, is that the only way we will ever
understand what brain-like is, is to pay tremendous attention experimentally
and in our models to the actual detailed anatomy and physiology of the
brains circuits and cells. 

I saw none of that at NIPS - and in fact, I see less and less of that at the
CNS meeting as well. 

All too easy to simplify, pontificate, and sell. 

So, I sympathize with Juyang Wang’s frustration. 

If there is any better evidence that we are still in the dark, it is that we
are still having the same debate 30 years later, with the same ruffled
feathers, the same bold assertions (mine included) and the same seeming lack
of progress. 

If anyone is interested, here is a chapter I recently wrote of the book I
edited on “20 years of progress in computational neuroscience (Springer) on
the last 40 years trying to understand the workings of a single neuron (The
cerebellar Purkinje cell), using models.
https://www.dropbox.com/s/5xxut90h65x4ifx/272602_1_En_5_DeltaPDF%20copy.pdf 

Perhaps some sense of how far we have yet to go. 

Jim Bower 

On Jan 24, 2014, at 4:00 PM, Ralph Etienne-Cummings
<ralph.etiennecummings at gmail.com> wrote: 

Hey, I am happy when our taxpayer money, of which I contribute way more than
I get back, funds any science in all branches of the government.  

Neuromorphic and brain-like computing is on the rise ... Let's please not
shoot ourselves in the foot with in-fighting!!

Thanks,
Ralph's Android

On Jan 24, 2014 4:13 PM, "Juyang Weng" <weng at cse.msu.edu> wrote: 

Yes, Gary, you are correct politically, not to upset the "emperor" since he
is always right and he never falls behind the literature.  

But then no clear message can ever get across.   Falling behind the
literature is still the fact.  More, the entire research community that does
brain research falls behind badly the literature of necessary disciplines.
The current U.S. infrastructure of this research community does not fit at
all the brain subject it studies!  This is not a joking matter.  We need to
wake up, please. 

Azriel Rosenfeld criticized the entire computer vision filed in his invited
talk at CVPR during early 1980s: "just doing business as usual" and "more or
less the same" .   However, the entire computer vision field still has not
woken up after 30 years!   As another example, I respect your colleague
Terry Sejnowski, but I must openly say that I object to his "we need more
data" as the key message for the U.S. BRAIN Project.  This is another
example of "just doing business as usual" and so everybody will not be
against you.    

Several major disciplines are closely related to the brain, but the
scientific community is still very much fragmented, not willing to wake up.
Some of our government officials only say superficial worlds like "Big Data"
because we like to hear.   This cost is too high for our taxpayers. 

-John   

On 1/24/14 2:19 PM, Gary Cottrell wrote:

Hi John - 

It's great that you have an over-arching theory, but if you want people to
read it, it would be better not to disrespect people in your emails. You say
you respect Matthew, but then you accuse him of falling behind in the
literature because he hasn't read your book. Politeness (and modesty!) will
get you much farther than the tone you have taken. 

g. 

On Jan 24, 2014, at 6:27 PM, Juyang Weng <weng at cse.msu.edu> wrote: 

Dear Matthew:

My apology if my words are direct, so that people with short attention spans
can quickly get my points.  I do respect you.

You wrote: "to build hardware that works in a more brain-like way than
conventional computers do.  This is not what is usually meant by research in
neural networks."

Your statement is absolutely not true.  Your term "brain-like way" is as old
as "brain-like computing".  Read about the 14 neurocomputers built by 1988
in Robert Hecht-Nielsen, "Neurocomputing: picking the human brain", IEEE
Spectrum 25(3), March 1988, pp. 36-41.  Hardware will not solve the
fundamental problems of the current human severe lack in understanding the
brain, no matter how many computers are linked together.  Neither will the
current "Big Data" fanfare from NSF in U.S..  The IBM's brain project has
similar fundamental flaws and the IBM team lacks key experts.  

Some of the NSF managers have been turning blind eyes to breakthrough work
on brain modeling for over a decade, but they want to waste more taxpayer's
money into its "Big Data" fanfare and other "try again" fanfares.  It is a
scientific shame for NSF in a developed country like U.S. to do that
shameful politics without real science, causing another large developing
country like China to also echo "Big Data".  "Big Data" was called "Large
Data", well known in Pattern Recognition for many years.  Stop playing
shameful politics in science!  

You wrote: "Nobody is claiming a `brain-scale theory that bridges the wide
gap,' or even close." 

To say that, you have not read the book: Natural and Artificial Intelligence
<http://www.brain-mind-institute.org/press.html> .  You are falling behind
the literature so bad as some of our NSF project managers.  With their lack
of knowledge, they did not understand that the "bridge" was in print on
their desks and in the literature.     

-John

On 1/23/14 6:15 PM, Matthew Cook wrote:

Dear John, 

I think all of us on this list are interested in brain-like computing, so I
don't understand your negativity on the topic. 

Many of the speakers are involved in efforts to build hardware that works in
a more brain-like way than conventional computers do.  This is not what is
usually meant by research in neural networks.  I suspect the phrase
"brain-like computing" is intended as an umbrella term that can cover all of
these efforts. 

I think you are reading far more into the announcement than is there.
Nobody is claiming a "brain-scale theory that bridges the wide gap," or even
close.  To the contrary, the announcement is very cautious, saying that
intense research is "gradually increasing our understanding" and "beginning
to shed light on the human brain".  In other words, the research advances
slowly, and we are at the beginning.  There is certainly no claim that any
of the speakers has finished the job. 

Similarly, the announcement refers to "successful demonstration of some of
the underlying principles [of the brain] in software and hardware", which
implicitly acknowledges that we do not have all the principles.  There is
nothing like a claim that anyone has enough principles to "explain highly
integrated brain functions". 

You are concerned that this workshop will avoid the essential issue of the
wide gap between neuron-like computing and highly integrated brain
functions.  What makes you think it will avoid this?  We are all interested
in filling this gap, and the speakers (well, the ones who I know) all either
work on this, or work on supporting people who work on this, or both. 

This looks like it will be a very nice workshop, with talks from leaders in
the field on a variety of topics, and I wish I were able to attend it. 

Matthew 

On Jan 23, 2014, at 7:08 PM, Juyang Weng wrote: 

Dear Anders,

Interesting topic about the brain!  But Brain-Like Computing is misleading
because neural networks have been around for at least 70 years.

I quote: "We are now approaching the point when our knowledge will enable
successful demonstrations of some of the underlying principles in software
and hardware, i.e. brain-like computing."

What are the underlying principles?  I am concerned that projects like
"Brain-Like Computing" avoid essential issues: 
the wide gap between neuron-like computing and well-known highly integrated
brain functions.
Continuing this avoidance would again create bad names for "brain-like
computing", just such behaviors did for "neural networks".

Henry Markram criticized IBM's brain project which does miss essential brain
principles, but has he published such principles?
Modeling individual neurons more and more precisely will explain highly
integrated brain functions?   From what I know, definitely not, by far. 

Has any of your 10 speakers published any brain-scale theory that bridges
the wide gap?  Are you aware of any such published theories? 

I am sorry for giving a CC to the list, but many on the list said that they
like to hear discussions instead of just event announcements. 

-John

On 1/13/14 12:14 PM, Anders Lansner wrote:

Workshop on Brain-Like Computing, February 5-6 2014 

The exciting prospects of developing brain-like information processing is
one of the Deans Forum focus areas.
As a means to encourage progress in this research area a Workshop is
arranged February 5th-6th 2014 on KTH campus in Stockholm. 

The human brain excels over contemporary computers and robots in processing
real-time unstructured information and uncertain data as well as in
controlling a complex mechanical platform with multiple degrees of freedom
like the human body. Intense experimental research complemented by
computational and informatics efforts are gradually increasing our
understanding of underlying processes and mechanisms in small animal and
mammalian brains and are beginning to shed light on the human brain. We are
now approaching the point when our knowledge will enable successful
demonstrations of some of the underlying principles in software and
hardware, i.e. brain-like computing.

This workshop assembles experts, from the partners and also other leading
names in the field, to provide an overview of the state-of-the-art in
theoretical, software, and hardware aspects of brain-like computing.

List of speakers 

Speaker

Affiliation

Giacomo Indiveri

ETH Zürich

Abigail Morrison

Forschungszentrum Jülich

Mark Ritter

IBM Watson Research Center

Guillermo Cecchi

IBM Watson Research Center

Anders Lansner

KTH Royal Institute of Technology

Ahmed Hemani

KTH Royal Institute of Technology

Steve Furber

University of Manchester

Kazuyuki Aihara

University of Tokyo

Karlheinz Meier

Heidelberg University

Andreas Schierwagen

Leipzig University

For signing up to the Workshop please use the registration form found at
http://bit.ly/1dkuBgR

You need to sign up before January 28th.

Web page:
http://www.kth.se/en/om/internationellt/university-networks/deans-forum/work
shop-on-brain-like-computing-1.442038 

******************************************

Anders Lansner

Professor in Computer Science, Computational biology

School of Computer Science and Communication

Stockholm University and Royal Institute of Technology (KTH)

ala at kth.se, +46-70-2166122 <tel:%2B46-70-2166122> 

  _____  

 <http://www.avast.com/> 

Detta epostmeddelande innehåller inget virus eller annan skadlig kod för
avast! Antivirus <http://www.avast.com/>  är aktivt. 

-- 
--
Juyang (John) Weng, Professor
Department of Computer Science and Engineering
MSU Cognitive Science Program and MSU Neuroscience Program
428 S Shaw Ln Rm 3115
Michigan State University
East Lansing, MI 48824 USA
Tel: 517-353-4388
Fax: 517-432-1061
Email: weng at cse.msu.edu
URL: http://www.cse.msu.edu/~weng/ <http://www.cse.msu.edu/%7Eweng/> 
----------------------------------------------

-- 
--
Juyang (John) Weng, Professor
Department of Computer Science and Engineering
MSU Cognitive Science Program and MSU Neuroscience Program
428 S Shaw Ln Rm 3115
Michigan State University
East Lansing, MI 48824 USA
Tel: 517-353-4388
Fax: 517-432-1061
Email: weng at cse.msu.edu
URL: http://www.cse.msu.edu/~weng/ <http://www.cse.msu.edu/%7Eweng/> 
----------------------------------------------

[I am in Dijon, France on sabbatical this year. To call me, Skype works best
(gwcottrell), or dial +33 788319271 <tel:%2B33%20788319271> ] 

Gary Cottrell 858-534-6640 FAX: 858-534-7029 

My schedule is here: http://tinyurl.com/b7gxpwo 

Computer Science and Engineering 0404 

IF USING FED EX INCLUDE THE FOLLOWING LINE:      
CSE Building, Room 4130
University of California San Diego
9500 Gilman Drive # 0404
La Jolla, Ca. 92093-0404

Things may come to those who wait, but only the things left by those who
hustle. -- Abraham Lincoln 

"Of course, none of this will be easy. If it was, we would already know
everything there was about how the brain works, and presumably my life would
be simpler here. It could explain all kinds of things that go on in
Washington." -Barack Obama 

"Probably once or twice a week we are sitting at dinner and Richard says,
'The cortex is hopeless,' and I say, 'That's why I work on the worm.'" Dr.
Bargmann said.

"A grapefruit is a lemon that saw an opportunity and took advantage of it."
- note written on a door in Amsterdam on Lijnbaansgracht.

"Physical reality is great, but it has a lousy search function." -Matt Tong

"Only connect!" -E.M. Forster

"You always have to believe that tomorrow you might write the matlab program
that solves everything - otherwise you never will." -Geoff Hinton 

"There is nothing objective about objective functions" - Jay McClelland

"I am awaiting the day when people remember the fact that discovery does not
work by deciding what you want and then discovering it."
-David Mermin

Email: gary at ucsd.edu
Home page: http://www-cse.ucsd.edu/~gary/ <http://www-cse.ucsd.edu/%7Egary/>

-- 
--
Juyang (John) Weng, Professor
Department of Computer Science and Engineering
MSU Cognitive Science Program and MSU Neuroscience Program
428 S Shaw Ln Rm 3115
Michigan State University
East Lansing, MI 48824 USA
Tel: 517-353-4388
Fax: 517-432-1061
Email: weng at cse.msu.edu
URL: http://www.cse.msu.edu/~weng/
----------------------------------------------

Dr. James M. Bower Ph.D.

Professor of Computational Neurobiology

Barshop Institute for Longevity and Aging Studies.

15355 Lambda Drive

University of Texas Health Science Center 

San Antonio, Texas  78245

Phone:  210 382 0553

Email: bower at uthscsa.edu

Web: http://www.bower-lab.org <http://www.bower-lab.org/> 

twitter: superid101

linkedin: Jim Bower

CONFIDENTIAL NOTICE:

The contents of this email and any attachments to it may be privileged or
contain privileged and confidential information. This information is only
for the viewing or use of the intended recipient. If you have received this
e-mail in error or are not the intended recipient, you are hereby notified
that any disclosure, copying, distribution or use of, or the taking of any
action in reliance upon, any of the information contained in this e-mail, or

any of the attachments to this e-mail, is strictly prohibited and that this
e-mail and all of the attachments to this e-mail, if any, must be

immediately returned to the sender or destroyed and, in either case, this
e-mail and all attachments to this e-mail must be immediately deleted from
your computer without making any copies hereof and any and all hard copies
made must be destroyed. If you have received this e-mail in error, please
notify the sender by e-mail immediately.

-- 

Dr. James M. Bower Ph.D.

Professor of Computational Neurobiology

Barshop Institute for Longevity and Aging Studies.

15355 Lambda Drive

University of Texas Health Science Center 

San Antonio, Texas  78245

Phone:  210 382 0553

Email: bower at uthscsa.edu

Web: http://www.bower-lab.org

twitter: superid101

linkedin: Jim Bower

CONFIDENTIAL NOTICE:

The contents of this email and any attachments to it may be privileged or
contain privileged and confidential information. This information is only
for the viewing or use of the intended recipient. If you have received this
e-mail in error or are not the intended recipient, you are hereby notified
that any disclosure, copying, distribution or use of, or the taking of any
action in reliance upon, any of the information contained in this e-mail, or

any of the attachments to this e-mail, is strictly prohibited and that this
e-mail and all of the attachments to this e-mail, if any, must be

immediately returned to the sender or destroyed and, in either case, this
e-mail and all attachments to this e-mail must be immediately deleted from
your computer without making any copies hereof and any and all hard copies
made must be destroyed. If you have received this e-mail in error, please
notify the sender by e-mail immediately.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/connectionists/attachments/20140125/966dac50/attachment.html>