Connectionists: Brain-like computing fanfare and big data fanfare

[Danny Silver]

I would echo Geoff Hinton's  comment.  This is a large and exciting area of research.

Many points of view are necessary and many avenues should be explored.


However, I would like to suggest that we sharpen our efforts in the area of problem definition as we proceed on our individual or collective efforts. Unlike the physical sciences we have had few well defined problems in the area of "how does the brain work" to which the current knowledge  of machine learning can be applied.  If you look up open problems in neuroscience  (try - wikipedia or  23 Problems in Systems Neuroscience, edited by L. Van Hemmen and T. Sejnowslti) you will find large sweeping problems such as "what is consciousness" and "how do we represent time in the brain".  These are important and significant problems, however (at this time) it is challenging to wrap them in specific requirements that would make them well defined (ie. provide a shared common understanding of the problem) and amenable to the formulation of competing hypotheses from the machine learning community.


Subsequently, machine learning researchers have tended to explore more specific areas beginning with a problem statement they "feel" is important to brain function --  such as "how can we learn invariants in a visual or auditory system", "how does one retain and transfer knowledge from one task to another".   Once there is agreement on a problem then many researchers can work on that problem and argue the merits of their solutions.


So .. If we wish to sharpen our focus, let us do so in the area of shared well defined problems upon which we can make meaningful headway.

Asking good questions that come with well developed requirements is the starting point to good science.  At least that is what we tell our graduate students.


.. Danny


=======================
Daniel L. Silver, Ph.D.       danny.silver at acadiau.ca<mailto:danny.silver at acadiau.ca>
Professor,  Jodrey School of Computer Science,   Acadia University
Office 314, Carnegie Hall,     Wolfville, NS  Canada  B4P 2R6
p:902-585-1413              f:902-585-1067


From: Geoffrey Hinton <geoffrey.hinton at gmail.com<mailto:geoffrey.hinton at gmail.com>>
Date: Sunday, 26 January, 2014 3:43 PM
To: Brad Wyble <bwyble at gmail.com<mailto:bwyble at gmail.com>>
Cc: Connectionists list <connectionists at cs.cmu.edu<mailto:connectionists at cs.cmu.edu>>
Subject: Re: Connectionists: Brain-like computing fanfare and big data fanfare

I can no longer resist making one point.

A lot of the discussion is about telling other people what they should NOT be doing. I think people should just get on and do whatever they think might work.  Obviously they will focus on approaches that make use of their particular skills. We won't know until afterwards which approaches led to major progress and which were dead ends. Maybe a fruitful approach is to  model every connection in a piece of retina in order to distinguish between detailed theories of how cells get to be direction selective. Maybe its building huge and very artificial neural nets that are much better than other approaches at some difficult task.  Probably its both of these and many others too. The way to really slow down the expected rate of progress in understanding how the brain works is to insist that there is one right approach and nearly all the money should go to that approach.

Geoff



On Sat, Jan 25, 2014 at 3:00 PM, Brad Wyble <bwyble at gmail.com<mailto:bwyble at gmail.com>> wrote:
I am extremely pleased to see such vibrant discussion here and my thanks to Juyang for getting the ball rolling.

Jim, I appreciate  your comments and I agree in large measure, but I have always disagreed with you as regards the necessity of simulating everything down to a lowest common denominator .  Like you, I enjoy drawing lessons from the history of other disciplines, but unlike you, I don't think the analogy between neuroscience and physics is all that clear cut.  The two fields deal with vastly different levels of complexity and therefore I don't think it should be expected that they will (or should) follow the same trajectory.

To take your Purkinje cell example, I imagine that there are those who view any such model that lacks an explicit simulation of the RNA as being incomplete.  To such a person, your models would also be unfit for the literature. So would we then change the standards such that no model can be published unless it includes an explicit simulation of the RNA?  And why stop there?  Where does it end?  In my opinion, we can't make effective progress in this field if everyone is bound to the molecular level.

I really think that neuroscience presents a fundamental challenge that is not present in physics, which is that progress can only occur when theory is developed at different levels of abstraction that overlap with one another.  The challenge is not how to force everyone to operate at the same level of formal specificity, but how to allow effective communication between researchers operating at different levels.

In aid of meeting this challenge, I think that our field should take more inspiration from engineering, a  model-based discipline that already has to work simultaneously at many different scales of complexity and abstraction.


Best,
Brad Wyble




On Sat, Jan 25, 2014 at 9:59 AM, james bower <bower at uthscsa.edu<mailto:bower at uthscsa.edu>> wrote:
Thanks for your comments Thomas, and good luck with your effort.

I can’t refrain myself from making the probably culturist remark that this seems a very practical approach.

I have for many years suggested that those interested in advancing biology in general and neuroscience in particular to a ‘paradigmatic’ as distinct from a descriptive / folkloric science, would benefit from understanding this transition as physics went through it in the 15th and 16th centuries.  In many ways, I think that is where we are today, although with perhaps the decided disadvantage that we have a lot of physicists around who, again in my view, don’t really understand the origins of their own science.  By that, I mean, that they don’t understand how much of their current scientific structure, for example the relatively clean separation between ‘theorists’ and ‘experimentalists’, is dependent on the foundation build by those (like Newton) who were both in an earlier time.  Once you have a sold underlying computational foundation for a science, then you have the luxury of this kind of specialization - as there is a framework that ties it all together.  The Higgs effort being a very visible recent example.

Neuroscience has nothing of the sort.  As I point out in the article I linked to in my first posting - while it was first proposed 40 years ago (by Rodolfo Llinas) that the cerebellar Purkinje cell had active dendrites (i.e. that there were non directly-synaptically associated voltage dependent ion channels in the dendrite that governed its behavior), and 40 years of anatomically and physiologically realistic modeling has been necessary to start to understand what they do - many cerebellar modeling efforts today simply ignore these channels.  While that again, to many on this list, may seem too far buried in the details, these voltage dependent channels make the Purkinje cell the computational device that it is.

Recently, I was asked to review a cerebellar modeling paper in which the authors actually acknowledged that their model lacked these channels because they would  have been too computationally expensive to include.  Sadly for those authors, I was asked to review the paper for the usual reason - that several of our papers were referenced accordingly.  They likely won’t make that mistake again - as after of course complementing them on the fact that they were honest (and knowledgable) enough to have remarked on the fact that their Purkinje cells weren’t really Purkinje cells - I had to reject the paper for the same reason.

As I said, they likely won’t make that mistake again - and will very likely get away with it.

Imagine a comparable situation in a field (like physics) which has established a structural base for its enterprise.  “We found it computational expedient to ignore the second law of thermodynamics in our computations - sorry”.  BTW, I know that details are ignored all the time in physics as one deals with descriptions at different levels of scale - although even there, the field clearly would like to have a way to link across different levels of scale.   I would claim, however, that that is precisely the “trick’ that biology uses to ‘beat’ the second law - linking all levels of scale together - another reason why you can’t ignore the details in biological models if  you really want to understand how biology works.  (too cryptic a comment perhaps).

Anyway, my advice would be to consider how physics made this transition many years ago, and ask the question how neuroscience (and biology) can now.  Key points I think are:
- you need to produce students who are REALLY both experimental and theoretical (like Newton).  (and that doesn’t mean programs that “import” physicists and give them enough biology to believe they know what they are doing, or programs that link experimentalists to physicists to solve their computational problems)
- you need to base the efforts on models (and therefore mathematics) of sufficient complexity to capture the physical reality of the system being studied (as Kepler was forced to do to make the sun centric model of the solar system even as close to as accurate as the previous earth centered system)
- you need to build a new form of collaboration and communication that can support the complexity of those models.  Fundamentally, we continue to use the publication system (short papers in a journal) that was invented as part of the transformation for physics way back then.  Our laboratories are also largely isolated and non-cooperative, more appropriate for studying simpler things (like those in physics).  Fortunate for us, we have a new communication tool (the Internet) although, as can be expected, we are mostly using it to reimplement old style communication systems (e-journals) with a few twists (supplemental materials).
- funding agencies need to insist that anyone doing theory needs to be linked to the experimental side REALLY, and vice versa.  I proposed a number of years ago to NIH that they would make it into the history books if they simply required the following monday,  that any submitted experimental grant include a REAL theoretical and computational component - Sadly, they interpreted that as meaning that P.I.s should state "an hypothesis" - which itself is remarkable, because most of the ‘hypotheses’ I see stated in Federal grants are actually statements of what the P.I. believes to be true.  Don’t get me started on human imaging studies.  arggg
- As long as we are talking about what funding agencies can do, how about the following structure for grants - all grants need to be submitted collaboratively by two laboratories who have different theories (better models) about how a particular part of the brain works.  The grant should support at set of experiments, that both parties agree distinguish between their two points of view.  All results need to be published with joint authorship.  In effect that is how physics works - given its underlying structure.
- You need to get rid, as quickly as possible, the pressure to “translate” neuroscience research explicitly into clinical significance - we are not even close to being able to do that intentionally - and the pressure (which is essentially a give away to the pharma and bio-tech industries anyway) is forcing neurobiologists to link to what is arguably the least scientific form of research there is - clinical research.  It just has to be the case that society needs to understand that an investment in basic research will eventually result in all the wonderful outcomes for humans we would all like, but this distortion now is killing real neuroscience just at a critical time, when we may finally have the tools to make the transition to a paradigmatic science.
As some of you know, I have been all about trying to do these things for many years - with the GENESIS project, with the original CNS graduate program at Caltech, with the CNS meetings, (even originally with NIPS) and with the first  ‘Methods in Computational Neuroscience Course" at the Marine Biological laboratory, whose latest incarnation in Brazil (LASCON) is actually wrapping up next week, and of course with my own research and students.  Of course, I have not been alone in this, but it is remarkable how little impact all that has had on neuroscience or neuro-engineering.  I have to say, honestly, that the strong tendency seems to be for these efforts to snap back to the non-realistic, non-biologically based modeling and theoretical efforts.

Perhaps Canada, in its usual practical and reasonable way (sorry) can figure out how to do this right.

I hope so.

Jim

p.s. I have also been proposing recently that we scuttle the ‘intro neuroscience’ survey courses in our graduate programs (religious instruction)  and instead organize an introductory course built around the history of the discovery of the origin of the axon potential that culminated in the first (and last) Nobel prize work in computational neuroscience for the Hodkin Huxley model.  The 50th anniversary of that prize was celebrated last year, and the year before I helped to organize a meeting celebrating the 60th anniversary of the publication of the original papers (which I care much more about anyway).  That meeting was, I believe, the first meeting in neuroscience ever organized around a single (mathematical) model or theory - and in organizing it, I required all the speakers to show the HH model on their first slide, indicating which term or feature of the model their work was related to.  Again, a first - but possible, as this is about the only “community model’ we have.

Most Neuroscience textbooks today don’t include that equation (second order differential) and present the HH model primarily as a description of the action potential.   Most theorists regard the HH model as a prime example of how progress can be made by ignoring the biological details.  Both views and interpretations are historically and practically incorrect.  In my opinion, if you can’t handle the math in the HH model, you shouldn’t be a neurobiologist, and if you don’t understand the profound impact of HH’s knowledge and experimental study of the squid giant axon on the model,  you shouldn’t be a neuro-theorist either.  just saying.   :-)


On Jan 25, 2014, at 6:58 AM, Thomas Trappenberg <tt at cs.dal.ca<mailto:tt at cs.dal.ca>> wrote:


James, enjoyed your writing.

So, what to do? We are trying to get organized in Canada and are thinking how we fit in with your (US) and the European approaches and big money. My thought is that our advantage might be flexibility by not having a single theme but rather a general supporting structure for theory and theory-experimental interactions. I believe the ultimate place where we want to be is to take theoretical proposals more seriously and try to make specific experiments for them; like the Higgs project. (Any other suggestions? Canadians, see http://www.neuroinfocomp.ca<http://www.neuroinfocomp.ca/>  if you are not already on there.)

Also, with regards to big data, I believe that one very fascinating thing about the brain is that it can function with 'small data'.

Cheers, Thomas


On 2014-01-25 12:09 AM, "james bower" <bower at uthscsa.edu<mailto:bower at uthscsa.edu>> wrote:
Ivan thanks for the response,

Actually, the talks at the recent Neuroscience Meeting about the Brain Project either excluded modeling altogether  -  or declared we in the US could leave it to the Europeans.  I am not in the least bit nationalistic - but, collecting data without having models (rather than imaginings) to indicate what to collect, is simply foolish, with many examples from history to demonstrate the foolishness.  In fact, one of the primary proponents (and likely beneficiaries) of this Brain Project, who gave the big talk at Neuroscience on the project (showing lots of pretty pictures), started his talk by asking: “what have we really learned since Cajal, except that there are also inhibitory neurons?”  Shocking, not only because Cajal actually suggested that there might be inhibitory neurons - in fact.  To quote “Stupid is as stupid does”.

Forbes magazine estimated that finding the Higgs Boson cost over $13BB, conservatively.  The Higgs experiment was absolutely the opposite of a Big Data experiment - In fact, can you imagine the amount of money and time that would have been required if one had simply decided to collect all data at all possible energy levels?   The Higgs experiment is all the more remarkable because it had the nearly unified support of the high energy physics community, not that there weren’t and aren’t skeptics, but still, remarkable that the large majority could agree on the undertaking and effort.  The reason is, of course, that there was a theory - that dealt with the particulars and the details - not generalities.  In contrast, there is a GREAT DEAL of skepticism (me included) about the Brain Project - its politics and its effects (or lack therefore), within neuroscience.  (of course, many people are burring their concerns in favor of tin cups - hoping).  Neuroscience has had genome envy for ever - the connectome is their response - who says its all in the connections? (sorry ‘connectionists’)  Where is the theory?  Hebb?  You should read Hebb if you haven’t - rather remarkable treatise.  But very far from a theory.

If you want an honest answer to your question - I have not seen any good evidence so far that the approach works, and I deeply suspect that the nervous system is very much NOT like any machine we have built or designed to date. I don’t believe that Newton would have accomplished what he did, had he not, first, been a remarkable experimentalist, tinkering with real things.  I feel the same way about Neuroscience.  Having spent almost 30 years building realistic models of its cells and networks (and also doing experiments, as described in the article I linked to) we have made some small progress - but only by avoiding abstractions and paying attention to the details.  OF course, most experimentalists and even most modelers have paid little or no attention.  We have a sociological and structural problem that, in my opinion, only the right kind of models can fix, coupled with a real commitment to the biology - in all its complexity.  And, as the model I linked tries to make clear - we also have to all agree to start working on common “community models’.  But like big horn sheep, much safer to stand on your own peak and make a lot of noise.

You can predict with great accuracy the movement of the planets in the sky using circles linked to other circles - nice and easy math, and very adaptable model (just add more circles when you need more accuracy, and invent entities like equant points, etc).  Problem is, without getting into the nasty math and reality of ellipses- you can’t possible know anything about gravity, or the origins of the solar system, or its various and eventual perturbations.

As I have been saying for 30 years:  Beware Ptolemy and curve fitting.

The details of reality matter.

Jim





On Jan 24, 2014, at 7:02 PM, Ivan Raikov <ivan.g.raikov at gmail.com<mailto:ivan.g.raikov at gmail.com>> wrote:


I think perhaps the objection to the Big Data approach is that it is applied to the exclusion of all other modelling approaches. While it is true that complete and detailed understanding of  neurophysiology and anatomy is at the heart of neuroscience, a lot can be learned about signal propagation in excitable branching structures using statistical physics, and a lot can be learned about information representation and transmission in the brain using mathematical theories about distributed communicating processes. As these modelling approaches have been successfully used in various areas of science, wouldn't you agree that they can also be used to understand at least some of the fundamental properties of brain structures and processes?

  -Ivan Raikov

On Sat, Jan 25, 2014 at 8:31 AM, james bower <bower at uthscsa.edu<mailto:bower at uthscsa.edu>> wrote:
[snip]
An enormous amount of engineering and neuroscience continues to think that the feedforward pathway is from the sensors to the inside - rather than seeing this as the actual feedback loop.  Might to some sound like a semantic quibble,  but I assure you it is not.

If you believe as I do, that the brain solves very hard problems, in very sophisticated ways, that involve, in some sense the construction of complex models about the world and how it operates in the world, and that those models are manifest in the complex architecture of the brain - then simplified solutions are missing the point.

What that means inevitably, in my view, is that the only way we will ever understand what brain-like is, is to pay tremendous attention experimentally and in our models to the actual detailed anatomy and physiology of the brains circuits and cells.




Dr. James M. Bower Ph.D.
Professor of Computational Neurobiology
Barshop Institute for Longevity and Aging Studies.
15355 Lambda Drive
University of Texas Health Science Center
San Antonio, Texas  78245

Phone:  210 382 0553<tel:210%20382%200553>
Email: bower at uthscsa.edu<mailto:bower at uthscsa.edu>
Web: http://www.bower-lab.org<http://www.bower-lab.org/>
twitter: superid101
linkedin: Jim Bower

CONFIDENTIAL NOTICE:
The contents of this email and any attachments to it may be privileged or contain privileged and confidential information. This information is only for the viewing or use of the intended recipient. If you have received this e-mail in error or are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of, or the taking of any action in reliance upon, any of the information contained in this e-mail, or
any of the attachments to this e-mail, is strictly prohibited and that this e-mail and all of the attachments to this e-mail, if any, must be
immediately returned to the sender or destroyed and, in either case, this e-mail and all attachments to this e-mail must be immediately deleted from your computer without making any copies hereof and any and all hard copies made must be destroyed. If you have received this e-mail in error, please notify the sender by e-mail immediately.





Dr. James M. Bower Ph.D.
Professor of Computational Neurobiology
Barshop Institute for Longevity and Aging Studies.
15355 Lambda Drive
University of Texas Health Science Center
San Antonio, Texas  78245

Phone:  210 382 0553<tel:210%20382%200553>
Email: bower at uthscsa.edu<mailto:bower at uthscsa.edu>
Web: http://www.bower-lab.org
twitter: superid101
linkedin: Jim Bower

CONFIDENTIAL NOTICE:
The contents of this email and any attachments to it may be privileged or contain privileged and confidential information. This information is only for the viewing or use of the intended recipient. If you have received this e-mail in error or are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of, or the taking of any action in reliance upon, any of the information contained in this e-mail, or
any of the attachments to this e-mail, is strictly prohibited and that this e-mail and all of the attachments to this e-mail, if any, must be
immediately returned to the sender or destroyed and, in either case, this e-mail and all attachments to this e-mail must be immediately deleted from your computer without making any copies hereof and any and all hard copies made must be destroyed. If you have received this e-mail in error, please notify the sender by e-mail immediately.





--
Brad Wyble
Assistant Professor
Psychology Department
Penn State University

http://wyblelab.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.srv.cs.cmu.edu/pipermail/connectionists/attachments/20140127/49c3bfdb/attachment.html>