Connectionists: New paper on why modules evolve, and how to evolve modular neural networks

Tue Feb 26 15:04:01 EST 2013

Wonderful to see a real conversation on connectionists (instead of an endless stream of meeting and postdoc announcements)

Reminds me of the old days :-) 

Speaking of which, now historical and written for a more general audience, but might be of interest to some:

Nelson. M.E. and  Bower, J.M. 1990 Brain maps and parallel computers. Trends in Neuroscience   13: 403-408.  

Also, I think it is probably important to consider the extent to which we, scientists, like modularity and all it implies and allows.

If you look carefully at the literature, including, even for example recent data published by the Blue Brain Project itself, it turns out that a number of assumptions about local connectivity in neo-cortical networks may not hold.  

to quote:  "Contrary to expectations, we found that the average number of connections in groups of six or more neurons initially increased rather than decreased monotonically with mean inter-somatic distance". 

Perin, Berger, Markram, PNAS 108: 13, 5419-5424.

While these authors still extrapolated their data to support the the existence of a columnar structure, it is very likely that, even in neocortex,  the large majority of synaptic inputs to individual neurons may be "extra-collumnar".  Never the less, we remain committed to the functional concept of cortical columns.  Why?

For more than 30 years now I (and others) have been suggesting that perhaps 3-layered cortex (including olfactory structures especially) may be a better model for thinking about cortical architecture (and core algorithms)  than neo (and especially maybe visual) cortex.  There are good reasons to suggest so evolutionarily - and even computationally.  The number of sparse but distant connections is no surprise to those of us who study 'old cortex', where, despite many efforts, there is no evidence for a columnar structure.   In the long run, it is my guess that the concept of 'modules' and accordingly cortical columns will be seen as theoretically (financially??) convenient rather than a reflection of cortical reality.  

Jim Bower

Beware Ptolemy

On Feb 26, 2013, at 5:01 AM, A.S. <schierwa at informatik.uni-leipzig.de> wrote:

> Dear colleagues, 
> When we look for reasons why modules are formed, we have assumed from the outset that modularity is an ubiquitous property of the brain (neural systems) as a cognitive system.  Next the localization principle often comes into play, assuming an one-to-one relationship between the corresponding structural and functional modules. The hypothesis of the columnar organization of the cerebral cortex basically rests on this idea: columns are structural modules computing certain basis functions, and the columnar network computes any reasonable (cognitive) function (e.g. Maas and Markram´s 2006  model of how cortical microcircuits compute cognitive functions).
> 
> Along these lines of thinking the method of reverse engineering works:
> 
> 1. Capacity analysis: Specify a certain cognitive capacity which is assumed to be produced through the cortex by computing a certain function.
> 
> 2. Decompositional analysis: 
> (a) Functional (computational) analysis: Select a set of basis functions which might serve as functional components or computational units in the cortex.
> (b) Structural analysis: Identify a set of anatomical components of the cortex. Provide evidence that cortical microcircuits are the anatomical components of the cortex.
> 
> 3. Localization: Provide evidence for the functional / computational components being linked with the anatomical components.
> 
> 4. Synthesis:
> (a) Modeling:
> i. Establish a structurally adequate functional model of the computational unit (the presumed ’canonical circuit’) which computes the basis functions of step 2.(a).
> ii. Build a structurally adequate network model of the cortex (or some subsystem) composed of the canonical circuit models.
> (b) Simulation: Prove that the specific cognitive capacity or function under study is computed by the network of circuit models, i.e. through superposition of the specified basis functions.
> 
> This `recipe´ - if reasonable - would be fine. We know, however, there are serious problems with this method. In short:
> 
> @ 1. Specification of a cognitive capacity: 
> Requires a taxonomy of cognitive processes which is out of sight, as is obvious from recent attempts to build cognitive ontologies.
> @ 2.-3. Decomposition -Localization:
> It has been impossible to find the cortical microcircuit that computes a specific basis function. No genetic mechanism has been deciphered that designates how to construct a column. The column structures encountered in many species (but not in all) seem to represent spandrels.
> @ 4. Synthesis / Proof by simulation:
> Sure, producing and understanding complex phenomena from the interaction of simple nonlinear elements like artificial neurons or cellular automata is possible. One expects then, that this would also work for cortical circuits which are recognized as nonlinear devices, and theories could be applied (or developed, if not yet available) that would guide us to which model setup might have generated a given network behavior. 
> However, inverse problems in complex systems  (which processes caused a specific complex behavior of a given system?) are hard because of  ill-posedness. Thus,  from observed activity or function  of cortical circuits and networks we cannot, in principle, infer the internal organization, and the proof is not possible that the particular cognitive capacity under study is generated by the network model. 
> 
> My conclusion is: In cognitive / computational neuroscience we (should) deal with complex, integrated systems. This means, there is no "natural" way to decompose or modularize the brain, neither structurally nor functionally! 
> 
> Details of the arguments can be found here:
> 
> Schierwagen, A.: On Reverse Engineering in the Cognitive and Brain  Sciences.  Natural Computing:  11 (2012), 141-150, doi:10.1007/s11047-012-9306-0 
> 
> 
> Best wishes,
> 
> Andreas  
> ---------------------------------------------------------------
> Prof. Dr. Andreas Schierwagen 
> Universität Leipzig, Institut für Informatik, Germany
>  http://www.informatik.uni-leipzig.de/~schierwa/ 
> 
> 
> 
> Am 24.02.2013 16:05, schrieb Tony Prescott:
>> Dear colleagues,
>> 
>> The Clune et al. article we are discussing mentions that selection for
>> reduced connectivity could be a "spandrel" (the consequence of
>> selection for something else) but does not explore this possibility in
>> much depth.  In the case of biological brains it is hard to see why
>> low connectivity should be directly selected rather than arising
>> through the need to keep a lid on the size and metabolic cost of
>> maintaining the brain.  A 1991 paper by Ringo
>> (http://www.ncbi.nlm.nih.gov/pubmed/1657274) shows that larger brains
>> cannot maintain the same degree of inter-connectedness as smaller ones
>> and therefore long-range connections are necessary sparser if
>> increased an in neuron count is not going to give rise to an
>> exponential increase in brain size.  Reduced connectivity is therefore
>> an architectural constraint for larger brains in not too dissimilar
>> way to the need for spandrels in cathedral domes (as discussed by
>> Gould, 1979).
>> 
>> An important consideration for biological brains is connection length.
>>  Leise 1990 (http://www.ncbi.nlm.nih.gov/pubmed/2194614) provides a
>> useful summary of the reasons why nervous systems are composed of
>> physically modular components with a high number of short-range
>> connections and low number of longer range ones.  As the literature on
>> small world networks show, however, it is important not to assume that
>> physical modularity requires functional modularity.  Appropriate
>> sparse connectivity can allow fast communication and synchronisation
>> across large  networks that can support distributed functional
>> modules.
>> 
>> Regards,
>> 
>> Tony Prescott
>> 
>> 
>> 
>> On 24 February 2013 03:49, Terry Sejnowski <terry at salk.edu> wrote:
>>> G. Mitchison, Neuronal branching patterns and the economy of cortical wiring, Proc. Roy. Soc. London
>>> B Biol. Sci. 245 (1991) 151{158
>>> 
>>> D.B. Chklovskii, C.F. Stevens, Wiring optimization in the brain, Neural Information Processing Systems
>>> (1999)
>>> 
>>> Koulakov AA, Chklovskii DB. Orientation preference patterns in mammalian visual cortex: a wire length minimization approach.  Neuron. 2001 Feb;29(2):519-27.
>>> 
>>> Chklovskii DB, Schikorski T, Stevens CF. Wiring optimization in cortical circuits.
>>> Neuron. 2002 Apr 25;34(3):341-7.
>>> 
>>> Terry
>>> 
>>> -----
>>> 
>>>> The paper mentions that Santiago Ram<F3>n y Cajal already pointed out
>>>> that evolution has created mostly short connections in animal brains.
>>>> 
>>>> Minimization of connection costs should also encourage modularization,
>>>> e.g., http://arxiv.org/abs/1210.0118 (2012).
>>>> 
>>>> But who first had such a wire length term in an objective function to
>>>> be minimized by evolutionary computation or other machine learning
>>>> methods?
>>>> I am aware of pioneering work by Legenstein and Maass:
>>>> 
>>>> R. A. Legenstein and W. Maass. Neural circuits for pattern recognition
>>>> with small total wire length. Theoretical Computer Science,
>>>> 287:239-249, 2002.
>>>> R. A. Legenstein and W. Maass. Wire length as a circuit complexity
>>>> measure. Journal of Computer and System Sciences, 70:53-72, 2005.
>>>> 
>>>> Is there any earlier relevant work? Pointers will be appreciated.
>>>> 
>>>> Juergen Schmidhuber
>>>> http://www.idsia.ch/~juergen/whatsnew.html
>> 
>> 
> 
> 

Dr. James M. Bower Ph.D.

Professor of Computational Neurobiology

Barshop Institute for Longevity and Aging Studies.

15355 Lambda Drive

University of Texas Health Science Center 

San Antonio, Texas  78245

Phone:  210 382 0553

Email: bower at uthscsa.edu

Web: http://www.bower-lab.org

twitter: superid101

linkedin: Jim Bower

CONFIDENTIAL NOTICE:

The contents of this email and any attachments to it may be privileged or contain privileged and confidential information. This information is only for the viewing or use of the intended recipient. If you have received this e-mail in error or are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of, or the taking of any action in reliance upon, any of the information contained in this e-mail, or

any of the attachments to this e-mail, is strictly prohibited and that this e-mail and all of the attachments to this e-mail, if any, must be

immediately returned to the sender or destroyed and, in either case, this e-mail and all attachments to this e-mail must be immediately deleted from your computer without making any copies hereof and any and all hard copies made must be destroyed. If you have received this e-mail in error, please notify the sender by e-mail immediately.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.srv.cs.cmu.edu/mailman/private/connectionists/attachments/20130226/7ab13ae1/attachment-0001.html>