Connectionists: New paper on why modules evolve, and how to evolve modular neural networks

Mon Apr 1 22:00:25 EDT 2013

Dear John Weng, 

The modularity definition you quote is not our definition of modularity. I'm not sure why you think it is, as ours is quite different and we do not cite Calabretta & Parisi, 2005. Please read our paper to understand the notion of modularity we cite, describe, and investigate.  

Here is the PDF: http://jeffclune.com/publications/2013-CluneEtAl-EvolutionaryOriginsModularity-RoyalSociety.pdf

A relevant snippet from the first paragraph: "Networks are modular if they contain highly connected clusters of nodes that are sparsely connected to nodes in other clusters [4,8,9]."

4. Wagner GP, Pavlicev M, Cheverud JM. 2007 The road to modularity. Nat. Rev. Genet. 8, 921 – 931. (doi:10.1038/nrg2267)
8. Lipson H, 2007 Principles of modularity, regularity, and hierarchy for scalable systems. J. Biol. Phys. Chem. 7, 125 – 128. 
9. Striedter G. 2005 Principles of brain evolution. Sunderland, MA: Sinauer Associates.

Best regards,
Jeff Clune

Assistant Professor
Computer Science
University of Wyoming
jeffclune at uwyo.edu
jeffclune.com

On Apr 1, 2013, at 4:57 PM, Juyang Weng <weng at cse.msu.edu> wrote:

> Dear Jeff Clune:
> 
> Thank you for your email.  Since this subject seems to be interesting to many people on this list, I take the liberty of giving the list a CC.  Those who do not like to read can simply delete.  Many people told me that they like this type of discussion, in addition to many announcements. 
> 
> The definition of modularity you quoted (from Calabretta & Parisi, 2005) is
> "In a modular architecture each weight is always involved in a single task: Modules are sets of ‘proprietary’ connections that are only used to accomplish a single task.”
> 
> The above seems to be grossly wrong according to known biological mechanisms of a cell.  It is well known that each area (e.g., each Bodmann area in the brain) and any set of connections in the brain are ALL involved in many many tasks.
> 
> Why?  I give an intuitive explanation.  
> 
> (a) Every receptor in the retina (or every pixel in a camera) is involved in many many tasks.  E.g., it can
> be the projection of a part of a human face now, but next it is the projection of a part of a bush.  
> 
> (b) Every muscle (effector) in the body is involved in many many tasks. For example, consider a motor neuron that drives a muscle in your upper left arm.  Then this neuron is involved in many many tasks that the left arm performs (e.g., dancing, working, fighting, exercising).  
> 
> (c) Since every neurons inside the brain serves for the connections among receptors and effectors (according to the DN theory) among other services (e.g., neuromodulation), the above two reasons have determined that there exists no neuron in the brain, or any set of neurons in the brain, that is "only used to accomplish a single task".
> 
> In other words, the well-accepted theory of modularity, at least as defined above, is fundamentally wrong.  The main reason for this mistake seems to be a lack of computational understanding of biological cell mechanisms and how individual and autonomous cells communicate with one another.  This process of autonomous communications seems to be sufficient to give rise to impressive array of brain functions.  
> This rise is in the absence of any "central controller".   The automata theory in computer science has been used to explain how, although the DN theory is very different from the traditional symbolic automata theory. 
> 
> -John
> 
> On 4/1/13 5:07 PM, Jeff Clune wrote:
>> Dear John Weng,
>> 
>> Our paper on the evolutionary origins of modularity focuses on modularity in biological networks in general, not just neural networks. The dynamics you describe are not relevant to all biological networks (e.g. metabolic networks, genetic regulatory networks, protein-protein interaction networks, etc.). However, we are particularly interested in the evolution of neural modularity, and neural networks have more obvious connection costs than many biological networks, so we do think our paper sheds light on the origins of neural modularity as well. 
>> 
>> On that front, while I of course agree that each connection in a complex neural wiring diagrams is not genetically specified, genes ultimately encode the rules that govern neural development. Evolution may thus favor modularity via selection for certain types of developmental programs (e.g. those that tend to produce fewer, shorter connections). Just because development plays a substantial role does not mean that genes do not as well. One can easily imagine selection for developmental programs that lead to fully connected neural networks, but that did not occur in nature. A major force that prevented that from happening is likely the many different costs that would be incurred for all those connections (including the cranial space to house them). Our work suggests that minimizing connection costs leads to modularity; that minimization could be accomplished via genetically-encoded developmental rules. 
>> 
>> I think our results are thus complementary with work investigating how neural development is biased towards creating modular connectivity patterns, and may even suggest a reason why there was selection for such developmental biases in the first place.  
>> 
>> 
>> Best regards,
>> Jeff Clune
>> 
>> Assistant Professor
>> Computer Science
>> University of Wyoming
>> jeffclune at uwyo.edu
>> jeffclune.com
>> 
>> On Mar 30, 2013, at 1:21 PM, Juyang Weng <weng at cse.msu.edu> wrote:
>> 
>>> Dear Jeff Clune:
>>> 
>>> Thank you for pointing to the URL.  I quote some statements below in two paragraphs.   Although I agree that the genome has made a "best guess" when a zygote forms, it is simple-minded to attribute the modularity of the brain, even at the birth time, primarily to "evolution of modularity" as you put it.  In other words, unlike the zygote, the brain of a new born is no longer simply the "best guess" of the genome.  The body of the new born has played a fundamental role in the formation of the modularity inside the newborn's brain.   Namely, the "emergence" or development, is the key process for brain's modularity in the newborn and of course also in the later life.  
>>> If you have a chance to read our computational model of the DEVELOPMENT of a brain-inspired network DN, at least computationally DN does not need to attribute its emergence of modularity to anything other than a set of cell mechanisms.  This is because of the cell-centered role of the genes, known as genomic equivalence.  For example, each cell grows and connects according to signals from other cells in its neighborhood (not primarily genes!).  Many biological experiments have shown how autonomous cells (whose properties are
>>> to some degree genome specified) communicate to migrate, differentiate, form tissues (e.g., cortex), and connect.  In our DN model, such cell behaviors give rise to surprising brain-like capabilities when sensory and motor signals are present. 
>>> By attention to "emergence" in the paragraphs I quoted below. 
>>> -John 
>>> "The existence of modules is recognized at all levels of the biological hierarchy. In order to understand what modules are, why and how they emerge and how they change, it would be necessary to start a joint effort by researchers in different disciplines (evolutionary and developmental biology, comparative anatomy, physiology, neuro- and cognitive science). This is made difficult by disciplinary specialization. [...] we claim that, because of the strong similarities in the intellectual agenda of artificial life and evolutionary biology and of their common grounding in Darwinian evolutionary theory, a close interaction between the two fields could easily take place. Moreover, by considering that artificial neural networks draw an inspiration from neuro- and cognitive science, an artificial life approach to the problem could theoretically enlarge the field of investigation." (Calabretta et al., 1998)
>>> 
>>> A general definition of modularity and nonmodularity in neural networks can be the following: “modular systems can be defined as systems made up of structurally and/or functionally distinct parts. While non-modular systems are internally homogeneous, modular systems are segmented into modules, i.e., portions of a system having a structure and/or function different from the structure or function of other portions of the system. [...] In a nonmodular architecture one and the same connection weight may be involved in two or more tasks. In a modular architecture each weight is always involved in a single task: Modules are sets of ‘proprietary’ connections that are only used to accomplish a single task.” (Calabretta & Parisi, 2005, Fig. 14.4; see also Calabretta et al., 2003).
>>> 
>>> On 3/29/13 8:30 PM, Jeff Clune wrote:
>>>> Hello Christos,
>>>> 
>>>> Rafael Calabretta keeps a list of papers on the subject of the evolution of modularity. 
>>>> 
>>>> http://gral.ip.rm.cnr.it/rcalabretta/modularity.html
>>>> 
>>>> I like your idea of a wiki too. It could be a great resource for the field. We could even start fleshing out this page, which is currently nearly empty: http://en.wikipedia.org/wiki/Modularity_(biology)
>>>> 
>>>> PS. Thanks to everyone who has participated in the discussion of our paper The Evolutionary Origins of Modularity. Some of the papers that have been mentioned we reference in our paper, and others are new to us. We have enjoyed learning about the various different studies and opinions on this subject, and look forward to more great work to come. 
>>>> 
>>>> 
>>>> Best regards,
>>>> Jeff Clune
>>>> 
>>>> Assistant Professor
>>>> Computer Science
>>>> University of Wyoming
>>>> jeffclune at uwyo.edu
>>>> jeffclune.com
>>>> 
>>>> On Mar 29, 2013, at 3:31 PM, Christos Dimitrakakis <christos.dimitrakakis at gmail.com> wrote:
>>>> 
>>>>> Dear all,
>>>>> 
>>>>> Is there no survey or taxonomy that discusses this line of work in one 
>>>>> place?
>>>>> If not, I have a suggestion. Why not start up a wiki to begin with? That 
>>>>> would also be of tremendous aid to any newcomers.
>>>>> 
>>>>> Best,
>>>>> Christos
>>>>> 
>>>>> -- 
>>>>> Dr. Christos Dimitrakakis
>>>>> http://lia.epfl.ch/People/dimitrak/
>>>>> 
>>>> 
>>> 
>>> -- 
>>> --
>>> Juyang (John) Weng, Professor
>>> Department of Computer Science and Engineering
>>> MSU Cognitive Science Program and MSU Neuroscience Program
>>> 428 S Shaw Ln Rm 3115
>>> Michigan State University
>>> East Lansing, MI 48824 USA
>>> Tel: 517-353-4388
>>> Fax: 517-432-1061
>>> Email: weng at cse.msu.edu
>>> URL: http://www.cse.msu.edu/~weng/
>>> ----------------------------------------------
>>> 
>> 
> 
> -- 
> --
> Juyang (John) Weng, Professor
> Department of Computer Science and Engineering
> MSU Cognitive Science Program and MSU Neuroscience Program
> 428 S Shaw Ln Rm 3115
> Michigan State University
> East Lansing, MI 48824 USA
> Tel: 517-353-4388
> Fax: 517-432-1061
> Email: weng at cse.msu.edu
> URL: http://www.cse.msu.edu/~weng/
> ----------------------------------------------
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.srv.cs.cmu.edu/mailman/private/connectionists/attachments/20130401/5efce33e/attachment.html>