Connectionists: New paper on why modules evolve, and how to evolve modular neural networks
Juyang Weng
weng at cse.msu.edu
Tue Apr 2 10:42:33 EDT 2013
> "Networks are modular if they contain highly connected clusters of
nodes that are sparsely connected to nodes in other clusters [4,8,9]."
Then, it should not be called modularity. Almost all networks have this
property, except for some trivial ones. Where are the boundaries of
your modules?
-John
On 4/1/13 10:00 PM, Jeff Clune wrote:
> Dear John Weng,
>
> The modularity definition you quote is not our definition of
> modularity. I'm not sure why you think it is, as ours is quite
> different and we do not cite Calabretta & Parisi, 2005. Please read
> our paper to understand the notion of modularity we cite, describe,
> and investigate.
>
> Here is the PDF:
> http://jeffclune.com/publications/2013-CluneEtAl-EvolutionaryOriginsModularity-RoyalSociety.pdf
>
> A relevant snippet from the first paragraph: "Networks are modular if
> they contain highly connected clusters of nodes that are sparsely
> connected to nodes in other clusters [4,8,9]."
>
> 4. Wagner GP, Pavlicev M, Cheverud JM. 2007 The road to
> modularity. Nat. Rev. Genet. 8, 921 – 931. (doi:10.1038/nrg2267)
> 8. Lipson H, 2007 Principles of modularity, regularity, and hierarchy
> for scalable systems. J. Biol. Phys. Chem. 7, 125 – 128.
> 9. Striedter G. 2005 Principles of brain evolution. Sunderland, MA:
> Sinauer Associates.
>
>
>
> Best regards,
> *Jeff Clune*
>
> Assistant Professor
> Computer Science
> University of Wyoming
> jeffclune at uwyo.edu <mailto:jeffclune at uwyo.edu>
> jeffclune.com
>
> On Apr 1, 2013, at 4:57 PM, Juyang Weng <weng at cse.msu.edu
> <mailto:weng at cse.msu.edu>> wrote:
>
>> Dear Jeff Clune:
>>
>> Thank you for your email. Since this subject seems to be interesting
>> to many people on this list, I take the liberty of giving the list a
>> CC. Those who do not like to read can simply delete. Many people
>> told me that they like this type of discussion, in addition to many
>> announcements.
>>
>> The definition of modularity you quoted (from Calabretta & Parisi
>> <http://gral.ip.rm.cnr.it/rcalabretta/tedarwin/Altenberg.pdf>, 2005) is
>> "In a modular architecture each weight is always involved in a single
>> task: Modules are sets of ‘proprietary’ connections that are only
>> used to accomplish a single task.”
>>
>> The above seems to be grossly wrong according to known biological
>> mechanisms of a cell. It is well known that each area (e.g., each
>> Bodmann area in the brain) and any set of connections in the brain
>> are ALL involved in many many tasks.
>>
>> Why? I give an intuitive explanation.
>>
>> (a) Every receptor in the retina (or every pixel in a camera) is
>> involved in many many tasks. E.g., it can
>> be the projection of a part of a human face now, but next it is the
>> projection of a part of a bush.
>>
>> (b) Every muscle (effector) in the body is involved in many many
>> tasks. For example, consider a motor neuron that drives a muscle in
>> your upper left arm. Then this neuron is involved in many many tasks
>> that the left arm performs (e.g., dancing, working, fighting,
>> exercising).
>>
>> (c) Since every neurons inside the brain serves for the connections
>> among receptors and effectors (according to the DN theory) among
>> other services (e.g., neuromodulation), the above two reasons have
>> determined that there exists no neuron in the brain, or any set of
>> neurons in the brain, that is "only used to accomplish a single task".
>>
>> In other words, the well-accepted theory of modularity, at least as
>> defined above, is fundamentally wrong. The main reason for this
>> mistake seems to be a lack of computational understanding of
>> biological cell mechanisms and how individual and autonomous cells
>> communicate with one another. This process of autonomous
>> communications seems to be sufficient to give rise to impressive
>> array of brain functions.
>> This rise is in the absence of any "central controller". The
>> automata theory in computer science has been used to explain how,
>> although the DN theory is very different from the traditional
>> symbolic automata theory.
>>
>> -John
>>
>> On 4/1/13 5:07 PM, Jeff Clune wrote:
>>> Dear John Weng,
>>>
>>> Our paper on the evolutionary origins of modularity focuses on
>>> modularity in biological networks in general, not just neural
>>> networks. The dynamics you describe are not relevant to all
>>> biological networks (e.g. metabolic networks, genetic regulatory
>>> networks, protein-protein interaction networks, etc.). However, we
>>> are particularly interested in the evolution of neural modularity,
>>> and neural networks have more obvious connection costs than many
>>> biological networks, so we do think our paper sheds light on the
>>> origins of neural modularity as well.
>>>
>>> On that front, while I of course agree that each connection in a
>>> complex neural wiring diagrams is not genetically specified, genes
>>> ultimately encode the rules that govern neural development.
>>> Evolution may thus favor modularity via selection for certain types
>>> of developmental programs (e.g. those that tend to produce fewer,
>>> shorter connections). Just because development plays a substantial
>>> role does not mean that genes do not as well. One can easily imagine
>>> selection for developmental programs that lead to fully connected
>>> neural networks, but that did not occur in nature. A major force
>>> that prevented that from happening is likely the many different
>>> costs that would be incurred for all those connections
>>> (including the cranial space to house them). Our work suggests that
>>> minimizing connection costs leads to modularity; that minimization
>>> could be accomplished via genetically-encoded developmental rules.
>>>
>>> I think our results are thus complementary with work investigating
>>> how neural development is biased towards creating modular
>>> connectivity patterns, and may even suggest a reason why there was
>>> selection for such developmental biases in the first place.
>>>
>>>
>>> Best regards,
>>> *Jeff Clune*
>>>
>>> Assistant Professor
>>> Computer Science
>>> University of Wyoming
>>> jeffclune at uwyo.edu <mailto:jeffclune at uwyo.edu>
>>> jeffclune.com <http://jeffclune.com>
>>>
>>> On Mar 30, 2013, at 1:21 PM, Juyang Weng <weng at cse.msu.edu
>>> <mailto:weng at cse.msu.edu>> wrote:
>>>
>>>> Dear Jeff Clune:
>>>>
>>>> Thank you for pointing to the URL. I quote some statements below
>>>> in two paragraphs. Although I agree that the genome has made a
>>>> "best guess" when a zygote forms, it is simple-minded to attribute
>>>> the modularity of the brain, even at the birth time, primarily to
>>>> "evolution of modularity" as you put it. In other words, unlike the
>>>> zygote, the brain of a new born is no longer simply the "best
>>>> guess" of the genome. The body of the new born has played a
>>>> fundamental role in the formation of the modularity inside the
>>>> newborn's brain. Namely, the "emergence" or development, is the
>>>> key process for brain's modularity in the newborn and of course
>>>> also in the later life.
>>>>
>>>> If you have a chance to read our computational model of the
>>>> DEVELOPMENT of a brain-inspired network DN, at least
>>>> computationally DN does not need to attribute its emergence of
>>>> modularity to anything other than a set of cell mechanisms. This
>>>> is because of the cell-centered role of the genes, known as genomic
>>>> equivalence. For example, each cell grows and connects according
>>>> to signals from other cells in its neighborhood (not primarily
>>>> genes!). Many biological experiments have shown how autonomous
>>>> cells (whose properties are
>>>> to some degree genome specified) communicate to migrate,
>>>> differentiate, form tissues (e.g., cortex), and connect. In our DN
>>>> model, such cell behaviors give rise to surprising brain-like
>>>> capabilities when sensory and motor signals are present.
>>>>
>>>> By attention to "emergence" in the paragraphs I quoted below.
>>>>
>>>> -John
>>>>
>>>> "The existence of modules is recognized at all levels of the
>>>> biological hierarchy. In order to understand what modules are, why
>>>> and how they emerge and how they change, it would be necessary to
>>>> start a joint effort by researchers in different disciplines
>>>> (evolutionary and developmental biology, comparative anatomy,
>>>> physiology, neuro- and cognitive science). This is made difficult
>>>> by disciplinary specialization. [...] we claim that, because of the
>>>> strong similarities in the intellectual agenda of artificial life
>>>> and evolutionary biology and of their common grounding in Darwinian
>>>> evolutionary theory, a close interaction between the two fields
>>>> could easily take place. Moreover, by considering that artificial
>>>> neural networks draw an inspiration from neuro- and cognitive
>>>> science, an artificial life approach to the problem could
>>>> theoretically enlarge the field of investigation." (Calabretta /et
>>>> al./, 1998
>>>> <http://laral.istc.cnr.it/rcalabretta/calabretta.modul3.pdf>)
>>>>
>>>> *A general definition of modularity and nonmodularity in neural
>>>> networks can be the following*: “modular systems can be defined as
>>>> systems made up of structurally and/or functionally distinct parts.
>>>> While non-modular systems are internally homogeneous, modular
>>>> systems are segmented into modules, i.e., portions of a system
>>>> having a structure and/or function different from the structure or
>>>> function of other portions of the system. [...] In a /nonmodular/
>>>> architecture one and the same connection weight may be involved in
>>>> two or more tasks. In a /modular/ architecture each weight is
>>>> always involved in a single task: /Modules are sets of
>>>> ‘proprietary’ connections that are only used to accomplish a single
>>>> task./” (Calabretta & Parisi
>>>> <http://gral.ip.rm.cnr.it/rcalabretta/tedarwin/Altenberg.pdf>,
>>>> 2005, Fig. 14.4; see also Calabretta /et al./, 2003
>>>> <http://gral.ip.rm.cnr.it/rcalabretta/WhatDoesItTake.pdf>).
>>>>
>>>> On 3/29/13 8:30 PM, Jeff Clune wrote:
>>>>> Hello Christos,
>>>>>
>>>>> Rafael Calabretta keeps a list of papers on the subject of the
>>>>> evolution of modularity.
>>>>>
>>>>> http://gral.ip.rm.cnr.it/rcalabretta/modularity.html
>>>>>
>>>>> I like your idea of a wiki too. It could be a great resource for
>>>>> the field. We could even start fleshing out this page, which is
>>>>> currently nearly empty:
>>>>> http://en.wikipedia.org/wiki/Modularity_(biology)
>>>>> <http://en.wikipedia.org/wiki/Modularity_%28biology%29>
>>>>>
>>>>> PS. Thanks to everyone who has participated in the discussion of
>>>>> our paper The Evolutionary Origins of Modularity. Some of the
>>>>> papers that have been mentioned we reference in our paper, and
>>>>> others are new to us. We have enjoyed learning about the various
>>>>> different studies and opinions on this subject, and look forward
>>>>> to more great work to come.
>>>>>
>>>>>
>>>>> Best regards,
>>>>> *Jeff Clune*
>>>>>
>>>>> Assistant Professor
>>>>> Computer Science
>>>>> University of Wyoming
>>>>> jeffclune at uwyo.edu <mailto:jeffclune at uwyo.edu>
>>>>> jeffclune.com <http://jeffclune.com/>
>>>>>
>>>>> On Mar 29, 2013, at 3:31 PM, Christos Dimitrakakis
>>>>> <christos.dimitrakakis at gmail.com
>>>>> <mailto:christos.dimitrakakis at gmail.com>> wrote:
>>>>>
>>>>>> Dear all,
>>>>>>
>>>>>> Is there no survey or taxonomy that discusses this line of work
>>>>>> in one
>>>>>> place?
>>>>>> If not, I have a suggestion. Why not start up a wiki to begin
>>>>>> with? That
>>>>>> would also be of tremendous aid to any newcomers.
>>>>>>
>>>>>> Best,
>>>>>> Christos
>>>>>>
>>>>>> --
>>>>>> Dr. Christos Dimitrakakis
>>>>>> http://lia.epfl.ch/People/dimitrak/
>>>>>>
>>>>>
>>>>
>>>> --
>>>> --
>>>> Juyang (John) Weng, Professor
>>>> Department of Computer Science and Engineering
>>>> MSU Cognitive Science Program and MSU Neuroscience Program
>>>> 428 S Shaw Ln Rm 3115
>>>> Michigan State University
>>>> East Lansing, MI 48824 USA
>>>> Tel: 517-353-4388
>>>> Fax: 517-432-1061
>>>> Email:weng at cse.msu.edu
>>>> URL:http://www.cse.msu.edu/~weng/
>>>> ----------------------------------------------
>>>>
>>>
>>
>> --
>> --
>> Juyang (John) Weng, Professor
>> Department of Computer Science and Engineering
>> MSU Cognitive Science Program and MSU Neuroscience Program
>> 428 S Shaw Ln Rm 3115
>> Michigan State University
>> East Lansing, MI 48824 USA
>> Tel: 517-353-4388
>> Fax: 517-432-1061
>> Email:weng at cse.msu.edu
>> URL:http://www.cse.msu.edu/~weng/
>> ----------------------------------------------
>>
>
--
--
Juyang (John) Weng, Professor
Department of Computer Science and Engineering
MSU Cognitive Science Program and MSU Neuroscience Program
428 S Shaw Ln Rm 3115
Michigan State University
East Lansing, MI 48824 USA
Tel: 517-353-4388
Fax: 517-432-1061
Email: weng at cse.msu.edu
URL: http://www.cse.msu.edu/~weng/
----------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.srv.cs.cmu.edu/mailman/private/connectionists/attachments/20130402/45123ecd/attachment.html>
More information about the Connectionists
mailing list