Connectionists: New paper on why modules evolve, and how to evolve modular neural networks

[Eric Mjolsness]

Here are two old and possibly relevant references that don't quite
answer Jurgen's question on neural wirelength optimization
in evolutionary computation and machine learning.

Wirelength minimization was a key criterion for the hand-design
of an artificial neural network architecture in my 1985 PhD thesis:
"Neural networks, pattern recognition, and fingerprint hallucination",
PhD thesis, posted at: http://emj.ics.uci.edu/?page_id=93 .

Later my collaborators and I experimented with genotype-phenotype
automated design of artificial neural network architectures [1989],
but we replaced the wirelength objective with a more easily
computable "parsimony cost" that measured genotypic information
content required to specify a network. We suggested but
didn't adopt an additional L1 wiring sparseness objective.
[1989] "Scaling, Machine Learning, and Genetic Neural Nets", Eric Mjolsness,
David H. Sharp, and Bradley K. Alpert. Advances 
in Applied Mathematics, June 1989, available at
http://computableplant.ics.uci.edu/papers/1989/ScalingMachineLearningGNN.pdf .

- Eric Mjolsness


>The paper mentions that Santiago Ramón y Cajal 
>already pointed out that evolution has created 
>mostly short connections in animal brains.
>
>Minimization of connection costs should also 
>encourage modularization, e.g., 
>http://arxiv.org/abs/1210.0118 (2012).
>
>But who first had such a wire length term in an 
>objective function to be minimized by 
>evolutionary computation or other machine 
>learning methods?
>I am aware of pioneering work by Legenstein and Maass:
>
>R. A. Legenstein and W. Maass. Neural circuits 
>for pattern recognition with small total wire 
>length. Theoretical Computer Science, 
>287:239-249, 2002.
>R. A. Legenstein and W. Maass. Wire length as a 
>circuit complexity measure. Journal of Computer 
>and System Sciences, 70:53-72, 2005.
>
>Is there any earlier relevant work? Pointers will be appreciated.
>
>Jürgen Schmidhuber
>http://www.idsia.ch/~juergen/whatsnew.html
>
>
>
>
>On Feb 10, 2013, at 3:14 AM, Jeff Clune wrote:
>
>>Hello all,
>>
>>I believe that many in the neuroscience 
>>community will be interested in a new paper 
>>that sheds light on why modularity evolves in 
>>biological networks, including neural networks. 
>>The same discovery also provides AI researchers 
>>a simple technique for evolving neural networks 
>>that are modular and have increased 
>>evolvability, meaning that they adapt faster to 
>>new environments.
>>
>>Cite: Clune J, Mouret J-B, Lipson H (2013) The 
>>evolutionary origins of modularity. Proceedings 
>>of the Royal Society B. 280: 20122863. 
>>http://dx.doi.org/10.1098/rspb.2012.2863 (pdf)
>>
>>Abstract: A central biological question is how 
>>natural organisms are so evolvable (capable of 
>>quickly adapting to new environments). A key 
>>driver of evolvability is the widespread 
>>modularity of biological networks-their 
>>organization as functional, sparsely connected 
>>subunits-but there is no consensus regarding 
>>why modularity itself evolved. Although most 
>>hypotheses assume indirect selection for 
>>evolvability, here we demonstrate that the 
>>ubiquitous, direct selection pressure to reduce 
>>the cost of connections between network nodes 
>>causes the emergence of modular networks. 
>>Computational evolution experiments with 
>>selection pressures to maximize network 
>>performance and minimize connection costs yield 
>>networks that are significantly more modular 
>>and more evolvable than control experiments 
>>that only select for performance. These results 
>>will catalyse research in numerous disciplines, 
>>such as neuroscience and genetics, and enhance 
>>our ability to harness evolution for 
>>engineering pu!
>>rposes.
>>
>>Video: http://www.youtube.com/watch?feature=player_embedded&v=SG4_aW8LMng
>>
>>There has been some nice coverage of this work 
>>in the popular press, in case you are 
>>interested:
>>
>>* National Geographic: 
>>http://phenomena.nationalgeographic.com/2013/01/30/the-parts-of-life/
>>* MIT's Technology Review: 
>>http://www.technologyreview.com/view/428504/computer-scientists-reproduce-the-evolution-of-evolvability/
>>* Fast Company: 
>>http://www.fastcompany.com/3005313/evolved-brains-robots-creep-closer-animal-learning
>>* Cornell Chronicle: 
>>http://www.news.cornell.edu/stories/Jan13/modNetwork.html
>>* ScienceDaily: http://www.sciencedaily.com/releases/2013/01/130130082300.htm
>>
>>I hope you enjoy the work. Please let me know if you have any questions.
>>
>>Best regards,
>>Jeff Clune
>>
>>Assistant Professor
>>Computer Science
>>University of Wyoming
>>jeffclune at uwyo.edu
>>jeffclune.com


-- 
Eric Mjolsness
Professor of Computer Science and Mathematics
Director, Center for Computational Morphodynamics
University of California, Irvine
emj at uci.edu
http://www.ics.uci.edu/~emj
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.srv.cs.cmu.edu/mailman/private/connectionists/attachments/20130225/7dde4168/attachment.html>