Connectionists Learning - Some New Ideas/Questions

[Asim Roy]

This is an attempt to respond to some of the questions raised in
regards to the network design issue in our new learning theory. I have
included all of the responses relevant to all of the remaining issues,
except one by Daniel Crespin, which was too long.
 
A. "Task A. Perform Network Design Task"
 
There is criticism of our learning theory on the grounds that humans
inherit a pre-designed learning architecture and that this architecture
has been designed and structured through the process of evolution
over millions of years. I think this is a biological fact beyond dispute.
The relevant question is what parts of the learning system can feasibly
come pre-designed and what parts cannot. For example, we know that
the architecture of the brain includes separate areas (partitions) for
vision, emotion and long term memory and so on. Thus we inherit a
partition of the brain based on the functions it is expected to perform.
This is a level of the architecture that is indeed pre-designed.
A learning mechanism is also prepackaged with this architecture that
knows this functional organization of the brain. And within these
partitions is available a collection of biological neurons (cells) and their
connections. There is also preprocessing in certain parts of the system
like the vision system. I don't think our theory is in conflict with these
basic facts/ideas at all. Our learning theory relates to "work"
performed by the learning mechanism within this functional
organization of the brain. For example, on a lighter side of this
argument, this learning mechanism may have to design and train
appropriate networks that incorporate knowledge about Windows 95.
None of the Windows 95 knowledge could have been inherited from
our biological ancestors, despite their millions of years of learning. So
the net for Windows 95 could not have come pre-designed to us. No
system, biological or otherwise, can design an appropriate net for a
problem about which it has no knowledge. When I was born, my
parents knew nothing about computers. So neither they nor their
ancestors could have pre-designed nets for me to learn about
computers, unless we bring up the notion of "fixed general purpose
nets/modules" being available in the brain for any kind of learning
situation. These fixed general purpose nets could come with a fixed set
of neurons. But the basic problem with that notion is its conflict with
the idea of "learning" itself. The essence of "learning" is
"generalization" and was discussed in the previous response on issues
related to generalization. Since learning is generalization and since
generalization is attempting to design the smallest possible net, the
idea of "fixed pre-designed" nets is incompatible with the notion of
"learning", whether it is in biological systems or otherwise. Learning
within fixed pre-designed nets is not "learning" at all and can be
dangerous indeed. Since we run the risk of simply over or underfitting
the training data in fixed size nets, we may not "learn" anything in
such pre-designed nets and we might be in grave danger as a species
if we did so on a continuous basis. We could not have survived this
long as a species by doing this - that is, by not being able to "generalize
and  learn".
 
So, in general, problem-specific nets could not feasibly come pre-
designed to us.