shift invariance

[Laurenz Wiskott]

	Dear connectionists,

here is an attempt to put the arguments so far into order (and to to
add a bit).  I have put it into the form of a list of statements,
which are, of course, subjective.  You can skip the indented comments
in the first reading.

---------------------------------------------------------------------------

1) With respect to shift invariance, there are two types of artificial
neural nets (ANNs):
  a) ANNs with a build in concept of spatial order (e.g. neocognitron
and other weight sharing ANNs, neural shifter circuits, dynamic link
matching), let me call these ANNs structured.
  b) ANNs without any build in concept of spatial order (e.g. fully
connected back-propagation), let me call these ANNs isotropic.

	(This distinction is important.  For instance, Jerry
	Feldman's statement referred to isotropic ANNs while Rolf
	W"urtz' disagreement was based on structured ANNs.)

2) Structured ANNs can DO shift invariant pattern discrimination, but
they do not LEARN it.

	(It is important to note that structured ANNs for shift
	invariant pattern recognition usually do NOT LEARN the shift
	invariance. It is already build in (I'd be glad to see a
	counterexample).  What they learn is pattern discrimination,
	under the constraint that, whatever they do, it is shift
	invariant.)
	
3) The isotropic ANNs can learn shift invariant pattern recognition,
given that during training the patterns are presented at ALL
locations.  This is not surprising and not what we are asking for.

	(This is what Christopher Lee pointed out:>>... for a small
	"test problem"-like space, if given an appropriate number of
	nodes a network could simply "memorize" all the configurations
	of an object at all locations. Clearly, this isn't what one
	would normally considering "learning" shift invariance.<<)
	
4) What we are asking for is generalization.  I see two types of
generalization:
  a) generalization of pattern recognition from one part of the input
plain to another.
  b) generalization of shift invariance from one pattern to another.

	(4a example: training on patterns A {101} and B {011} in the
	left half-plane, i.e. {101000, 010100, 011000, 001100}, and
	testing on patterns A and B in the right half-plane,
	e.g. {000101, 000011}.
	4b example: training on some patterns {111, 011, 010} in all
	possible locations, i.e. {111000, 011100, 001110, 000111,
	011000, ..., 000010}, and on pattern A {101} in the left
	half-plane, i.e {101000, 010100}, and testing on pattern A in
	the right half-plane, e.g. {000101}.
	This is again an important distinction.  For instance,
	Jerry Feldman's statement referred to generalization
	4a and Geoffrey Hinton's disagreement referred to
	generalization 4b.)

5) Generalization 4a seems to be impossible for an isotropic ANN.

	(This was illustrated by Jerry Feldman and, more elaborately,
	by Georg Dorffner.)

6) Generalization 4b is possible.

	(This has been demonstrated by the model of Geoffrey Hinton.)

7) Models which generalize according to 4b usually loose discriminative
power, because patterns with the same set of features but in different
spatial order get confused.

	(This has been pointed out by Shlomo Geva.  
	This also holds for some structured ANNs, such as the
	neocognitron and other weight sharing ANNs, but does not hold
	for the neural shifter circuits and dynamic link matching.
	The loss of discriminative power can be avoided by
	using sufficiently complex features, which has its own
	drawbacks.)

---------------------------------------------------------------------------

	Best regards,

		Laurenz Wiskott.

===========================================================================

 .---.           .---.  Dr.  L a u r e n z   W i s k o t t
 |   |. S A L K .|   |  Computational Neurobiology Laboratory     
 |   ||. C N L .||   |	The Salk Institute for Biological Studies 
 |   |||  L W  |||   |	mail: PO Box 85800, San Diego, CA 92186-5800
 |   |||=======|||   |	street, parcels: 10010 North Torrey Pines Road
 `---'''   "   ```---'          La Jolla, CA 92037
 phone: +1 (619) 453-4100 ext 1463; fax: +1 (619) 587-0417; 
 email: wiskott at salk.edu; WWW: http://www.cnl.salk.edu/~wiskott/

===========================================================================