Simple pictures, tough problems.

[Steve Lehar]

> The problem is that I want the architecture  to generalize. This means
> that it should group small and  large circles,  rather than circles of
> size X with squares of size nearly X.
> ...
>
> the net generalizes not  on second-order properties like  form, but on
> first-order properties (roughly: where the black spots in  the picture
> reside). Thus, circles are  seen similar  to squares  when their sizes
> "match".

Maybe the solution is to take  a cue from nature.  How  does the brain
represent visual information?  Hubel and Wiesel [1] found simple cells
which respond to the very  simplest visual primitives-  oriented edges
or moving edges.  They also found complex cells, which generalized the
simple cell responses "spatially",  but not "featurally".  This  is an
important point, and is, I  believe,  the key to understanding how the
brain solves the generalization problem.

Say you have a simple cell  that fires in response  to a vertical edge
in a very specific location.  A complex cell might fire for a vertical
edge in a much larger range of locations (spatial generalization), but
this is not due to the complex cell having a coarser representation of
the world,  because the complex cell will  not fire in  response  to a
cruder fuzzier edge, it is every bit as  specific  about the sharpness
of that vertical edge as was the simple cell- i.e. we have NO featural
generalization, just spatial.

When you move up  to complex and  hyper complex cells, you   get cells
that respond  to even  more  specialized  features, such   as end-stop
detectors that fire for a vertical edge in a large  region but only if
it terminates, not if it  goes straight  through, and corner detectors
which  fire  for  two   end-stop detectors,   one  vertical  and   one
horizontal.  Notice the trend- as we become more general spatially, we
become more    specific  featurally.    This  is  what    I  call  the
spatial/featural hierarchy, and one can posit that at  the pinnacle of
the hierarchy would be found very specific detectors that respond, for
example, to your  grandmother's face, wherever  it  may  appear in the
visual field.   This  is  the basic  idea behind the Neocognitron [2],
although  I believe   that that   model is lacking   in one  important
element, that  being resonant  feedback  between  the  levels   of the
hierarchy,  which Grossberg [3] shows is  so  important to maintain  a
consistancy between different levels of the representation.  I discuss
a resonant spatial/featural hierarchy and how it may be implemented in
[4] and [5].

Now you might argue that the construction of such a hierarchy would be
very  expensive in both  space    and  time  (memory and  computation)
especially  if it is implemented  as I propose, with resonant feedback
between all  the layers of  the hierarchy.  My  response would be that
the problem of vision is by  no means trivial, and  that until we come
up with a better solution, we cannot presume to do better than nature,
and if nature deems it necessary to  create such a  hierarchy,  then I
strongly suspect that that hierarchy is  an essential prerequisite for
featural generalization.

[1] Hubel & Wiesel RECEPTIVE FIELDS AND FUNCTIONAL ARCHITECTURE IN TWO
    NONSTRIATE   VISUAL   AREAS   OF   THE  CAT(1965)    Journal    of
    Neurophysiology 28 229-289

[2] Fukushima   & Miyake  NEOCOGNITRON:   A  NEW ALGORITHM FOR  PATTERN
    RECOGNITION TOLERANT OF DEFORMATIONS AND SHIFTS IN POSITION.(1982)
    Pattern Recognition 15, 6 455-469

[3] Grossberg,   Stephen  & Mingolla,   Ennio.    NEURAL  DYNAMICS  OF
    PERCEPTUAL    GROUPING:    TEXTURES,   BOUNDARIES   AND   EMERGENT
    SEGMENTATIONS Perception & Psychophysics (1985), 38 (2), 141-171.

[4] Lehar S.,  Worth A. MULTI   RESONANT BOUNDARY CONTOUR  SYSTEM,  Boston
    University,   Center   for  Adaptive   Systems   technical  report   
    CAS/CNS-TR-91-017.  To get a copy, write to...

      Boston University
      Center for Adaptive Systems
      111 Cummington Street, Second Floor
      Boston, MA 02215
      (617) 353-7857,7858

[5] Lehar S., Worth A. MULTIPLE RESONANT BOUNDARY CONTOUR SYSTEM.  In:
    PROGRESS IN NEURAL NETWORKS volume 3 (Ed. by Ablex Publishing Corp.)
    In print. (i.e. not available yet)