locality, RBF's, receptive fields, and all that

[john moody]

Concerning the questions of terminology raised by Tom Dietterich:

After we wrote "Learning with localized receptive fields" (Proceedings of
the 1988 Connectionist Models Summer School, Touretzky, Hinton, and Sejnowski,
eds., Morgan Kaufmann), we found that our choice of terminology was confusing
to some. We tried to remedy this confusion when we wrote "Fast learning in
networks of locally-tuned processing units" (Neural Computation 1(2)281-294,
1989).

The point of confusion is whether "local receptive field" should refer to the
network connectivity pattern (or the afferent connectivity of a single unit)
or the form of the response function of a processing unit. 

We agree with Dietterich that the term "local receptive field" should be 
reserved for internal units whose afferent connections come primarily from a 
local neighborhood of units in the preceding layer.  We believe that this term 
is best NOT used to describe RBF type units, which typically (but not always)
have global connectivity to their input layer.

To describe localized unit response functions, we favor the term "locally-
tuned processing unit" (LTPU). These include RBFs, but are not limited to 
units whose response functions are radially symmetric { R(x,y) = R(|x-y|) }. 
[Indeed, we have found that non-radial LTPUs often work better than the 
standard RBFs.]

The distinction between "local receptive field" and "locally tuned processing
unit" can become blurred when one considers "effective variables". These 
variables are *not* the activations of individual input units, but are rather
implicitly encoded for by the population of input units.  

For example, the orientation selective cells of V1 have "local receptive 
fields" as determined by their afferent connection patterns. However, they 
also "respond locally" in the *effective variables* angular orientation and 
retinal position. The locality of response in angular orientation depends on 
the *values* of the afferent connections, while the locality of response to 
retinal position is due to the locality of the receptive fields. [The 
localities of response to the effective variables could be modeled by a 
network of LTPUs with three input variables. Of course, such a network would 
not be responsive to more that one object in the scene or to different 
levels of illumination.] 

Provided that one is clear about which variables one is referring to,
whether input unit activations or effective variables, confusion between
local connectivity patterns ("local receptive fields") and localized
response functions ("LTPUs") can be avoided.


--John Moody and Christian Darken
-------