Layers

[Dave.Touretzky@B.GP.CS.CMU.EDU]

>  From: Michael.Witbrock at F.GP.CS.CMU.EDU

>  Let the distance between two units be defined as the *minimal* number of
>  modifiable weights forming a path between them (i.e. the number of
>  weights on the shortest path between the two nodes) .

>  Then the Layer in which a unit lies is the minimal distance between it
>  and an input unit.

I think you meant to use MAXIMAL distance in the definition of which layer
a unit lies in.  If one uses minimal distance, then in a net with direct
connections from input to output, the output layer would always be layer 1,
even if there were hidden units forming layers 2, 3, etc.  For this definition
to make sense, it should always be the case that if unit i has a connection
to unit j, then Layer(i) <= Layer(j).

> The number of layers in the network is the maximum value of the
   	 distance between any unit and an input unit.

We should tighten this up by specifying that it's ONE PLUS the maximum
distance between any unit and an input unit, EXCLUDING CYCLES.  This
definition is fine for feed-forward nets, but it isn't very satisfying for
recurrent nets like Pineda's.  Imagine a recurrent backprop net in which
every unit was connected to every other.  If such a net has N units, then
by Michael's definition it has N layers.  What's really strange is that
layers 1 through N-1 are empty, and layer N has N units in it.

The notion of layers is just not as useful in recurrent networks.  It is
perhaps better to speak in terms of modules.  A module might be defined as
a set of units with similar connectivity patterns, or as a set of units
that are densely connected to each other and less densely connected to
units in other modules.  This isn't a nice, clean, graph-theoretic
definition, but then whoever said life was as simple as graph theory?

-- Dave