reply to the open letter to Amari

[Shun-ichi Amari]

[[ Editor's Note:  In a personal note, I thanked Dr. Amari for his
response.  I had assumed, incorrectly, that Dr. Pellionisz had sent a
copy to Dr. Amari who is not a Neuron Digest subscriber.  I'm sure all
readers will remember that Neuron Digest is not a peer-referreed journal
but an informal forum for electronic communication. I hope the debate can
come to a fruitful conclusion -PM ]]

Dear Editor : 
     Professor Usui at Toyohashi Institute of Technology and Science
kindly let me know that there is an "open letters to Amari" in Neuron
Digest.  I was so surprised that an open letter to me was published
without sending it to me.  Moreover, the letter requires me to answer
repeatedly what I have already answered to Dr. Pellionisz.  I again try
to repeat my answer in more detail.

Reply to Dr. Pellionisz
                                   by Shun-ichi Amari

1.  Dr. Pellionisz accused me that I have two contradictory opinions : 1)
My work is a generalization of his and 2) my approach is nothing to do
with his.  This is incorrect.  Once one reads my paper ("Dualistic
geometry of the manifold of higher-order neurons", Neural Networks, vol.
4 (1991), pp. 443-451; see also another paper "Information geometry of
Boltzmann machines" by S. Amari, K. Kurata and H. Nagaoka, IEEE Trans. on
Neural Networks, March 1992), it is immediately clear 1) that my work is
never a generalization of his and 2) more strongly that it has nothing to
do with Pellionisz' work.  Dr. Pellionisz seems accusing me without
reading or understanding my paper at all. I would like to ask the readers
to read my paper.
     For those readers who have not yet read my paper, I would like to
compare his work with mine in the following, because this is what Dr.
Pellionisz has carefully avoided.

2.  We can summarize his work in that the main function of the cerebellum
is a transformation of a covariant vector to a contravariant vector in a
metric Euclidean space since non-orthogonal reference bases are used in
the brain.  He mentioned verbally non-linear generalizations and so on,
but nothing scientific has been done along this line.

3.  In my 1991 paper, I proposed a geometrical theory of the manifold of
parameterized non-linear systems, with special reference to the manifold
of non-linear higher order neurons.  I did not focus on any functions of
a neural network but the mutual relations among different neural networks
such as the distance of two different neural networks, the curvature of a
family of neural networks and its role, etc. Here the method of
information geometry plays a fundamental role.  It uses a dual pair of
affine connections, which is a new concept in differential geometry, and
has been proved to be very useful for analyzing statistical inference
problems, multiterminal information theory, the manifold of linear
control systems, and so on (see S.Amari, Differential Geometrical Methods
of Statistics, Springer Lecture Notes in Statistics, vol.28, 1985 and
many papers referred to in its second printing).  Now a number of
mathematicians are studying on this new subject.  I have shown that the
same method of information geometry is applicable to the manifold of
neural networks, elucidating the capabilities and limitations of a family
of neural networks in terms of their architecture.
     I have opened, I believe, a new fertile field of studying, not the
behaviors of single neural networks, but the collective properties of the
set or the manifold of neural networks in terms of new differential
geometry.

4.  Now we can discuss the point.  Is my theory a generalization of his
theory?  Definitely No.  If A is a generalization of B, A should include
B as a special example.  My theory does never include any of his
tensorial transformations.  A network is merely a point of the manifold
in my theory.  I have studied a collective behaviors of the manifold but
have not studied properties of points.

5.  The second point.  One may ask that, even if my theory is not a
generalization of his theory, it might have something to do with his
theory so that I should have referred to his work.  The answer is again
no.  Dr. Pellionisz insists that he is a pioneer of tensor theory and my
theory is also tensorial.  This is not true.  My theory is
differential-geometrical, but it does not require any tensorial notation.
Modern differential geometry has been constructed without using tensorial
notations, although it is sometimes convenient to use them.  As one sees
from my paper, its essential part is described without tensor notations.
In differential geometry, what is important is intrinsic structures of
manifolds such as affine connections, parallel transports, curvatures,
and so on.  The Pellionisz theory has nothing to do with these
differential-geometrical concepts.  He used the tensorial notation to
point out that the role of the cerebellum is a special type of linear
transformations, namely a covariant-contravariant linear transformation,
which M.A.Arbib and myself have criticized.

6.  Dr. Pellionisz claims that he is the pioneer of the tensorial theory
of neural networks.  Whenever one uses a tensor, should he refer to
Pellionisz'?  This is rediculus.  Who does claim that he is the pioneer
of using differential equations, linear algebra, probability theory, etc.
in neural network theory?  It is just a commonly used method.  Moreover,
the tensorial method itself had been used since the old time in neural
network research.  For example, in my 1967 paper (S. Amari, A
mathematical theory of adaptive pattern classifiers, IEEE Trans. on EC,
vol.16, pp.299-307) where I proposed the general stochastic gradient
learning method for multilayer networks, I used the metric tensor C
(p.301) in order to transform a covariant gradient vector to the
corresponding contravariant learning vector.  But I suppressed the tensor
notation there.  However, in p.303, I explicitly used the tensorial
notation in order to analyze the dynamic behavior of modifiable parameter
vectors.
     I never claim that Dr. Pellionisz should refer to this old paper,
because the tensorial method itself is of common use to all applied
mathematicians, and my old theory is nothing to do with his except that
both used a covariant-contravariant transformation and tensorial
notations, a common mathematical concept.

7.  I do not like non-productive, time-consuming and non-scientific
discussions like this.  If one reads my paper, everything will be melted
away.  This is nothing to do with the fact that I am unfortunately a
coeditor-in-chief of Neural Networks, a threaten on the intellectual
properties (of tensorial theory), the world-wide competition of
scientific research, etc.  which Dr. Pellionisz hinted as if such were in
the background.
     Instead, this reminded me of the horrible days when Professor M. A.
Arbib and myself were preparing the righteous criticism to his theory
(not a criticism to using the tensor concept but to his theory itself).
I had received astonishing interference repeatedly, which I hope would
never happen again.

     I will disclose my e-mail letter to Pellionisz in the following,
hoping that he discloses his first letter including his unbelievable
request, because it makes the situation and his desire clear.  The reader
would understand why I do not want to continue fruitless discussions with
him.  I also request him to read my paper and to point out which concepts
or theories in my paper are generalizations of his.

The folowing is my old reply to Dr.Pellionisz which he partly referred to
in his "open letter to Amari".

Dear Dr. Pellionisz:

Thank you for your e-mail remarking my recent paper entitled "Dualistic
geometry of the manifold of higher-order neurons".

     As you know very well, we criticized your idea of tensorial approach
in our memorial joint paper with M.Arbib.  The point is that, although
the tensorial approach is welcome, it is too restrictive to think that
the brain function is merely a transformation between contravarian
vectors and covariant vectors; even if we use linear approximations, the
transformation should be free of the positivity and symmetry.  As you may
understand these two are the essential restrictions of
covariant-contravariant transformations.

     You have interests in analyzing a general but single neural network.
Of course this is very important.  However, what I am interested in is to
know a geometrical structures of a set of neural networks (in other
words, a set of brains).  This is a new object of research.  Of course, I
did some work along this line in statistical neurodynamics where a
probability measure is introduced in a manifold of neural networks, and
physicists later have followed a similar idea (E.Gardner and others).
However, a geometrical structure is implicit.

    As you noted, I have written that my paper opens a new fertile field
of neural network research, in the following two senses: First, that we
are treating a set of networks, not the behavior of a single network.
There are vast number of researches on single networks by analytical,
stochastic, tensorial and many other mathematical methods.  The point is
to treat a new object of research, a manifold of neural networks.
Secondly, I have proposed a new concept of dual affine connections, which
mathematician have recently been studying in more detail as mathematical
research.

    So if you have studied the differential geometrical structure of a
manifold of neural networks, I should refer to it.  If you have proposed
a new concept of duality in affine connections, I should refer to it.  If
you are claiming that you used tensor analysis in analyzing behaviors of
single neural networks, it is nothing to do with the field which I have
opened.

    Indeed, when I wrote that paper, I thought to refer to your paper.
But if I did so, I could only state that it is nothing to do with this
new approach.  Moreover, I need to repeat our memorial criticism again.
I do not want to do such irrelevant discussions.

    If you read my paper, I think you understand what is newly opened by
this approach.  Since our righteous criticism to your memorable approach
has been published, we do not need to repeat it again and again.

    I do hope your misunderstanding is resolved by this mail and by
reading my paper.

    Sincerely yours,          Shun-ichi Amari


------------------------------
Neuron Digest   Wednesday, 25 Mar 1992
                Volume 9 : Issue 14

Today's Topics:
       Is it time to change our referring? ("Open Letter" debate)

------------------------------