No subject

[SCHOLTES]

TR Available on Recurrent Self-Organization in NLP:
 
 
 
 
       Kohonen Feature Maps in Natural Language Processing
 
                        J.C. Scholtes
                    University of Amsterdam
 
 
Main points: showing the possibilities of Kohonen feature maps in symbolic
             applications by pushing self-organization.
 
             showing a different technique in Connectionist NLP by using
             only (unsupervised) self organization.
 
Although the model is tested in a NLP context, the linguistic aspects of
these experiments are probably less interesting than the connectionist ones.
People inquiring a copy should be aware of this.
 
 
                         Abstract
 
In the 1980s, backpropagation (BP) started the connectionist bandwagon in
Natural Language Processing (NLP). Although initial results were good, some
critical notes must be made towards the blind application of BP. Most such
systems add contextual and semantical features manually by structuring the
input set. Moreover, these models form a small subtract of the brain
structures known from neural sciences. They do not adapt smoothly to a
changing environment and can only learn input/output pairs.
 
Although these disadvantages of the backpropagation algorithm are commonly
known and accepted, other more plausible learning algorithms, such as
unsupervised learning techniques are still rare in the field of NLP. Main
reason is the highly increasing complexity of unsupervised learning methods
when applied in the already complex field of NLP. However, recent efforts
implementing unsupervised language learning have been made, resulting in
interesting conclusions (Elman and Ritter). Sequencing this earlier work,
a recurrent self-organizing model (based on an extension of the Kohonen
feature map), capable to derive contextual (and some semantical) information
from scratch, is presented in detail. The model implements a first step
towards an overall unsupervised language learning system. Simple linguistic
tasks such as single word clustering (representation on the map), syntactical
group formation, derivation of contextual structures, string prediction,
grammatical correctness checking, word sense disambiguation and structure
assigning are carried out in a number of experiments. The performance of the
model is as least as good as achieved in recurrent backpropagation, and at
some points even better (e.g. unsupervised derivation of word classes and
syntactical structures).
 
Although premature, the first results are promising and show possibilities
for other even more biologically-inspired language
processing techniques such as real Hebbian, Genetic or Darwinistic models.
Forthcoming research must overcome limitations still present in the extended
Kohonen model, such as the absence of within layer learning, restricted
recurrence, no look-ahead functions (absence of distributed or unsupervised
buffering mechanisms) and a limited support for an increased number of layers.
 
 
A copy can be obtained by sending a Email message to SCHOLTES at ALF.LET.UVA.NL
Please indicate whether you want a hard copy or a postscript file being send
to you.