TR: neural statistical language model beats trigram

[Yoshua Bengio]

Hello,

The following tech. report that is a long version of the recently
presented NIPS'2000 oral is now available on the web:

            A Neural Probabilistic Language Model
             Y. Bengio, R. Ducharme, P. Vincent
         Tech. Rep. 1178, Dept. of CS&OR / CRM, U of Montreal


A goal of statistical language modeling is to learn the joint probability
function of sequences of words in a language. This is intrinsically
difficult because of the curse of dimensionality: a word sequence on which
the model will be tested is likely to be different from all the word
sequences seen during training.  Traditional but very successful approaches
based on N-grams obtain generalization by gluing very short sequences seen
in the training set.  Instead, we propose to fight the curse of
dimensionality with its own weapons. In the proposed approach one learns
simultaneously (1) a distributed representation for each word along with
(2) the probability function for word sequences, expressed in terms of
these representations. Generalization is obtained because a sequence of
words that has never been seen before gets high probability if it is made
of words that are similar to words forming an already seen sentence. We
report on experiments using neural networks for the probability function,
showing on two text corpora that the proposed approach very significantly
improves on a state-of-the-art trigram model, and that the proposed
approach allows to take advantage of much longer context.

postscript file available at: 
   http://www.iro.umontreal.ca/~lisa/pointeurs/TR1178.ps
(or through my web page: follow publications -> tech. reports)
 
-- 
Yoshua Bengio 
Professeur agrg
Dpartement d'Informatique et Recherche Oprationnelle
Universit de Montral, 
adresse postale: C.P. 6128 Succ. Centre-Ville, Montral, Qubec, Canada H3C 3J7
adresse civique: 2920 Chemin de la Tour, Montral, Qubec, Canada H3T 1J8, #2194
Tel: 514-343-6804. Fax: 514-343-5834. Bureau 3339.
http://www.iro.umontreal.ca/~bengioy
http://www.iro.umontreal.ca/~lisa