Input to Past Tense Net

[Steve Pinker]

Dear Jay,

We of course agree with you completely that there's a lot of work to
be done in exploring both the properties of nets and the relevant
empirical data.

On the input/output of nets for the past tense: We agree that some of
the problems with RM'86 can be attributed to its using distributed
phonological representations of the stem as input. We also agree that
by using different a different kind of input some of those problems
would be diminished. But models that "take as input a distributed
representation of the intended meaning, and generate as output a
description of the phonological properties of the utterance that
expresses the meaning" is on the wrong track. As we showed in OLC (pp.
110-114), the crucial aspects of the input are not its semantic
properties, but whether the root of its lexical entry is marked as
'irregular', which in turn often depends on the grammatical category
of the root. Two words with different roots will usually have
different meanings, but the difference is epiphenomenal -- there's no
*systematic*, generalization-supporting pattern between verb semantics
and regularity. As we noted, there are words with high semantic
similarity and different past tense forms ('hit/hit', 'strike/struck',
'slap/slapped') and words with low semantic similarity and the same
past tense forms ('come=arrive/came'; 'come=have an organism/came',
'become/became', 'overcome/overcame', 'come to one's senses/came to
one's senses', etc.).

On flying-out: We're not sure what the Elman anecdote is supposed to
imply. The phenomena are quite clear: a word of Category X that is
transparently derived from a word of Category Y is regular with
respect to inflectional rules applying to X. That is why the vast
majority of time one hears 'flied out', not 'flew out' ('flew out' is
a vanishingly rare anecdote worthy of an e-mail message; 'flied-out'
usages would over-run mboxes if we bothered to publicly document every
instance). That's also why all the other examples of unambiguous
cross-category conversion one can think of are regular (see OLC p.
111). That's also why you can add to this list of regulars that are
homophonous with an irregular indefinitely (e.g. 'Mary
out-Sally-Rided/*out-Sally-Rode Sally Ride'; 'Alcatraz
out-Sing-Singed/*out-sang-sang Sing Sing', etc.). And that's why you
find the phenomenon in different categories ('Toronto Maple Leafs')
and in other languages.  In other words we have an absolutely
overwhelming empirical tendency toward overregularizing
cross-categorially derived verbs and an extremely simple and elegant
explanation for it (OLC 111-112). If one is also interested in
accounting for one-shot violations like the Elman anecdote there are
numerous hypotheses to test (an RM86-like model that doesn't apply the
majority of the time (?); a speech error (OLC n.32); hypercorrection
(OLC. p. 127); derivational ambiguity (OLC n. 16), and no doubt
others.)

In general: What the facts are telling us is that the right way to set
up a net for the past tense is to have an input vector that encodes
grammatical cateogry, root/derived status, etc. Perhaps such a net
would "merely implement" a traditional grammar, but perhaps it would
shed new light on the problem, solving some previous difficulties.
What baffles us is why this obvious step would be anathema to so many
connectionists. There seems to be a puzzling trend in connectionist
approaches to language -- the goal of exploring the properties of nets
as psycholinguistic models is married to the goal of promoting a
particular view of language that eschews grammatical representations
of any sort at any cost and tries to use knowledge-driven processing,
associationist-style learning, or both as a substitute. In practice
the empirical side of this effort often relies on isolated anecdotes
and examples and ignores the vast amount of systematic research on the
phenomena at hand. There's no reason why connectionist work on
language has to proceed this way, as Paul Smolensky for one has
pointed out. Why not exploit the discoveries of linguistics and
psycholinguistics, instead of trying to ignore or rewrite them? Our
understanding of both connectionism and of language would be the
better for it.

Steve Pinker
Alan Prince