A summary on Catastrophic Interference

[Phil A. Hetherington]

>        * Reduce overlapping in hidden unit representations.  French (1991,
>1994), Kruschke (1992), Murre (1992), Fahlman (1991).
>
>        * Prior training. Assuming later patterns are drawn from the same
>underlying function as earlier patterns, prior training of the general task
>can reduce RI (McRae and Hetherington, 1993).  This proposal will not
>work if later patterns have little to do with previously trained patterns.

Actually, prior training is a method of reduction of overlap at the 
hidden layer, as it achieves the same goal naturally, and will work if 
later patterns have little to do with previously trained patterns.  McRae 
and Hetherington demonstrated that the method was sucessful with 
autoencoders and then replicated this with a network that computed 
arbitrary associations (i.e., random).  Given that prior training also 
consisted of random associations, there was no general function the net 
could derive (i.e., later patterns had little to do with previous 
patterns).  The training merely had the effect of 'sharpening' the 
hidden unit responses so that later items would not be distributed across 
all or most units, as is the case in a naive net.  I would make an 
empirical prediction that in a net trained sequentially to recall faces, 
prior training on white-noise would probably suffice to reduce CI.  

Cheers,
Phil Hetherington