Composite networks

[pratt@cs.rutgers.edu]

P.J. Hampson asks:

  >From STAY8026 at iruccvax.ucc.ie Mon Mar  2 09:45:00 1992
  >Subject: Composite networks
  >Hi,
  > 
  >I am interested in modelling tasks in which invariant information from
  >previous input-output pairs is brought to bear on the acquisition of current
  >input-output pairs.  Thus I want to use previously extracted regularity to
  >influence current processing.  Does anyone think this is feasible??
  >...

Papers on how networks can be constructed modularly (source and target have
different topologies, are responsible for different classes) include:
[Waibel et al., IEEASSP], [Pratt & Kamm, IJCNN91], [Pratt et al., AAAI91], 
  [Pratt, CLNL92].  

The working title and abstract for my upcoming PhD thesis (to be described 
in [Pratt, 1992b]) are as follows:

	      Transferring Previously Learned Back-Propagation
		   Neural Networks to New Learning Tasks 

				Lori Pratt


      Neural network learners traditionally extract most of their information
      from a set of training data.  If training data is in short supply, the
      learned classifier may perform poorly. Although this problem can be
      addressed partly by carefully choosing network parameters, this process
      is ad hoc and requires expertise and manual intervention by a system
      designer.

      Several symbolic and neural network inductive learners have explored
      how a domain theory which supplements training data can be
      automatically incorporated into the training process to bias learning.
      However, research to date in both fields has largely ignored an
      important potential knowledge source: classifiers that have been
      trained previously on related tasks.  If new classifiers were able to
      build directly on previous results, then training speed, performance,
      and the ability to effectively utilize small amounts of training data
      could potentially be substantially improved.  This thesis introduces
      the problem of {\em transfer} of information from a trained learner to
      a new learning task.  It also presents an algorithm for transfer
      between neural networks.  Empirical results from several domains
      demonstrate that this algorithm can improve learning speed on a 
      variety of tasks.

This will be published in part as [Pratt, 1992].
						    --Lori

--------------------------------------------------------------------------------
References:
@article{ waibel-89b,
MYKEY           = " waibel-89b : .bap .unr .unb .tem .spc .con ",
TITLE           = "Modularity and Scaling in Large Phonemic Neural
                    Networks",
AUTHOR          = "Alexander Waibel and Hidefumi Sawai and Kiyohiro Shikano",
journal         = "IEEE Transactions on Acoustics, Speech, and Signal
                   Processing",
VOLUME          = 37,
NUMBER          = 12,
MONTH           = "December",
YEAR            = 1989,
PAGES           = {1888-1898}
}

@inproceedings{ pratt-91,
MYKEY           = " pratt-91 : .min .bap .app .spc .con ",
AUTHOR          = "Lorien Y. Pratt and Jack Mostow and Candace A. Kamm",
TITLE           = "{Direct Transfer of Learned Information among Neural
                    Networks}",
BOOKTITLE       = "Proceedings of the Ninth National Conference on
                   Artificial Intelligence (AAAI-91)",
PAGES           = {584--589},
ADDRESS         = "Anaheim, CA",
YEAR            = 1991,
}

@inproceedings{ pratt-91b,
MYKEY           = " pratt-91b : .min .bap .app .spc .con ",
AUTHOR          = "Lorien Y. Pratt and Candace A. Kamm",
TITLE           = "Improving a Phoneme Classification Neural Network through
                   Problem Decomposition",
YEAR            = 1991,
MONTH           = "July",
BOOKTITLE       = "Proceedings of the International Joint Conference on Neural
                   Networks (IJCNN-91)",
ADDRESS         = "Seattle, WA",
PAGES           = {821--826},
ORGANIZATION    = "IEEE",
}

@incollection{ pratt-92,
MYKEY           = " pratt-92 : .min .bap .app .spc .con ",
AUTHOR          = "Lorien Y. Pratt",
TITLE           = "Experiments on the Transfer of Knowledge Between Neural
                   Networks",
BOOKTITLE       = "Computational Learning Theory and Natural Learning Systems,
                   Constraints and Prospects",
EDITOR          = "S. Hanson and G. Drastal and R. Rivest",
YEAR            = 1992,
PUBLISHER       = "MIT Press",
CHAPTER         = "4.1",
NOTE            = "To appear",
}

@incollection{ pratt-92b,
MYKEY           = " pratt-92b : .min .bap .app .spc .con ",
AUTHOR          = "Lorien Y. Pratt",
TITLE           = "Non-literal information transfer between neural networks",
BOOKTITLE       = "Neural Networks: Theory and Applications {II}",
EDITOR          = "R.J.Mammone and Y. Y. Zeevie",
YEAR            = 1992,
PUBLISHER       = "Academic Press",
NOTE            = "To appear",
}

-------------------------------------------------------------------
L. Y. Pratt                      ,_~o      Computer Science Department
pratt at cs.rutgers.edu           _-\_<,      Rutgers University
                              (*)/'(*)     Hill Center  
(908) 932-4974 (CoRE building office)      New Brunswick, NJ  08903, USA
(908) 846-4766 (home)