Papers on Rule-Extraction from trained ANN

[Prof invite]

The following papers are available via anonymous ftp:


	An Evaluation And Comparison Of Techniques For Extracting And
	Refining Rules From Artificial Neural Networks

			Robert Andrews* **
			Russell Cable*
			Joachim Diederich*
			Shlomo Geva*
			Mostefa Golea*
			Ross Hayward*
			Chris Ho-Stewart*
			Alan B. Tickle* **


		Neurocomputing Research Centre*
		  School of Information Systems**
	      Queensland University of Technology
		  Brisbane Q 4001 Australia

		      QUTNRC-96-01-04.ps.Z	
			
			   Abstract

	It is becoming increasingly apparent that without some form of
explanation capability, the full potential of trained Artificial Neural
Networks (ANNs) may not be realised. The primary purpose of this report is
to survey techniques which have been developed to redress this situation.
Specifically the survey focuses on mechanisms, procedures, and algorithms
designed to insert knowledge into ANNs (knowledge initialisation), extract
rules from trained ANNs (rule extraction), and utilise ANNs to refine
existing rule bases (rule refinement). The survey also introduces a new
taxanomy for classifying the various techniques, discusses their modus
operandi, and delineates criteria for evaluating their efficacy. An
additional facet of the report is a comparative evaluation of the
performance of a set of techniques developed at the Neurocomputing Research
Centre at QUT to extract knowledge from trained ANNs as a set of symbolic
rules.

Note: This is an extended version of: Andrews, R.; Diederich, J.; Tickle, A.B.
A Survey and Critique of Techniques for Extracting Rules from Trained
Artificial Neural Networks. KNOWLEDGE-BASED SYSTEMS 8 (1995) 6, 373-389.
This version includes first empirical results and is distributed with
permission of the editor and publisher.

*******************************************************************************	

	DEDEC: Decision Detection by Rule Extraction from Neural
		           Networks

			Alan B. Tickle* **
			Marian Orlowski* **
			Joachim Diederich*

		Neurocomputing Research Centre*
		  School of Information Systems**
	      Queensland University of Technology
		  Brisbane Q 4001 Australia

		      QUTNRC-95-01-03.ps.Z	
			
			   Abstract									  														           								
	A clearly recognised impediment to the realisation of the full
potential of Artificial Neural Networks is an inherent inability to explain
in a comprehensible form (e.g. as a set of symbolic rules), the process by
which an ANN arrived at a particular conclusion/decision/result. While a
variety of techniques have already appeared to address this limitation, a
substantial number of the more successful approaches are dependent on
specialised ANN architectures. The DEDEC technique is a generic approach to
rule extraction from trained ANNs which is designed to be applicable across
a broad range of ANN architectures. The DEDEC technique is a generic
approach to rule extraction from trained ANNs which is designed to be
applicable across a broad range of ANN architectures. The basic motif
adopted is to utilise the generalisation capability of a trained ANN to
generate a set of examples from the problem domain which may include
examples beyond the initial training set. These examples are then presented
to a symbolic induction algorithm and the requisite rule set extracted.
However an important innovation over other rule-extraction techniques of
this ('pedagogical') type is that the DEDEC technique utilises information
extracted from an analysis of the weight vectors of the trained ANN to rank
the input variables (rule antecedents) in terms of their relative
importance. This additional information is used to focus the search of the
solution space on those examples from the problem domain which are deemed
to be of most significance. The paper gives a detailed description of one
possible implementation of the DEDEC technique and discusses results
obtained on both a set of structured sample problems and 'real world'
problems.

*******************************************************************************

		DEDEC: A Methodology For Extracting Rules
		From Trained Artificial Neural Networks

			Alan B. Tickle* **
			Marian Orlowski
			Joachim Diederich*

		Neurocomputing Research Centre*
		  School of Information Systems**
	      Queensland University of Technology
		  Brisbane Q 4001 Australia

		      QUTNRC-96-01-05.ps.Z	
			
			   Abstract

	A recognised impediment to the more widespread utilisation of
Artificial Neural Networks (ANNs) is the absence of a capability to
explain, in a human comprehensible form, either the process by which a
specific decision/result has been reached or, in general, the totality of
knowledge embedded within the ANN. Currently, one of the most promising
approaches to redressing this situation is to extract the knowledge
embedded in the trained ANN as a set of symbolic rules. In this paper we
describe the DEDEC methodology for rule-extraction which is applicable to a
broad class of multilayer, feedforward ANNs trained by the
'back-propogation' method. Central to the DEDEC approach is the
identification of the functional dependencies between the ANN inputs (i.e.
the attribute values of the data) and the ANN outputs (e.g. the
classification decision). However the key motif of the DEDEC methodology is
the utilisation of information extracted from analysing the weight vectors
in the trained ANN to focus the process of determining these functional
dependencies. In addition, if required, DEDEC exploits the capability of a
trained ANN to generalise beyond the data used in the ANN training phase.
The paper illustrates one of a number of possible implementations of the
DEDEC methodology, discusses results obtained on both a set of structured
sample problems and a "real world" problem, and provides a comparison with
other rule extraction techniques.

*****************************************************************************

	Artificial Intelligence Meets Artificial Insemination

The Importance and Application of Symbolic Rule Extraction From Trained
		Artificial Neural Networks

			Robert Andrews* **
			Joachim Diederich*
			Emanoil Pop*
			Alan B Tickle* **

		Neurocomputing Research Centre*
		  School of Information Systems**
	      Queensland University of Technology
		  Brisbane Q 4001 Australia

		      QUTNRC-96-01-01.ps.Z	
			
			   Abstract

	In a recent article Andrews et al.[1995] describe a schema for
classifying neural network rule extraction techniques as either
decompositional, eclectic, or pedagogical. Decompositional techniques
require knowledge of the neural network architecture and weights. Each
hidden and output unit is interpreted as a Boolean rule with the
antecedents being a set of incoming links whose summed weights guarantee to
exceed the unit's bias regardless of the activations of the other incoming
links. Pedagogical techniques on the other hand treat the underlyling
neural network as a 'black box' using it to both classify examples and to
generate examples which a symbolic algorithm then converts to rules.
Eclectic techniques combine elements of the two basic categories. In this
paper we describe some reasons why rule extraction is an important area of
research. We then briefly describe three rule extraction algorithms, RULEX,
DEDEC & RULENEG, these being representative of each of the abovementioned
groups. We test these algorithms using two classification problems; the
first being a laboratory benchmarking problem while the second is drawn
from real life. For each problem, each of the rule extraction techniques
previously described is applied to a trained neural network and the
resulting rules presented.

********************************************************************************

	     Rule Extraction From CASCADE-2 Networks			

			Ross Hayward
			Emanoil Pop
			Joachim Diederich

		Neurocomputing Research Centre
	      Queensland University of Technology
		  Brisbane Q 4001 Australia

		      QUTNRC-96-01-02.ps.Z	
			
			   Abstract

	Rule extraction from feed forward neural networks is a topic that
is gaining increasing interest. Any symbolic representation of how a
network arrives at a particular decision is important not only for user
acceptance, but also for rule refinement and network learning. This paper
describes a new method of extracting rules that predict the firing of
single units within a feed forward neural network. The extraction technique
is applied to networks constructed by the Cascade 2 algorithm each of which
solve a different benchmark problem. The hidden and output units within
each of the networks are shown to represent distinct rules which govern the
classification of patterns. Since a discrete rule set can be obtained for
each of the units within the network, a logical mapping between input and
output values can be achieved.

********************************************************************************

		Feasibility of Incremental Learning in 
		Biologically Plausible Networks

			James M. Hogan
			Joachim Diederich

		Neurocomputing Research Centre
	      Queensland University of Technology
		  Brisbane Q 4001 Australia

		      QUTNRC-96-01-03.ps.Z	
			
			   Abstract

	The feasibility of incremental learning within a feed-forward
network is examined under the constraint of biologically plausible
connectivity. A randomly connected network (of physiologically plausible
global connection probability) is considered under the assumption of a
local connection probability which decays with distance between nodes. The
representation of the function XOR is chosen as a test problem, and the
likelihood of its recruitment is discussed with reference to the
probability of occurrence of a subnetwork suitable for implementation of
this function, assuming a uniform initial weight distribution.

******************************************************************************

These papers are available from

ftp.fit.qut.edu.au

cd to /pub/NRC/tr/ps