Summary (long): pattern recognition comparisons

[Vasant Honavar]

>The whole point of using comparatively inefficient NN setups (such as fully
>interconnected backprop nets) is that they are general enough to solve
>complex problems without built-in heuristics.  

	While I know of theoretical results that show that a feedforward
	neural net exists that can adequately encode any arbitrary
 	real-valued function (Hornick, Stinchcombe, & White, 1988;
	Cybenko, 1988; Carrol & Dickinson, 1989), I am not aware of
	any results that suggest that such nets can LEARN any real-vauled
	function using backpropagation (ignoring the issue of 
	computational tractability). 

	Heuristics (or architectural constraints) like those used
	by some researchers for some vision problems - locally linked 
	multi-layer converging nets (probably one of
	the most successful demonstrations is the work of LeCun et al. 
	on handwritten zip code recognition) are interesting because
	they constrain (or bias) the network to develop particular types of
	representations. Also, they might enable efficient learning
	to take place in tasks that exhibit a certain intrinsic structure.
	
	The choice of a particular fixed neural network architecture 
	(even if it is fully interconnected backprop net) implies the 
	use of a corresponding representational bias. 
	Whether such a representational bias is in any sense more
	general than some other (e.g., a network of nodes with limited 
	fan-in but sufficient depth) is questionable (For any given
	completely interconnected feedforward network, there exists
	a functionally equivalent feedforward network of nodes with
	limited fan in - and for some problems, the latter may be
	more efficient).

	On a different note, how does one go about assessing the 
	"generality" of a learning algorithm/architecture in practice?
	I would like to see a discussion on this issue.

	Vasant Honavar (honavar at cs.wisc.edu)