preprints available
Klaus Obermayer
oby at cs.tu-berlin.de
Tue Jul 6 11:04:39 EDT 1999
Dear connectionists,
attached please find abstracts and preprint locations of four manuscripts on
ANN theory and one short manuscript on visual cortex modelling:
---
1. self-organizing maps for similarity data & active learning (book chapter)
2. support vector learning for ordinal data (conference paper)
3. classification on proximity data with LP-machines (conference paper)
4. neural networks in economics (review)
---
5. contrast response and orientation tuning in a mean field model of visual
cortex (conference paper)
---
Comments are welcome!
Cheers
Klaus
-----------------------------------------------------------------------------
Prof. Dr. Klaus Obermayer phone: 49-30-314-73442
FR2-1, NI, Informatik 49-30-314-73120
Technische Universitaet Berlin fax: 49-30-314-73121
Franklinstrasse 28/29 e-mail: oby at cs.tu-berlin.de
10587 Berlin, Germany http://ni.cs.tu-berlin.de/
=============================================================================
Active Learning in Self-Organizing Maps
M. Hasenj\"ager^1, H. Ritter^1 and K. Obermayer^2
^1 Technische Fakult\"at, Universit\"at Bielefeld,
^2 Fachbereich Informatik, Technische Universitaet Berlin
The self-organizing map (SOM) was originally proposed by
T. Kohonen in 1982 on biological grounds and has since then become a
widespread tool for explanatory data analysis. Although introduced as a
heuristic, SOMs have been related to statistical methods in recent years,
which led to a theoretical foundation in terms of cost functions as well as to
extensions to the analysis of pairwise data, in particular of dissimilarity
data. In our contribution, we first relate SOMs to probabilistic autoencoders,
re-derive the SOM version for dissimilarity data, and review part of the
above-mentioned work. Then we turn our attention to the fact, that
dissimilarity-based algorithms scale with O($D^2$), where {\it D} denotes the
number of data items, and may therefore become impractical for real-world
datasets. We find that the majority of the elements of a dissimilarity matrix
are redundant and that a sparse matrix with more than 80% missing values
suffices to learn a SOM representation of low cost. We then describe a strategy
how to select the most informative dissimilarities for a given set of
objects. We suggest to select (and measure) only those elements whose
knowledge maximizes the expected reduction in the SOM cost function. We find
that active data selection is computationally expensive, but may reduce the
number of necessary dissimilarities by more than a factor of two compared to
a random selection strategy. This makes active data selection a viable
alternative when the cost of actually measuring dissimilarities between data
objects comes high.
in: Kohonen Maps (Eds. E. Oja and S. Kaski), Elsevier, pp. 57-70 (1999).
available at: http://ni.cs.tu-berlin.de/publications/#conference
-----------------------------------------------------------------------------
Support vector learning for ordinal regression
R. Herbrich, T. Graepel, and K. Obermayer
Fachbereich Informatik, Technische Universit\"at Berlin
We investigate the problem of predicting variables of ordinal
scale. This task is referred to as {\em ordinal regression} and is
complementary to the standard machine learning tasks of classification
and metric regression. In contrast to statistical models we present a
distribution independent formulation of the problem together with
uniform bounds of the risk functional. The approach presented is based
on a mapping from objects to scalar utility values. Similar to Support
Vector methods we derive a new learning algorithm for the task of
ordinal regression based on large margin rank boundaries. We give
experimental results for an information retrieval task: learning the
order of documents w.r.t.\ an initial query. Experimental results
indicate that the presented algorithm outperforms more naive
approaches to ordinal regression such as Support Vector classification
and Support Vector regression in the case of more than two ranks.
in: International Conference for Artificial Neural Networks 1999 (accepted
for publication)
available at: http://ni.cs.tu-berlin.de/publications/#conference
-----------------------------------------------------------------------------
Classification on proximity data with LP-machines
T. Graepel^1, R. Herbrich^1, B. Sch\"ollkopf^2, A. Smola^2, P. Bartlett^3,
K. M\"uller^2, K. Obermayer^1, and R. Williamson^3
^1 Fachbereich Informatik, Technische Universit\"at Berlin
^2 GMD FIRST
^3 Australian National University
We provide a new linear program to deal with classification of data in the case
of data given in terms of pairwise proximities. This allows to avoid the
problems inherent in using feature spaces with indefinite metric in Support
Vector Machines, since the notion of a margin is purely needed ininput space
where the classification actually occurs. Moreover in our approach we can
enforce sparsity in the proximity representation by sacrificing training error.
This turns out to be favorable for proximity data. Similar to $\nu$--SV methods,
the only parameter needed in the algorithm is the (asymptotical) number of data
points being classified with a margin. Finally, the algorithm is successfully
compared with $\nu$--SV learning in proximity space and $K$--nearest-neighbors
on real world data from Neuroscience and molecular biology.
in: International Conference for Artificial Neural Networks 1999 (accepted
for publication)
available at: http://ni.cs.tu-berlin.de/publications/#conference
-----------------------------------------------------------------------------
Neural Networks in Economics: Background, Applications and New Developments
R. Herbrich, M. Keilbach, T. Graepel, and K. Obermayer
Fachbereich Informatik, Technische Universitaet Berlin
Neural Networks were developed in the sixties as devices for
classification and regression. The approach was originally inspired
from Neuroscience. Its attractiveness lies in the ability to
learn, i.e. to generalize to as yet unseen observations. One aim
of this paper is to give an introduction to the technique of Neural
Networks and an overview of the most popular architectures. We start
from statistical learning theory to introduce the basics of
learning. Then, we give an overview of the general principles of
neural networks and of their use in the field of Economics. A second
purpose is to introduce a recently developed Neural Network Learning
technique, so called Support Vector Network Learning, which is an
application of ideas from statistical learning theory. This approach
has shown very promising results on problems with a limited amount of
training examples. Moreover, utilizing a technique that is known as
the kernel trick, Support Vector Networks can easily be adapted to
nonlinear models. Finally, we present an economic application of this
approach from the field of preference learning.
in: Computational Techniques for Modelling Learning in Economics
(Ed. T. Brenner), Kluwer Academics, in press (1999)
available at: http://ni.cs.tu-berlin.de/publications/#books
-----------------------------------------------------------------------------
On the Influence of Threshold Variability in a Model of the Visual Cortex
H. Bartsch, M. Stetter, and K. Obermayer
Fachbereich Informatik, Technische Universit\"at Berlin
Orientation--selective neurons in monkeys and cats show contrast saturation
and contrast--invariant orientation tuning. Recently proposed models for
orientation selectivity predict contrast invariant orientation tuning but no
contrast saturation at high strength of recurrent intracortical coupling,
whereas at lower coupling strengths the contrast response saturates but the
tuning widths are contrast dependent. In the present work we address the
question, if and under which conditions the incorporation of a stochastic
distribution of activation thresholds of cortical neurons leads to the
saturation of the contrast response curve as a network effect. We find
that contrast saturation occurs naturally if two different classes of
inhibitory inter-neurons are combined. Low threshold inhibition keeps
the gain of the cortical amplification finite, whereas high threshold
inhibition causes contrast saturation.
in: International Conference for Artificial Neural Networks 1999 (accepted
for publication)
available at: http://ni.cs.tu-berlin.de/publications/#conference
More information about the Connectionists
mailing list