Learning Team Strategies: Soccer Case Studies

[Rafal Salustowicz]

            LEARNING TEAM STRATEGIES: SOCCER CASE STUDIES

Rafal Salustowicz           Marco Wiering          Juergen Schmidhuber

                    IDSIA, Lugano (Switzerland)

                 Revised Technical Report IDSIA-29-97
           To appear in the Machine Learning Journal (1998)

We use simulated soccer   to  study multiagent learning.  Each  team's
players   (agents)  share  action  set and    policy,  but  may behave
differently due to position-dependent  inputs.  All agents making up a
team are  rewarded  or punished collectively  in   case of goals.   We
conduct  simulations  with varying  team   sizes, and compare  several
learning algorithms: TD-Q learning with linear neural networks (TD-Q),
Probabilistic Incremental Program Evolution (PIPE), and a PIPE version
that learns   by coevolution (CO-PIPE).    TD-Q is  based on  learning
evaluation functions  (EFs)  mapping  input/action  pairs  to expected
reward.  PIPE  and CO-PIPE  search  policy  space directly.  They  use
adaptive    probability distributions    to  synthesize programs  that
calculate action  probabilities from current  inputs. Our results show
that   linear  TD-Q  encounters    several  difficulties in   learning
appropriate shared EFs. PIPE and  CO-PIPE, however,  do not depend  on
EFs  and  find good policies  faster  and more reliably. This suggests
that in some  multiagent learning scenarios   direct search in  policy
space can offer advantages over EF-based approaches.

                              http://www.idsia.ch/~rafal/research.html
                             ftp://ftp.idsia.ch/pub/rafal/soccer.ps.gz

Related papers by the same authors:

Evolving soccer strategies. In N. Kasabov, R. Kozma, K. Ko, R. O'Shea,
G. Coghill,  and T. Gedeon,  editors,  Progress in Connectionist-based
Information Systems:Proc. of the 4th Intl. Conf. on Neural Information
Processing ICONIP'97, pages 502-505, Springer-Verlag, Singapore, 1997.
		      ftp://ftp.idsia.ch/pub/rafal/ICONIP_soccer.ps.gz

On learning soccer strategies.  In W. Gerstner, A. Germond, M. Hasler,
and J.-D.  Nicoud, editors, Proc. of the 7th Intl. Conf. on Artificial
Neural Networks (ICANN'97),  volume 1327 of Lecture Notes in  Computer
Science,  pages 769-774, Springer-Verlag Berlin Heidelberg, 1997.
                       ftp://ftp.idsia.ch/pub/rafal/ICANN_soccer.ps.gz

**********************************************************************
                        Rafal Salustowicz
 Istituto Dalle Molle di Studi sull'Intelligenza Artificiale (IDSIA)
 Corso Elvezia 36                   e-mail: rafal at idsia.ch
 6900 Lugano                                ==============
 Switzerland                                raf at cs.tu-berlin.de
 Tel (IDSIA) : +41 91 91198-38              raf at psych.stanford.edu
 Tel (office): +41 91 91198-32
 Fax         : +41 91 91198-39         WWW: http://www.idsia.ch/~rafal
**********************************************************************