We provide a broad survey of the cooperative multi-agent learning literature. Previous surveys of this area have largely focused on issues common to specific subareas (for example, reinforcement learning or robotics). In this survey we attempt to draw from multi-agent learning work in a spectrum of areas, including reinforcement learning, evolutionary computation, game theory, complex systems, agent modeling, and robotics.

We find that this broad view leads to a division of the work into two categories, each with its own special issues: applying a single learner to discover joint solutions to multi-agent problems (team learning), or using multiple simultaneous learners, often one per agent (concurrent learning). Additionally, we discuss direct and indirect communication in connection with learning, plus open issues in task decomposition, scalability, and adaptive dynamics. We conclude with a presentation of multi-agent learning problem domains, and a list of multi-agent learning resources.

Despite the relative youth of the field, there are already several hundreds of papers trying to answer interesting questions in domains where learning affects the behavior of more than a single agent. Such questions target issues as varied as learning to communicate, modeling other agents in the environment, learning to compete or cooperate with the other agents, analysis of optimality for the learning algorithms, and many others. The large variety of papers generates a desirability for good surveys categorizing previous work.

We argue that there are two alternatives for learning in multiagent systems. A first direction is to have each individual agent learn how to improve its performance. The alternative is to have a single learning process that improves the behavior of the entire team of agents. Because of scalability problems with respect to increased numbers of agents, the majority of machine learning techniques can not be realistically applied to learn team behaviors. We discovered that most previous surveys concentrate on individual agents' learning, but neglect approaches at the team level.

In this survey, we propose a new arrangement for multiagent learning papers. As such, there is an entire category of papers dealing with learning behaviors for the entire team and issues associated with scalability to non-trivial numbers of agents, such as the heterogeneity of the team. A second class of papers is concerned with individual agent learning in multiagent domains. Based on their main research focus, the papers are categorized in sections dealing with optimality of learned behaviors, impact of locality of reward information, cooperation or competition relations among agents, and modeling other agents.

Additionally, we identified two issues relatively perpendicular to the team or individual levels of learning: problem decomposition and communication. The former is concerned with techniques for decomposing either the problem to be solved or the collective behavior into simpler independent components that can be more easily handled by the learning process. An in-depth survey of the literature on the relationship between multiagent learning and communication revealed three approaches, each with its own particularities. The three such methods involve communication via either rapidly decaying information, slowly decaying information, or embodiment.

Experiments have been performed over a 2000 km^2 area comprising a part of the continental boundary between the African and Eurasian plate, roughly corresponding to Italy and Greece, one of the most seismically active regions. Using a 365-days-long time series, we identified three signals associated with seismic events. Additionally, it was possible to establish that the extent of the signal does not propagate further than 600 km from the epicenter of the earthquake.

RoboCup 2009 provides an opportunity for the Autonomous Robotics Lab at George Mason University to apply its expertise in evolutionary algorithms to robotics. Our research goals in developing the RoboPatriots have focused on three elements: first, we are interested in the multiagent coordination aspects of the problem. Second, we are simplifying the design of robot motion behaviors by gathering human motion data, then converting this data to the humanoid servos. Third, we are optimizing these motion behaviors in a massively distributed evo- lutionary computation system attached to a custom humanoid robot simulator of our own devising.