Evaluating Architectures for Intelligence 2007

Cognitive architectures form an integral part of robots and agents. Architectures structure and organize the knowledge used by the agents to select actions in dynamic environments, plan and solve problems, learn, and coordinate with others. Architectures serve to integrate general capabilities expected of an intelligent agent (e.g. planning and learning), to implement and test theories about agent cognition, and to explore domain-independent mechanisms for intelligence.

As AI research has improved in formal and empirical rigor, traditional evaluation methodologies for architectures have sometimes proved insufficient. Formal analysis has often proved elusive; we seem to be missing the notation required for proving properties of architectures. Experiments which demonstrate generality are notoriously expensive to perform, and are not sufficiently informative. And at a high-level, evaluation is difficult because the criteria are not well defined: Is it generality? Ease of programmability? Compatibility with data from biology and psychology? There are no established evaluation methodologies and only a handful established evaluation criteria.

Recognizing that scientific progress depends on the ability to conduct informative evaluation (by experiment or formal analysis), this workshop will address the methodologies needed for evaluating architectures. The focus is on methodology, rather than specific architectures. The workshop has two goals: To promote discussion and to propose evaluation criteria that will be accepted by the research community as recognized evaluation guidelines. We intend to publish the results in a special issue of an international journal and to archive presentation slides and explanatory material on an active web site.

Workshop Details

• Technical Program.
• Research topics and intended audience.
• Call For Papers in various formats.
• Organizing Committee.

• What are the underlying research hypotheses one explores with architectures?
• Which functions/characteristics turn an architecture into an architecture supporting intelligence?
• How are architectures to be compared in an informative manner?
• What evaluation methods are needed for different types of cognitive architectures?
• What are the criteria and scales of evaluation?
• How should we validate the design of a cognitive architecture?
• Are there any relevant formal methods? Can we prove properties of architectures?
• Can we develop a common ontology for describing architectures and/or the various sets of requirements against which they can be evaluated?
• How can data-sets and benchmarks (standardized tasks) be used to evaluate architectures? Are there useful case-studies?
• How can we determine what architectures to use for different tasks or environments? Are there any trade-offs involved?

Workshop Format and Submissions

The workshop will be composed of invited and contributed talks on evaluation methodologies, interleaved with panels, and moderated discussions. We seek submission of extended abstracts (2 pages) and short position papers (4 pages) that discuss evaluation methodologies for architectures. See the call for papers for submission instructions. Submission deadline is April 15, 2007.

• Kevin Gluck, Air Force Research Laboratory, USA
• Pat Langley, Stanford University, USA
• Brian Logan, University of Nottingham, UK
• Ralf Mikut, Karlsruhe Institute for Technology, Germany
• Praveen Paritosh, Northwestern University, USA
• Bilge Say, Middle East Technical University, Turkey
• Robert Wray, Soar Technology, Inc., USA

For additional information