Exploring the Future of Artificial Intelligence Evaluation: A Guest Post by José Hernández-Orallo

We are thrilled to have guest blogger José Hernández-Orallo, a Professor at Technical University of Valencia, share his insights on the recent advancements in measuring artificial intelligence. Professor Hernández-Orallo has been a pioneer in the field of metrics of machine intelligence for over two decades, and his expertise sheds light on the current state of AI evaluation platforms and challenges.

In his blog post, Professor Hernández-Orallo reflects on the progress made in the field of artificial intelligence evaluation over the past year. He notes the increasing interest in assessing artificial general intelligence (AGI) systems, which are capable of finding diverse solutions for a range of tasks. This shift towards evaluating general-purpose AI systems poses new challenges, as traditional task-oriented evaluations are no longer sufficient.

One of the key challenges identified by Professor Hernández-Orallo is the ability of AI agents to reuse representations and skills from one task to new ones, enabling faster learning with minimal examples. This concept of “compositionality” is crucial for building new concepts and skills over previous ones, a skill that is well-documented in humans from early childhood.

Professor Hernández-Orallo highlights two AI evaluation platforms, Malmö and CommAI-env, as being well-suited for testing compositionality in AI agents. Malmö provides a 3D gaming environment where agents must combine previous concepts and skills to create more complex solutions. On the other hand, CommAI-env focuses on communication skills through a binary interaction interface, emphasizing the importance of simple interactions in evaluating gradual learning.

The General AI Challenge’s decision to use CommAI-env for their warm-up round is praised by Professor Hernández-Orallo, as it allows participants to focus on reinforcement learning without the complexities of vision and navigation. By starting with a minimal interface, participants are challenged to evaluate whether their agents can learn incrementally, addressing an essential open problem in general AI.

The modified CommAI-env used in the warm-up round introduces 8-bit characters for task definition, simplifying the interface and allowing for more intuitive task design. This simple, symbolic sequential interface opens the challenge to various AI techniques beyond deep reinforcement learning, such as natural language processing, evolutionary computation, and inductive programming.

Overall, Professor Hernández-Orallo sees the warm-up round of the General AI Challenge as a unique competition that will push the boundaries of AI evaluation. He looks forward to seeing how participants integrate and invent new techniques to solve the sequence of micro and mini-tasks, hoping that this challenge will propel us closer to understanding and advancing artificial intelligence.

We are grateful to Professor José Hernández-Orallo for sharing his expertise and insights on the future of artificial intelligence evaluation. His perspective provides valuable insights into the current challenges and opportunities in the field of AI, and we look forward to further advancements in measuring machine intelligence.