Automated Design of Agentic Systems

id:: 2408.08435
Authors:: Shengran Hu, Cong Lu, Jeff Clune
Published:: 2024-08-15
arXiv:: https://arxiv.org/abs/2408.08435
PDF:: https://arxiv.org/pdf/2408.08435
DOI:: N/A
Journal Reference:: N/A
Primary Category:: cs.AI
Categories:: cs.AI
Comment:: Website: https://shengranhu.com/ADAS
github_url:: _

abstract

Researchers are investing substantial effort in developing powerful general-purpose agents, wherein Foundation Models are used as modules within agentic systems (e.g. Chain-of-Thought, Self-Reflection, Toolformer). However, the history of machine learning teaches us that hand-designed solutions are eventually replaced by learned solutions. We formulate a new research area, Automated Design of Agentic Systems (ADAS), which aims to automatically create powerful agentic system designs, including inventing novel building blocks and/or combining them in new ways. We further demonstrate that there is an unexplored yet promising approach within ADAS where agents can be defined in code and new agents can be automatically discovered by a meta agent programming ever better ones in code. Given that programming languages are Turing Complete, this approach theoretically enables the learning of any possible agentic system: including novel prompts, tool use, control flows, and combinations thereof. We present a simple yet effective algorithm named Meta Agent Search to demonstrate this idea, where a meta agent iteratively programs interesting new agents based on an ever-growing archive of previous discoveries. Through extensive experiments across multiple domains including coding, science, and math, we show that our algorithm can progressively invent agents with novel designs that greatly outperform state-of-the-art hand-designed agents. Importantly, we consistently observe the surprising result that agents invented by Meta Agent Search maintain superior performance even when transferred across domains and models, demonstrating their robustness and generality. Provided we develop it safely, our work illustrates the potential of an exciting new research direction toward automatically designing ever-more powerful agentic systems to benefit humanity.

premise

outline

quotes

notes

summary

Automated Design of Agentic Systems (ADAS) is a new research area that aims to automatically design powerful agentic systems. The paper introduces Meta Agent Search, an algorithm that uses a meta-agent to iteratively program new agents in code, leveraging an ever-growing archive of previous discoveries. Experiments show that agents generated by Meta Agent Search consistently outperform state-of-the-art hand-designed agents across various domains and demonstrate robustness when transferred across domains and foundation models. The work highlights the potential of ADAS as a faster and more efficient path to developing powerful agents.

Brief Overview

This research paper introduces Automated Design of Agentic Systems (ADAS), a new research area focused on automatically generating powerful agentic systems. The authors propose a novel algorithm called Meta Agent Search which leverages large language models (LLMs) as meta-agents to iteratively generate, test, and refine agents defined entirely in code. Extensive experiments across multiple domains demonstrate the superior performance and generalizability of the automatically generated agents compared to hand-designed baselines. The study also emphasizes the importance of safe development in this new field.

Key Points

ADAS is formulated as a search problem with three key components: search space (code), search algorithm (Meta Agent Search), and evaluation function.
Meta Agent Search uses LLMs as meta-agents to iteratively generate novel agents in code, evaluating them and adding them to an archive which informs future generations.
Experiments across diverse domains (logic puzzles, reading comprehension, math, science) show significant performance improvements compared to hand-designed baselines.
Automatically generated agents exhibit strong robustness and generalizability, maintaining superior performance even when transferred across different domains and foundation models.
The research underscores the importance of developing ADAS safely, addressing the potential risks associated with untrusted model-generated code.

Notable Quotes

(No notable quotes were identified in the provided text.)

Primary Themes

Automation of Agentic System Design: The core theme is the automation of the design process for complex agentic systems, moving away from manual design and tuning.
Code as the Search Space: A significant contribution is the use of code as the search space for exploring possible agentic system designs.
Meta-Learning and Open-Endedness: Meta Agent Search employs a meta-learning approach with features of open-endedness, progressively discovering more complex and capable agents.
Generalizability and Robustness: The generated agents demonstrate remarkable generalizability and robustness across various tasks and models.
Safety and Ethical Considerations: The research acknowledges the crucial need for safe and responsible development of ADAS, emphasizing the need for robust safety mechanisms.