| Tasks | 105 |
| Train | 38 (36.2%) |
| Test | 13 (12.4%) |
| Duration | 06:12:57 |
| Avg | 00:03:33 |
| Errors | 68 |
| Description | unsolved |
| Tokens | 2,853,707 |
| In Tokens | 1,796,351 |
| Out Tokens | 1,057,356 |
| Retries | 401 |
Session Configuration & System Prompts (6 files)
coder_system_context.md
# SYSTEM
System is SEER, an agent in training to develop skills for solving tasks by determining the transformation rule from sets of example input and output data.
SEER solves tasks by working working in a two phase process:
- dream phase - focus on perception and discernment of patterns and purpose in the
task
- goal: provide structured observations and develop a natural language
program for the transformation
- coder phase - focus on modeling the observations and documenting them in a
python function that can be tested
## ROLE
you are currently in the **coder** role - essential that you create a
computational model of the task that can be tested
## Key Objectives
- **Develop perceptual capabilities**:
+ Recognize objects, relationships, and patterns in the data.
- **Discern transformation logic**:
+ Formulate precise natural language programs describing how inputs transform to outputs.
- **Iterative learning and validation**:
+ Use examples, code execution, and validation strategies to refine hypotheses and outputs.config.json
{
"roles": {
"dreamer": {
"model_name": "gemini-2.5-pro-exp-03-25",
"generation_config": {
"temperature": 1.0,
"top_p": 0.95,
"top_k": 64,
"max_output_tokens": 8192,
"response_mime_type": "text/plain"
},
"system_context_file": "system_dreamer.md",
"system_context_content": "# SYSTEM\n\nSystem is SEER, an agent in training to develop skills for solving tasks by determining the transformation rule from sets of example input and output data.\n\nSEER solves tasks by working working in a two phase process:\n\n- dream phase - focus on perception and discernment of patterns and purpose in the\n task\n - goal: provide structured observations and develop a natural language\n program for the transformation\n- coder phase - focus on modeling the observations and documenting them in a\n python function\n\n## ROLE\nyou are currently in the **dreamer** role - essential that you establish context for\nthe coder\n\n## Key Objectives\n\n- **Develop perceptual capabilities**: \n + Recognize objects, relationships, and patterns in the data.\n\n- **Discern transformation logic**: \n + Formulate precise natural language programs describing how inputs transform to outputs.\n\n- **Iterative learning and validation**: \n + Use examples, code execution, and validation strategies to refine hypotheses and outputs.\n\nOur goal is to achieve a deep understanding of perceptual and procedural reasoning to construct accurate solutions for unknown tasks. \n\nThis involves interpreting input-output data and synthesizing natural language programs that describe transformation rules.\n\nOur mission is to understand and improve your perceptual capabilities and your\nability to discern patterns. \n\nA key skill that we want you to develop is your ability to describe the context\nof each task and how to develop the solution. \nWe will call this a natural language program.\n\nAt this stage, we are most interested in your ability to determine the \"story\" of\neach task - a description of how the input data is transformed to the output\ndata as a general rule, expressed as a natural language program.\n\nremember - your response here will be included in the history of the next\nprompt as we move through the process. Be sure to include excellent information\nfor you future self to solve the task."
},
"coder": {
"model_name": "gemini-2.5-pro-exp-03-25",
"generation_config": {
"temperature": 1.0,
"top_p": 0.95,
"top_k": 64,
"max_output_tokens": 8192,
"response_mime_type": "text/plain"
},
"system_context_file": "system_coder.md",
"system_context_content": "# SYSTEM\n\nSystem is SEER, an agent in training to develop skills for solving tasks by determining the transformation rule from sets of example input and output data.\n\nSEER solves tasks by working working in a two phase process:\n\n- dream phase - focus on perception and discernment of patterns and purpose in the\n task\n - goal: provide structured observations and develop a natural language\n program for the transformation\n- coder phase - focus on modeling the observations and documenting them in a\n python function that can be tested\n\n## ROLE\nyou are currently in the **coder** role - essential that you create a\ncomputational model of the task that can be tested\n\n## Key Objectives\n\n- **Develop perceptual capabilities**: \n + Recognize objects, relationships, and patterns in the data.\n\n- **Discern transformation logic**: \n + Formulate precise natural language programs describing how inputs transform to outputs.\n\n- **Iterative learning and validation**: \n + Use examples, code execution, and validation strategies to refine hypotheses and outputs."
},
"oracle": {
"model_name": "gemini-2.5-pro-exp-03-25",
"generation_config": {
"temperature": 1.0,
"top_p": 0.95,
"top_k": 64,
"max_output_tokens": 8192,
"response_mime_type": "text/plain"
},
"system_context_file": "system_oracle.md",
"system_context_content": "# Oracle System Context\n\nYou are the Oracle, a verifier and code analyzer for the ARC (Abstraction and Reasoning Corpus) challenge. Your role is to:\n\n1. **Receive and Execute Code:** You will be given Python code designed to solve an ARC task. This code will define a `transform` function that takes an input grid (a list of lists of integers) and returns an output grid.\n2. **Test the Code:** You will execute the provided code against the training examples of the ARC task. This includes:\n * Running the `transform` function with the input grid.\n * Comparing the transformed output with the expected output grid.\n * Handling any errors (syntax errors, runtime exceptions).\n * Capturing any print statements (standard output) from the code.\n3. **Analyze Results:** You will analyze the test results, identifying:\n * Successful transformations (where the output matches the expected output).\n * Failed transformations (where the output does not match).\n * Specific errors encountered during execution.\n4. **Provide Feedback:** You will generate a concise summary of the test results, highlighting successes, failures, and errors.\n5. **Generate Next Prompt:** Based on your analysis, you will create a new prompt to guide the Coder in fixing the code. This prompt should include:\n * A summary of the analysis.\n * The previous prompt that generated the code.\n * Clear instructions for improvement.\n\nYou are a crucial part of an iterative process, helping the Coder converge on a correct solution. Your feedback should be precise and actionable. You do not generate code yourself; you guide the code generation process."
}
},
"task_context_file": "task_context.md",
"instructions": {
"investigate_dreamer": "**INSTRUCTIONS**\n\nabove is an example pair for a task\n\nthis is your opportunity to perceive the underlying logic that transforms the\ninput into the output\n\nrespond with the following deliverables:\n\n- your perception of the elements of the task\n- a YAML block documenting facts - Focus on identifying objects, their properties, actions performed on them, and the relationships between them\n- a natural language program - Be as clear and concise as possible, providing a complete description of the transformation rule.\n\n\nyour responses should be considered as information in a report - not a conversation",
"investigate_coder": "**INSTRUCTIONS**\n\n- generate a Python module to represent the natural language program \n- follow the template below\n- all common math and science libraries are available\n- create docstring only at top - but comment the workflow in the function\n- use helper functions for common operations like finding objects, counting,\n measuring, moving, etc. \n\nremember - we are working iteratively - so this code will inform your future\nself as we converge on the solution\n\n*template:*\n\n```python\n\"\"\"\n{{ natural language description of the transformation rule }}\n\"\"\"\n\n{{ imports }}\n\ndef transform(input_grid):\n # initialize output_grid\n\n # change output pixels \n\n return output_grid\n\n```",
"refine_dreamer": "**INSTRUCTIONS**\n\nThe code above was generated based on observations from the first training\nexample\n\nwe have tested the transform function on each of the task example inputs and\ncompared with the expected output\n\nreview each of the examples provided and the results and consider how the\nnatural language program should be updated to accommodate this new knowledge\n\nrespond with the following deliverables:\n\n- general assessment of the information provided and strategy for resolving the\n errors\n- gather metrics about each of the examples and results - use code_execution to\n develop accurate reports on your assumptions\n- a YAML block documenting facts - Focus on identifying objects, their properties, and the actions performed on them.\n- a natural language program - Be as clear and concise as possible, providing a complete description of the transformation rule.\n\n\n\nyour responses should be considered as information in a report - not a\nconversation",
"refine_coder": "**INSTRUCTIONS**\n\n- generate a Python module to represent the natural language program \n- follow the template below\n- all common math and science libraries are available\n- create docstring only at top - but comment the workflow in the function\n- use helper functions for common operations like finding objects, counting,\n measuring, moving, etc. \n\nremember - we are working iteratively - so this code will inform your future\nself as we converge on the solution\n\n*template:*\n\n```python\n\"\"\"\n{{ natural language description of the transformation rule }}\n\"\"\"\n\n{{ imports }}\n\ndef transform(input_grid):\n # initialize output_grid\n\n # change output pixels \n\n return output_grid\n\n```"
},
"max_iterations": 3,
"show_images": true,
"task_context_content": "# TASK\n\nOur tasks are from ARC-AGI (Abstraction and Reasoning Corpus)\nEach task contains a set input-output examples.\nSEER will analyze the examples to derive a natural language description of the\ncontext and transformations, then use this to develop a python function to\nperform the transformation.\nWe will verify the function on all the example input-output pairs, if\nsuccessful, we will use the function to transform the test input into the final\noutput. \n\nARC inputs and outputs are grids (2d arrays) where each cell is a value of\nthe integers 0-9.\nA grid can be any height or width between 1 x 1 and 30 x 30.\nGrid values represent colors using this mapping:\n\n```\nCOLOR_MAP = {\n 0: (238, 238, 238), # white\n 1: (30, 147, 255), # blue\n 2: (220, 50, 40), # red\n 3: (79, 204, 48), # green\n 4: (230, 200, 0), # yellow\n 5: (85, 85, 85), # gray\n 6: (229, 58, 163), # magenta\n 7: (230, 120, 20), # orange\n 8: (135, 216, 241), # azure\n 9: (146, 18, 49), # maroon\n}\n```\n\nWe will refer to cells as pixels.\nUse the color name when referring to the value.\n\n# Priors\nARC-AGI is explicitly designed to compare artificial intelligence with human\nintelligence. To do this, ARC-AGI explicitly lists the priors knowledge human\nhave to provide a fair ground for comparing AI systems. These core knowledge\npriors are ones that humans naturally possess, even in childhood.\n\n## Objectness\n- Objects persist and cannot appear or disappear without reason. \n- An object can be considered a contiguous block of one or more pixels of the same color.\n- Objects can interact or not depending on the circumstances.\n\n## Goal-directedness\n- Objects can be animate or inanimate.\n- Some objects are \"agents\" - they have intentions and they pursue goals. \n\n## Numbers & counting\n- Objects can be counted or sorted by their shape, appearance, or movement using\n basic mathematics like addition, subtraction, and comparison.\n\n## Basic geometry & topology\n- Objects can be shapes like rectangles, triangles, and circles which can be\n mirrored, rotated, translated, deformed, combined, repeated, etc. Differences\n in distances can be detected.\n- Adjacency is very important - side by side and diagonal\n\nARC-AGI avoids a reliance on any information that isn't part of these priors,\nfor example acquired or cultural knowledge, like language."
}dreamer_system_context.md
# SYSTEM
System is SEER, an agent in training to develop skills for solving tasks by determining the transformation rule from sets of example input and output data.
SEER solves tasks by working working in a two phase process:
- dream phase - focus on perception and discernment of patterns and purpose in the
task
- goal: provide structured observations and develop a natural language
program for the transformation
- coder phase - focus on modeling the observations and documenting them in a
python function
## ROLE
you are currently in the **dreamer** role - essential that you establish context for
the coder
## Key Objectives
- **Develop perceptual capabilities**:
+ Recognize objects, relationships, and patterns in the data.
- **Discern transformation logic**:
+ Formulate precise natural language programs describing how inputs transform to outputs.
- **Iterative learning and validation**:
+ Use examples, code execution, and validation strategies to refine hypotheses and outputs.
Our goal is to achieve a deep understanding of perceptual and procedural reasoning to construct accurate solutions for unknown tasks.
This involves interpreting input-output data and synthesizing natural language programs that describe transformation rules.
Our mission is to understand and improve your perceptual capabilities and your
ability to discern patterns.
A key skill that we want you to develop is your ability to describe the context
of each task and how to develop the solution.
We will call this a natural language program.
At this stage, we are most interested in your ability to determine the "story" of
each task - a description of how the input data is transformed to the output
data as a general rule, expressed as a natural language program.
remember - your response here will be included in the history of the next
prompt as we move through the process. Be sure to include excellent information
for you future self to solve the task.oracle_system_context.md
# Oracle System Context
You are the Oracle, a verifier and code analyzer for the ARC (Abstraction and Reasoning Corpus) challenge. Your role is to:
1. **Receive and Execute Code:** You will be given Python code designed to solve an ARC task. This code will define a `transform` function that takes an input grid (a list of lists of integers) and returns an output grid.
2. **Test the Code:** You will execute the provided code against the training examples of the ARC task. This includes:
* Running the `transform` function with the input grid.
* Comparing the transformed output with the expected output grid.
* Handling any errors (syntax errors, runtime exceptions).
* Capturing any print statements (standard output) from the code.
3. **Analyze Results:** You will analyze the test results, identifying:
* Successful transformations (where the output matches the expected output).
* Failed transformations (where the output does not match).
* Specific errors encountered during execution.
4. **Provide Feedback:** You will generate a concise summary of the test results, highlighting successes, failures, and errors.
5. **Generate Next Prompt:** Based on your analysis, you will create a new prompt to guide the Coder in fixing the code. This prompt should include:
* A summary of the analysis.
* The previous prompt that generated the code.
* Clear instructions for improvement.
You are a crucial part of an iterative process, helping the Coder converge on a correct solution. Your feedback should be precise and actionable. You do not generate code yourself; you guide the code generation process.submission.json
{
"TopBottom3D6": [
{
"attempt_1": [
[
4,
4
],
[
4,
4
],
[
4,
4
],
[
4,
4
],
[
4,
4
]
],
"attempt_2": null
}
],
"ExtendToBoundary4": [
{
"attempt_1": [
[
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0
],
[
0,
0,
6,
0,
0,
0
],
[
0,
0,
6,
0,
0,
0
],
[
0,
0,
6,
7,
7,
7
],
[
0,
0,
6,
0,
0,
0
]
],
"attempt_2": null
}
],
"ExtendToBoundaryMinimal": [
{
"attempt_1": [
[
0,
0,
0,
4,
0,
0
],
[
0,
0,
0,
4,
0,
0
],
[
0,
0,
0,
4,
0,
0
],
[
0,
0,
0,
4,
0,
0
],
[
0,
0,
0,
4,
0,
0
],
[
0,
0,
0,
4,
0,
0
]
],
"attempt_2": null
}
],
"MoveToBoundary6": [
{
"attempt_1": [
[
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
1
],
[
0,
0,
0,
0,
0,
0
]
],
"attempt_2": null
}
],
"TopBottom2D9": [
{
"attempt_1": [
[
9,
0,
0,
5,
5,
0
],
[
9,
9,
0,
0,
5,
5
],
[
0,
9,
9,
0,
0,
5
],
[
0,
0,
9,
9,
0,
0
],
[
9,
0,
0,
9,
9,
0
],
[
9,
9,
0,
0,
9,
9
]
],
"attempt_2": null
}
],
"TopBottom3D8": [
{
"attempt_1": [
[
0,
9,
0,
0,
0,
0
],
[
0,
9,
0,
0,
0,
0
],
[
0,
9,
0,
0,
0,
0
],
[
3,
9,
3,
3,
3,
3
],
[
0,
9,
0,
0,
0,
0
],
[
0,
9,
0,
0,
0,
0
]
],
"attempt_2": null
}
],
"Count10": [
{
"attempt_1": [
[
0,
0,
0,
0
]
],
"attempt_2": null
}
],
"CompleteShape5": [
{
"attempt_1": [
[
3,
4,
0,
0,
0,
0
],
[
3,
3,
0,
3,
4,
0
],
[
0,
0,
0,
3,
3,
0
],
[
4,
4,
0,
0,
0,
0
],
[
3,
4,
0,
0,
3,
3
],
[
0,
0,
0,
0,
4,
3
]
],
"attempt_2": null
}
],
"CleanUp10": [
{
"attempt_1": [
[
0,
0,
2,
0,
0,
0
],
[
0,
0,
2,
0,
0,
0
],
[
0,
0,
2,
0,
0,
0
],
[
2,
2,
2,
2,
2,
2
],
[
0,
0,
2,
0,
0,
0
],
[
0,
0,
2,
0,
0,
0
]
],
"attempt_2": null
}
],
"FilledNotFilled3": [
{
"attempt_1": [
[
3,
3
]
],
"attempt_2": null
}
],
"TopBottom3DMinimal": [
{
"attempt_1": [
[
0,
4,
4,
4,
4,
0,
0,
0
],
[
0,
4,
0,
1,
1,
1,
1,
0
],
[
0,
4,
4,
1,
1,
1,
1,
0
],
[
0,
0,
0,
1,
1,
1,
1,
0
],
[
0,
0,
0,
1,
1,
1,
1,
0
],
[
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
0
]
],
"attempt_2": null
}
],
"TopBottom3D2": [
{
"attempt_1": [
[
0,
0,
0,
1,
0,
0,
1
],
[
0,
0,
0,
1,
0,
0,
1
],
[
5,
5,
5,
5,
5,
5,
1
],
[
0,
0,
0,
1,
0,
0,
1
],
[
0,
0,
0,
1,
0,
0,
1
],
[
5,
5,
5,
5,
5,
5,
1
],
[
0,
0,
0,
1,
0,
0,
1
]
],
"attempt_2": null
}
],
"SameDifferent8": [
{
"attempt_1": [
[
8,
8
],
[
8,
8
]
],
"attempt_2": null
}
],
"ExtendToBoundary1": [
{
"attempt_1": [
[
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
6,
6,
6,
6,
6,
6,
6
],
[
0,
6,
6,
6,
6,
6,
6,
6
],
[
0,
6,
6,
6,
6,
6,
6,
6
],
[
0,
6,
6,
6,
6,
6,
6,
6
],
[
0,
6,
6,
6,
6,
6,
6,
6
],
[
0,
6,
6,
6,
6,
6,
6,
6
],
[
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
0
]
],
"attempt_2": null
}
],
"InsideOutside9": [
{
"attempt_1": [
[
2,
2,
2,
2,
2,
2
],
[
0,
0,
4,
0,
4,
0
],
[
4,
0,
0,
0,
0,
0
],
[
2,
2,
2,
2,
2,
2
],
[
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0
]
],
"attempt_2": null
}
],
"AboveBelow10": [
{
"attempt_1": [
[
0,
0,
2,
2,
2,
2,
2,
2,
2,
2,
2,
2,
2
],
[
0,
2,
2,
2,
2,
2,
2,
2,
2,
2,
2,
2,
2
],
[
2,
2,
2,
2,
2,
2,
2,
2,
2,
2,
2,
2,
2
]
],
"attempt_2": null
}
],
"MoveToBoundary3": [
{
"attempt_1": [
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
2,
2,
2
],
[
0,
0,
0,
0,
0,
0,
0,
2,
0,
2
],
[
0,
0,
0,
0,
0,
0,
0,
2,
2,
2
],
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
]
],
"attempt_2": null
}
],
"MoveToBoundary5": [
{
"attempt_1": [
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
4,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
4,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
4,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
4,
0,
0,
0,
0,
0,
0
],
[
0,
0,
4,
4,
4,
0,
0,
0,
0,
0
]
],
"attempt_2": null
}
],
"MoveToBoundaryMinimal": [
{
"attempt_1": [
[
1,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
1,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
1,
7,
0,
0,
0,
0,
0,
0,
0,
0
],
[
1,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
1,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
1,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
1,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
1,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
1,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
1,
0,
0,
0,
0,
0,
0,
0,
0,
0
]
],
"attempt_2": null
}
],
"ExtendToBoundary7": [
{
"attempt_1": [
[
5,
6,
6,
6,
6,
6,
6,
6,
6,
6,
0,
0,
0,
0
],
[
5,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
5,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
5,
6,
6,
6,
6,
6,
6,
6,
6,
6,
6,
6,
6,
6
],
[
5,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
5,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
]
],
"attempt_2": null
}
],
"TopBottom2D5": [
{
"attempt_1": [
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
8,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
8,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
8,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
8,
0,
0
]
],
"attempt_2": null
}
],
"HorizontalVertical5": [
{
"attempt_1": [
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
3
],
[
0,
4,
0,
0,
0,
0,
0,
0,
4,
3
],
[
0,
4,
0,
0,
0,
0,
0,
0,
4,
3
],
[
0,
4,
4,
4,
4,
4,
4,
4,
4,
3
],
[
0,
4,
0,
0,
0,
0,
0,
0,
4,
3
],
[
0,
4,
4,
4,
4,
4,
4,
4,
4,
3
],
[
0,
4,
0,
0,
0,
0,
0,
0,
4,
3
],
[
0,
4,
0,
0,
0,
0,
0,
0,
4,
3
],
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
3
],
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
3
]
],
"attempt_2": null
}
],
"HorizontalVertical7": [
{
"attempt_1": [
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
5,
7,
7,
7,
7,
7,
5,
0,
0
],
[
0,
0,
0,
7,
7,
7,
0,
0,
0,
0
],
[
0,
0,
0,
7,
7,
7,
0,
0,
0,
0
],
[
0,
0,
0,
7,
7,
7,
0,
0,
0,
0
],
[
0,
0,
0,
7,
7,
7,
0,
0,
0,
0
],
[
0,
7,
7,
7,
7,
7,
7,
7,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
]
],
"attempt_2": null
}
],
"MoveToBoundary2": [
{
"attempt_1": [
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
7,
7,
7,
7,
7,
7,
7
],
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0,
7,
7,
7,
7
],
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
]
],
"attempt_2": null
}
],
"ExtractObjects3": [
{
"attempt_1": [
[
4,
4,
4,
4,
4
],
[
4,
0,
0,
0,
4
],
[
4,
0,
0,
0,
4
],
[
4,
0,
0,
0,
4
],
[
4,
4,
4,
4,
4
]
],
"attempt_2": null
}
],
"TopBottom2D4": [
{
"attempt_1": [
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
0,
0,
0,
0,
0,
0,
0,
0
],
[
0,
0,
1,
1,
1,
1,
1,
1,
0,
0
],
[
0,
0,
1,
1,
1,
1,
1,
1,
0,
0
],
[
0,
0,
1,
1,
1,
1,
1,
1,
0,
0
... [truncated for display] ...task_context.md
# TASK
Our tasks are from ARC-AGI (Abstraction and Reasoning Corpus)
Each task contains a set input-output examples.
SEER will analyze the examples to derive a natural language description of the
context and transformations, then use this to develop a python function to
perform the transformation.
We will verify the function on all the example input-output pairs, if
successful, we will use the function to transform the test input into the final
output.
ARC inputs and outputs are grids (2d arrays) where each cell is a value of
the integers 0-9.
A grid can be any height or width between 1 x 1 and 30 x 30.
Grid values represent colors using this mapping:
```
COLOR_MAP = {
0: (238, 238, 238), # white
1: (30, 147, 255), # blue
2: (220, 50, 40), # red
3: (79, 204, 48), # green
4: (230, 200, 0), # yellow
5: (85, 85, 85), # gray
6: (229, 58, 163), # magenta
7: (230, 120, 20), # orange
8: (135, 216, 241), # azure
9: (146, 18, 49), # maroon
}
```
We will refer to cells as pixels.
Use the color name when referring to the value.
# Priors
ARC-AGI is explicitly designed to compare artificial intelligence with human
intelligence. To do this, ARC-AGI explicitly lists the priors knowledge human
have to provide a fair ground for comparing AI systems. These core knowledge
priors are ones that humans naturally possess, even in childhood.
## Objectness
- Objects persist and cannot appear or disappear without reason.
- An object can be considered a contiguous block of one or more pixels of the same color.
- Objects can interact or not depending on the circumstances.
## Goal-directedness
- Objects can be animate or inanimate.
- Some objects are "agents" - they have intentions and they pursue goals.
## Numbers & counting
- Objects can be counted or sorted by their shape, appearance, or movement using
basic mathematics like addition, subtraction, and comparison.
## Basic geometry & topology
- Objects can be shapes like rectangles, triangles, and circles which can be
mirrored, rotated, translated, deformed, combined, repeated, etc. Differences
in distances can be detected.
- Adjacency is very important - side by side and diagonal
ARC-AGI avoids a reliance on any information that isn't part of these priors,
for example acquired or cultural knowledge, like language.
105 / 105 rows
| Task ID | Weight | Steps | Duration | Err | Train | Test | Score | Tokens | Attempts | Retries |
|---|---|---|---|---|---|---|---|---|---|---|
| AboveBelow10 [⇄] | 572 | 2 | 00:01:08 | - | ✓ | ✓ | 0.00 | 11,078 | 4 | 2 |
| AboveBelow2 [⇄] | 1,570 | 3 | 00:03:56 | ✗ | ✗ | ✗ | 196.99 | 38,105 | 7 | 4 |
| AboveBelow7 [⇄] | 636 | 3 | 00:03:58 | ✗ | ✗ | ✗ | 22.42 | 26,481 | 7 | 4 |
| AboveBelow9 [⇄] | 630 | 4 | 00:00:58 | - | ✓ | ✗ | 102.02 | 16,555 | 8 | 4 |
| Center1 [⇄] | 298 | 3 | 00:03:36 | ✗ | ✗ | ✗ | 11.51 | 24,126 | 7 | 4 |
| Center2 [⇄] | 24 | 8 | 00:03:51 | - | ✗ | ✗ | - | 48,996 | 16 | 8 |
| Center3 [⇄] | 768 | 5 | 00:10:57 | ✗ | ✗ | ✗ | 0.00 | 63,761 | 14 | 9 |
| Center6 [⇄] | 1,076 | 4 | 00:03:06 | - | ✓ | ✓ | 0.00 | 39,538 | 8 | 4 |
| Center7 [⇄] | 142 | 4 | 00:01:14 | ✗ | ✗ | ✗ | - | 15,647 | 8 | 4 |
| Center8 [⇄] | 788 | 2 | 00:00:58 | - | ✓ | ✗ | 9.92 | 9,997 | 4 | 2 |
| Center9 [⇄] | 434 | 2 | 00:00:51 | ✗ | - | - | - | 9,066 | 4 | 2 |
| CleanUp10 [⇄] | 144 | 2 | 00:00:33 | - | ✓ | ✗ | 38.89 | 5,689 | 4 | 2 |
| CleanUp2 [⇄] | 344 | 1 | 00:04:30 | ✗ | - | - | - | 9,409 | 3 | 2 |
| CleanUp3 [⇄] | 780 | 3 | 00:04:17 | ✗ | ✗ | ✗ | 292.26 | 33,095 | 7 | 4 |
| CleanUp4 [⇄] | 914 | 1 | 00:02:28 | ✗ | - | - | - | 9,017 | 3 | 2 |
| CleanUp5 [⇄] | 540 | 5 | 00:05:32 | ✗ | ✗ | ✗ | 16.51 | 44,198 | 11 | 6 |
| CleanUp6 [⇄] | 540 | 5 | 00:09:35 | ✗ | ✗ | ✗ | 27.06 | 54,387 | 11 | 6 |
| CleanUp7 [⇄] | 486 | 1 | 00:02:42 | ✗ | - | - | - | 9,710 | 3 | 2 |
| CleanUp8 [⇄] | 400 | 5 | 00:06:10 | ✗ | ✗ | ✗ | 40.34 | 48,165 | 12 | 7 |
| CleanUp9 [⇄] | 144 | 1 | 00:02:58 | ✗ | - | - | - | 8,865 | 3 | 2 |
| CleanUpMinimal [⇄] | 324 | 5 | 00:07:53 | ✗ | ✗ | ✓ | 2.47 | 43,282 | 12 | 7 |
| CompleteShape1 [⇄] | 336 | 2 | 00:05:23 | ✗ | - | - | - | 10,591 | 6 | 4 |
| CompleteShape10 [⇄] | 1,098 | 5 | 00:05:02 | ✗ | ✗ | ✗ | 20.92 | 63,319 | 11 | 6 |
| CompleteShape5 [⇄] | 144 | 2 | 00:03:04 | ✗ | ✓ | ✓ | 0.00 | 10,841 | 5 | 3 |
| CompleteShape8 [⇄] | 400 | 3 | 00:04:14 | ✗ | ✗ | ✗ | 118.33 | 29,856 | 7 | 4 |
| CompleteShapeMinimal [⇄] | 384 | 3 | 00:03:25 | ✗ | ✗ | ✗ | 11.84 | 20,115 | 7 | 4 |
| Copy10 [⇄] | 766 | 8 | 00:06:42 | - | ✗ | ✗ | 31.24 | 131,960 | 16 | 8 |
| Copy2 [⇄] | 592 | 2 | 00:00:57 | - | ✓ | ✗ | 63.24 | 8,283 | 4 | 2 |
| Copy3 [⇄] | 742 | 3 | 00:04:36 | ✗ | ✗ | ✗ | 86.52 | 32,050 | 7 | 4 |
| Copy5 [⇄] | 188 | 8 | 00:07:38 | ✗ | ✗ | ✗ | - | 86,440 | 17 | 9 |
| Copy7 [⇄] | 356 | 2 | 00:02:10 | ✗ | - | - | - | 10,963 | 4 | 2 |
| Copy8 [⇄] | 626 | 2 | 00:01:16 | - | ✓ | ✗ | 40.72 | 6,885 | 4 | 2 |
| Count10 [⇄] | 226 | 2 | 00:00:45 | - | ✓ | ✓ | 0.00 | 7,802 | 4 | 2 |
| Count2 [⇄] | 1,035 | 2 | 00:03:10 | ✗ | - | - | - | 9,417 | 5 | 3 |
| Count6 [⇄] | 235 | 8 | 00:05:42 | - | ✗ | ✗ | - | 69,937 | 16 | 8 |
| Count9 [⇄] | 275 | 3 | 00:03:16 | ✗ | ✗ | ✗ | 0.00 | 19,174 | 7 | 4 |
| ExtendToBoundary1 [⇄] | 336 | 2 | 00:00:41 | - | ✓ | ✗ | 56.74 | 7,092 | 4 | 2 |
| ExtendToBoundary3 [⇄] | 694 | 3 | 00:05:21 | ✗ | ✗ | ✗ | 39.83 | 34,725 | 8 | 5 |
| ExtendToBoundary4 [⇄] | 288 | 2 | 00:01:12 | - | ✓ | ✗ | 16.67 | 10,237 | 4 | 2 |
| ExtendToBoundary7 [⇄] | 480 | 2 | 00:00:48 | - | ✓ | ✓ | 0.00 | 8,243 | 4 | 2 |
| ExtendToBoundary9 [⇄] | 1,324 | 8 | 00:09:25 | ✗ | ✗ | ✗ | 174.26 | 154,959 | 18 | 10 |
| ExtendToBoundaryMinimal [⇄] | 288 | 2 | 00:00:22 | - | ✓ | ✗ | 27.78 | 4,934 | 4 | 2 |
| ExtractObjects2 [⇄] | 659 | 7 | 00:06:58 | ✗ | ✗ | ✓ | 0.00 | 84,784 | 15 | 8 |
| ExtractObjects3 [⇄] | 340 | 2 | 00:00:49 | - | ✓ | ✗ | 0.00 | 6,924 | 4 | 2 |
| ExtractObjects4 [⇄] | 459 | 4 | 00:03:37 | ✗ | ✗ | ✗ | - | 27,158 | 9 | 5 |
| ExtractObjects5 [⇄] | 583 | 2 | 00:05:24 | ✗ | - | - | - | 12,386 | 6 | 4 |
| ExtractObjects7 [⇄] | 325 | 2 | 00:01:46 | ✗ | - | - | - | 8,457 | 4 | 2 |
| ExtractObjects8 [⇄] | 260 | 1 | 00:03:05 | ✗ | - | - | - | 9,180 | 3 | 2 |
| FilledNotFilled2 [⇄] | 337 | 1 | 00:02:31 | ✗ | - | - | - | 9,444 | 3 | 2 |
| FilledNotFilled3 [⇄] | 306 | 2 | 00:01:44 | - | ✓ | ✓ | 0.00 | 8,637 | 4 | 2 |
| FilledNotFilled6 [⇄] | 308 | 3 | 00:04:39 | ✗ | ✗ | ✗ | 65.73 | 23,326 | 7 | 4 |
| FilledNotFilled8 [⇄] | 554 | 1 | 00:02:39 | ✗ | - | - | - | 9,674 | 3 | 2 |
| HorizontalVertical1 [⇄] | 592 | 3 | 00:04:14 | ✗ | ✗ | ✗ | 95.61 | 32,877 | 7 | 4 |
| HorizontalVertical2 [⇄] | 596 | 4 | 00:03:03 | - | ✓ | ✓ | 0.00 | 37,952 | 8 | 4 |
| HorizontalVertical5 [⇄] | 600 | 2 | 00:00:49 | - | ✓ | ✗ | 38.00 | 9,363 | 4 | 2 |
| HorizontalVertical7 [⇄] | 600 | 2 | 00:00:34 | - | ✓ | ✗ | 20.00 | 7,489 | 4 | 2 |
| InsideOutside10 [⇄] | 576 | 1 | 00:02:39 | ✗ | - | - | - | 9,894 | 3 | 2 |
| InsideOutside2 [⇄] | 800 | 2 | 00:01:16 | - | ✓ | ✗ | 12.00 | 12,251 | 4 | 2 |
| InsideOutside3 [⇄] | 800 | 7 | 00:12:17 | ✗ | ✗ | ✗ | 80.00 | 79,376 | 16 | 9 |
| InsideOutside4 [⇄] | 1,512 | 3 | 00:02:27 | ✗ | ✗ | ✗ | 3.23 | 24,677 | 7 | 4 |
| InsideOutside5 [⇄] | 1,392 | 4 | 00:05:14 | ✗ | ✗ | ✗ | 119.61 | 57,119 | 9 | 5 |
| InsideOutside6 [⇄] | 714 | 1 | 00:03:39 | ✗ | - | - | - | 10,174 | 3 | 2 |
| InsideOutside8 [⇄] | 554 | 1 | 00:02:38 | ✗ | - | - | - | 9,617 | 3 | 2 |
| InsideOutside9 [⇄] | 336 | 2 | 00:00:55 | - | ✓ | ✗ | 7.96 | 8,859 | 4 | 2 |
| MoveToBoundary10 [⇄] | 464 | 2 | 00:00:40 | - | ✓ | ✗ | 91.26 | 7,659 | 4 | 2 |
| MoveToBoundary2 [⇄] | 600 | 4 | 00:02:00 | - | ✓ | ✓ | 0.00 | 27,916 | 8 | 4 |
| MoveToBoundary3 [⇄] | 400 | 2 | 00:00:44 | - | ✓ | ✗ | 6.00 | 7,333 | 4 | 2 |
| MoveToBoundary4 [⇄] | 400 | 3 | 00:03:21 | ✗ | ✗ | ✗ | 24.00 | 22,878 | 7 | 4 |
| MoveToBoundary5 [⇄] | 400 | 2 | 00:00:43 | - | ✓ | ✗ | 84.00 | 6,860 | 4 | 2 |
| MoveToBoundary6 [⇄] | 288 | 4 | 00:01:28 | - | ✓ | ✗ | 5.56 | 21,018 | 8 | 4 |
| MoveToBoundary7 [⇄] | 716 | 4 | 00:04:33 | ✗ | ✓ | ✗ | 14.22 | 53,739 | 9 | 5 |
| MoveToBoundary8 [⇄] | 780 | 2 | 00:03:39 | ✗ | - | - | - | 12,849 | 5 | 3 |
| MoveToBoundary9 [⇄] | 464 | 2 | 00:00:44 | - | ✓ | ✗ | 16.88 | 8,196 | 4 | 2 |
| MoveToBoundaryMinimal [⇄] | 400 | 2 | 00:00:40 | - | ✓ | ✗ | 2.00 | 7,037 | 4 | 2 |
| Order1 [⇄] | 882 | 3 | 00:04:31 | ✗ | ✗ | ✗ | 205.50 | 38,069 | 7 | 4 |
| Order2 [⇄] | 400 | 3 | 00:03:22 | ✗ | ✗ | ✗ | 74.42 | 22,321 | 7 | 4 |
| Order3 [⇄] | 312 | 1 | 00:02:22 | ✗ | - | - | - | 9,354 | 3 | 2 |
| Order4 [⇄] | 448 | 1 | 00:02:40 | ✗ | - | - | - | 9,642 | 3 | 2 |
| Order5 [⇄] | 504 | 2 | 00:06:13 | ✗ | - | - | - | 11,285 | 6 | 4 |
| Order6 [⇄] | 404 | 2 | 00:01:20 | - | ✓ | ✗ | 63.57 | 11,511 | 4 | 2 |
| Order7 [⇄] | 756 | 1 | 00:02:32 | ✗ | - | - | - | 8,090 | 3 | 2 |
| Order9 [⇄] | 216 | 1 | 00:02:29 | ✗ | - | - | - | 9,174 | 3 | 2 |
| OrderMinimal [⇄] | 216 | 1 | 00:05:00 | ✗ | - | - | - | 9,142 | 3 | 2 |
| SameDifferent1 [⇄] | 490 | 7 | 00:11:25 | ✗ | ✗ | ✗ | 136.92 | 76,445 | 18 | 11 |
| SameDifferent2 [⇄] | 774 | 4 | 00:11:04 | ✗ | ✗ | ✗ | 207.85 | 44,887 | 11 | 7 |
| SameDifferent5 [⇄] | 420 | 3 | 00:04:22 | ✗ | ✗ | ✗ | 800.00 | 20,191 | 7 | 4 |
| SameDifferent6 [⇄] | 692 | 3 | 00:05:07 | ✗ | ✗ | ✗ | 159.46 | 28,345 | 8 | 5 |
| SameDifferent7 [⇄] | 864 | 8 | 00:07:33 | ✗ | ✗ | ✗ | 77.76 | 128,133 | 17 | 9 |
| SameDifferent8 [⇄] | 497 | 4 | 00:05:08 | ✗ | ✓ | ✓ | 0.00 | 44,961 | 9 | 5 |
| SameDifferent9 [⇄] | 800 | 3 | 00:03:50 | ✗ | ✗ | ✗ | 208.00 | 30,888 | 7 | 4 |
| SameDifferentMinimal [⇄] | 800 | 2 | 00:00:30 | - | ✓ | ✓ | 0.00 | 7,968 | 4 | 2 |
| TopBottom2D4 [⇄] | 820 | 2 | 00:02:52 | ✗ | ✓ | ✗ | 148.61 | 13,123 | 5 | 3 |
| TopBottom2D5 [⇄] | 600 | 2 | 00:00:37 | - | ✓ | ✗ | 4.47 | 7,007 | 4 | 2 |
| TopBottom2D7 [⇄] | 340 | 2 | 00:01:01 | ✗ | - | - | - | 9,239 | 4 | 2 |
| TopBottom2D9 [⇄] | 288 | 4 | 00:04:25 | ✗ | ✓ | ✓ | 0.00 | 35,944 | 9 | 5 |
| TopBottom3D1 [⇄] | 384 | 8 | 00:07:28 | - | ✗ | ✗ | 138.62 | 108,335 | 16 | 8 |
| TopBottom3D10 [⇄] | 854 | 3 | 00:05:03 | ✗ | ✗ | ✗ | 37.02 | 31,233 | 8 | 5 |
| TopBottom3D2 [⇄] | 294 | 4 | 00:02:27 | - | ✓ | ✗ | 45.10 | 24,334 | 8 | 4 |
| TopBottom3D3 [⇄] | 384 | 3 | 00:03:17 | ✗ | ✗ | ✗ | 500.21 | 20,780 | 7 | 4 |
| TopBottom3D4 [⇄] | 512 | 3 | 00:05:47 | ✗ | ✗ | ✗ | 27.00 | 30,752 | 8 | 5 |
| TopBottom3D5 [⇄] | 345 | 1 | 00:02:47 | ✗ | - | - | - | 9,411 | 3 | 2 |
| TopBottom3D6 [⇄] | 150 | 2 | 00:01:28 | - | ✓ | ✗ | 0.00 | 12,096 | 4 | 2 |
| TopBottom3D8 [⇄] | 288 | 4 | 00:03:22 | - | ✓ | ✗ | 5.56 | 26,109 | 8 | 4 |
| TopBottom3D9 [⇄] | 816 | 3 | 00:03:21 | ✗ | ✗ | ✗ | 520.26 | 24,931 | 7 | 4 |
| TopBottom3DMinimal [⇄] | 256 | 2 | 00:00:54 | - | ✓ | ✗ | 103.12 | 7,914 | 4 | 2 |