Addressing the Abstraction and Reasoning Corpus via Procedural Example Generation

id:

2404.07353

Authors:

Michael Hodel

Published:

2024-04-10

arXiv:

https://arxiv.org/abs/2404.07353

PDF:

https://arxiv.org/pdf/2404.07353

DOI:

N/A

Journal Reference:

N/A

Primary Category:

cs.LG

Categories:

cs.LG, cs.AI

Comment:

N/A

github_url:

_

abstract

This work presents code to procedurally generate examples for the ARC training tasks. For each of the 400 tasks, an example generator following the transformation logic of the original examples was created. In effect, the assumed underlying distribution of examples for any given task was reverse engineered by implementing a means to sample from it. An attempt was made to cover an as large as reasonable space of possible examples for each task. That is, whenever the original examples of a given task may be limited in their diversity e.g. by having the dimensions of the grids, the set of symbols or number of objects constant or within tight bounds, even though the transformation does not require it, such constraints were lifted. Having access to not just a few examples per task, as the case for ARC, but instead very many, should enable a wide range of experiments that may be important stepping stones towards making leaps on the benchmark.

premise

outline

quotes

notes

summary

  1. Brief Overview

This paper presents code that procedurally generates examples for the Abstraction and Reasoning Corpus (ARC) training tasks. The code reverse engineers the underlying distribution of examples for each task, aiming to create a larger and more diverse set of examples than originally provided. This expanded dataset enables a wider range of experiments, particularly focusing on sample-efficient learning.

  1. Key Points

  • Procedurally generates examples for all 400 ARC tasks.

  • Reverse engineers the underlying distribution of examples for each task.

  • Aims for a larger and more diverse set of examples than the original ARC dataset.

  • Provides verifiers for each task to ensure the quality of generated examples.

  • Allows control over example difficulty through parameters.

  • Offers two metrics (RNG-Difficulty and PSO-Difficulty) for assessing example difficulty, but suggests post-hoc filtering for more robust control.

  • Highlights limitations, including the challenge of perfectly defining the underlying transformation logic for each task.

  1. Notable Quotes

None explicitly presented in the provided text warrant inclusion as notable quotes.

  1. Primary Themes

  • Data Augmentation for ARC: The primary focus is on expanding the ARC dataset to improve research into sample-efficient learning.

  • Procedural Generation Techniques: The paper details the methods used to generate examples, including random sampling and techniques to control for difficulty.

  • Benchmarking AI: The work aims to contribute to a better understanding of AI capabilities by providing tools to improve benchmarking on a challenging dataset.

  • Sample Efficiency in Machine Learning: A core theme is addressing the limitations of existing datasets for efficiently training machine learning models.