Reasoning Abilities of Large Language Models: In-Depth Analysis on the Abstraction and Reasoning Corpus

id:

2403.11793

Authors:

Seungpil Lee, Woochang Sim, Donghyeon Shin, Wongyu Seo, Jiwon Park, Seokki Lee, Sanha Hwang, Sejin Kim, Sundong Kim

Published:

2024-03-18

arXiv:

https://arxiv.org/abs/2403.11793

PDF:

https://arxiv.org/pdf/2403.11793

DOI:

N/A

Journal Reference:

N/A

Primary Category:

cs.CL

Categories:

cs.CL, cs.AI, cs.ET, cs.SC

Comment:

N/A

github_url:

_

abstract

The existing methods for evaluating the inference abilities of Large Language Models (LLMs) have been results-centric, making it difficult to assess the inference process. We introduce a new approach using the Abstraction and Reasoning Corpus (ARC) dataset to evaluate the inference and contextual understanding abilities of large language models in a process-centric manner. ARC demands rigorous logical structures for problem-solving, making it a benchmark that facilitates the comparison of model inference abilities with humans. Experimental results confirm that while large language models possess weak inference abilities, they still lag in terms of logical coherence, compositionality, and productivity. Our experiments highlight the reasoning capabilities of LLMs, proposing development paths for achieving human-level reasoning.

premise

outline

quotes

notes

summary

  1. Brief Overview

This paper presents an in-depth analysis of the reasoning abilities of Large Language Models (LLMs) using the Abstraction and Reasoning Corpus (ARC) dataset. The existing methods for evaluating LLM inference abilities are criticized as being results-centric, hindering a proper understanding of the inference process. The authors propose a process-centric approach leveraging the ARC dataset to evaluate LLMs’ inference and contextual understanding. The study focuses on three key aspects of human reasoning, as defined by the Language of Thought Hypothesis (LoTH): logical coherence, compositionality, and productivity. The experiments reveal that while LLMs demonstrate some capabilities, they significantly lag behind humans in all three aspects, highlighting the need for further development.

  1. Key Points

  • Existing LLM evaluation methods are primarily results-centric, neglecting the process.

  • The study uses the ARC dataset for a process-centric evaluation of LLM reasoning abilities.

  • Three aspects of human reasoning (LoTH) are investigated: logical coherence, compositionality, and productivity.

  • Experiments show LLMs have weak inference abilities and lag significantly behind humans in logical coherence, compositionality, and productivity.

  • The study proposes development paths for achieving human-level reasoning in LLMs.

  • ARC’s limitations are discussed, and alternative benchmarks and evaluation methods are suggested.

  1. Notable Quotes

No notable quotes were identified in the provided PDF excerpt.

  1. Primary Themes

  • Process-centric evaluation of LLM reasoning: Shifting the focus from results to the reasoning process itself.

  • Language of Thought Hypothesis (LoTH): Utilizing the LoTH framework to analyze the three essential characteristics of human reasoning: logical coherence, compositionality, and productivity.

  • Limitations of LLMs in reasoning: Identifying and quantifying the weaknesses of LLMs in logical inference, sequential planning, and generating unseen images.

  • Future directions for improving LLM reasoning: Proposing and discussing potential research avenues to enhance LLM reasoning capabilities. This includes exploring alternative benchmarks, quantitative analysis of reasoning processes, and the development of new evaluation methods to compare LLM and human approaches more effectively.