Benchmarking Defeasible Reasoning with Large Language Models

The prominence of Large Language Models in the field of AI has been established due to their exceptional performance in various natural language tasks. However, their capacity for nonmonotonic reasoning remains largely underexplored. This paper introduces a benchmark designed to evaluate LLMs across a range of defeasible rule-based reasoning patterns. To develop the benchmark, we extended an existing one for defeasible logic reasoners by translating its rules into textual prompts suitable for LLMs. We conducted experiments with LLMs and Defeasible Logic theories of increasing size and compared their outputs with the expected outputs under Defeasible Logic. Experimental results indicate that LLMs cannot yet reliably substitute a traditional symbolic defeasible reasoner.