Complex Logical Instruction Generation

Instruction following has catalyzed the recent era of Large Language Models (LLMs) and is the foundational skill underpinning more advanced capabilities such as reasoning and agentic behaviors. As tasks grow more challenging, the logic structures embedded in natural language instructions becomes increasingly intricate. However, how well LLMs perform on such logic-rich instructions remains under-explored. We propose LogicIFGen and LogicIFEval. LogicIFGen is a scalable, automated framework for generating verifiable instructions from code functions, which can naturally express rich logic such as conditions, loops, and function calls. We further curate a collection of complex code functions and use LogicIFGen to construct LogicIFEval, a benchmark comprising 426 verifiable logic-rich instructions. Our experiments demonstrate that current state-of-the-art LLMs still struggle to correctly follow the instructions in LogicIFEval. Most LLMs can only follow fewer than 60% of the instructions, revealing significant deficiencies in the instruction-following ability. Code and Benchmark: https://github.com/mianzhang/LogicIF

💡 Research Summary

The paper addresses a notable gap in the evaluation of large language models (LLMs): while instruction‑following has become a cornerstone capability of modern LLMs, existing benchmarks largely focus on shallow constraints such as output length, style, or format. They rarely test a model’s ability to faithfully execute instructions that embed rich logical structures—nested conditionals, loops, recursion, and multi‑function coordination. To fill this gap, the authors introduce two complementary contributions: LogicIFGen, an automated pipeline that converts code functions into verifiable natural‑language instructions, and LogicIFEval, a benchmark comprising 426 such logic‑rich instructions together with 3,050 test cases.

LogicIFGen operates in three stages. First, the original code is anonymized: variable and function names are replaced with generic placeholders to eliminate domain‑specific cues. Simultaneously, “state trackers” are injected into the code to record intermediate execution metrics (e.g., loop iteration counts, number of times a conditional branch is taken, heap size). These trackers provide a fine‑grained ground truth that a model must reproduce, ensuring that success is not merely a correct final output but also a correct internal reasoning trace. Second, an LLM is prompted to translate the anonymized function into a step‑by‑step natural‑language instruction. The instruction adopts a conversational style (“Now you need to…”, “Next, go through…”) and explicitly enumerates input formats, control‑flow decisions, and data transformations. Third, a multi‑turn verification and refinement loop lets another LLM review the generated instruction, flag missing logic (e.g., omitted loop boundaries or edge‑case handling), and iteratively improve the description. After up to three refinement rounds, human experts verified that 97 % of the instructions fully capture the underlying code logic, with inter‑annotator agreement of 97.79 %.

To quantify instruction difficulty, the authors parse each anonymized function with Python’s AST and compute four metrics: cyclomatic complexity (C), nesting depth (D), function‑call count (F), and line count (L). A weighted sum (Score = D × 3 + F × 2 + C × 1 + L × 0) yields a composite difficulty score, which is then used to split the benchmark into Easy, Medium, and Hard tiers (142, 145, and 139 instructions respectively).

LogicIFEval’s seed functions are drawn from high‑difficulty implementation problems on Codeforces (rating > 1700) and challenging simulation tasks on POJ. Duplicate or near‑duplicate functions are removed using text‑embedding similarity (threshold 0.7), and test cases are filtered to avoid extreme values, excessive precision, malformed state‑tracker dictionaries, or inputs that could cause runaway loops. After this two‑stage filtering, 426 unique functions remain, each paired with multiple validated test cases. The final instructions average 3,428 characters, reflecting substantial logical depth.

The authors evaluate a suite of state‑of‑the‑art LLMs, including OpenAI’s GPT‑4.1, Anthropic’s Claude‑4‑Sonnet, Google’s Gemini‑2.5‑Flash, as well as open‑source models such as LLaMA‑3‑70B, Qwen‑3‑32B, and DeepSeek variants. Overall accuracy on LogicIFEval is below 60 %, with an average of 48 % across models. Performance degrades sharply with increasing difficulty: Hard instructions see accuracies under 30 %. Introducing an “Explicit Thinking” prompt (asking the model to reason before answering) yields modest gains of 5–10 % for some models but does not close the gap. Error analysis identifies three dominant failure modes: (1) misinterpretation of conditional branches, (2) omission or incorrect handling of loop termination conditions, and (3) inaccurate reproduction of state‑tracker values, indicating that models often shortcut to plausible final answers without faithfully simulating the prescribed logic.

The paper’s contributions are significant. It provides a scalable, reproducible method for generating high‑quality, logic‑rich instruction data, and releases a benchmark that can become a standard testbed for future instruction‑following research. The inclusion of state trackers is a novel verification mechanism that pushes evaluation beyond surface‑level correctness. The empirical findings highlight a clear limitation of current LLMs: despite impressive language generation abilities, they struggle with step‑wise logical reasoning when forced to operate solely from natural‑language instructions.

Future directions suggested include (a) enriching state trackers to capture finer‑grained computational states, (b) incorporating logic‑rich instruction data into pre‑training or fine‑tuning pipelines to improve internal reasoning capabilities, and (c) exploring human‑in‑the‑loop correction mechanisms where models can query for clarification when ambiguous logic is encountered. Overall, this work advances the field by redefining how we assess LLMs’ ability to follow complex, algorithmic instructions, and it sets a clear agenda for bridging the gap between language understanding and logical problem solving.