A Problem-Oriented Perspective and Anchor Verification for Code Optimization

Large Language Models (LLMs) have shown remarkable capabilities in solving various programming tasks, such as code generation. However, their potential for code optimization, particularly in performance enhancement, remains largely unexplored. This paper investigates the capabilities of LLMs in optimizing code for minimal execution time, addressing a critical gap in current research. The recently proposed code optimization methods construct program optimization pairs based on iterative submissions from the same programmer for the same problem. However, this approach confines LLMs to local performance improvements, neglecting global algorithmic innovation. To overcome this limitation, we adopt a completely different perspective by reconstructing the optimization pairs into a problem-oriented approach. This allows for the integration of various ideas from multiple programmers tackling the same problem. Furthermore, we observe that code optimization presents greater challenges compared to code generation, often accompanied by “optimization tax”. Recognizing the inherent trade-offs in correctness and efficiency, we introduce a novel anchor verification framework to mitigate this “optimization tax”. Ultimately, the problem oriented perspective combined with the anchor verification framework significantly enhances both the correct optimization ratio and speedup to new levels.

💡 Research Summary

The paper tackles a largely unexplored area: using large language models (LLMs) to automatically improve the runtime performance of existing code. Existing work on code optimization (e.g., the PIE dataset) builds “optimization pairs” from sequential submissions of a single programmer. While this captures a natural refinement process, it is limited to incremental, local changes and suffers from data scarcity because each user’s trajectory provides only a few pairs. Moreover, LLM‑generated faster code often introduces correctness bugs—a phenomenon the authors call the “optimization tax”.

To overcome these issues the authors propose two complementary innovations:

1. Problem‑oriented optimization pairs – Instead of grouping submissions by user, all correct solutions for the same problem are pooled together, sorted by measured runtime, and paired along this global ordering. This creates pairs that combine diverse algorithmic ideas from many programmers, enabling large‑scale structural and algorithmic changes (e.g., swapping sorting algorithms, replacing naïve loops with vectorized libraries). The authors analytically show that the number of pairs grows quadratically with the total number of submissions per problem, yielding an order‑of‑magnitude increase when ten or more users contribute. Empirically, a balanced subset of 78 K pairs from the new PCO dataset exhibits a much higher average Graph Edit Distance (GED) between “slow” and “fast” programs and a more dispersed semantic embedding space, indicating richer, global optimizations compared with the user‑oriented PIE pairs.

2. Anchor verification framework – To mitigate the optimization tax, the authors treat the slower but already‑correct code as an “anchor”. An LLM first generates plausible test inputs by interpreting the slow code. Those inputs are executed on the slow version to obtain ground‑truth outputs, forming verified test cases. The optimized (fast) code is then run on the same inputs; any mismatch flags a correctness error, prompting further refinement. This approach differs from prior test‑case synthesis because it leverages actual execution results, guaranteeing that the test suite faithfully reflects the problem’s semantics without expensive manual labeling.

The experimental pipeline fine‑tunes a GPT‑4‑Turbo‑based model on both the user‑oriented (PIE) and problem‑oriented (PCO) datasets. Results are striking:

• Optimization ratio (percentage of programs improved by ≥10 % while remaining correct) rises from 31.24 % (PIE) to 58.90 % (PCO). Adding anchor verification pushes it further to 71.06 %.
• Speedup improves from 2.95× (PIE) to 5.22× (PCO), and to 6.08× with anchor verification.
• Correctness (fraction of optimized programs that pass all tests) climbs from 61.55 % to 74.54 % after applying the anchor framework.

Ablation studies confirm that without anchor verification the correctness drop is severe, underscoring the framework’s role in controlling the optimization tax. Human labeling of sampled pairs shows that problem‑oriented pairs contain a majority of global algorithmic changes, whereas user‑oriented pairs are dominated by minor local tweaks.

The paper also discusses limitations. The anchor approach assumes the slow code is bug‑free; any latent error propagates into the test suite. Moreover, the quality of generated inputs depends on the LLM’s ability to exercise diverse program paths, which may introduce bias. Future work could incorporate multi‑anchor ensembles (using several slow variants) and automated static analysis to filter erroneous anchors.

In summary, by re‑thinking how optimization data are constructed (problem‑oriented) and by introducing a pragmatic verification loop (anchor verification), the authors demonstrate that LLMs can achieve substantial, reliable performance gains on real code. This work paves the way for more robust, large‑scale automated code optimization pipelines that balance speed and correctness.