What to Cut? Predicting Unnecessary Methods in Agentic Code Generation

Agentic Coding, powered by autonomous agents such as GitHub Copilot and Cursor, enables developers to generate code, tests, and pull requests from natural language instructions alone. While this accelerates implementation, it produces larger volumes of code per pull request, shifting the burden from implementers to reviewers. In practice, a notable portion of AI-generated code is eventually deleted during review, yet reviewers must still examine such code before deciding to remove it. No prior work has explored methods to help reviewers efficiently identify code that will be removed.In this paper, we propose a prediction model that identifies functions likely to be deleted during PR review. Our results show that functions deleted for different reasons exhibit distinct characteristics, and our model achieves an AUC of 87.1%. These findings suggest that predictive approaches can help reviewers prioritize their efforts on essential code.

💡 Research Summary

The paper tackles a newly emerging bottleneck in modern software development: the review burden introduced by agentic coding tools such as GitHub Copilot and Cursor. While these autonomous agents dramatically reduce implementation time by generating code, tests, and pull‑request scaffolding from natural‑language prompts, they also tend to produce larger, sometimes redundant code bases. Consequently, reviewers must sift through a substantial amount of AI‑generated code, a portion of which is later removed during the pull‑request (PR) revision process. No prior work has attempted to predict which of the newly added methods will be deleted before the PR is merged, a capability that could help reviewers focus their attention on truly essential changes.

Data collection and labeling
The authors built their dataset from the AIDev corpus, which contains 33,596 PRs created by five major AI agents. They filtered for Python‑only changes (the refactoring detection tool they employ only supports Python) and for PRs that were eventually merged, yielding 1,664 PRs spanning 197 projects. For each PR they identified three revisions: the base (branch creation), the PR creation revision, and the merge revision. Using Python’s ast module they extracted every function (FunctionDef and AsyncFunctionDef) at each revision, uniquely identified by file path, class name, and function name. To distinguish genuine additions/deletions from refactorings (renames, moves, extracts, etc.), they ran ActRef, a state‑of‑the‑art Python refactoring detector, on every adjacent revision pair. Methods that appeared in the PR‑creation revision but vanished by the merge revision (and were not accounted for by a rename/move operation) were labeled “Deleted”; those that persisted (including renamed or moved methods) were labeled “Survived”. This process produced 12,343 added methods, of which 323 (2.6 %) were deleted overall, but the deletion rate rose to 9.9 % when considering only PRs that actually underwent revisions (397 PRs, 3,257 methods).

Feature engineering
For each added method the authors computed 23 quantitative features, grouped into three categories:

• Size – lines of code (code_loc), character length, token count, number of words in the method name, number of words in the docstring, etc.
• Method Type – binary flags such as “is_getter”, “is_setter”, “is_private”, “is_test”, “is_dunder_method”, etc.
• Contents – number of parameters, number of local variables, count of call expressions, number of print statements, comment‑to‑code ratio, cyclomatic complexity, Halstead volume, maximum nesting depth, presence of try‑except blocks, usage of constants, etc.

These features are standard in software‑quality research and were extracted via the radon library (for complexity metrics), the tokenize module, and regular‑expression analyses of identifiers.

Prediction model
The authors framed the task as binary classification (Deleted vs. Survived) and chose a Random Forest (RF) classifier because it handles heterogeneous features, provides interpretable importance scores, and has a strong track record in software‑engineering classification tasks. To address the severe class imbalance (≈ 97 % survived), they undersampled the majority class to match the minority class size. Model evaluation employed 10‑fold cross‑validation, reporting median values across folds for Accuracy, Precision, Recall, F1‑Score, and Area Under the ROC Curve (AUC).

Results

• Model performance – The RF model achieved an AUC of 0.871, Accuracy of 0.735, Recall of 0.825, Precision of 0.049, and F1‑Score of 0.092. The high recall indicates that the model successfully flags most methods that will later be deleted, albeit at the cost of many false positives (low precision).
• Baselines – Two baselines were used: (1) random guessing according to the class distribution (AUC = 0.500, Accuracy = 0.498) and (2) a commercial LLM (GPT‑4o) prompted to predict deletion. GPT‑4o obtained higher Accuracy (0.836) but a dismal Recall (0.026) and an AUC of 0.621, reflecting a tendency to label almost everything as “Survived”. An illustrative example showed GPT‑4o praising code quality while the method was actually deleted, highlighting that LLMs may conflate local code quality with system‑level necessity.
• Feature importance – The top ten features (by median importance) were: method_name_words (0.16), char_length (0.14), tokens (0.09), code_loc (0.07), call_expression_count (0.07), docstring_words (0.06), comment_ratio (0.06), halstead_volume (0.05), param_count (0.05), and number_of_variable (0.05). Size‑related metrics dominate, while method‑type flags such as is_private or is_getter contributed little, suggesting that simple naming conventions do not reliably indicate future deletion.

Threats to validity

1. Domain limitation – Only Python code was examined; results may not transfer to other languages with different idioms or tooling.
2. Selection bias – The study only considered merged PRs; deleted PRs (e.g., rejected or abandoned) could contain additional patterns of unnecessary code.
3. Refactoring detection accuracy – ActRef’s reported precision (78 %) and recall (91 %) imply that some rename/move operations may be mis‑classified, potentially contaminating the “Deleted” label.
4. Undersampling effects – Balancing the dataset by discarding many survived examples could lead to overfitting on the minority class and limit generalizability.

Implications and future work
The findings demonstrate that a lightweight, feature‑based model can reliably flag methods likely to be removed, offering a practical tool for reviewers to prioritize their effort. By surfacing high‑risk methods early, teams could reduce cognitive load, shorten review cycles, and possibly guide AI agents to generate more concise code. Future research directions include extending the approach to other programming languages, incorporating dynamic information (e.g., test coverage, runtime profiling), exploring more sophisticated imbalance‑handling techniques (e.g., SMOTE, cost‑sensitive learning), and integrating the predictor directly into IDEs or CI pipelines for real‑time feedback to AI code generators.