Mining Type Constructs Using Patterns in AI-Generated Code

Artificial Intelligence (AI) increasingly automates various parts of the software development tasks. Although AI has enhanced the productivity of development tasks, it remains unstudied whether AI essentially outperforms humans in type-related programming tasks, such as employing type constructs properly for type safety, during its tasks. Moreover, there is no systematic study that evaluates whether AI agents overuse or misuse the type constructs under the complicated type systems to the same extent as humans. In this study, we present the first empirical analysis to answer these questions in the domain of TypeScript projects. Our findings show that, in contrast to humans, AI agents are 9x more prone to use the ‘any’ keyword. In addition, we observed that AI agents use advanced type constructs, including those that ignore type checks, more often compared to humans. Surprisingly, even with all these issues, Agentic pull requests (PRs) have 1.8x higher acceptance rates compared to humans for TypeScript. We encourage software developers to carefully confirm the type safety of their codebases whenever they coordinate with AI agents in the development process.

💡 Research Summary

The paper presents the first large‑scale empirical study of how AI coding agents use TypeScript’s type system compared with human developers. Using the publicly available AIDev dataset, the authors extracted 38,979 pull requests (PRs) and applied a two‑stage filtering pipeline. The first stage employed custom regular‑expression parsers to quickly capture PRs that touched TypeScript files and contained type‑related modifications. Because regex‑based filtering can generate many false positives, a second stage used a multi‑agent LLM workflow (both a classifier and a validator, each powered by OpenAI’s gpt‑4o) to refine the selection. After both stages, the final corpus comprised 545 AI‑generated PRs and 269 human‑authored PRs.

Three research questions guided the analysis. RQ1 asked whether AI agents introduce type‑related issues and how often. The authors measured the insertion of the unsafe ‘any’ keyword. AI PRs added ‘any’ on average 2.16 times per PR, whereas humans added it only 0.24 times—a nine‑fold difference. A Mann‑Whitney U test confirmed statistical significance (p ≈ 2.33 × 10⁻⁷, Cohen’s d = 0.32). The study also broke down behavior by model: the Cursor model had the highest any_additions/any_removals ratio (2.46), while GitHub Copilot tended to remove ‘any’ more than it added.

RQ2 examined the breadth of advanced type constructs and anti‑patterns. The authors defined a set of “type‑related patterns” (e.g., generics, union/intersection types, mapped types) and “type‑related anti‑patterns” (non‑null assertions, type assertions) that can bypass compile‑time checks. AI agents used an average of 5.5–6.7 advanced type features per PR, compared with only 2.66 for humans. This gap was highly significant (p < 5.5 × 10⁻⁵, Cohen’s d = 1.45). Moreover, AI PRs introduced anti‑patterns at a substantially higher rate, indicating a propensity to sacrifice type safety for rapid code generation.

RQ3 compared acceptance rates. Counter‑intuitively, despite introducing more unsafe constructs, AI‑generated PRs were merged at a higher rate: 45.8 % acceptance versus 25.3 % for human PRs (χ² = 27.52, p < 0.0001, Cramér’s V = 0.32). The rejection rate for AI PRs was 42.6 % versus 2.6 % for humans. The authors suggest that reviewers may be more lenient toward AI‑generated changes, or that AI PRs may be concentrated on lower‑risk tasks such as minor bug fixes or refactorings, which could explain the higher acceptance despite poorer type‑safety practices.

The paper discusses several threats to validity. The reliance on regex parsers may miss unconventional or deeply nested type syntax, leading to false negatives. Although the subsequent LLM validation achieved perfect precision on the sampled data, LLMs can introduce their own biases, potentially affecting which PRs survive the filtering pipeline. The authors also note that their dataset is limited to TypeScript and that results may not generalize to other statically typed languages.

Ethical considerations are addressed: the study uses only publicly available PR data, does not collect personal identifiers, and aggregates findings to avoid profiling individual developers.

In conclusion, AI coding agents are markedly more likely to employ unsafe ‘any’ types and a broader suite of advanced (and sometimes unsafe) type constructs than human developers. Yet, AI‑generated PRs enjoy a higher acceptance rate, suggesting that current code review practices may not adequately penalize type‑safety violations introduced by AI. The authors recommend that teams using AI‑assisted programming enforce additional static analysis and manual review steps to guard against accumulating technical debt. Future work should explore AST‑based detection, extend the analysis to other languages, and investigate how different prompting strategies affect AI’s type‑system usage.