Automated Customization of LLMs for Enterprise Code Repositories Using Semantic Scopes

Code completion (CC) is a task frequently used by developers when working in collaboration with LLM-based programming assistants. Despite the increased performance of LLMs on public benchmarks, out of the box LLMs still have a hard time generating code that aligns with a private code repository not previously seen by the model’s training data. Customizing code LLMs to a private repository provides a way to improve the model performance. In this paper we present our approach for automated LLM customization based on semantic scopes in the code. We evaluate LLMs on real industry cases with two private enterprise code repositories with two customization strategies: Retrieval-Augmented Generation (RAG) and supervised Fine-Tuning (FT). Our mechanism for ingesting the repository’s data and formulating the training data pairs with semantic scopes helps models to learn the underlying patterns specific to the repository, providing more precise code to developers and helping to boost their productivity. The code completions of moderately sized customized models can be significantly better than those of uncustomized models of much larger capacity. We also include an analysis of customization on two public benchmarks and present opportunities for future work.

💡 Research Summary

The paper addresses a practical problem faced by enterprises: out‑of‑the‑box large language models (LLMs) struggle to generate code that conforms to the style, conventions, and domain‑specific patterns of private code repositories that were never seen during pre‑training. To bridge this gap, the authors propose an automated pipeline that extracts “semantic scopes” from a repository, converts them into prefix‑label training pairs, and then customizes the LLM using either Retrieval‑Augmented Generation (RAG) or supervised fine‑tuning (FT).

A “semantic scope” is defined as a contiguous block of code that carries a coherent meaning, independent of the underlying syntax. In Java and C++ projects, the authors approximate scopes by the bodies delimited by matching braces or parentheses, then filter candidates by size (50–1 000 bytes), nesting depth, amount of preceding context (≥200 bytes), and recency. Each selected scope is split into a “query” (the code prefix up to the start of the scope) and a “label” (the full scope), terminated with a special end‑of‑text token. Random offsets are also introduced to generate additional pairs, improving robustness.

The pipeline produces a large, high‑quality dataset without any human annotation. For RAG, these pairs are indexed in a vector database; at inference time the query is embedded, the top‑N nearest‑neighbor labels are retrieved, and the retrieved snippets are concatenated to the prompt. For FT, the same dataset is used to continue training a pre‑trained model, allowing the model to internalize the repository’s naming conventions, error‑handling idioms, and overall “dialect.”

Experiments were conducted on two sizable proprietary repositories (one Java, one C++) and on several public repository‑level benchmarks (Repobench‑C, CrossCodeEval, HumanEval). Evaluation metrics included exact token‑level accuracy, an “effort‑to‑value” ratio (how much developer time is saved per prediction length), and latency. The results show that FT consistently outperforms RAG and off‑the‑shelf models. A fine‑tuned 8‑billion‑parameter model achieved higher accuracy than a 120‑billion‑parameter baseline while delivering predictions in roughly one second, compared to 30–100 seconds for RAG and up to 100 seconds for the large baseline. FT also handled optional arguments and discretionary choices (e.g., severity levels) more reliably, whereas RAG often produced out‑of‑order or missing arguments due to imperfect nearest‑neighbor retrieval.

Key insights include: (1) semantic scopes are an effective unit for teaching a model repository‑specific style; (2) careful filtering of scope size, depth, and surrounding context dramatically improves downstream performance; (3) moderate‑sized fine‑tuned models can surpass much larger generic models, offering a cost‑effective solution for enterprises; (4) RAG’s dependence on vector similarity introduces latency and correctness bottlenecks that limit its suitability for real‑time code completion.

Limitations are acknowledged: the current scope extraction relies on language‑specific parsers, making extension to languages without robust AST tools non‑trivial; complex cross‑file dependencies are not captured by single‑file scopes; and the approach assumes that the repository’s “dialect” can be learned from static code alone. Future work proposes integrating static analysis with dynamic execution traces to define richer scopes, incorporating documentation and comments for multimodal learning, and developing automated heuristics for optional‑argument generation.

In conclusion, the authors demonstrate that an automated, semantics‑driven data preparation pipeline combined with supervised fine‑tuning provides the most effective path to customizing LLMs for enterprise code bases, delivering higher accuracy, lower latency, and substantial developer productivity gains while keeping computational costs modest.