"ENERGY STAR" LLM-Enabled Software Engineering Tools

The discussion around AI-Engineering, that is, Software Engineering (SE) for AI-enabled Systems, cannot ignore a crucial class of software systems that are increasingly becoming AI-enhanced: Those used to enable or support the SE process, such as Computer-Aided SE (CASE) tools and Integrated Development Environments (IDEs). In this paper, we study the energy efficiency of these systems. As AI becomes seamlessly available in these tools and, in many cases, is active by default, we are entering a new era with significant implications for energy consumption patterns throughout the Software Development Lifecycle (SDLC). We focus on advanced Machine Learning (ML) capabilities provided by Large Language Models (LLMs). Our proposed approach combines Retrieval-Augmented Generation (RAG) with Prompt Engineering Techniques (PETs) to enhance both the quality and energy efficiency of LLM-based code generation. We present a comprehensive framework that measures real-time energy consumption and inference time across diverse model architectures ranging from 125M to 7B parameters, including GPT-2, CodeLlama, Qwen 2.5, and DeepSeek Coder. These LLMs, chosen for practical reasons, are sufficient to validate the core ideas and provide a proof of concept for more in-depth future analysis.

💡 Research Summary

The paper investigates the energy efficiency of AI‑enhanced software engineering tools, specifically CASE tools and IDEs, which are increasingly embedding large language models (LLMs) for code generation and assistance. Recognizing that AI is becoming a default feature throughout the software development lifecycle, the authors ask whether Retrieval‑Augmented Generation (RAG) combined with Prompt Engineering Techniques (PET) can reduce the power consumption and inference latency of such tools.

Four LLMs of varying scale—GPT‑2 (125 M parameters), CodeLlama (7 B), Qwen 2.5 (7 B), and DeepSeek Coder (7 B)—are evaluated on two benchmark datasets: CONCODE (Java) and a Kaggle natural‑language‑to‑Python collection. The RAG pipeline works as follows: (1) the natural‑language query is encoded with Sentence‑BERT (all‑MiniLM‑L6‑v2) to obtain a dense vector; (2) FAISS performs cosine‑similarity search against a pre‑built code snippet repository; (3) the top‑k retrieved examples (typically 2–3) are concatenated to the original prompt, respecting the model’s token limit; (4) the augmented prompt is fed to the LLM for generation. Energy consumption is measured in real time using the CodeCarbon library, while inference time is recorded per query.

Key findings:

• RAG yields mixed results. GPT‑2 and CodeLlama both show modest energy reductions (≈9 % and ≈11 % respectively) and CodeLlama enjoys a 25 % speed‑up. In contrast, Qwen 2.5 and DeepSeek Coder experience higher energy use and longer latency when RAG is enabled, likely due to increased memory overhead from the retrieval step.
• Across models, the smallest model (GPT‑2) is the most power‑efficient, followed by CodeLlama; the two 7 B models consume roughly three times more energy. Without RAG, Qwen 2.5 is the fastest in raw inference, but its overall energy footprint remains the highest.
• No clear correlation emerges between model size and RAG‑driven energy savings. The benefit appears to depend more on architectural specifics than on parameter count.
• RAG can enable smaller models to achieve code quality comparable to larger counterparts while using significantly less energy. For example, on the Kaggle dataset, GPT‑2 with RAG attains a quality score of 0.6, matching DeepSeek Coder’s performance while consuming only about 28 % of the energy.

The authors acknowledge limitations: experiments were conducted on a single physical server, limiting generalizability to diverse cloud environments; code quality assessment relied on BLEU‑style metrics without deeper static or dynamic analysis; and the energy measurement does not account for the carbon intensity of the electricity source.

Future work includes scaling experiments to varied cloud infrastructures, integrating robust quality metrics such as CodeBLEU, static analysis, and dynamic testing, exploring the Model Context Protocol (MCP) in conjunction with RAG, and extending the study to quantum‑computing SDK code generation to assess energy implications in emerging domains.

Overall, the study demonstrates that RAG can be a viable strategy for reducing the environmental impact of AI‑augmented development tools, especially when applied to smaller, more efficient LLMs, but its effectiveness is highly model‑dependent and warrants further investigation across broader hardware and software contexts.