Agent Skill Framework: Perspectives on the Potential of Small Language Models in Industrial Environments

Agent Skill framework, now widely and officially supported by major players such as GitHub Copilot, LangChain, and OpenAI, performs especially well with proprietary models by improving context engineering, reducing hallucinations, and boosting task accuracy. Based on these observations, an investigation is conducted to determine whether the Agent Skill paradigm provides similar benefits to small language models (SLMs). This question matters in industrial scenarios where continuous reliance on public APIs is infeasible due to data-security and budget constraints requirements, and where SLMs often show limited generalization in highly customized scenarios. This work introduces a formal mathematical definition of the Agent Skill process, followed by a systematic evaluation of language models of varying sizes across multiple use cases. The evaluation encompasses two open-source tasks and a real-world insurance claims data set. The results show that tiny models struggle with reliable skill selection, while moderately sized SLMs (approximately 12B - 30B) parameters) benefit substantially from the Agent Skill approach. Moreover, code-specialized variants at around 80B parameters achieve performance comparable to closed-source baselines while improving GPU efficiency. Collectively, these findings provide a comprehensive and nuanced characterization of the capabilities and constraints of the framework, while providing actionable insights for the effective deployment of Agent Skills in SLM-centered environments.

💡 Research Summary

The paper investigates whether the Agent Skill framework—originally designed for large proprietary language models—can deliver comparable benefits when applied to small, open‑source language models (SLMs) in industrial settings where reliance on external APIs is undesirable for security and cost reasons. The authors first formalize the Agent Skill process as a partially observable Markov decision process (POMDP). Each skill k is represented as a triple (dₖ, πₖ, ρₖ) where dₖ is a textual description, πₖ is an intra‑skill policy (an “option” that generates a sequence of actions), and ρₖ is a reference mechanism that can reveal additional context or tools on demand. The agent maintains a belief state bₜ over the hidden task state, and at each step can (i) select a skill, (ii) spend a cost to reveal extra information (reveal(ρₖ)), or (iii) execute the chosen skill. This “progressive disclosure” mirrors the classic result that optimal value functions for finite‑horizon POMDPs are piecewise‑linear and convex, implying that the agent should only acquire information when the expected value of information outweighs its cost.

To evaluate the practical impact of this formulation, the authors construct three benchmark tasks of increasing complexity: (1) binary sentiment classification on a filtered subset of IMDB reviews, (2) financial XBRL tagging (FiNER) with 139 label types, and (3) a proprietary insurance‑claims dataset (InsurBench) containing long, noisy, multilingual email threads that require a recommendation (continue, act further, or close the claim). For each task they create a temporary “skill repository” containing 4–6 distractor skills plus the ground‑truth skill, thereby forcing the model to perform non‑trivial skill routing.

Three context‑engineering strategies are compared:

• Direct Instruction (DI) – a minimal prompt that mimics raw user input.
• Full‑Skill Instruction (FSI) – the entire skill repository is supplied up front, requiring the model to pick the correct skill from many.
• Agent Skill Instruction (ASI) – the model decides whether additional skill details are needed, retrieves them on demand, and then answers conditioned on the retrieved information. ASI implements the progressive‑disclosure principle.

The experimental suite spans a wide range of open‑source models: 270 M Gemma‑3‑270M‑it, 4 B Gemma‑3‑4B‑it, 12 B Gemma‑3‑12B‑it, 30 B Qwen3‑30B‑Instruct, and three 80 B variants (Qwen3‑80B‑Instruct, Qwen3‑80B‑Thinking, Qwen3‑80B‑Coder). A closed‑source baseline, gpt‑4o‑mini, is also evaluated where data‑privacy permits. Metrics include classification accuracy (Cls ACC), F1 score (Cls F1), skill‑selection accuracy (Skill ACC), average processing time per task (Avg GT min), and a composite GPU‑cost metric (Avg VRAM Time = GPU‑memory × minutes), reflecting real‑world production billing.

Key findings

1. Mid‑size models (≈12 B–30 B) benefit most. When using ASI, Qwen3‑30B‑Instruct’s skill‑selection accuracy on FiNER jumps from 0.198 (DI) to 0.654, and classification F1 improves markedly. Similar gains appear on InsurBench, indicating that the progressive‑disclosure mechanism effectively compensates for the limited context window of these models.

2. Tiny models (≤4 B) struggle. Gemma‑3‑270M‑it and Gemma‑3‑4B‑it show only modest improvements over DI, and their skill‑selection accuracy remains low, leading to overall poor downstream performance. The authors attribute this to insufficient in‑context learning capacity to parse and prioritize skill descriptors.

3. Large code‑specialized models (80 B) approach closed‑source performance. Qwen3‑80B‑Coder attains classification accuracy and F1 scores on FiNER comparable to gpt‑4o‑mini, while ASI reduces unnecessary token consumption. However, the “Thinking” variant, despite the highest accuracy, incurs prohibitive GPU memory and time costs (≈40 GB · min), limiting its practical deployment.

4. Efficiency gains. Across all model sizes, ASI reduces average processing time by roughly 10–30 % and cuts the GPU‑cost metric by 15–40 % relative to DI, confirming that on‑demand context loading can lower operational expenses in production pipelines.

Limitations and future work

• The evaluation is confined to three datasets; broader domain coverage (e.g., medical, legal) would strengthen generalizability claims.
• Skill repositories are synthetically constructed with a fixed number of distractors; sensitivity analyses varying distractor count and description complexity are absent.
• Cost metrics focus on GPU memory and wall‑clock time, omitting cloud‑specific pricing nuances (instance types, spot pricing, batch scheduling) that affect real‑world budgets.
• No systematic ablation of prompt engineering or fine‑tuning is performed, leaving open the question of how much performance stems from the Agent Skill framework versus model‑specific optimizations.

Practical implications

• For enterprises constrained by data‑privacy and API costs, deploying a 12 B–30 B open‑source model with an Agent Skill layer offers a sweet spot between accuracy and resource consumption.
• In highly specialized tasks (e.g., insurance claim triage), investing in an 80 B code‑oriented model can yield near‑state‑of‑the‑art results while still benefiting from ASI’s reduced token usage.
• The formal POMDP framing provides a principled basis for future extensions such as learned reveal policies, dynamic skill‑library updates, or reinforcement‑learning‑based skill routing.

In sum, the study demonstrates that the Agent Skill paradigm is not exclusive to massive proprietary LLMs; when paired with appropriately sized open‑source models, it can substantially improve task performance, lower hallucinations, and cut operational costs—making it a viable strategy for secure, cost‑effective AI deployment in industrial environments.