Augmenting Parameter-Efficient Pre-trained Language Models with Large Language Models

Training AI models in cybersecurity with help of vast datasets offers significant opportunities to mimic real-world behaviors effectively. However, challenges like data drift and scarcity of labelled data lead to frequent updates of models and the risk of overfitting. To address these challenges, we used parameter-efficient fine-tuning techniques for pre-trained language models wherein we combine compacters with various layer freezing strategies. To enhance the capabilities of these pre-trained language models, in this work we introduce two strategies that use large language models. In the first strategy, we utilize large language models as data-labelling tools wherein they generate labels for unlabeled data. In the second strategy, large language modes are utilized as fallback mechanisms for predictions having low confidence scores. We perform comprehensive experimental analysis on the proposed strategies on different downstream tasks specific to cybersecurity domain. We empirically demonstrate that by combining parameter-efficient pre-trained models with large language models, we can improve the reliability and robustness of models, making them more suitable for real-world cybersecurity applications.

💡 Research Summary

The paper tackles two persistent challenges in applying natural‑language models to cybersecurity: the high cost of fine‑tuning large pre‑trained language models (PLMs) and the chronic shortage of labeled data. The authors first introduce “CompFreeze,” a parameter‑efficient fine‑tuning framework that inserts low‑rank adapters called Compacters into selected transformer layers while freezing the remaining layers. Four layer‑selection strategies are explored—Odd‑LC, Even‑LC, Upper‑LC, and Lower‑LC—applied to three cybersecurity‑specific PLMs (CyBERT, SecureBERT, CySecBERT). By limiting trainable parameters to roughly half of the model, CompFreeze achieves comparable accuracy to full fine‑tuning while dramatically reducing memory usage and training time.

To further mitigate data scarcity and improve reliability, the paper proposes two large language model (LLM) integration strategies. In the first, state‑of‑the‑art LLMs (e.g., GPT‑4, LLaMA) are used as zero‑shot labelers for large unlabeled corpora; the generated pseudo‑labels augment the training set for CompFreeze models, effectively expanding the labeled data without human effort. In the second, a confidence‑based fallback mechanism forwards inputs that receive low confidence scores from the CompFreeze model to an LLM, which supplies auxiliary predictions. This approach leverages the LLM’s strong generalization to handle rare or emerging threat categories that the compact model may struggle with.

The authors evaluate the combined system on three representative cybersecurity tasks: spam detection, domain‑generation‑algorithm (DGA) classification, and entity extraction from cyber‑threat‑intelligence (CTI) sources. Results show that pseudo‑label augmentation improves spam‑detection F1 by 3.2 percentage points, while the confidence‑fallback boosts low‑confidence DGA classification accuracy by 12 percentage points. CTI entity extraction benefits from a 5 percentage‑point increase in entity‑level F1. Across all tasks, the total number of trainable parameters is reduced by roughly 68 % compared with conventional full fine‑tuning, and inference latency is similarly lowered.

The study also discusses limitations, including potential domain bias in LLM outputs, the need for careful prompt engineering, and the necessity of validating pseudo‑labels. Future work is outlined: pre‑training domain‑specific LLMs, developing confidence‑calibration models, and extending the framework to multimodal cybersecurity data (e.g., logs combined with network traffic). Overall, the paper demonstrates that coupling parameter‑efficient PLMs with the flexibility of large language models yields a more robust, scalable, and cost‑effective solution for real‑world cybersecurity applications.