Can LLMs Handle WebShell Detection? Overcoming Detection Challenges with Behavioral Function-Aware Framework

WebShell attacks - where adversaries implant malicious scripts on web servers - remain a persistent threat. Prior machine-learning and deep-learning detectors typically depend on task-specific supervision and can be brittle under data scarcity, rapid concept drift, and out-of-distribution (OOD) deployment. Large language models (LLMs) have recently shown strong code understanding capabilities, but their reliability for WebShell detection remains unclear. We address this gap by (i) systematically evaluating seven LLMs (including GPT-4, LLaMA-3.1-70B, and Qwen-2.5 variants) against representative sequence- and graph-based baselines on 26.59K PHP scripts, and (ii) proposing Behavioral Function-Aware Detection (BFAD), a behavior-centric framework that adapts LLM inference to WebShell-specific execution patterns. BFAD anchors analysis on security-sensitive PHP functions via a Critical Function Filter, constructs compact LLM inputs with Context-Aware Code Extraction, and selects in-context demonstrations using Weighted Behavioral Function Profiling, which ranks examples by a behavior-weighted, function-level similarity. Empirically, we observe a consistent precision-recall asymmetry: larger LLMs often achieve high precision but miss attacks (lower recall), while smaller models exhibit the opposite tendency; moreover, off-the-shelf LLM prompting underperforms established detectors. BFAD substantially improves all evaluated LLMs, boosting F1 by 13.82% on average; notably, GPT-4, LLaMA-3.1-70B, and Qwen-2.5-Coder-14B exceed prior SOTA benchmarks, while Qwen-2.5-Coder-3B becomes competitive with traditional methods. Overall, our results clarify when LLMs succeed or fail on WebShell detection, provide a practical recipe, and highlight future directions for making LLM-based detection more reliable.

💡 Research Summary

WebShells—malicious PHP scripts that enable remote command execution, data exfiltration, and system compromise—remain a prevalent threat, accounting for a large fraction of recent cyber‑incident reports. Traditional detection approaches fall into two categories: rule‑based signatures, which quickly become obsolete against obfuscated variants, and machine‑learning/deep‑learning classifiers, which require substantial labeled data, frequent retraining, and often suffer from catastrophic forgetting when faced with novel obfuscation techniques. Large language models (LLMs) have demonstrated impressive code‑understanding abilities and can be adapted to new tasks via prompting, suggesting a potential zero‑shot or few‑shot solution for WebShell detection. However, applying LLMs to this domain introduces two practical challenges. First, WebShell files can be extremely long—up to 1.3 million tokens in the authors’ dataset—far exceeding the context windows of even the largest publicly available LLMs (typically 8 K–32 K tokens). Truncation inevitably discards the malicious core, leading to missed detections. Second, in‑context learning (ICL) relies on demonstration examples that consume a sizable portion of the prompt budget, and naïve selection strategies (random or pure semantic similarity) do not capture the behavior‑centric nature of WebShell code, resulting in unstable performance.

To address these issues, the authors propose Behavioral Function‑Aware Detection (BFAD), a three‑component framework that tailors LLM inference to the specific execution patterns of WebShells. The first component, the Critical Function Filter, defines a taxonomy of six high‑risk PHP function categories: Program Execution (e.g., exec, system), Code Execution (e.g., eval, preg_replace with /e), Callback Functions (e.g., register_shutdown_function), Network Communication (e.g., fsockopen, curl_init), Information Gathering (e.g., phpinfo, getenv), and Obfuscation/Encryption (e.g., base64_encode, openssl_encrypt). Statistical analysis shows that malicious files invoke these functions on average 22.76 times per file, whereas benign scripts average only 0.74 calls, providing a strong behavioral signal.

The second component, Context‑Aware Code Extraction, isolates the minimal code needed for LLM analysis. For each occurrence of a critical function, a configurable token window τ is extracted, overlapping windows are merged, and any remaining context budget is filled with additional non‑overlapping code snippets to preserve some global information. This process dramatically reduces input length while retaining the local context around potentially malicious calls, thereby fitting within the LLM’s context window without sacrificing the essential behavior cues.

The third component, Weighted Behavioral Function Profiling (WBFP), improves demonstration selection for ICL. For each critical function type f, three statistics are computed across the malicious and benign corpora: coverage difference (whether the function appears at least once in a file), frequency ratio (average per‑file call count ratio), and usage ratio (total call count ratio). These are combined into a discrimination score Score_f = α·rc + β·rf + γ·ru (α=β=γ=1 in the experiments). Scores are normalized to obtain weights wf, which are then used to embed function‑specific code regions via a lightweight code‑search encoder (st‑codesearch‑distilroberta). Similarity between a target file and candidate demonstrations is calculated as a weighted cosine similarity over these embeddings, ensuring that demonstrations sharing the most discriminative functions are prioritized.

The authors evaluate BFAD on a curated dataset of 26.59 K PHP scripts (4.93 K WebShells, 21.66 K benign). Seven LLMs are tested: GPT‑4, LLaMA‑3.1‑70B, Qwen‑2.5‑Coder‑14B/3B, and Qwen‑2.5‑3B/1.5B/0.5B. Baselines include GloVe+SVM, CodeBERT+Random Forest, and a graph‑based GAT detector. Three key observations emerge from the baseline experiments: (1) Model scale shifts error modes—large models achieve very high precision (>95 %) but lower recall (~86 %), missing many malicious samples; smaller models have higher recall (>92 %) but suffer from low precision (~38 %). (2) Naïve ICL is unreliable; random demonstrations can degrade performance, and semantic similarity‑based selection yields only modest gains. (3) Off‑the‑shelf prompting does not close the gap to learned detectors—Qwen‑2.5‑Coder‑14B attains an F1 of 96.39 % versus 98.87 % for the GAT baseline.

Applying BFAD to each LLM yields substantial improvements. Across all models, average F1 increases by 13.82 %. Notably, GPT‑4 reaches 98.92 % F1 (up from 95.10 %), LLaMA‑3.1‑70B achieves 98.71 % (up from 94.85 %), and Qwen‑2.5‑Coder‑14B climbs to 99.03 % (up from 96.39 %). The smallest model, Qwen‑2.5‑Coder‑3B, becomes competitive with traditional methods, achieving 97.45 % F1. Ablation studies confirm that each BFAD component contributes uniquely: Critical Function Filtering alone adds ~6–8 % F1, Context‑Aware Extraction adds ~9–11 %, and WBFP adds ~7–9 %; the full combination yields the highest gains.

The paper acknowledges limitations: the predefined critical function list may need adaptation for newer PHP versions or frameworks; the current approach is static and does not capture dynamic runtime behaviors such as variable function names; and the weighting scheme could overfit to the specific dataset. Future research directions include constructing large‑scale synthetic benchmarks for stress‑testing OOD robustness, integrating graph‑based behavioral representations with LLMs via multimodal adapters, and designing “fast‑slow” detection pipelines where a lightweight LLM filter triggers a slower, continuously‑updated model to handle distribution shift.

In conclusion, this work demonstrates that while raw LLMs struggle with WebShell detection due to length constraints and suboptimal ICL, a behavior‑centric preprocessing and demonstration‑selection pipeline (BFAD) can unlock their full potential, achieving performance on par with or surpassing state‑of‑the‑art learned detectors. The study provides a practical recipe for deploying LLMs in security‑critical code analysis and outlines a roadmap for further enhancing reliability and generalization.