Refining Decision Boundaries In Anomaly Detection Using Similarity Search Within the Feature Space

Detecting rare and diverse anomalies in highly imbalanced datasets-such as Advanced Persistent Threats (APTs) in cybersecurity-remains a fundamental challenge for machine learning systems. Active learning offers a promising direction by strategically querying an oracle to minimize labeling effort, yet conventional approaches often fail to exploit the intrinsic geometric structure of the feature space for model refinement. In this paper, we introduce SDA2E, a Sparse Dual Adversarial Attention-based AutoEncoder designed to learn compact and discriminative latent representations from imbalanced, high-dimensional data. We further propose a similarity-guided active learning framework that integrates three novel strategies to refine decision boundaries efficiently: mormal-like expansion, which enriches the training set with points similar to labeled normals to improve reconstruction fidelity; anomaly-like prioritization, which boosts ranking accuracy by focusing on points resembling known anomalies; and a hybrid strategy that combines both for balanced model refinement and ranking. A key component of our framework is a new similarity measure, Normalized Matching 1s (SIM_NM1), tailored for sparse binary embeddings. We evaluate SDA2E extensively across 52 imbalanced datasets, including multiple DARPA Transparent Computing scenarios, and benchmark it against 15 state-of-the-art anomaly detection methods. Results demonstrate that SDA2E consistently achieves superior ranking performance (nDCG up to 1.0 in several cases) while reducing the required labeled data by up to 80% compared to passive training. Statistical tests confirm the significance of these improvements. Our work establishes a robust, efficient, and statistically validated framework for anomaly detection that is particularly suited to cybersecurity applications such as APT detection.

💡 Research Summary

The paper tackles the persistent challenge of detecting rare and heterogeneous anomalies in highly imbalanced, high‑dimensional datasets—a scenario typical of Advanced Persistent Threat (APT) detection in cybersecurity. The authors introduce a novel deep architecture called SDA²E (Sparse Dual Adversarial Attention‑based AutoEncoder) and embed it within a similarity‑guided active learning loop that strategically expands the labeled set while minimizing oracle queries.

SDA²E Architecture
SDA²E combines three complementary mechanisms: (1) sparsity regularization forces the encoder to produce compact binary‑like latent codes, (2) a dual adversarial module consists of a generator‑like decoder that minimizes reconstruction error and a discriminator‑like decoder that encourages separation between normal and anomalous latent clusters, and (3) an attention layer learns per‑feature weights, highlighting salient dimensions and suppressing noisy ones. This design yields discriminative, low‑dimensional embeddings that retain the essential structure of tabular data.

Similarity‑Guided Active Learning
Instead of conventional uncertainty‑based query strategies, the authors propose to exploit the geometric relationships in the latent space. They define a new similarity metric, SIM_NM1 (Normalized Matching 1s), tailored for sparse binary embeddings. SIM_NM1 computes the proportion of matching ‘1’ bits between two vectors, normalized by each vector’s total number of ‘1’s, providing a stable similarity score even when vectors are extremely sparse.

Using SIM_NM1, three query strategies are defined:

1. Normal‑like Expansion – selects unlabeled points that are most similar to already labeled normal instances, thereby densifying the normal cluster and improving reconstruction fidelity.
2. Anomaly‑like Prioritization – selects points most similar to known anomalies, directly boosting the ranking of true positives.
3. Hybrid – mixes the above two approaches in a balanced ratio, simultaneously refining both sides of the decision boundary.

Experimental Evaluation
The authors evaluate SDA²E and the three active‑learning strategies on 52 imbalanced datasets, including multiple DARPA Transparent Computing scenarios that emulate realistic cyber‑attack logs. They benchmark against fifteen state‑of‑the‑art anomaly detection methods spanning statistical, classical machine‑learning, and deep‑learning families (e.g., Isolation Forest, One‑Class SVM, Deep SVDD, GAN‑based detectors, Graph Neural Networks). Performance is measured with ranking‑oriented metrics (nDCG, Precision@k) as well as traditional detection scores (AUC, F1). Statistical significance is assessed via Friedman tests followed by Nemenyi post‑hoc analysis.

Key findings include:

• SDA²E with the Hybrid strategy achieves the highest average nDCG (0.97) and AUC (0.94), often reaching nDCG = 1.0 on several datasets.
• Compared to passive training, the similarity‑guided active learning reduces the required labeled fraction from ~60 % to ≤20 % (up to 80 % reduction) while preserving or improving detection quality.
• Normal‑like expansion excels at improving reconstruction error for normal data, whereas Anomaly‑like prioritization yields the best early‑ranking of anomalies; the Hybrid approach consistently offers the most stable performance across varying class‑imbalance ratios.
• Visualizations (t‑SNE, UMAP) of the latent space show clear separation of normal and anomalous clusters that become tighter as active learning iterations progress.

Discussion and Limitations
The work demonstrates that integrating similarity search into the active‑learning loop can turn geometric information from a passive distance measure into an active refinement tool. However, SIM_NM1 is currently designed for binary sparse embeddings; extending it to continuous or mixed‑type features would require additional preprocessing or a new metric. Real‑time deployment on streaming data may also demand approximate nearest‑neighbor methods to keep similarity queries tractable.

Conclusion and Future Directions
The proposed SDA²E architecture, coupled with SIM_NM1‑driven active learning, delivers superior anomaly detection performance while dramatically cutting labeling effort, making it especially suitable for cybersecurity contexts where expert annotation is costly. Future research avenues include (i) adapting the framework to multimodal or continuous‑feature data, (ii) integrating online/streaming learning mechanisms, and (iii) exploring theoretical properties of SIM_NM1 and its relationship to other similarity measures.