A Practical Solution to Systematically Monitor Inconsistencies in SBOM-based Vulnerability Scanners

Software Bill of Materials (SBOM) provides new opportunities for automated vulnerability identification in software products. While the industry is adopting SBOM-based Vulnerability Scanning (SVS) to identify vulnerabilities, we increasingly observe inconsistencies and unexpected behavior, that result in false negatives and silent failures. In this work, we present the background necessary to understand the underlying complexity of SVS and introduce SVS-TEST, a method and tool to analyze the capability, maturity, and failure conditions of SVS-tools in real-world scenarios. We showcase the utility of SVS-TEST in a case study evaluating seven real-world SVS-tools using 16 precisely crafted SBOMs and their respective ground truth. Our results unveil significant differences in the reliability and error handling of SVS-tools; multiple SVS-tools silently fail on valid input SBOMs, creating a false sense of security. We conclude our work by highlighting implications for researchers and practitioners, including how organizations and developers of SVS-tools can utilize SVS-TEST to monitor SVS capability and maturity. All results and research artifacts are made publicly available and all findings were disclosed to the SVS-tool developers ahead of time.

💡 Research Summary

The paper addresses the emerging reliability challenges of Software Bill of Materials (SBOM)‑based Vulnerability Scanners (SVS). While SBOMs have become a cornerstone of modern software supply‑chain security—driven by policy mandates such as the 2021 US Executive Order and the EU Cyber Resilience Act—the authors observe that many SVS tools produce inconsistent results, silent failures, and false negatives. To systematically evaluate and monitor these tools, the authors introduce SVS‑TEST, a methodology and open‑source framework that automates the testing of SVS tools against a curated set of edge‑case SBOMs.

The background section first explains SBOM formats (CycloneDX and SPDX) and the two dominant component identifiers: Common Platform Enumeration (CPE) and package URLs (purl). It highlights that both identifiers lack a universally enforced matching algorithm, leading to divergent implementations across tools. The authors also discuss ancillary challenges such as incomplete SBOM generation, delayed vulnerability database updates, version string ambiguity, and wildcard handling, all of which can cause mismatches and missed vulnerabilities.

SVS‑TEST consists of five phases: (1) a test‑case library containing 16 precisely crafted SBOMs and machine‑readable expectations (e.g., regex patterns, explicit error messages); (2) scenario assembly, where experts group related test cases into scenarios using a fractional factorial design to isolate error causes; (3) test execution, where each of seven representative SVS tools (both commercial and open‑source) processes every SBOM; (4) output evaluation, which automatically compares tool outputs to the ground‑truth expectations, flagging both false positives/negatives and “silent” failures where a tool returns no result or crashes; and (5) scenario interpretation, which translates quantitative findings into actionable insights about tool maturity, assumptions, and regression risks.

The empirical study evaluates seven SVS tools against the 16 SBOM scenarios. Results reveal substantial variability: some tools miss vulnerabilities when CPE entries are malformed, others ignore purl‑based components, and several tools fail to parse certain SBOM formats (e.g., SPDX XML) without emitting an error, effectively giving a false sense of security. Regression testing also uncovered version‑specific behavior changes, underscoring the need for continuous monitoring.

All artifacts—including the test cases, execution scripts, and detailed results—are publicly released on Zenodo, and the authors disclosed findings to the tool developers before publication, prompting prompt patches. The paper concludes with practical recommendations: organizations should adopt SVS‑TEST to benchmark and continuously monitor their SVS pipelines, especially when using SVS‑as‑a‑Service offerings that may change behavior without notice. Developers are encouraged to integrate SVS‑TEST into CI/CD pipelines to catch regressions early. Future work will explore automated generation of SBOM edge cases, broader vulnerability database integration, and real‑time dashboard visualizations for ongoing SVS health monitoring.