Operationalizing Research Software for Supply Chain Security

Empirical studies of research software are hard to compare because the literature operationalizes ``research software’’ inconsistently. Motivated by the research software supply chain (RSSC) and its security risks, we introduce an RSSC-oriented taxonomy that makes scope and operational boundaries explicit for empirical research software security studies. We conduct a targeted scoping review of recent repository mining and dataset construction studies, extracting each work’s definition, inclusion criteria, unit of analysis, and identification heuristics. We synthesize these into a harmonized taxonomy and a mapping that translates prior approaches into shared taxonomy dimensions. We operationalize the taxonomy on a large community-curated corpus from the Research Software Encyclopedia (RSE), producing an annotated dataset, a labeling codebook, and a reproducible labeling pipeline. Finally, we apply OpenSSF Scorecard as a preliminary security analysis to show how repository-centric security signals differ across taxonomy-defined clusters and why taxonomy-aware stratification is necessary for interpreting RSSC security measurements.

💡 Research Summary

The paper tackles a fundamental problem in empirical research software (RS) security studies: the lack of a consistent definition and operational boundary for “research software.” Because prior works have used disparate criteria—linking repositories to publications, using funding information, or applying text‑based classification—their sampled software populations differ substantially, making cross‑study comparisons unreliable. To address this, the authors develop a research‑software‑supply‑chain (RSSC)‑oriented taxonomy that makes scope choices explicit and enables systematic stratification of RS datasets.

The taxonomy is derived from Okafor et al.’s software‑supply‑chain model, which distinguishes actors, operations, and artifacts. The authors adapt this to the research context and define four dimensions: (1) Actor Unit (who maintains the software – research groups/labs, individual maintainers, institutions, community/governance bodies), (2) Supply Chain Role (application software, dependency artifact, infrastructure, unknown), (3) Research Role (direct research execution, research‑support tooling, incidental/general‑purpose, unknown), and (4) Distribution Pathway (package registry, source repository, installer/binary, network service, container, build‑and‑release pipeline, unknown).

To validate and refine the taxonomy, a targeted scoping review of papers published between 2020 and 2025 in IEEE Xplore and ACM DL was performed. The review extracted each study’s definition of RS, inclusion/exclusion criteria, unit of analysis, and identification heuristics, then mapped these to the four dimensions. This process revealed that many existing studies rely on proxies that either over‑narrow or over‑broaden the RS population, confirming the need for a unified framework.

The refined taxonomy was then operationalized on the Research Software Encyclopedia (RSE) corpus, comprising 6,966 entries. For each entry, repository metadata and README content were collected. An automated labeling pipeline powered by OpenAI’s GPT‑5.1 was built; a small manual validation set showed >92 % agreement with human labels. The authors release the labeled dataset, the labeling codebook, and the reproducible pipeline, thereby supporting reuse and replication.

With the taxonomy‑annotated corpus in hand, the authors conduct a preliminary security assessment using the OpenSSF Scorecard. Scorecard evaluates ten repository‑centric checks (e.g., branch protection, CI configuration, security policy) and returns a score from 0 to 10, or –1 when a check cannot be evaluated. The analysis covered 5,937 of the 6,966 RSE entries (85.2 % coverage). Median overall Scorecard scores for RS were 2.9, compared with 3.9 for a baseline set of Apache Software Foundation (ASF) projects. Missingness rates (fraction of –1 scores) were similar (22 % for RS vs 28 % for ASF), indicating that the lower RS scores are not merely due to lack of evaluable data but reflect fewer adopted security practices.

Crucially, when the RS population is stratified by the taxonomy, substantial variation emerges. Community‑governed projects achieve a median score of 3.6 (Δ = –0.3 relative to ASF) and exhibit low missingness (11 %). Individual maintainers score 2.7 (Δ = –1.2) with missingness comparable to ASF, suggesting that the gap stems from weaker adoption of best practices rather than evaluation limitations. Research groups/labs and institutions fall in between, while “unknown” or mixed‑responsibility projects show the lowest scores and highest missingness. These findings demonstrate that treating RS as a monolithic group masks meaningful differences in governance, maturity, and security posture.

The paper discusses threats to validity, including the ad‑hoc nature of the scoping review, potential biases in LLM‑based labeling, and the limited scope of Scorecard (which does not capture non‑repository activities such as external CI pipelines or supply‑chain provenance beyond the repository). Nonetheless, the authors argue that the RSSC‑oriented taxonomy provides a necessary scaffold for future comparative security research, policy formulation, and risk mitigation strategies in the research software ecosystem.

In summary, the contributions are threefold: (1) a validated, RSSC‑focused taxonomy that harmonizes divergent operationalizations of research software; (2) an openly released, taxonomy‑labeled RSE dataset with reproducible labeling tools; and (3) an illustrative, taxonomy‑aware security analysis using OpenSSF Scorecard that underscores the importance of explicit scope and stratification for interpreting security measurements. This work lays the groundwork for more rigorous, comparable, and actionable research software supply‑chain security studies.