Stratified Bootstrap Test Package

The Stratified Bootstrap Test (SBT) provides a nonparametric, resampling-based framework for assessing the stability of group-specific ranking patterns in multivariate survey or rating data. By repeatedly resampling observations and examining whether a group’s top-ranked items remain among the highest-scoring categories across bootstrap samples, SBT quantifies ranking robustness through a non-containment index. In parallel, the stratified bootstrap test extends this framework to formal statistical inference by testing ordering hypotheses among population means. Through resampling within groups, the method approximates the null distribution of ranking-based test statistics without relying on distributional assumptions. Together, these techniques enable both descriptive and inferential evaluation of ranking consistency, detection of aberrant or adversarial response patterns, and rigorous comparison of groups in applications such as survey analysis, item response assessment, and fairness auditing in AI systems.

💡 Research Summary

The paper introduces the Stratified Bootstrap Test (SBT), a non‑parametric, resampling‑based framework for assessing the stability of group‑specific ranking patterns in multivariate survey or rating data, and for formally testing ordering hypotheses among population means. The authors argue that traditional parametric methods such as ANOVA or t‑tests are ill‑suited for data with complex dependencies, missingness, or non‑normal distributions, especially when the research question concerns the robustness of rank order rather than absolute mean differences.

SBT operates by repeatedly drawing bootstrap samples within each group (i.e., stratified resampling with replacement). For each bootstrap replicate, group‑wise item means are recomputed, and two types of statistics are derived:

1. Non‑containment index – For a given group g and a target set of the top i items (indices T_{g,i}), the method computes the proportion of bootstrap replicates in which the target set is not fully contained in the bootstrap‑derived top‑i set. The index is defined as 1 – (# of replicates where containment holds)/B, where B is the number of bootstrap iterations. Values near 0 indicate highly stable rankings; values near 1 signal instability.

2. Ordering hypothesis test – The observed sample means (\bar{x}1 \ge \bar{x}2 \ge \dots \ge \bar{x}G) may be a product of sampling variability. For a pre‑specified split g (1 ≤ g < G), the null hypothesis is (H_0: \min{i\le g}\mu_i \le \max{j>g}\mu_j) versus the alternative (H_1: \min{i\le g}\mu_i > \max_{j>g}\mu_j). In each bootstrap replicate the event (E^{(b)} = {\min_{i\le g}\bar{x}^{(b)}i > \max{j>g}\bar{x}^{(b)}j}) is checked, and the Monte‑Carlo estimate (\hat p = \frac{1}{B}\sum{b=1}^B \mathbf{1}{E^{(b)}}) serves as a p‑value. The same machinery can test a strict total ordering (\mu_1 > \mu_2 > \dots > \mu_G).

The authors provide an R package that implements two primary functions:

• SingleStratifiedBootstrap() – Takes a numeric matrix, a set of target column indices, and bootstrap parameters, returning the non‑containment rate for that specific target set. The user can specify the summary function (mean, median, etc.), whether to sort in decreasing order, handling of missing values, and parallel execution via the n_cores argument.

• GetSBT() – Accepts a vector of group labels and a response matrix, computes (a) a table of group‑level means (or proportions for binary data) and (b) a matrix of non‑containment rates for each top‑i (i = 1,…,k). The function supports Likert, binary, and continuous response types, with an optional mapping table to convert textual Likert responses into numeric scores. Additional arguments control minimum group size, bootstrap sample size, replacement, and reproducibility (seed).

A reproducible example simulates a 5‑item Likert questionnaire for 100 respondents, randomly assigns them to “Woman” or “Man”, and runs GetSBT() with n_boot = 500. The output shows group mean scores (e.g., women = 3.42, men = 3.15) and non‑containment rates for top‑1 through top‑5 items (e.g., top‑1 = 0.02, top‑5 = 0.35). Low non‑containment values indicate that the top‑ranked items for a group are consistently reproduced across bootstrap samples, suggesting genuine ranking stability.

The paper discusses practical considerations:

• Sample size – Very small groups (< 10) yield noisy bootstrap estimates; the package issues warnings and allows the user to set a min_group_size threshold.
• Computational cost – With B = 10 000, runtime can be substantial; parallelization via n_cores or external back‑ends (e.g., future.apply) is recommended.
• Data preprocessing – Textual responses should be cleaned (case normalization, trimming) before conversion; a likert_map argument facilitates custom coding schemes.
• Interpretation – Non‑containment rates near 0 imply stable rankings, while rates near 1 flag potential data quality issues, adversarial responses, or genuine heterogeneity.

The methodological contribution lies in (1) introducing a direct, non‑parametric measure of ranking stability, (2) extending bootstrap resampling to test order‑based hypotheses without relying on distributional assumptions, and (3) delivering an easy‑to‑use software implementation. Potential applications span survey validation, detection of low‑quality or adversarial responses, fairness auditing of AI systems (e.g., gender or racial bias in visual labeling), and any domain where the consistency of top‑ranked items across subpopulations matters.

Limitations are acknowledged: the approach can be computationally intensive for large k or G, multiple testing corrections for examining many i or g values are not built‑in, and the current resampling scheme assumes independence within groups. Future work could explore block or hierarchical bootstrap variants, integrate false‑discovery rate adjustments, and compare SBT’s non‑containment index with Bayesian posterior rank probabilities.

In summary, the Stratified Bootstrap Test package offers a robust, assumption‑free toolkit for quantifying how reliably group‑specific rankings persist under sampling variability and for formally testing whether observed orderings reflect true population differences. Its blend of methodological rigor and practical implementation makes it a valuable addition to the toolbox of statisticians, data scientists, and researchers dealing with complex, multivariate rating data.