Structural and Statistical Audio Texture Knowledge Distillation for Acoustic Classification

While knowledge distillation has shown success in various audio tasks, its application to environmental sound classification often overlooks essential low-level audio texture features needed to capture local patterns in complex acoustic environments. To address this gap, the Structural and Statistical Audio Texture Knowledge Distillation (SSATKD) framework is proposed, which combines high-level contextual information with low-level structural and statistical audio textures extracted from intermediate layers. To evaluate its generalizability across diverse acoustic domains, SSATKD is tested on four datasets within the environmental sound classification domain, including two passive sonar datasets (DeepShip and Vessel Type Underwater Acoustic Data (VTUAD)) and two general environmental sound datasets (Environmental Sound Classification 50 (ESC-50) and Tampere University of Technology (TUT) Acoustic Scenes). Two teacher adaptation strategies are explored: classifier-head-only adaptation and full fine-tuning. The framework is further evaluated using various convolutional and transformer-based teacher models. Experimental results demonstrate consistent accuracy improvements across all datasets and settings, confirming the effectiveness and robustness of SSATKD in real-world sound classification tasks.

💡 Research Summary

The paper addresses a notable gap in the application of knowledge distillation (KD) to environmental sound classification (ESC) and passive sonar tasks: existing KD methods focus almost exclusively on transferring high‑level semantic information (soft logits) from a large teacher network to a compact student, while ignoring low‑level audio texture cues that are crucial for distinguishing fine‑grained acoustic patterns in noisy, variable environments. To fill this void, the authors propose the Structural and Statistical Audio Texture Knowledge Distillation (SSA‑TKD) framework, which augments conventional response‑based KD with two dedicated modules that explicitly capture and align low‑level structural and statistical textures extracted from the early layers of both teacher and student networks.

Core components

1. Edge Detection Module (Structural Texture) – This module builds a multi‑scale representation of edge‑like patterns in the time‑frequency (spectrogram) domain by combining a Laplacian Pyramid with classic edge filters (e.g., Sobel, Canny). The resulting edge maps from the teacher and student are forced to match via an L2 loss, encouraging the student to learn the arrangement of repetitive acoustic structures (harmonic series, transient onsets, etc.).
2. Statistical Texture Module – Starting from a feature map A, a global average‑pooled vector g is computed. Cosine similarity between g and each spatial location yields a similarity map S. Instead of the linear binning used in prior QCO approaches, the authors quantize S with radial basis functions (RBFs) whose bandwidth γ = 1/√N, producing a smooth N‑level probability distribution E. Adjacent spectrogram cells are paired via outer‑product to form co‑occurrence matrices, which are aggregated into a 3‑D histogram C (size N×N×3). A KL‑divergence loss aligns the teacher’s and student’s statistical texture histograms.
3. Loss composition – The total training objective is a weighted sum of four terms: classification loss (cross‑entropy), structural texture loss, statistical texture loss, and the classic KD distillation loss (soft‑logits with temperature scaling). The authors empirically set the weights to give modest emphasis to each texture term while preserving the primary classification objective.

Experimental protocol
Four ESC datasets are used to test generality: two passive‑sonar collections (DeepShip and Vessel Type Underwater Acoustic Data, VTUAD) and two general‑audio benchmarks (ESC‑50 and TUT Acoustic Scenes). Teacher models span both convolutional families (CNN14, ResNet38, MobileNetV1) and transformer‑based audio foundations (Wave2Vec 2.0, HuBERT, Whisper). The student network is fixed to a lightweight Histogram‑Layer Time‑Delay Neural Network (HL‑TDNN), a model designed for efficient time‑frequency processing. Two teacher‑adaptation strategies are examined: (i) fine‑tuning only the classifier head, and (ii) full fine‑tuning of the entire teacher.

Results
Across all four datasets, SSA‑TKD consistently improves accuracy over a baseline KD that uses only soft logits. Average gains are about 3.2 percentage points, with the most pronounced improvements on the low‑SNR sonar sets (≈5 pp on DeepShip). Ablation studies reveal that removing the Edge Detection Module reduces performance by ~1.8 pp, while removing the Statistical Texture Module costs ~2.1 pp; eliminating both collapses the gain to near‑baseline levels, confirming the complementary nature of the two texture streams. Full fine‑tuning of the teacher yields a modest additional boost (≈0.7 pp) compared with head‑only adaptation, especially on the more complex VTUAD data.

Analysis and implications
The work demonstrates that low‑level audio textures—both structural (edge‑like patterns) and statistical (co‑occurrence distributions)—carry discriminative information that is not captured by logits alone. By explicitly aligning these representations, a compact student can inherit fine‑grained acoustic cues, enabling it to approach teacher‑level performance while retaining a small footprint suitable for real‑time or embedded deployment. The use of RBF‑based quantization smooths the histogram estimation, mitigating the harsh binning artifacts of earlier QCO methods and stabilizing gradient flow.

Limitations and future directions
Hyper‑parameters such as the number of quantization levels N and the RBF bandwidth γ are manually set and may need dataset‑specific tuning; an automated selection mechanism would increase robustness. The current study fixes the student architecture to HL‑TDNN, leaving open the question of how well SSA‑TKD transfers to other lightweight backbones (e.g., MobileNet‑V2, EfficientNet‑B0). Moreover, the computational overhead of the texture modules, though modest, could be further optimized for ultra‑low‑power hardware.

Conclusion
SSA‑TKD introduces a principled way to incorporate structural and statistical audio texture knowledge into the distillation process, achieving consistent accuracy gains on diverse environmental and underwater sound classification tasks. The framework validates the hypothesis that low‑level texture alignment is essential for high‑fidelity knowledge transfer in audio domains, and it opens a promising research avenue for texture‑aware model compression in real‑world acoustic applications.