Exploring Frequency-Domain Feature Modeling for HRTF Magnitude Upsampling

Accurate upsampling of Head-Related Transfer Functions (HRTFs) from sparse measurements is crucial for personalized spatial audio rendering. Traditional interpolation methods, such as kernel-based weighting or basis function expansions, rely on measurements from a single subject and are limited by the spatial sampling theorem, resulting in significant performance degradation under sparse sampling. Recent learning-based methods alleviate this limitation by leveraging cross-subject information, yet most existing neural architectures primarily focus on modeling spatial relationships across directions, while spectral dependencies along the frequency dimension are often modeled implicitly or treated independently. However, HRTF magnitude responses exhibit strong local continuity and long-range structure in the frequency domain, which are not fully exploited. This work investigates frequency-domain feature modeling by examining how different architectural choices, ranging from per-frequency multilayer perceptrons to convolutional, dilated convolutional, and attention-based models, affect performance under varying sparsity levels, showing that explicit spectral modeling consistently improves reconstruction accuracy, particularly under severe sparsity. Motivated by this observation, a frequency-domain Conformer-based architecture is adopted to jointly capture local spectral continuity and long-range frequency correlations. Experimental results on the SONICOM and HUTUBS datasets demonstrate that the proposed method achieves state-of-the-art performance in terms of interaural level difference and log-spectral distortion.

💡 Research Summary

This paper addresses the problem of upsampling head‑related transfer functions (HRTFs) from a limited set of sparsely measured directions to a dense directional grid, a prerequisite for personalized spatial‑audio rendering. Traditional approaches—distance‑weighted interpolation (e.g., bilinear, spherical‑triangle) and basis‑function expansions (spherical harmonics, PCA)—operate solely on the target subject’s data and are fundamentally constrained by the spatial sampling theorem; performance collapses when the number of measured directions is small. Recent learning‑based methods mitigate this by exploiting multi‑subject datasets, but they largely focus on modeling spatial relationships (directional coordinates) while treating the frequency axis either independently per bin or with shallow aggregation. Consequently, the rich spectral structure of HRTF log‑magnitude responses—pinna resonances, notches, and long‑range cross‑frequency correlations—is under‑utilized.

The authors first quantify these spectral dependencies using Pearson correlation matrices computed over all subjects, directions, and ears in the SONICOM dataset. The matrices reveal strong local continuity (high correlation near the diagonal) as well as significant long‑range correlations across distant frequency bins. This observation motivates a systematic exploration of frequency‑domain modeling strategies. Four families of architectures are examined: (1) per‑frequency multilayer perceptrons (MLPs) that treat each frequency independently; (2) 1‑D convolutions that capture local continuity; (3) dilated convolutions that expand the receptive field without increasing parameters, thereby modeling long‑range dependencies; and (4) self‑attention mechanisms that directly learn global inter‑frequency relationships. Experiments across varying sparsity levels (from 5 % to 30 % of the full directional set) consistently show that any explicit frequency‑domain modeling improves log‑spectral distortion (LSD) and interaural level difference (ILD) error relative to coordinate‑conditioned CNN baselines, with the greatest gains observed under severe sparsity.

Building on these findings, the paper proposes the Frequency‑Domain Conformer (FD‑Conformer). The input tensor comprises M measured directions, two ears, and F frequency bins (size M × 2 × F). For each frequency bin three binaural features are extracted: left‑ear magnitude, right‑ear magnitude, and their difference (L‑R), yielding a 3 M‑dimensional vector per bin. Stacking across all bins forms a spectral feature map S ∈ ℝ^{3M × F}. S is linearly projected to a latent dimension C, and sinusoidal positional encodings are added to preserve frequency ordering. The core of the model consists of N stacked Conformer blocks, each comprising: (i) a feed‑forward network (FFN) for point‑wise nonlinearity, (ii) multi‑head self‑attention (MHSA) to capture global frequency‑wise dependencies, and (iii) depth‑wise convolution to model local continuity. Residual connections and layer normalization ensure stable training. After the N blocks, a direction‑expansion head maps the frequency‑domain representation back to the full directional grid, producing a dense log‑magnitude estimate ˆH_{freq}. A parallel spatial‑mapping module performs a simple linear sparse‑to‑dense transformation per frequency (ignoring inter‑frequency structure). The final prediction is the sum of the spatial and frequency components, allowing the network to exploit both spatial and spectral cues.

Evaluation is conducted on two public datasets, SONICOM and HUTUBS, using a 2‑fold cross‑validation protocol. Baselines include traditional interpolation, spherical‑harmonic reconstruction, and state‑of‑the‑art learning‑based methods (coordinate‑conditioned CNNs, graph neural networks, retrieval‑augmented networks such as RANF). Performance is measured by mean absolute ILD error (in dB) and LSD (in dB) across all directions, ears, and subjects. FD‑Conformer achieves ILD errors below 1.2 dB and LSD below 1.8 dB, outperforming all baselines by a margin of 1–3 dB. Notably, when only 10 % of directions are available, the proposed model maintains an ILD error of ~1.5 dB, whereas the best competing learning‑based method degrades to >3 dB. Ablation studies confirm that both the Conformer’s attention and convolution components contribute uniquely: removing attention harms long‑range spectral fidelity, while removing convolution degrades local smoothness.

In summary, the paper makes three key contributions: (1) a quantitative analysis of frequency‑domain correlations in HRTF log‑magnitude spectra, establishing the need for explicit spectral modeling; (2) a comprehensive design space study of frequency‑domain architectures, demonstrating the superiority of models that jointly capture local continuity and global dependencies; and (3) the introduction of FD‑Conformer, a Conformer‑based network that achieves state‑of‑the‑art upsampling performance across multiple datasets and sparsity regimes. The authors release their code and pretrained models (GitHub link) to facilitate reproducibility and encourage further research on frequency‑centric approaches for personalized spatial audio.