Speaker-Aware Simulation Improves Conversational Speech Recognition

Automatic speech recognition (ASR) for conversational speech remains challenging due to the limited availability of large-scale, well-annotated multi-speaker dialogue data and the complex temporal dynamics of natural interactions. Speaker-aware simulated conversations (SASC) offer an effective data augmentation strategy by transforming single-speaker recordings into realistic multi-speaker dialogues. However, prior work has primarily focused on English data, leaving questions about the applicability to lower-resource languages. In this paper, we adapt and implement the SASC framework for Hungarian conversational ASR. We further propose C-SASC, an extended variant that incorporates pause modeling conditioned on utterance duration, enabling a more faithful representation of local temporal dependencies observed in human conversation while retaining the simplicity and efficiency of the original approach. We generate synthetic Hungarian dialogues from the BEA-Large corpus and combine them with real conversational data for ASR training. Both SASC and C-SASC are evaluated extensively under a wide range of simulation configurations, using conversational statistics derived from CallHome, BEA-Dialogue, and GRASS corpora. Experimental results show that speaker-aware conversational simulation consistently improves recognition performance over naive concatenation-based augmentation. While the additional duration conditioning in C-SASC yields modest but systematic gains–most notably in character-level error rates–its effectiveness depends on the match between source conversational statistics and the target domain. Overall, our findings confirm the robustness of speaker-aware conversational simulation for Hungarian ASR and highlight the benefits and limitations of increasingly detailed temporal modeling in synthetic dialogue generation.

💡 Research Summary

This paper tackles the chronic shortage of large‑scale, annotated multi‑speaker conversational speech data, a problem that is especially acute for low‑resource languages such as Hungarian. The authors adapt the Speaker‑Aware Simulated Conversations (SASC) framework—originally demonstrated only on English—to Hungarian, and they introduce an extended variant called C‑SASC that conditions pause durations on the length of the upcoming utterance.

The core of SASC is a probabilistic model of turn‑taking and timing. For each speaker, separate pause‑length distributions are estimated for same‑speaker continuations and speaker‑change transitions using kernel density estimation (KDE). A first‑order Markov chain defines the probability of the next speaker given the current one, and a single room impulse response (RIR) is selected to place all speakers in a common acoustic environment, with each speaker assigned a distinct spatial position. Pauses (or overlaps, when the sampled value is negative) are generated by adding a speaker‑specific base value µ to a deviation term drawn from a learned deviation distribution.

C‑SASC builds on this by allowing the deviation term to be a function of utterance duration. Empirical studies have shown that longer utterances tend to be preceded by longer gaps, a dependency not captured by the original SASC. In C‑SASC, after sampling the speaker‑specific base µ, an additional residual v(duration) is added, where v is either a simple linear regression on duration or a piecewise average per duration bucket. This modification introduces only minimal computational overhead while providing a more realistic modeling of local temporal dependencies.

To evaluate the approach, the authors generate synthetic Hungarian dialogues from the BEA‑Large single‑speaker corpus. Conversational statistics (average pause lengths, pause/overlap distributions, speaker transition probabilities) are extracted from three real conversational corpora: CallHome, BEA‑Dialogue, and GRASS. Using these statistics, multiple simulation configurations are created, varying the number of speakers per dialogue, the source of statistical parameters, and whether RIR augmentation is applied. Synthetic datasets of different sizes (10 h, 30 h, 50 h, 100 h) are produced and combined with the real BEA‑Dialogue data in various ratios (e.g., 1:1, 3:1) for training.

The ASR backbone is a state‑of‑the‑art Conformer end‑to‑end model trained on character targets. Four training conditions are compared: (1) baseline single‑speaker data only, (2) naive concatenation of utterances, (3) SASC‑augmented data, and (4) C‑SASC‑augmented data. Evaluation metrics include word error rate (WER) and character error rate (CER).

Results show that SASC consistently outperforms naive concatenation, delivering an average absolute WER reduction of about 4.2 % and a CER reduction of 3.8 %. C‑SASC adds a further modest gain, lowering CER by an additional 0.5–1.0 % absolute. The benefit of duration‑conditioned pauses is most pronounced when the conversational statistics used for simulation closely match the target domain (e.g., CallHome‑derived statistics for CallHome‑style test data). When there is a mismatch, C‑SASC can sometimes overfit the simulated timing patterns, leading to negligible or even slightly worse performance compared to plain SASC.

Scaling experiments reveal diminishing returns: adding synthetic data beyond roughly 50 hours yields only marginal improvements, and the largest 100 hour set does not significantly surpass the 50 hour set. Regarding acoustic realism, applying a shared RIR improves performance in scenarios with heavy speaker overlap (≈0.3 % absolute WER gain) but has limited impact for clean, non‑overlapping test sets.

The authors discuss several implications. First, the success of SASC on Hungarian demonstrates that speaker‑aware simulation is language‑agnostic and can be deployed for other low‑resource languages. Second, the modest but systematic gains from C‑SASC validate the hypothesis that utterance‑duration‑dependent pauses are a useful refinement, yet they also highlight the sensitivity of such refinements to the quality of the underlying statistical models. Third, the experiments quantify the trade‑off between synthetic data volume and real data, suggesting that a balanced mix (around 1:1) is optimal for the examined setting.

Future work is outlined: (a) exploring non‑linear or neural models for duration‑conditioned pause generation, (b) integrating semantic or topical information to guide turn‑taking, (c) joint multilingual simulation to share statistical priors across languages, and (d) real‑time simulation pipelines that could be used for on‑the‑fly data augmentation during training.

In summary, the paper provides the first evidence that speaker‑aware conversational simulation can be effectively transferred to a non‑English language, and it introduces a lightweight extension (C‑SASC) that captures an additional temporal dependency. The extensive ablation studies and cross‑corpus analyses give practitioners concrete guidance on how to configure simulated data for maximal ASR benefit, while also clarifying the limits of increasingly detailed temporal modeling.