An End-to-End Conversational Style Matching Agent

We present an end-to-end voice-based conversational agent that is able to engage in naturalistic multi-turn dialogue and align with the interlocutor’s conversational style. The system uses a series of deep neural network components for speech recognition, dialogue generation, prosodic analysis and speech synthesis to generate language and prosodic expression with qualities that match those of the user. We conducted a user study (N=30) in which participants talked with the agent for 15 to 20 minutes, resulting in over 8 hours of natural interaction data. Users with high consideration conversational styles reported the agent to be more trustworthy when it matched their conversational style. Whereas, users with high involvement conversational styles were indifferent. Finally, we provide design guidelines for multi-turn dialogue interactions using conversational style adaptation.

💡 Research Summary

The paper presents a fully integrated, voice‑based conversational agent that can adapt its linguistic and prosodic behavior to match the conversational style of its human interlocutor in real time. The system combines four core neural components: (1) speech recognition using Microsoft’s Bing API, (2) paralinguistic feature extraction (fundamental frequency and RMS energy) via a digital‑signal‑processing pipeline, (3) dialogue generation that blends a large‑scale neural language model trained on Twitter data with intent detection from Microsoft LUIS, and (4) speech synthesis driven by SSML, allowing fine‑grained control over pitch, loudness, and speech rate.

The authors ground their style model in Tannen’s “consideration‑involvement” axis, distinguishing High Consideration (HC) – slower speech, longer pauses, moderate prosody – from High Involvement (HI) – faster, louder, overlapping speech. Four content‑level variables (word choice, sentence length, lexical repetition, affective vocabulary) and two prosodic variables (pitch, loudness) are computed on the fly from the user’s utterance. A rule‑based dialogue manager maps these measurements onto a predefined set of style‑matching transformations, adjusting the agent’s SSML parameters so that the generated response mirrors the user’s style.

The architecture is implemented on the open‑source Platform for Situated Interaction (PSI), ensuring low‑latency data flow between modules. The agent can operate autonomously without human intervention, handling open‑ended chit‑chat via the neural model and task‑oriented queries via scripted LUIS‑driven responses.

To evaluate the impact of style matching, a user study with 30 adult participants was conducted. Each participant conversed with the system for 15–20 minutes, yielding over eight hours of interaction data. Participants were randomly assigned to either a “style‑matched” condition (the agent adjusted its prosody and lexical choices in real time) or a “non‑matched” baseline (the agent used default prosody). After the interaction, participants completed questionnaires measuring perceived trust, rapport, and overall satisfaction.

Results show a clear interaction between user style and the benefit of matching. Participants identified as HC reported significantly higher trust scores when the agent matched their style (mean increase of 1.2 points on a 7‑point Likert scale, p < 0.05). In contrast, HI participants showed no statistically significant difference between conditions, suggesting that style matching is particularly valuable for users who prefer a more considerate, slower conversational rhythm. Overall satisfaction was high in both conditions, indicating that the core dialogue capabilities were adequate, but the added style adaptation provided a measurable trust boost for a specific user segment.

Based on these findings, the authors propose five design guidelines for future conversational systems: (1) select style‑related features that can be extracted in real time without heavy computational overhead; (2) limit prosodic adjustments to a modest range (five discrete levels) to avoid unnatural sounding speech; (3) implement style‑matching logic asynchronously to preserve natural turn‑taking latency; (4) incorporate user profiling to apply style‑matching preferentially for HC‑type users; and (5) establish a continuous feedback loop to refine mapping rules based on ongoing interaction data.

In conclusion, the study demonstrates that real‑time conversational style matching is technically feasible and can enhance perceived trust, especially for users who value considerate communication. It also highlights that style adaptation is not universally beneficial; designers must consider individual user preferences and possibly combine style matching with other social cues (e.g., small talk, empathy) to achieve broader acceptance. Future work could explore richer multimodal cues (facial expression, gesture), more sophisticated affect detection, and long‑term adaptation across repeated sessions.