Unlocking Fine-Grained and Within-Utterance Speaking Style Control in Prompt-Based Text-to-Speech Models

Article Type: Research

The Attention Bottleneck in Autoregressive TTS

The paper identifies a fundamental architectural constraint in autoregressive TTS decoders: a strong attention bias toward early tokens. This bias causes the initial audio realization, often determined by the first few style prompt tokens, to disproportionately dominate the entire subsequent generation. This effect makes it exceptionally difficult to realize a time-varying style transition within a single utterance, as the model's "mind" is stubbornly anchored to the beginning. The proposed solution—KV-cache swapping and sliding-window attention masking—is a direct intervention to mitigate this bias, forcing the decoder to pay more attention to later segments of the style prompt sequence to enable dynamic shifts.

Moving Beyond Monolithic Style Transfer

Current prompt-based TTS models effectively apply a single, global style vector derived from a text prompt to an entire utterance. This work's core contribution is reframing style control from a static transfer problem to a dynamic interpolation and transition problem. The inter-utterance method treats style embeddings not as isolated points but as part of a navigable vector space, where direction vectors between contrastive prompts define paths for continuous interpolation. This enables applications like gradually morphing a speaker's gender or emotional intensity across multiple sentences, which is a qualitative leap beyond switching between fixed style presets.

Quantifying the Fluidity of Voice

The experimental metrics emphasize control fidelity and perceptual quality. For interpolation between utterances, the high success rates (99-100%) in gender conversion demonstrate robust controllability along that specific axis. The reported ranges for pitch variation (up to 36 Hz) and speed change (1.6 syllables/second) provide concrete bounds on the model's interpolatable style space. For intra-utterance transitions, the metrics shift to measuring stability and naturalness: speaker similarity scores of 0.81-0.91 confirm the core identity is preserved even during stylistic flux, while perceptual smoothness scores of 3.48-4.48 (presumably on a 1-5 scale) indicate the transitions are not jarring but rather integrated into the speech output.

A Shift Toward Compositional and Temporal Style Control

This work implicitly argues that practical TTS requires compositional and temporally granular control. The ability to interpolate styles inter-utterance allows for narrative pacing and emotional arcs. The ability to transition styles intra-utterance is crucial for expressive prosody, such as a speaker's voice rising in excitement mid-sentence or adopting a different tone for a quoted phrase. By solving the technical challenge of the initial-token attention bottleneck, the research opens the door to synthesizing speech that mirrors the complex, moment-to-moment stylistic variations present in natural human speech, moving beyond the flat, uniform tone of early TTS systems.