GRAFT: Grafted Reference Audio for Fine-grained Pronunciation in Zero-shot Text-to-Speech
GRAFT introduces a per-word pronunciation conditioning mechanism for neural codec language modeling in zero-shot text-to-speech systems. The method uses a short spoken sample of a specific word, encoded by the model’s own tokenizer, to disambiguate rare proper nouns and technical terms. Voice conversion during training disentangles the hint speaker from the target speaker, allowing pronunciation hints from any voice while maintaining the target identity. GRAFT reduces target-word phoneme error r
Analysis
TL;DR
- GRAFT introduces a per-word pronunciation conditioning mechanism for neural codec language modeling in zero-shot text-to-speech systems.
- The method uses a short spoken sample of a specific word, encoded by the model’s own tokenizer, to disambiguate rare proper nouns and technical terms.
- Voice conversion during training disentangles the hint speaker from the target speaker, allowing pronunciation hints from any voice while maintaining the target identity.
- GRAFT reduces target-word phoneme error rates by 22-39% compared to text-only backbones and outperforms existing open-source zero-shot systems on pronunciation accuracy.
- Human listeners ranked GRAFT first in blind studies, judging its rendering of difficult words closest to the reference recordings.
Why It Matters
This advancement addresses a critical limitation in current zero-shot TTS systems: the inability to reliably pronounce rare or ambiguous words without extensive fine-tuning or manual phoneme annotation. By enabling precise, per-word control over pronunciation while preserving speaker identity and naturalness, GRAFT significantly enhances the utility of TTS for professional applications involving specialized terminology, names, or multilingual content.
Technical Details
- Mechanism: GRAFT binds a spoken audio snippet of a target word to its corresponding position in the text prompt, using the model's internal speech tokenizer for encoding.
- Training Strategy: Employs voice conversion during dataset construction to separate the pronunciation source (hint speaker) from the target voice, ensuring the model learns to transfer pronunciation characteristics independently of speaker identity.
- Benchmarking: Evaluated on a five-language objective benchmark, demonstrating superior performance in target-word pronunciation over both phoneme-conditioned and text-conditioned baselines.
- Performance Metrics: Achieved a 22-39% reduction in phoneme error rate for target words compared to the identical text-only backbone model.
- User Evaluation: Won blind listening tests with human raters, confirming high perceptual quality and accuracy in reproducing difficult words.
Industry Insight
- Enhanced Professional Usability: TTS systems can now be deployed in contexts requiring high precision for names, technical jargon, or loanwords without the need for costly manual phoneme labeling or speaker-specific fine-tuning.
- Standardization of Pronunciation Control: The ability to inject pronunciation hints dynamically suggests a shift toward more modular TTS architectures where pronunciation and identity are decoupled, offering greater flexibility for developers.
- Competitive Advantage: Systems integrating GRAFT-like capabilities will likely set a new standard for zero-shot TTS quality, particularly in multilingual and specialized domain applications, pushing competitors to adopt similar fine-grained conditioning techniques.
Disclaimer: The above content is generated by AI and is for reference only.