Workload-Driven Optimization for On-Device Real-Time Subtitle Translation
Optimizes on-device English-to-Traditional-Chinese subtitle translation for short inputs with strict latency and privacy constraints. Replaces the standard 151k-token vocabulary with a domain-specific 64k-token tokenizer to reduce decode-time projection costs after GGUF quantization. Achieves a 1.63x speedup on Apple M2 Metal hardware compared to the baseline vocabulary size. Demonstrates superior quality against Google Translate, securing a 59.2% win rate in pairwise evaluations by GPT-4o on sh
Analysis
TL;DR
- Optimizes on-device English-to-Traditional-Chinese subtitle translation for short inputs with strict latency and privacy constraints.
- Replaces the standard 151k-token vocabulary with a domain-specific 64k-token tokenizer to reduce decode-time projection costs after GGUF quantization.
- Achieves a 1.63x speedup on Apple M2 Metal hardware compared to the baseline vocabulary size.
- Demonstrates superior quality against Google Translate, securing a 59.2% win rate in pairwise evaluations by GPT-4o on short cues.
Why It Matters
This research highlights critical optimization strategies for deploying Large Language Models on edge devices where computational resources are limited. By focusing on workload-specific constraints like short sequences and batch-size-one inference, it offers a practical blueprint for improving real-time performance without sacrificing accuracy. This approach is particularly relevant for developers building privacy-centric, low-latency applications such as live captioning tools.
Technical Details
- Model Architecture & Base: Starts with the LMT-60-0.6B model, utilizing GGUF quantization to reduce the relative cost of Transformer blocks, thereby making vocabulary projection the dominant bottleneck during decoding.
- Vocabulary Optimization: Migrates from a 151k-token vocabulary to a specialized 64k-token subtitle-domain tokenizer. This involves embedding space migration, embedding calibration, and full supervised fine-tuning to adapt the model to the new token distribution.
- Evaluation Metrics: Tested on a fixed 500-example subset of the OpenSubtitles2024 test set. Quality was assessed via GPT-4o pairwise judging, while performance was measured using Apple M2 Metal profiling for latency.
- Performance Results: The optimized model, named LocalSubs, showed a 1.63x speedup over the baseline. Quality-wise, it outperformed Google Translate with a 59.2% tie-excluded win rate, though performance degraded as input cue length increased.
Industry Insight
- Edge-Specific Tokenizers: For on-device deployments, reducing vocabulary size through domain-specific tokenization can yield significant latency improvements, especially when combined with quantization techniques that shift the computational bottleneck to the output layer.
- Short-Context Optimization: Standard LLM serving optimizations often target long-context or high-throughput scenarios. Developers should prioritize workload-driven optimizations tailored to specific use cases, such as short-input real-time translation, to maximize efficiency on consumer hardware.
- Privacy-Preserving Alternatives: The success of local inference models like LocalSubs suggests a growing market for on-device solutions that offer competitive quality to cloud-based services while ensuring user data privacy, a key selling point for enterprise and consumer applications.
Disclaimer: The above content is generated by AI and is for reference only.