Accelerating LLM Inference via Vector Index Based Output Embeddings
Large output embedding matrices in LLMs create a memory bandwidth bottleneck during autoregressive decoding, particularly for compact models with large multilingual vocabularies The authors reformulate the output projection and top-k token selection as a maximum inner product search problem, replacing dense vocabulary projection with an HNSW-based vector index The method retrieves only a small candidate set of high-scoring tokens and integrates into existing pipelines by scattering retrieved log
Analysis
TL;DR
- Large output embedding matrices in LLMs create a memory bandwidth bottleneck during autoregressive decoding, particularly for compact models with large multilingual vocabularies
- The authors reformulate the output projection and top-k token selection as a maximum inner product search problem, replacing dense vocabulary projection with an HNSW-based vector index
- The method retrieves only a small candidate set of high-scoring tokens and integrates into existing pipelines by scattering retrieved logits into a sparse full-vocabulary tensor
- On CPU inference with Gemma 3, Llama 3.2, and Qwen 3 models, the approach improves batch-size-one decoding throughput by up to 82% (Gemma 3 270M) while preserving generation quality on AlpacaEval
- Approximate retrieval via vector indexes is demonstrated as a practical alternative to dense output projections in latency-sensitive small-batch decoding scenarios
Why It Matters
This work addresses a critical bottleneck in LLM inference that becomes increasingly relevant as models deploy on edge devices and CPU-only environments where memory bandwidth is a limiting factor. By reframing token generation as an approximate nearest neighbor search problem, it opens a new optimization pathway that could influence how inference engines are designed for resource-constrained deployments.
Technical Details
- The core innovation replaces the standard dense output projection (matrix multiplication against the full vocabulary embedding matrix) with an HNSW (Hierarchical Navigable Small World) vector index built over token embeddings
- The output head performs maximum inner product search to retrieve a small candidate set of high-scoring tokens, then scatters the retrieved logits back into a sparse full-vocabulary tensor for standard top-k selection
- Evaluated on CPU inference across three model families: Gemma 3 (270M parameter variant), Llama 3.2, and Qwen 3, demonstrating the approach generalizes across architectures
- Generation quality was validated using AlpacaEval, confirming that the approximation introduced by vector index retrieval does not meaningfully degrade output quality
- The method is designed for seamless integration into existing decoding pipelines without requiring architectural modifications to the base model
Industry Insight
- As the industry pushes toward smaller, more efficient models for edge and on-device deployment, memory-bandwidth optimization techniques like this will become increasingly valuable compared to compute-centric optimizations
- The HNSW-based approach could be adopted by inference frameworks (vLLM, TensorRT-LLM, etc.) as a configurable optimization for CPU inference paths, particularly for multilingual models with vocabularies exceeding 100K tokens
- This work suggests a broader trend: treating LLM output projection as a retrieval problem rather than a linear algebra operation, which could inspire similar approximate methods for other expensive operations in the decoding pipeline
Disclaimer: The above content is generated by AI and is for reference only.