From CUDA to MLX: How K-Search Brings Decades of Kernel Expertise to Apple Silicon
The paper proposes a structured CUDA-to-MLX translation layer to transfer kernel optimization expertise from the mature CUDA ecosystem to newer hardware platforms like Apple Silicon, using the K-Search evolutionary framework. This approach enables automatic adaptation of hand-tuned CUDA kernels (e.g., attention, SSM) into high-performance MLX kernels without rebuilding from scratch, achieving near-expert performance levels. Results show up to 0.97x speedup over native MLX Attention kernels and u
Analysis
TL;DR
- The paper proposes a structured CUDA-to-MLX translation layer to transfer kernel optimization expertise from the mature CUDA ecosystem to newer hardware platforms like Apple Silicon, using the K-Search evolutionary framework.
- This approach enables automatic adaptation of hand-tuned CUDA kernels (e.g., attention, SSM) into high-performance MLX kernels without rebuilding from scratch, achieving near-expert performance levels.
- Results show up to 0.97x speedup over native MLX Attention kernels and up to 20x prefill speedup for Mamba SSM compared to community mlx-lm implementations.
- The method is generalizable beyond MLX to any architecture where CUDA knowledge can be meaningfully translated rather than copied instruction-for-instruction.
- K-Search uses an LLM-driven iterative search with a persistent world model tree to explore optimization paths, guided by hardware-specific specs that prevent invalid code generation.
Why It Matters
This work addresses a critical bottleneck in AI deployment: the fragmentation of hardware ecosystems and the loss of accumulated low-level optimization expertise when moving between platforms. By enabling automated transfer of GPU kernel knowledge, it reduces the engineering burden required to achieve peak performance on emerging architectures like Apple Silicon, democratizing access to high-efficiency computing without requiring years of specialized tuning experience per platform.
Technical Details
- Built upon K-Search, an evolutionary kernel optimization framework that uses LLMs to iteratively propose, compile, and benchmark GPU kernel optimizations based on hardware specifications.
- Introduced a novel structured translation layer that maps CUDA optimization patterns to MLX-native strategies rather than direct instruction copying, preserving semantic intent while adapting to architectural constraints.
- Utilized a single LLM (Gemini 3.5 Pro Preview) functioning as both reasoning engine ("GPU kernel performance engineer") and code generator within a decision-tree world model that tracks partial optimization plans with scores for overall rating, confidence, and hardware impact metrics.
- Applied domain-specific "Spec" documents encoding hardware rules, optimization patterns, and mathematical constraints to ensure generated kernels are valid and efficient, preventing hallucinations of unsupported primitives.
- Evaluated on two key workloads: Attention kernels (achieving 0.97x relative to native MLX) and Mamba State Space Model kernels (showing up to 20x prefill speedup over mlx-lm baseline).
Industry Insight
As hardware diversity accelerates—with custom AI chips from multiple vendors and frameworks like MLX gaining traction—this approach offers a scalable solution to avoid reinventing optimization wheels for each new platform. Organizations investing in non-NVIDIA hardware should consider adopting similar knowledge-transfer methodologies to rapidly close performance gaps without massive R&D投入, while open-source communities could build shared specification libraries to enable cross-platform kernel portability. The success of this translation strategy suggests future AI toolchains may increasingly include automated legacy-kernel migration layers as standard components.
Disclaimer: The above content is generated by AI and is for reference only.