Kernel Forge: An Agent Harness for LLM-based Generation and Optimization of CUDA Kernels Kernel Forge：基于LLM生成和优化CUDA内核的代理框架

TL;DR

• Kernel Forge is an open-source agentic harness that uses LLMs to automatically generate and optimize CUDA kernels for PyTorch models without requiring manual code integration.
• It supports vision, diffusion, and LLM workloads and employs Monte Carlo Tree Search (MCTS) to explore multiple optimization paths rather than relying on a single linear refinement chain.
• Evaluated on four PyTorch models across diverse domains, Kernel Forge achieved speedups of up to 2.83× over PyTorch eager mode after only 50 optimization iterations per kernel.
• The system includes a graphical user interface for monitoring progress, inspecting candidate kernels, and debugging failures, enhancing usability and transparency.
• Unlike prior tools that focus narrowly on isolated kernels or LLM-only models, Kernel Forge operates directly on unmodified PyTorch models, enabling broader applicability in real-world deployments.

Why It Matters

This work represents a significant step toward democratizing high-performance GPU kernel optimization by reducing reliance on expert-level CUDA programming. By integrating LLM-based agent systems with automated search strategies like MCTS, Kernel Forge lowers the barrier to entry for performance tuning while maintaining compatibility with existing ML workflows. Its ability to deliver measurable speedups across multiple model types suggests potential for widespread adoption in both research and production environments where latency and cost are critical concerns.

Technical Details

• Architecture: Kernel Forge functions as an end-to-end agentic framework that interfaces directly with unmodified PyTorch models, extracting compute kernels for optimization without requiring source code changes.
• Optimization Strategy: Instead of sequential refinement, it leverages Monte Carlo Tree Search (MCTS) to explore diverse kernel transformation paths simultaneously, increasing the likelihood of discovering high-performing variants.
• Supported Workloads: The tool handles three major categories—vision (e.g., ResNet-50), diffusion (e.g., Stable Diffusion 3.5 Medium), and large language models (e.g., Gemma 4 E2B, Qwen 3.5 35B-A3B)—demonstrating cross-domain versatility.
• Evaluation Setup: Experiments were conducted on an NVIDIA DGX Spark equipped with GB10 GPUs; performance was measured against PyTorch’s default eager execution mode.
• Performance Results: After 50 iterations per kernel, Kernel Forge improved throughput on key operations: adaptive_avgpool2d (+1.52×), group_norm (+1.70×), softmax in Gemma 4 E2B (+2.83×), and softmax in Qwen 3.5 35B-A3B (+1.54×).
• User Interface: A built-in GUI enables users to visualize optimization trajectories, examine generated CUDA code, identify bottlenecks, and troubleshoot failed optimizations interactively.

Industry Insight

The development of Kernel Forge signals a shift toward autonomous software optimization pipelines powered by AI agents, which could drastically reduce engineering overhead in deploying efficient deep learning systems. As organizations increasingly deploy complex multimodal models at scale, tools that automate low-level performance tuning will become essential for maintaining competitiveness in inference speed and energy efficiency. Furthermore, the open-sourcing of such frameworks encourages community-driven improvements and accelerates innovation in compiler-augmented ML stacks, potentially leading to standardized APIs for LLM-assisted kernel synthesis across hardware platforms.

TL;DR

• Kernel Forge 是一个基于大语言模型（LLM）的端到端智能体框架，用于生成和优化 CUDA 内核代码。
• 它支持多种工作负载（视觉、扩散模型和 LLM），并使用蒙特卡洛树搜索（MCTS）探索多条优化路径。
• 该工具提供图形用户界面以监控进度、检查候选内核并调试失败案例。
• 在四个 PyTorch 模型上进行了评估，结果显示在某些操作中性能提升显著，如 ResNet-50 中的 adaptive_avgpool2d 提升了 1.52 倍。
• 与现有工具相比，Kernel Forge 更易于集成到现有的 PyTorch 模型中，并且不需要手动重新整合生成的 CUDA 代码。

为什么值得看

对于 AI 从业者来说，Kernel Forge 提供了一种高效的方法来优化深度学习模型中的关键计算内核，从而减少延迟和成本。通过利用 LLM 的智能体系统，开发者可以大幅降低对高级 GPU 编程技能的需求，同时实现更高的性能优化效果。此外，其开源性质和丰富的功能集使得它成为研究和工业应用中的一个有力工具。

技术解析

• 核心架构: Kernel Forge 接受任何未经修改的 PyTorch 模型作为输入，并通过内置的智能体系统自动生成和优化相应的 CUDA 内核代码。这一过程减少了人工干预的需求，提高了开发效率。
• 优化策略: 使用蒙特卡洛树搜索（MCTS）算法来探索不同的优化路径，而不是传统的线性改进链。这种方法能够更全面地评估各种可能的优化方案，找到最优解的可能性更大。
• 用户体验: 配备了直观的图形用户界面（GUI），允许用户实时监控优化进程、查看不同候选内核的表现以及快速定位和解决出现的问题。这对于提高整个工作流程的透明度和可控性非常重要。
• 实验设置: 在 NVIDIA DGX Spark 平台上配备 GB10 GPU 的环境下对四个典型的 PyTorch 模型进行了测试，包括 ResNet-50、Stable Diffusion 3.5 Medium、Gemma 4 E2B 和 Qwen 3.5 35B-A3B。每个内核仅经过 50 次迭代优化即达到了超越默认 PyTorch eager mode 的效果。
• 性能成果: 具体而言，在几个关键操作上取得了令人印象深刻的加速比：adaptive_avgpool2d (ResNet-50) 达到 $1.52\times$；group_norm (Stable Diffusion 3.5 Medium) 达到 $1.70\times$；softmax (Gemma 4 E2B) 达到 $2.83\times$；softmax (Qwen 3.5 35B-A3B) 达到 $1.54\times$。

行业启示

• 自动化趋势: 随着人工智能技术的不断进步，越来越多的任务正趋向于自动化处理。Kernel Forge 的成功实践表明，在未来可能会有更多类似的工具出现，帮助非专业人士也能轻松完成原本需要专业知识才能做到的事情。
• 跨领域融合: 将机器学习特别是自然语言处理领域的先进技术应用于传统软件工程领域（如高性能计算内核优化），展示了跨学科合作带来的巨大潜力和发展空间。
• 开源生态的重要性: 作为一个开源项目，Kernel Forge 不仅促进了技术创新的速度，还鼓励了社区成员之间的交流与贡献，这对于构建健康的技术生态系统至关重要。