DataKernelBench: Can LLMs Optimize Database Queries on GPUs?
DataKernelBench is introduced as a new benchmark that translates SQL queries into PyTorch TorchPlan programs to evaluate LLMs optimizing database-style GPU kernels, filling a gap left by existing benchmarks focused on ML operators. The strongest full-query CUDA configuration achieves a 2.11× speedup over baseline at full pass rate across ten models on TPC-H SF10 with an H100 GPU. Higher-performing LLM implementations commonly leverage kernel fusion and execution-strategy changes, with stronger m
Analysis
TL;DR
- DataKernelBench is introduced as a new benchmark that translates SQL queries into PyTorch TorchPlan programs to evaluate LLMs optimizing database-style GPU kernels, filling a gap left by existing benchmarks focused on ML operators.
- The strongest full-query CUDA configuration achieves a 2.11× speedup over baseline at full pass rate across ten models on TPC-H SF10 with an H100 GPU.
- Higher-performing LLM implementations commonly leverage kernel fusion and execution-strategy changes, with stronger models benefiting most from full-query specialization.
- Workload context proves more impactful than hardware context in determining optimization quality.
- Scaling to larger datasets via Dask-cuDF integration on four H100 GPUs (TPC-H SF100) achieves a 2.54× speedup through on-demand partition loading.
Why It Matters
This work addresses a critical gap in LLM evaluation by shifting focus from ML-centric kernel benchmarks to irregular, data-movement-heavy database operators that dominate real-world GPU-accelerated query workloads. For AI practitioners and database researchers, it demonstrates that LLMs can meaningfully optimize production-grade query kernels, opening the door to automated performance tuning that previously required deep domain expertise.
Technical Details
- Benchmark pipeline: SQL queries are translated into validated PyTorch TorchPlan programs, which LLMs then optimize as either core tensor-bounded snippets or full queries, targeting CUDA or Triton implementations via execution-guided repair.
- Evaluation setup: Tested across ten proprietary and open-weight LLMs on TPC-H SF10 using a single H100 GPU, with baseline comparisons at full pass rate.
- Key optimization patterns: Successful implementations frequently employ kernel fusion and execution-strategy modifications rather than naive code generation.
- Large-scale extension: TorchPlan is extended with Dask-cuDF to handle datasets exceeding GPU memory, enabling on-demand partition loading across four H100 GPUs for TPC-H SF100.
- Cross-domain scope: The paper spans cs.CL, cs.AI, cs.DB, cs.LG, and cs.PL, reflecting its interdisciplinary nature at the intersection of language models, databases, and systems programming.
Industry Insight
- Automated LLM-driven kernel optimization could significantly reduce the engineering bottleneck in GPU-accelerated database systems, where hand-written kernels currently dominate performance-critical paths.
- The finding that workload context matters more than hardware context suggests that benchmarking and training LLMs for code generation should prioritize diverse, realistic query patterns over hardware-specific tuning.
- The successful integration of Dask-cuDF for out-of-GPU-memory workloads signals a practical path toward deploying LLM-optimized kernels in production environments with large-scale data, making this approach viable beyond controlled benchmark settings.
Disclaimer: The above content is generated by AI and is for reference only.