Function-Level Execution Feedback for Code Preference Optimization
STEP-KTODER introduces function-level process supervision for code generation by defining steps as module-level functions in decomposed multi-function programs Binary correctness labels are assigned via automatically generated unit tests, combining function-level process supervision with outcome-level feedback on the full program The framework is a code-specific instantiation of stepwise KTO (Kahneman-Tversky Optimization), addressing the lack of a standard notion of "step" in code generation ST
Analysis
TL;DR
- STEP-KTODER introduces function-level process supervision for code generation by defining steps as module-level functions in decomposed multi-function programs
- Binary correctness labels are assigned via automatically generated unit tests, combining function-level process supervision with outcome-level feedback on the full program
- The framework is a code-specific instantiation of stepwise KTO (Kahneman-Tversky Optimization), addressing the lack of a standard notion of "step" in code generation
- STEP-KTODER outperforms outcome-only KTO and DPO baselines across HumanEval(+), MBPP(+), BigCodeBench, and LiveCodeBench benchmarks
- LLM-as-a-judge annotations systematically over-predict function failures, corrupt positive step labels, and degrade downstream preference optimization, making execution-based labels essential
Why It Matters
This work addresses a critical gap in AI code generation by establishing a principled approach to process supervision in programming tasks, where intermediate steps lack the natural structure found in mathematical reasoning. For AI practitioners building code generation systems, it demonstrates that execution-based feedback significantly outperforms LLM-judged annotations, which introduces systematic biases that harm preference optimization.
Technical Details
- STEP-KTODER Framework: Defines "steps" as module-level functions within decomposed multi-function programs, enabling granular process supervision that was previously absent in code generation
- Automatic Unit Test Generation: Binary correctness labels for each function are assigned through automatically generated unit tests, providing objective and reliable execution-based feedback without human annotation
- Hybrid Supervision Signal: Combines function-level process supervision (intermediate correctness) with outcome-level feedback (full program correctness), mirroring the dual signal used in stepwise KTO for mathematical reasoning
- Benchmarks: Evaluated on HumanEval(+), MBPP(+), BigCodeBench, and LiveCodeBench, showing consistent improvements over outcome-only KTO and DPO
- LLM-as-a-Judge Failure Analysis: Systematic evaluation reveals that LLM judges over-predict function failures, which corrupts positive step labels and degrades downstream preference optimization performance
Industry Insight
- Execution-based evaluation should be the default for code preference optimization rather than LLM-as-a-judge approaches, which introduce systematic biases that actively harm model performance
- The function-level decomposition strategy provides a reusable blueprint for applying process supervision to other structured generation tasks beyond code, such as technical documentation or multi-step workflows
- As code generation models approach production readiness, the choice of preference optimization signal (execution vs. judgment) will become a decisive factor in closing the gap between benchmark performance and real-world reliability
Disclaimer: The above content is generated by AI and is for reference only.