Experiences with local models for coding
Local execution of small coding models (under 48GB RAM) is viable for agentic tasks but requires a strict multi-stage viability funnel assessing RAM fit, speed, tool calling, functional correctness, context handling, complexity, and code quality. Model performance varies significantly by task type; while Qwen Coder Next 80B achieved functional correctness, it suffered from runtime instability during extended conversations, whereas smaller models like Gemma 4 26B struggled with specific logic imp
Analysis
TL;DR
- Local execution of small coding models (under 48GB RAM) is viable for agentic tasks but requires a strict multi-stage viability funnel assessing RAM fit, speed, tool calling, functional correctness, context handling, complexity, and code quality.
- Model performance varies significantly by task type; while Qwen Coder Next 80B achieved functional correctness, it suffered from runtime instability during extended conversations, whereas smaller models like Gemma 4 26B struggled with specific logic implementations despite initial success.
- There is a notable discrepancy between manual human-in-the-loop evaluations and automated testing setups, with automated tools failing to replicate the nuanced success rates observed during manual interaction, particularly regarding edge cases in output formatting.
- The "text wall of doom" and context degradation issues in longer conversations highlight current limitations in local model context management, suggesting that presence penalties or similar mitigation strategies may be necessary for sustained agentic workflows.
Why It Matters
This analysis provides a realistic benchmark for developers considering self-hosted AI solutions, demonstrating that while local models can perform complex coding tasks, they currently lack the robustness and consistency of cloud-based counterparts for long-running agentic sessions. It highlights the critical importance of evaluating not just raw code generation capability but also stability, context retention, and the reliability of automated versus manual testing methodologies in local deployment scenarios.
Technical Details
- Hardware Environment: Tests were conducted on Apple Silicon machines, specifically an M3 Max with 48GB RAM and an M5 Pro with 64GB RAM, establishing a baseline for local viability constraints.
- Models Evaluated: The study compared several open-source models including Qwen3.6 35B MoE, Gemma 4 31B, Gemma 4 26B, and Qwen Coder Next 80B MoE, utilizing harnesses such as OpenCode and Pi.
- Evaluation Methodology: A three-phase approach was employed: Phase 1 involved manual iterative testing to gauge user experience; Phase 2 introduced an automated evaluation setup to gather quantitative data; Phase 3 focused on integrating the most promising model (Qwen3.6 35B MoE) into daily workflows.
- Task Complexity: Specific tasks included modifying JavaScript/TypeScript frontend components, such as sorting bar charts and calculating cumulative percentages on axes, requiring code search, file modification, and logical reasoning.
- Observed Failures: Key technical failures included runtime crashes in Qwen Coder Next during conversation continuation, functional inaccuracies in Gemma 4 26B regarding axis labeling, and the "text wall of doom" phenomenon indicating context overflow or repetition issues.
Industry Insight
- Organizations should prioritize stability and context management capabilities over raw parameter count when selecting local models for agentic coding, as larger models may offer better initial accuracy but suffer from runtime instability in long sessions.
- Developers must validate automated testing frameworks against manual human-in-the-loop results, as current automation tools may underestimate model performance in nuanced, interactive coding tasks, leading to incorrect viability assessments.
- The integration of local LLMs into daily workflows requires careful tuning of inference parameters (such as presence penalties) to mitigate context degradation, suggesting that out-of-the-box deployments may need significant configuration to achieve production-ready reliability.
Disclaimer: The above content is generated by AI and is for reference only.