Running OpenWiki for Real: Why the Model You Pick Decides Everything
Model selection for agentic documentation tools is critical for completion reliability, outweighing prose quality in headless, multi-turn workflows. Claude Sonnet failed repeatedly due to malformed tool calls in a strict schema environment, whereas Claude Opus 4.8 successfully generated a complete, accurate documentation set. True cost analysis must include completion rates and token waste from failed attempts, making reliable models potentially more economical than cheaper alternatives that cra
Analysis
TL;DR
- Model selection for agentic documentation tools is critical for completion reliability, outweighing prose quality in headless, multi-turn workflows.
- Claude Sonnet failed repeatedly due to malformed tool calls in a strict schema environment, whereas Claude Opus 4.8 successfully generated a complete, accurate documentation set.
- True cost analysis must include completion rates and token waste from failed attempts, making reliable models potentially more economical than cheaper alternatives that crash.
- Optimizing input scope by excluding build artifacts and dependencies significantly reduces token consumption and improves documentation relevance.
Why It Matters
This case study highlights a fundamental shift in evaluating Large Language Models for autonomous agents: reliability in structured tool execution is now a primary metric alongside generative quality. For AI practitioners, it demonstrates that prompt engineering cannot compensate for inherent model limitations in schema adherence during long-horizon tasks. It serves as a practical warning that "cheaper" models may incur higher total costs due to low success rates in automated, unattended workflows.
Technical Details
- Tooling: OpenWiki, a LangChain-based agentic documentation generator applied to a Spring Boot 3.1/Angular 20 monorepo.
- Failure Mode: Claude Sonnet produced intermittent malformed tool calls (missing required fields) that caused immediate crashes in the headless
--printmode, as the framework lacked recovery mechanisms for schema violations. - Success Case: Claude Opus 4.8 demonstrated strict adherence to tool schemas across dozens of turns, successfully executing sub-agent dispatches, file reads, and page writes without interruption.
- Cost Dynamics: The article emphasizes that token usage scales with repository depth and file count, noting that wasted tokens from crashed runs contribute to the final bill, necessitating a calculation of cost-per-completed-document rather than cost-per-token.
Industry Insight
- Prioritize Schema Adherence: When selecting models for agentic workflows involving strict JSON or tool schemas, benchmark for reliability and consistency over creative writing capabilities.
- Rethink Pricing Models: Implement cost-tracking metrics that account for failure rates; a higher-cost model with near-100% completion may offer better ROI than a low-cost model with frequent errors.
- Input Sanitization: Proactively filter repositories to exclude non-source artifacts (e.g.,
node_modules, build outputs) before initiating agentic processes to minimize token overhead and improve signal-to-noise ratio.
Disclaimer: The above content is generated by AI and is for reference only.