Teaching OpenWiki to Draw: From “No Diagrams” to an Upstream PR
OpenWiki, a LangChain-based tool for generating codebase documentation, initially produces text-only outputs lacking visual aids like diagrams. By leveraging Mermaid.js syntax within markdown blocks, the author successfully instructed an LLM to generate accurate sequence and entity-relationship diagrams based on source code analysis. A critical technical hurdle emerged where auto-generated text containing special characters (e.g., semicolons) broke Mermaid rendering, highlighting the need for va
Analysis
TL;DR
- OpenWiki, a LangChain-based tool for generating codebase documentation, initially produces text-only outputs lacking visual aids like diagrams.
- By leveraging Mermaid.js syntax within markdown blocks, the author successfully instructed an LLM to generate accurate sequence and entity-relationship diagrams based on source code analysis.
- A critical technical hurdle emerged where auto-generated text containing special characters (e.g., semicolons) broke Mermaid rendering, highlighting the need for validation passes in AI-generated code.
- The author identified that OpenWiki’s conservative "update" mode risks orphaning manually added diagrams, necessitating changes to the system prompt to maintain synchronization.
- Instead of forking, the contributor submitted an upstream Pull Request implementing an opt-in environment variable (
OPENWIKI_DIAGRAMS) to enable diagram generation safely.
Why It Matters
This case study demonstrates the practical limitations of current agentic documentation tools, specifically their tendency to overlook visual context which is crucial for complex architectural understanding. It highlights the importance of "source-grounded" AI agents that can read actual code logic rather than relying on plausible hallucinations, providing a blueprint for integrating visual documentation into automated workflows. Furthermore, it underscores the value of contributing fixes back to open-source projects to solve systemic gaps that affect the broader community of developers using these tools.
Technical Details
- Tooling & Format: The solution utilizes Mermaid.js, chosen for its native support in GitHub/GitLab markdown and its ability to remain version-controlled alongside code. The diagrams are embedded as fenced markdown blocks.
- Model Execution: The author used the Opus model to execute a targeted prompt instructing the agent to read specific source files (e.g., auth interceptors) and generate accurate sequence and ER diagrams, verifying details like token lifetimes and soft links.
- Syntax Validation Issue: A parsing error occurred because the LLM included semicolons in message labels (e.g.,
store new token; queue concurrent 401s), which Mermaid interprets as statement separators, breaking the render. This suggests the need for a linting step in CI pipelines for auto-generated diagrams. - System Prompt Engineering: Analysis of OpenWiki’s source revealed that the system prompt runs on every invocation (init, update, chat). The "update" mode is designed to be conservative to preserve structure, which inadvertently causes newly added diagrams to be ignored or desynchronized during subsequent runs.
- Upstream Contribution Design: The proposed fix involves an environment variable
OPENWIKI_DIAGRAMS=mermaidto opt-in to diagram generation. This approach ensures backward compatibility, prevents unexpected token costs, and modifies the system prompt to enforce diagram synchronization and label safety rules during updates.
Industry Insight
- Automated Documentation Requires Visuals: For complex systems, text-only documentation is insufficient. AI agents should be explicitly prompted or configured to generate visual representations (like UML or flowcharts) to aid rapid comprehension for new team members.
- Validation is Non-Negotiable for Generated Code/Syntax: Even when the semantic intent of AI-generated content is correct, syntactic errors (such as unescaped characters in markup languages) can cause total failure. Implementing automated validation or linting steps for AI outputs is essential for production reliability.
- Design for Maintainability in Agentic Workflows: Features added via one-off prompts often fail in iterative environments. Systematic solutions, such as modifying system prompts or using configuration flags, ensure that enhancements persist and stay synchronized with the evolving codebase, reducing technical debt in documentation.
Disclaimer: The above content is generated by AI and is for reference only.