Same Prompt, Same Model Different Results: Put 5 AI Harnesses to the Test
Five AI coding harnesses (Codex, PI agent, Jcode, OpenCode, Reasonix) were tested with the identical prompt and the same underlying model (DeepSeek V4), producing surprisingly different outputs Runtime feedback loops — the ability to run headless browsers, take screenshots, and iterate on observed results — mattered more than raw code generation capability Codex produced the most ambitious build with complex physics and transparency effects; OpenCode produced a static non-interactive result desp
Analysis
TL;DR
- Five AI coding harnesses (Codex, PI agent, Jcode, OpenCode, Reasonix) were tested with the identical prompt and the same underlying model (DeepSeek V4), producing surprisingly different outputs
- Runtime feedback loops — the ability to run headless browsers, take screenshots, and iterate on observed results — mattered more than raw code generation capability
- Codex produced the most ambitious build with complex physics and transparency effects; OpenCode produced a static non-interactive result despite looking polished
- The key metaphor: "If the model is the brain, the harness is the eyes and the hands" — tooling shapes output as much as the model itself
- Simple prompts do not guarantee simple outputs; the same request yielded multi-file apps, single-file HTML blobs, and everything in between
Why It Matters
This experiment directly challenges the common assumption that model choice is the primary lever for AI coding quality. For practitioners, it means that investing in or selecting the right agent framework may yield more significant improvements than upgrading to a newer or larger model. It also highlights that runtime feedback mechanisms — the ability for an agent to observe, debug, and iterate on its own output — are a critical differentiator in real-world coding performance.
Technical Details
- Experiment setup: A single non-technical prompt ("a tiny browser toy — a jelly blob you can poke, grab, and stretch") was given to five harnesses, all running DeepSeek V4, in sandboxed "YOLO" mode with no framework restrictions
- Harnesses tested: Codex (OpenAI, Rust-based), PI agent (minimal TypeScript agent), Jcode, OpenCode, and Reasonix (positioned as native to DeepSeek)
- Codex produced an over-engineered multi-file app with bouncy physics, mouse-following behavior, and transparency effects, requiring dev mode to run
- PI agent generated a simpler but visually cool result; however, the grab interaction broke after one movement, requiring a page reload
- Jcode demonstrated headless browser testing (preferring Firefox, falling back to Chromium), added unrequested gravity and multi-touch mobile support, and left debugging artifacts
- OpenCode bundled everything into a single HTML file, producing a visually appealing but entirely non-interactive static blob
- Reasonix initially outlined to-do items but struggled when context grew too large
- Core technical finding: Harnesses with runtime feedback (screenshot inspection, headless browser iteration) consistently outperformed those that generated code in a single pass
Industry Insight
- Agent framework selection should be a deliberate engineering decision, not an afterthought. Teams should evaluate harnesses based on their feedback loop capabilities — screenshot analysis, self-debugging, and iterative refinement — rather than focusing solely on the underlying model
- Runtime observability is a competitive advantage. The gap between top and bottom performers was not model quality but the ability to observe and correct output. Investing in tooling that enables agents to "see" their results will likely yield higher ROI than model upgrades
- Prompt simplicity does not equal output simplicity. Practitioners should not assume that straightforward prompts will produce straightforward code; harness architecture and iteration behavior will significantly shape the final product structure and complexity
Disclaimer: The above content is generated by AI and is for reference only.