Show HN: Wayfinder – A reference implementation for evaluating AI applications
Wayfinder is a reference implementation for building and understanding AI evaluation systems, evolving a single AI-powered flight search application through progressively sophisticated evaluation layers It covers five evaluation paradigms: rule-based, human, LLM-as-a-Judge, online evaluation, and experiment comparison — each building on the same application The project is designed alongside an article series on AI Engineering Fundamentals, providing runnable code for each evaluation concept intr
Analysis
TL;DR
- Wayfinder is a reference implementation for building and understanding AI evaluation systems, evolving a single AI-powered flight search application through progressively sophisticated evaluation layers
- It covers five evaluation paradigms: rule-based, human, LLM-as-a-Judge, online evaluation, and experiment comparison — each building on the same application
- The project is designed alongside an article series on AI Engineering Fundamentals, providing runnable code for each evaluation concept introduced
- It integrates with LangSmith for interaction tracing, evaluation scoring, and experiment comparison, enabling production-grade evaluation workflows
- The system addresses core AI engineering challenges: measuring improvement, detecting regressions, evaluating subjective qualities, scaling beyond manual review, and confidently shipping to production
Why It Matters
AI evaluation remains one of the most critical yet underdeveloped areas in AI engineering, and Wayfinder provides a practical, end-to-end reference that practitioners can study and adapt. It demonstrates how evaluation systems should evolve alongside AI applications — from simple rule-based checks to sophisticated online and comparative evaluation — giving teams a concrete roadmap rather than abstract theory.
Technical Details
- Architecture: Built around a single AI-powered flight search application ("Wayfinder") that serves as the consistent subject across all evaluation stages, allowing direct comparison of how different evaluation methods assess the same system
- Evaluation paradigms implemented: Rule-based evaluation (deterministic scoring), human evaluation (manual review workflows), LLM-as-a-Judge (reference-free automated scoring), online evaluation (real-time interaction tracing with LangSmith), and experiment comparison (repeated runs with statistical rigor)
- Tech stack: Python 3.12+, uv for dependency management, LangSmith for tracing and evaluation integration, OpenAI models for AI response generation and LLM-as-a-Judge scoring
- Project structure: Source code in
src/wayfinder/with agent, evaluators, models, services, and tools modules; examples organized by evaluation type inexamples/with dedicated directories for each paradigm - Reproducible milestones: Each GitHub release (v0.1.0 through v0.6.0) captures a specific evaluation stage, enabling version-controlled comparison between baseline and candidate implementations
Industry Insight
- Teams building production AI applications should adopt an iterative evaluation strategy that matures alongside the application — starting with rule-based checks and progressively incorporating human and LLM-based evaluation rather than attempting to implement all methods simultaneously
- The experiment comparison pattern (running the same dataset repeatedly across versions with statistical rigor) is essential for A/B testing AI systems and should be treated as a production requirement, not a nice-to-have
- Integrating evaluation into existing observability tooling (like LangSmith) from the start — rather than bolting it on later — dramatically reduces the friction of adopting online evaluation and continuous monitoring in production environments
Disclaimer: The above content is generated by AI and is for reference only.