Conversation With An Honest Agent
The author built P.U.C.K., an uncertainty-aware conversational agent that quantifies LLM confidence using both logprob averaging and self-reported confidence levels (low/mid/high buckets) The system combines a local Qwen2.5–7B-Instruct model served via llama.cpp with a Wikipedia-based knowledge base and external tool calls (weather, NHL results API) to ground responses A fact-checking layer cross-references model outputs against Wikipedia sources, triggering when self-reported confidence is low
Analysis
TL;DR
- The author built P.U.C.K., an uncertainty-aware conversational agent that quantifies LLM confidence using both logprob averaging and self-reported confidence levels (low/mid/high buckets)
- The system combines a local Qwen2.5–7B-Instruct model served via llama.cpp with a Wikipedia-based knowledge base and external tool calls (weather, NHL results API) to ground responses
- A fact-checking layer cross-references model outputs against Wikipedia sources, triggering when self-reported confidence is low
- The agent successfully retrieved factual information (e.g., Wayne Gretzky's NHL points record) but also exposed hallucination risks, including confidently stating incorrect information about the 2025/2026 Stanley Cup winner and failing to recognize a fabricated anecdote about a Scottish physicist sharing a name with Bobby Orr
- Logprob-based confidence and self-reported confidence did not always align, revealing that self-reporting is unreliable but useful as a diagnostic for consistency
Why It Matters
This project demonstrates a practical approach to uncertainty quantification in LLMs that practitioners can adapt for domain-specific applications, highlighting the gap between a model's self-perceived confidence and its actual factual accuracy. It also underscores the importance of external fact-checking layers and tool-augmented retrieval when deploying AI systems in areas where hallucination can have real consequences.
Technical Details
- Model & Infrastructure: A local llama.cpp server serves Qwen2.5–7B-Instruct-Q4_K_M, chosen for reproducibility and privacy; frontier model providers were excluded because they do not return logprobs
- Uncertainty Measurement: Two signals are combined—average logprob over generated tokens (measuring generation "flow") and self-reported confidence in low/mid/high buckets (measuring introspective confidence); these are displayed via a "b.s.-meter" gauge
- Knowledge Architecture: A user-selectable Wikipedia knowledge base provides ground truth, supplemented by two external tools (weather API, NHL results API); no direct web search is provided to test behavior when the model "doesn't know that it doesn't know"
- Fact-Checking Layer: Triggered on low self-reported confidence, it fetches relevant Wikipedia excerpts and compares them against the model's claim, returning one of three verdicts: supported, contradicted, or inconclusive
- UI Design: The P.U.C.K. interface separates chat and diagnostics into two panels, color-codes information sources (KB, training data, tool calls), and displays confidence gauges alongside source attribution for each response
Industry Insight
- Uncertainty quantification should not rely on a single signal; combining logprob-based metrics with self-reported confidence—and cross-validating both against external fact-checking—produces more trustworthy deployments than any one approach alone
- The Bobby Orr anecdote test reveals that even small-model agents will confabulate plausible-sounding responses to fabricated premises, making external verification layers essential for any production system handling user claims
- The observation that the model's training data cutoff appears to be 2022 for NHL content (despite the article being written in August 2026) highlights the risk of stale knowledge in sports and time-sensitive domains, reinforcing the need for real-time tool-augmented retrieval in agent architectures
Disclaimer: The above content is generated by AI and is for reference only.