Cite or Decline: A Strict Course-Grounded Chatbot for STEM Lecture Videos
A semester-long deployment of VideoPoints chatbot showed 70.5% of 833 student messages included timestamped citations, with zero cross-course boundary violations Chapter-summary-guided transcript ranking improved correct-lecture retrieval by 6.3 percentage points over dense-only retrieval on the EduVidQA benchmark Students primarily used the chatbot for quick lookups and exam review, with practice-question generation identified as the strongest unmet need The chatbot's strict "cite or decline" d
Analysis
TL;DR
- A semester-long deployment of VideoPoints chatbot showed 70.5% of 833 student messages included timestamped citations, with zero cross-course boundary violations
- Chapter-summary-guided transcript ranking improved correct-lecture retrieval by 6.3 percentage points over dense-only retrieval on the EduVidQA benchmark
- Students primarily used the chatbot for quick lookups and exam review, with practice-question generation identified as the strongest unmet need
- The chatbot's strict "cite or decline" design—refusing to answer when no lecture evidence matched—proved effective for maintaining answer reliability
- Successful deployment hinges on three factors: course isolation, supported citations, and alignment with students' actual study practices
Why It Matters
This work demonstrates that retrieval-augmented chatbots for educational video content can achieve high reliability when strict grounding constraints are enforced, directly addressing the hallucination problem that plagues general-purpose educational AI tools. For AI practitioners building domain-specific assistants, it provides empirical evidence that course isolation and citation requirements are not just UX features but critical correctness mechanisms.
Technical Details
- The system uses a retrieval-augmented generation (RAG) pipeline that restricts retrieval to the student's active course only, preventing cross-contamination between course materials
- Chapter summaries are leveraged to guide transcript ranking, improving retrieval accuracy by 6.3 percentage points over dense-vector-only approaches on the EduVidQA real-world test split
- The chatbot implements a strict "cite or decline" policy: when no lecture evidence matches a query, it refuses to answer rather than generating a potentially incorrect response
- Evaluated across 833 real student messages during a full semester deployment on the VideoPoints platform, with clickable timestamped citations as the primary output format
Industry Insight
- Strict grounding constraints (course isolation + citation requirements) should be treated as core architectural decisions, not optional features, when building educational AI systems—hallucination mitigation requires systemic design, not just better models
- The gap between student usage patterns (quick lookups, exam review) and unmet needs (practice-question generation) suggests that deployment feedback loops are essential for identifying feature priorities that benchmark evaluations alone cannot reveal
- Multimodal lecture-video QA remains an underexplored area with clear real-world demand; the 6.3-point improvement from a relatively simple ranking enhancement indicates that hybrid retrieval strategies (summary-guided + dense) offer strong ROI for educational AI investments
Disclaimer: The above content is generated by AI and is for reference only.