AnchorSIPS: A Synthetic Dataset and Evaluation Resource for Evidence-Supported Psychosis-Risk Symptom Measurement
AnchorSIPS is a synthetic dataset of 10,000 structured psychosis-risk interviews modeled on the Mini-SIPS clinical interview, designed to bypass privacy and consent barriers that limit real clinical data sharing. The dataset uses a plan-then-realize generation pipeline: a hidden case sheet defines the clinical state, a deterministic planner fixes interview structure, and an LLM generates only patient utterances under validation and bounded repair to avoid inter-turn inconsistencies. Every interm
Analysis
TL;DR
- AnchorSIPS is a synthetic dataset of 10,000 structured psychosis-risk interviews modeled on the Mini-SIPS clinical interview, designed to bypass privacy and consent barriers that limit real clinical data sharing.
- The dataset uses a plan-then-realize generation pipeline: a hidden case sheet defines the clinical state, a deterministic planner fixes interview structure, and an LLM generates only patient utterances under validation and bounded repair to avoid inter-turn inconsistencies.
- Every intermediate decision (symptom endorsements, delusion/hallucination classifications, frank-psychosis exclusion, and final APS diagnosis) is anchored to supporting transcript turns, enabling evidence-grounded evaluation.
- Seven LLM baselines recovered coarse diagnostic decisions but failed to extract follow-up evidence or cite supporting transcript turns, revealing that final-label accuracy overstates true interview competence.
- AnchorSIPS is positioned as a research resource for evidence extraction, transcript-grounded measurement, and studying model uncertainty under partial disclosure in clinical NLP.
Why It Matters
This work addresses a critical bottleneck in AI-driven mental health research: the scarcity of shareable clinical interview data due to privacy, governance, and consent constraints. By providing a rigorously structured synthetic dataset with transcript-anchored labels, AnchorSIPS enables reproducible evaluation of evidence extraction and diagnostic reasoning—capabilities essential for any AI system intended to support psychosis-risk assessment in real clinical settings.
Technical Details
- Dataset scale and structure: 10,000 synthetic interviews, each containing patient history, 24 symptom questions, follow-up evidence for affirmed items, symptom-class decisions (delusion-like, hallucination-like, disorganized communication), frank-psychosis exclusion checks, and a final Attenuated Psychosis Syndrome (APS) diagnosis.
- Generation pipeline: A three-stage plan-then-realize architecture: (1) a hidden case sheet specifies the patient's clinical state, (2) a deterministic planner fixes the interview structure and all diagnostic labels, and (3) an LLM generates only patient utterances under validation and bounded repair, preventing the inter-turn inconsistencies common in multi-turn LLM dialogue.
- Transcript grounding: Every intermediate decision is linked to specific supporting transcript turns, making the dataset suitable for evidence extraction and citation-based evaluation rather than simple label prediction.
- Evaluation: Seven LLM baselines were tested; models performed adequately on coarse diagnostic decisions but significantly underperformed on extracting follow-up details and citing supporting evidence, demonstrating a gap between label-level and evidence-level competence.
- Target research directions: Evidence extraction, transcript-grounded clinical measurement, and uncertainty quantification under partial disclosure.
Industry Insight
- Synthetic clinical datasets with rigorous grounding structures will become increasingly important as privacy regulations tighten; AnchorSIPS demonstrates a viable blueprint for plan-then-realize generation that prioritizes structural fidelity over raw fluency.
- The finding that label-level performance overstates interview competence is a cautionary signal for the mental health AI community: evaluation frameworks must demand evidence citation and intermediate reasoning, not just final diagnostic accuracy.
- Researchers building clinical NLP systems should treat AnchorSIPS as a benchmark for evidence extraction and consider adopting its transcript-anchoring methodology to ensure their models can justify decisions with supporting text, a requirement for any system approaching real-world clinical deployment.
Disclaimer: The above content is generated by AI and is for reference only.