Beyond Liars' Bench: The Impact of Lie Typology, Depth, and Sparsity on Deception Detection in LLMs
Deception detection in LLMs is highly representation-dependent, with performance varying significantly based on the typology of lies used in training data. Augmenting standard benchmarks with diverse deception types (fabrication, omission, exaggeration) reveals that optimal probe depth and expressivity are dataset-specific rather than universal. Sparse autoencoder features do not outperform dense hidden states for this task, and more complex probes offer only selective gains over simple linear b
Analysis
TL;DR
- Deception detection in LLMs is highly representation-dependent, with performance varying significantly based on the typology of lies used in training data.
- Augmenting standard benchmarks with diverse deception types (fabrication, omission, exaggeration) reveals that optimal probe depth and expressivity are dataset-specific rather than universal.
- Sparse autoencoder features do not outperform dense hidden states for this task, and more complex probes offer only selective gains over simple linear baselines.
Why It Matters
This research challenges the assumption that a single, robust detector can be trained for all forms of LLM deception, highlighting a critical gap in current safety evaluation methodologies. For AI practitioners, it underscores the necessity of diversifying training data to avoid brittle models that fail in out-of-domain scenarios. Researchers must reconsider how they design and evaluate interpretability tools, moving beyond generic benchmarks to address specific deception mechanisms.
Technical Details
- The study systematically evaluates seven different probe types to assess their effectiveness in detecting deceptive outputs from large language models.
- Researchers augmented standard benchmark datasets with a supplementary collection containing diverse lie typologies, specifically including fabrication, omission, and exaggeration examples.
- Key variables analyzed include representation depth, probe expressivity, and the use of sparse feature representations versus dense hidden states.
- Experimental results indicate that the optimal layer depth for detection varies heavily depending on the specific dataset and lie type, contradicting the idea of a fixed "best" layer across all contexts.
Industry Insight
- Developers building safety filters or alignment layers should prioritize diverse, multi-typology training data rather than relying on homogeneous datasets to improve generalization.
- The limited gain from complex probes suggests that simpler, linear classifiers may be sufficient for many deception detection tasks, reducing computational overhead and complexity.
- Future work in AI safety must focus on understanding the specific representational signatures of different deception types to create more robust and transferable detection mechanisms.
Disclaimer: The above content is generated by AI and is for reference only.