Silicon Sampling via Cross-Survey Transfer
Introduces "cross-survey transfer," a rigorous evaluation framework where LLMs predict answers to unseen survey questions based on a respondent's answers to other questions, moving beyond simple distributional matching. Zero-shot LLMs (27B-120B parameters) achieved 52% accuracy on genuinely unseen items, performing within 6 percentage points of a supervised random forest baseline trained on the same population. A stable hierarchy of construct predictability was identified, ranging from 67% accur
Analysis
TL;DR
- Introduces "cross-survey transfer," a rigorous evaluation framework where LLMs predict answers to unseen survey questions based on a respondent's answers to other questions, moving beyond simple distributional matching.
- Zero-shot LLMs (27B-120B parameters) achieved 52% accuracy on genuinely unseen items, performing within 6 percentage points of a supervised random forest baseline trained on the same population.
- A stable hierarchy of construct predictability was identified, ranging from 67% accuracy for partisan attitudes down to 23% for sovereignty issues.
- The study reveals that common LLM limitations like variance collapse and safety alignment effects are more nuanced, with variance collapse also impacting supervised models and alignment varying significantly across model families.
Why It Matters
This research provides a critical methodological advancement for "silicon sampling," offering a more robust way to evaluate whether LLMs can truly simulate individual human respondents rather than just mimicking aggregate statistical patterns. For AI practitioners and social scientists, it establishes concrete performance benchmarks and highlights specific cognitive domains where current models struggle, guiding future improvements in LLM alignment and reasoning capabilities for social simulation tasks.
Technical Details
- Evaluation Framework: Utilizes the Taiwan Election and Democratization Study (TEDS) 2024 dataset to test "cross-survey transfer," requiring models to infer responses to entirely different question sets from known answers within the same survey instrument.
- Models Tested: Evaluated three open-weight LLMs ranging from 27B to 120B parameters against supervised machine learning baselines, specifically a random forest trained on same-population data.
- Performance Metrics: Measured individual-level prediction accuracy, finding zero-shot LLMs at 52% overall accuracy, compared to the supervised baseline's higher performance, while noting a significant gap in predicting complex constructs like sovereignty (23%) versus partisan attitudes (67%).
- Analysis of Limitations: Investigated the impact of variance collapse and safety alignment, discovering that variance collapse is not exclusive to LLMs but also affects supervised models, and that alignment effects are highly dependent on the specific model family used.
Industry Insight
- Researchers should prioritize individual-level prediction metrics over aggregate distributional comparisons when validating LLM-based social simulations to avoid conflating pattern matching with genuine respondent simulation.
- The significant drop in accuracy for sovereignty-related questions suggests that LLMs currently lack the nuanced contextual understanding required for high-stakes political or sensitive social topics, indicating a need for specialized fine-tuning or domain-specific adapters.
- The finding that variance collapse affects supervised models as well implies that this is a broader statistical phenomenon in predictive modeling rather than solely an LLM artifact, urging caution in interpreting LLM limitations without proper comparative baselines.
Disclaimer: The above content is generated by AI and is for reference only.