Speech Signals Complement LLMs for Predicting Interpersonal Attraction in Speed Dating 语音信号补充大语言模型预测速配中的人际吸引力

TL;DR

• Large language models (LLMs) can predict interpersonal attraction from conversation transcripts, but the added value of speech signals remains unclear.
• Combining transcript-only LLM predictions with supervised speech predictors improves pairwise ranking accuracy across all evaluated conditions in Japanese speed-dating conversations.
• Gains in per-participant Pearson correlation vary by conversation round and rating direction, with no significant improvement after correction, suggesting conditional complementarity rather than universal benefit.
• Speech retains predictive value even when LLMs perform well on transcripts, indicating that the key question is not whether speech helps, but where its complementary signal emerges.

Why It Matters

This research addresses a critical gap in multimodal AI: understanding how non-textual signals like speech augment or complement text-based LLM predictions for complex social phenomena such as interpersonal attraction. For practitioners building emotion-aware systems, it highlights that simply adding audio features may not universally improve performance—contextual alignment between modalities matters more than mere fusion. The findings caution against assuming automatic gains from multimodal integration without evaluating modality-specific contributions within specific interaction dynamics.

Technical Details

• Dataset: Japanese speed-dating conversations involving multiple rounds of paired interactions with self-reported liking ratings.
• Models: Transcript-only LLM (likely fine-tuned for attraction prediction) combined with a supervised speech predictor trained on acoustic/prosodic features (e.g., pitch, tone, rhythm).
• Evaluation metrics: Pairwise ranking accuracy (consistently improved upon combination), per-participant Pearson correlation (variable gains, not statistically significant post-correction).
• Analysis approach: Retrospective assessment of which participants benefited most from speech input based on individual speech predictor accuracy relative to LLM output.
• Key insight: Complementarity is context-dependent—speech adds value primarily when the speech predictor outperforms the LLM on certain individuals or conversational phases.

Industry Insight

AI developers designing social interaction tools should avoid blanket assumptions about multimodal superiority; instead, they must conduct granular analyses of when and why auxiliary signals (like speech) enhance text-based models. This suggests investing in adaptive fusion mechanisms that dynamically weigh modalities based on real-time confidence scores or user profiles rather than fixed-weight combinations. Additionally, future work should explore whether similar conditional complementarity holds across cultures or relationship types beyond speed dating, informing broader deployment strategies for empathetic AI agents.

TL;DR

• 研究探讨了语音信号与大型语言模型（LLM）在预测速配约会中人际吸引力方面的互补性。
• 结合转录本LLM和语音预测器的联合方法显著提升了成对排名准确性，但皮尔逊相关系数的增益并不普遍显著。
• 语音信号的补充价值取决于具体情境，尤其在语音预测器表现更优的参与者中更为明显。

为什么值得看

该研究为多模态融合提供了实证支持，表明语音特征可作为文本分析的有效补充，尤其在情感或社交意图识别场景中。对于AI从业者而言，它强调了在构建预测模型时需谨慎评估不同模态的贡献边界，而非盲目叠加。

技术解析

1. 实验设计：基于日本速配对话数据集，使用转录本LLM单独预测吸引力，并引入一个监督式语音预测器作为辅助模块。
2. 性能指标：主要采用成对排名准确率（pairwise ranking accuracy）衡量整体效果，同时报告个体层面的皮尔逊相关系数（Pearson $r$）。
3. 结果差异：虽然组合模型在所有条件下均优于纯文本LLM，但在某些对话轮次和评分方向上，相关性提升未通过统计校正。
4. 条件依赖性：语音带来的额外信息量集中在那些语音预测器本身更准确的子群体身上，说明其效用具有选择性而非普适性。

行业启示

1. 在多模态应用中应优先验证各模态是否真正提供独立且有用的信息，避免冗余计算资源浪费。
2. 针对特定任务（如社交互动建模），可考虑开发自适应机制动态决定是否启用语音分支以优化效率与精度平衡。