Nine Emotion Centroids: A Label-Free Valence Axis That Transfers Across Four Modalities
A single internal direction (V-axis) encoding emotional valence can be extracted from LLMs using only 9 emotion category names and ~450 narrative paragraphs, requiring roughly 1,500 fewer labels than standard supervised methods The same valence direction transfers across four modalities—text, vision, audio, and human EEG—despite never being jointly trained, demonstrating cross-modal alignment of emotional representation Mechanistic ablation confirms the direction is causally responsible for sent
Analysis
TL;DR
- A single internal direction (V-axis) encoding emotional valence can be extracted from LLMs using only 9 emotion category names and ~450 narrative paragraphs, requiring roughly 1,500 fewer labels than standard supervised methods
- The same valence direction transfers across four modalities—text, vision, audio, and human EEG—despite never being jointly trained, demonstrating cross-modal alignment of emotional representation
- Mechanistic ablation confirms the direction is causally responsible for sentiment tracking, with accuracy drops of 5.5–37.2 percentage points versus at most 0.88 pp for matched random directions
- A lightweight 2-parameter classifier trained on text labels transfers to images (AUC 0.961), audio (AUC 0.764), and brain recordings (AUC 0.828) without any target-modality labels
- The approach is bounded to continuous attributes (categorical concept tests return near-chance) and steering effectiveness is family-specific (works on Llama/Mistral, not Qwen/Gemma)
Why It Matters
This work demonstrates that emotional valence is a robust, cross-modal feature encoded in AI systems, discoverable with dramatically fewer labels than conventional supervised approaches. For AI practitioners, it opens a low-cost pathway to inject emotional awareness into multimodal systems without expensive annotation pipelines. For researchers, it provides evidence that continuous psychological constructs like valence have shared representational geometry across fundamentally different input modalities.
Technical Details
- Method: Embed nine emotion-anchored story sets in a frozen encoder, compute the top principal component of the nine averaged embeddings to extract the V-axis direction
- Text evaluation: On SST-2 with Llama-3-8B-Instruct, the label-free V-axis achieves AUC 0.772 versus 0.828 for fully supervised baselines (93% of supervised performance)
- Cross-modal transfer: Correlation with human valence ratings on 11,811 EmoSet images at r=0.636; AUC 0.906 on ESC-50 audio (p<2.2e-15); AUC 0.720±0.055 on EEG from 123 subjects (p<3.65e-8)
- Ablation study: Removing the V-axis collapses sentiment accuracy by 5.5–37.2 pp across three LLMs, while matched random directions cause at most 0.88 pp drop (z>12), confirming mechanistic specificity
- Classifier transfer: A 2-parameter linear classifier trained on text labels transfers to images (AUC 0.961), audio (0.764), and brain recordings (0.828); a generic 16-D subspace performs at chance (0.525), ruling out dimensionality artifacts
- Limitations: Steering is family-specific (effective on Llama/Mistral, ineffective on Qwen/Gemma); the recipe is bounded to continuous attributes with near-chance results on seven categorical concept tests
Industry Insight
- The dramatic reduction in labeling requirements (~1,500 fewer labels) makes cross-modal emotional alignment economically viable for resource-constrained teams, potentially replacing expensive human annotation with lightweight narrative-based extraction
- Family-specific steering suggests that interpretability and controllability techniques cannot be assumed portable across model architectures; practitioners should validate mechanistic interventions on each model family before deployment
- The cross-modal transfer of a single continuous attribute raises the possibility that other psychological dimensions (arousal, dominance, etc.) may share similar transferable representations, warranting systematic exploration for building emotionally intelligent multimodal systems
Disclaimer: The above content is generated by AI and is for reference only.