From Profiles to Steering Vectors: Global Sparse Priors and Local Semantic Calibration for Personalized Text Generation
Proposes GLASS, a training-free framework for personalized text generation using Global-Local Activation Steering with Sparse priors. Utilizes sparse autoencoders to extract global user-style priors from historical responses and constructs local contrastive style vectors over clustered interaction scenarios. Achieves superior performance on LaMP and LongLaMP benchmarks compared to retrieval-, fine-tuning-, and steering-based baselines, with enhanced robustness to topic and length shifts. Demonst
Analysis
TL;DR
- Proposes GLASS, a training-free framework for personalized text generation using Global-Local Activation Steering with Sparse priors.
- Utilizes sparse autoencoders to extract global user-style priors from historical responses and constructs local contrastive style vectors over clustered interaction scenarios.
- Achieves superior performance on LaMP and LongLaMP benchmarks compared to retrieval-, fine-tuning-, and steering-based baselines, with enhanced robustness to topic and length shifts.
- Demonstrates effective disentanglement of stylistic information from semantic content, enabling context-aware personalization without retrieval or parameter updates.
Why It Matters
This work addresses a critical challenge in personalized text generation: efficiently capturing user-specific writing styles without compromising semantic content or incurring high computational costs. By introducing a training-free approach that separates style from semantics, GLASS offers a scalable and practical solution for real-world applications where user data is dynamic and model resources are constrained.
Technical Details
- Global Sparse Priors: Sparse autoencoders (SAEs) are employed to extract a global user-style prior from historical responses, capturing consistent stylistic patterns across diverse interactions.
- Local Contrastive Style Vectors: Local style vectors are constructed by clustering interaction scenarios and applying contrastive learning, enabling fine-grained adaptation to specific contexts.
- Layer-wise Injection: During inference, both global and local style vectors are injected into different layers of the language model, allowing for context-aware personalization without modifying model parameters.
- Evaluation Framework: Experiments are conducted on LaMP and LongLaMP datasets, with performance assessed using ROUGE metrics and LLM-as-judge evaluations, demonstrating GLASS's superiority over existing methods.
Industry Insight
- Efficiency and Scalability: GLASS's training-free nature makes it highly efficient for large-scale deployment, reducing the need for frequent retraining or storage of user-specific models.
- Robustness to Shifts: The SAE-based representations show resilience to topic and length shifts, suggesting that GLASS can maintain consistent performance even as user interactions evolve over time.
- Potential for Integration: The framework's ability to disentangle style from content opens opportunities for integration with existing personalization pipelines, enhancing user experience without significant infrastructure changes.
Disclaimer: The above content is generated by AI and is for reference only.